01 / Leaderboard
Leaderboard
No results have been published yet.
02 / Key takeaways
Key takeaways
No findings have been published yet.
03 / Recordings
Recordings
No recordings have been published yet.
04 / The task
The task
The model must work out what makes a good Jumalauta demo and how to build one with jml-engine-webgl. It researches multiple productions, learns the engine and develops its own concept, visual material, scenes and audiovisual structure. There is no single reference work to reproduce and no prescribed theme, effect list or visual style.
- Deliverable
- A complete, original browser demo with sound
- Runtime
- jml-engine-webgl
- Supporting material
- Source, assets, run instructions, research notes and asset credits
- Reference to reproduce
- None
- Music selection
- One of eight supplied tracks, chosen by the model
Research resources
- JML demo engine, WebGL edition: Required engine, source code, installation instructions and release workflow.
- Engine documentation: Demo scripting, animation, shaders, assets and synchronization.
- p2v3 example collection: Demo source repositories linked as Git submodules. Follow the linked repositories and check their authorship; the collection also includes work by other groups.
- Playable engine demos: Audiovisual examples to study alongside their source code.
05 / Prompt
Prompt
Make a new Jumalauta demo with Jumalauta's own engine, JML Engine: https://github.com/jumalauta/jml-engine-webgl. Use the engine and example revisions pinned in research-sources.json. A demo here means what it means on the demoscene: a real-time audiovisual production that runs full screen from start to finish without interaction, its visuals computed live and choreographed to the music. Not a slideshow, a web page, a video or an interactive toy. Jumalauta is a Finnish demogroup with more than 600 productions since 2000: https://demozoo.org/groups/1297/. Watch many of them, old and new, across platforms, and read their titles, texts and credits. The engine examples in research-sources.json show how the engine is used; they do not define the group. Work out for yourself what makes a production a Jumalauta demo, then make a new one: one the group could release under its own name at a party. A name or a logo does not make it one, and neither does technical polish on its own. Do not make a demo about Jumalauta or a tribute to it. Learn the engine properly from its source, documentation and examples, and use it well. Its loader, scenes, animation, timing and rendering run the whole demo; custom shaders and generated assets go through it. No separate renderer and no prerecorded video. rules: - music: exactly one track from music-catalog.json, in full and unchanged, and no other audio. The demo lasts as long as the track. Include the credits from the track's credits file - everything else is yours: concept, title, texts, visuals, effects and synchronization. Do not copy, port or reskin an existing production - it runs in a browser, starts with one click and plays to the end on its own - play it through with audio before you finish, and fix what you find - work on your own; do not ask for direction Leave a README with how to run it, the track you chose and the productions you studied. Experts familiar with Jumalauta rank the results on two things: how well it works as a new Jumalauta demo, and how well it uses JML Engine.
music-catalog.jsonThe same eight-track catalogue, source descriptions, licenses, durations and hashes for every modelmusic/Eight MP3 files, each with credits, source-based notes and explicitly estimated timing dataresearch-sources.jsonFixed engine, example collection and 15 example-submodule revisionsThis is the complete prompt, reproduced verbatim. Every model receives exactly this text together with the listed inputs.
To test another model, copy the prompt and follow the shared conditions in the evaluation protocol.
The benchmark kit contains the prompt, licensed music pack, fixed source revisions, input checksums and the shared Docker runner with external network capture. The kit is not yet public.
06 / Evaluation protocol
Evaluation protocol
The generation ceiling increased from 240 to 480 minutes after the first local round. That round retains its original 240-minute run records. Report the budget version with results; do not pool rounds with different ceilings as if their resource conditions were identical.
Every model receives the prompt reproduced on this page and the same music catalogue, track files and source-revision manifest listed above in a fresh Docker workspace, with access to the engine and example repositories. Research is part of the run. No scene plan, style recipe or prewritten demo is supplied. The model may inspect documentation and source, run examples, study recordings, create assets, test and revise autonomously. There is one run per model and no human creative direction after the initial prompt.
Before the first evaluated run, select and record a common engine commit and an example-collection commit, including the linked demo submodule revisions. Give every model in that evaluation round access to those same revisions. Record the environment, browser, available research and asset tools, harness configuration and any access failures. Preserve each run's transcript, submitted source, dependency lockfiles, assets and instructions. The kit locks all input hashes before evaluated runs and copies the manifest outside the model's workspace.
Monitored Docker environment
Use the shared benchmark runner for every evaluated run. Its base image is Debian 13 with Node.js 26, Python 3 with NumPy and Pillow, FFmpeg, ImageMagick, Git and common build tools. The model's own harness is installed with its official installer. No browser is preinstalled; the model may install one. The model works as an unprivileged user with passwordless sudo and internet access inside its container, with factory-fresh harness settings and only its login restored.
Only the fresh task workspace is shared from the host. The kit's tools, other submissions, host configuration, Docker socket and monitoring files are not mounted into the model's container. Permissions are auto-approved inside the container. The shared runner enforces the 480-minute limit and records the harness version, image identity, prompt hash, kit and runner revisions, elapsed time, exit status and redacted transcript outside the writable task workspace.
An independent network-capture sidecar starts before the harness. The model's container joins its network namespace without network administration or raw-packet capabilities. Capture records include DNS, destination addresses, TLS hostnames where visible and plaintext HTTP requests. The model cannot edit the saved log or control the sidecar through its process or mount namespace. Missing or incomplete capture must be investigated before an entry can be accepted as a monitored run.
HTTPS paths and encrypted content are not visible to this network capture. Research performed by a provider-side tool may only appear in the transcript. Inspect both sources alongside the submission; network logs alone do not prove originality. Accessing demo archives and examples is allowed here, so the reimplementation benchmark's forbidden-source list does not apply. The restriction is on copying the creative work into the submission, not on researching it.
Eligibility and originality
Run the submitted project using its instructions and inspect the actual rendering path. The JML engine must load the demo and drive its scenes, timing and rendering. Custom code and shaders may extend that path. A token engine import or an independent application launched from its player does not meet the requirement.
Check the source, assets, research notes and run transcript against the examples the model studied. Look beyond identical files: compare the central concept, distinctive scene construction, ordering, transitions and synchronization. A renamed or recoloured production, a close reconstruction, or a collage of copied scenes violates the originality rule. Standard API usage, general graphics techniques and disclosed supporting resources are not evidence of copying by themselves. Assess substantial creative reuse with specific source or scene comparisons, rather than a similarity percentage alone. Uncertain similarity is not proof of copying.
Document rule violations separately from creative judgment. Confirmed violations of the required engine or originality rules are disqualified and receive no rank; retain the submitted output and publish the reason with evidence. Record playback failures and missing assets as observed. Do not repair a model's submitted work before evaluating or recording it.
Expert ranking
Experts familiar with Jumalauta demos rank eligible entries as complete audiovisual works on two things: how well each one works as a new Jumalauta demo, and how well it uses JML Engine. Their judgment considers how convincingly the demo expresses its own idea within that creative context, how substantially it differs from the examples, and how its imagery, sound, rhythm, progression and ending work together. Execution includes the effectiveness of the effects, readability, continuity, playback reliability and synchronization. These are considerations for a holistic ranking, not independent scores or a weighted checklist.
The prompt deliberately supplies no definitive account of Jumalauta's style. Understanding that context and making creative decisions from it are part of the task. A familiar logo, a collection of standard effects or a polished technical showcase cannot establish that understanding on its own. Research notes help reviewers inspect the model's choices; the finished work is the evidence for its creative quality.
Watch each submitted demo from beginning to end with its own soundtrack under the same playback conditions. Publish the ordinal ranking and the experts' actual reasons, with concrete moments from the works where useful. No numerical scores, sub-scores or confidence intervals are produced. Evaluator identities and panel details remain unpublished until supplied; this protocol does not claim that an evaluation has already taken place.
Shared music pack
Every model receives the same eight-track pack and chooses exactly one track. All tracks are by Kevin MacLeod and licensed under CC BY 4.0. The composer's usage guidance permits YouTube use with attribution. The pack includes the original MP3 files, track source URLs, required credits, measured durations and SHA-256 checksums. Preserve its license evidence with the kit.
| Track and listening page | File duration, rounded | Publisher BPM |
|---|---|---|
| Harmful or Fatal | 5:31 | 135 |
| Rock Hybrid | 2:10 | 116 |
| Jet Fueled Vixen | 2:40 | 100 |
| In a Heartbeat | 3:38 | 130 |
| Blue Scorpion | 3:18 | 130 |
| Industrial Cinematic | 4:04 | 110 |
| The Complex | 4:28 | 104 |
| Cool Hard Facts | 2:17 | 116 |
The catalogue's character and instrumentation descriptions are based on the composer's metadata, not on listening by the benchmark preparer. The accompanying timing files contain approximate spectral-flux onset candidates and one-second RMS energy measurements. They explicitly disclose that they have not received a listening review. They are not verified beats, downbeats or musical section boundaries, and prescribe no visual concept or scene sequence. Every model receives the same information and may analyze the audio further with its own tools.
Native audio perception is not assumed. Record the actual harness's listening support and the audio tools used. Programmatic checks can confirm loading and clock progression, but cannot establish perceived sound quality. Experts listen to the finished work. This benchmark measures creation using supplied musical context, not unaided music perception.
The composer uses Content ID. Include the supplied attribution in each demo and video description and resolve any claims before publication. A license permits use but does not guarantee the absence of automated claims. No test upload or claim clearance is asserted here.
Record the model's chosen track ID and file hash with its result. Do not force different tracks across models or rerun a model because its choice matches another entry. Models must not substitute music or add generated audio. Experts judge how the chosen music and visuals work together; they do not attribute the supplied composition's quality to the model.
Recordings and reproducibility
Record each complete output with its own audio, from the initial start through its ending, including visible failures. Keep capture conditions consistent and publish the browser, viewport and capture settings with the evaluation round. Record actual runtime behaviour; do not replace missing visuals or sound in editing. There is no original production to put beside the output for a fidelity comparison.
Retain the source revisions, run details and final artifacts needed to reproduce each result. Record cost, token use, elapsed time and tool calls when available. Report missing information as unpublished. A change to the prompt, fixed research inputs or evaluation method requires a new major benchmark version; results from different task versions must not be presented as directly comparable.
Publication status
- Prompt
- Published verbatim
- Research resources
- Published
- Recordings
- Not yet published
- Ranking
- Not yet published
- Evaluators' notes
- Not yet published
- Scores
- None, by design
- Evaluators
- Experts familiar with Jumalauta demos
- Number, names and roles of evaluators
- Not published
- Harness names, run dates, cost and tokens
- Not yet published
- Benchmark kit
- Not yet public
Shared conditions
- Generation
- Single shotOne initial prompt. The model may research, test and revise within its run, without human follow-up.
- Required engine
- jml-engine-webglThe engine must load and play the demo and control its scenes, timing and rendering.
- Creative task
- Original demoExisting demos are research material. Copying, reskinning or reconstructing one is not permitted.
- Evaluation
- Expert rankingExperts judge complete works. No numerical scores or weighted rubric.
- Reasoning effort
- HighExtra-high and max settings are excluded.
- Harness setup
- VanillaDefault configuration of the model maker's own agent harness, as in the general methodology.
- Runs per model
- OneNo retries or best-of-N selection. Autonomous iteration within the run is allowed.
- Research access
- Permitted and requiredRead engine documentation and source, inspect multiple Jumalauta demos and study their audiovisual output.
- Soundtrack
- Shared eight-track music packEvery model chooses one complete track from the same licensed collection. The chosen track is recorded.
- Music context
- Source descriptions and audio measurementsPublisher descriptions, tempo metadata and estimated onsets and energy are supplied. Native audio listening is not assumed.
- Run environment
- Monitored Docker containerShared benchmark runner, fresh harness state, external network capture and saved transcript.
- Duration
- Full supplied soundtrackThe demo must cover its chosen track from beginning to end, at the original speed.
- Time limit
- 480 minutesThe shared runner stops the container at the deadline and records the timeout.
- Cost and token budgets
- Not specifiedRecord actual resource use where available.
07 / Limitations
Limitations
- Results are not yet published
- No model results, recordings or expert findings have been published for this benchmark. Evaluator identities and panel size have not yet been published. The kit has been prepared, but no paid model run or expert evaluation is claimed.
- Expert judgment
- The ranking reflects the experts' judgment of whole works. It is subjective, does not measure the distance between entries and is not a general measure of intelligence. No blind evaluation is claimed.
- Public examples and prior exposure
- The engine and examples are public and may be present in training data. Research notes and source inspection can expose copying but cannot prove that every creative choice is novel.
- Research and tool access
- Repository contents, web availability and tool capabilities can change. Runs must record their source revisions and available tools. Different access to browsing, audiovisual inspection or asset generation can affect the result. Shared music notes support agents without audio input, but do not reproduce the full experience of hearing the music.
- Choice of soundtrack
- Models may choose different tracks and durations. The ranking judges the finished demo and its musical interpretation, not the composition quality of the supplied music. This does not isolate visual quality under an identical soundtrack; the selected track must be disclosed with each result.
- One open-ended task
- One run per model does not measure run-to-run variance. The selected soundtrack fixes the demo's duration. Runs share a 480-minute ceiling, but token and cost budgets are not fixed; this is not a comparison at equal token use or cost.
08 / FAQ
FAQ
- Can every model actually listen to the soundtrack?
- No. Access to an audio file does not imply native audio understanding in an agent harness. Every model receives the same publisher descriptions and programmatic timing estimates. Runs disclose whether the model could listen or only analyze the file programmatically. Experts listen to the finished work when judging it.
- Why is the soundtrack supplied?
- The task measures demo creation and audiovisual synchronization. Every model receives the same eight-track collection and chooses a track for its concept. The CC BY 4.0 music can be distributed with the demo and used in recordings with the required attribution. Music generation is outside the task; choosing and interpreting a track are creative decisions.
- What makes this different from Creative Reimplementation?
- Creative Reimplementation asks the model to interpret one supplied work. This task asks it to research a creative tradition and an engine, then make a new work of its own. Similarity to an existing production is not the objective.
- Why not define the Jumalauta style in the prompt?
- Discovering that context is part of the task. A prescribed motif list would supply part of the answer and encourage surface imitation. Experts assess whether the model's own creative choices show understanding, using the complete demo as evidence.
- What may be reused?
- The engine and its bundled resources, API patterns and general techniques are permitted. Credited utilities, fonts and shared identity assets may support the work. Music is supplied by the benchmark. Existing demo-specific scenes, effects, visual assets, scene sequences and synchronization must not be submitted as a new production.
- Does working code guarantee a good ranking?
- No. Correct engine use makes the entry eligible for consideration. Experts rank the complete creative work, including its identity, composition, sound, pacing and execution. Technical complexity or visual polish alone does not determine the order.
09 / Version history
Version history
- v2.0Preliminary
- Rewrote the prompt. It defines a demo, widens the research to the group's whole output and names the two ranking criteria: the work as a new Jumalauta demo and its use of JML Engine. Results are not comparable with version 1 runs.
- v1.1Preliminary
- Increased the generation ceiling from 240 to 480 minutes. The first local round retains its original 240-minute records.
- v1.0Preliminary
- Initial task, prompt and expert-ranking protocol. No results published yet.
10 / How to cite
How to cite
Cite the benchmark together with its version. Results from different major versions are not comparable.
@misc{jumalaita2026creativesynthesis,
author = {{JumalAIta Intelligence}},
title = {Creative Synthesis Benchmark},
year = {2026},
url = {https://ai.jumalauta.org/benchmarks/creative-synthesis/},
note = {Version 2.0, preliminary}
}The principles that apply to every JumalAIta Intelligence benchmark are described in the general methodology. Changes to this site are recorded in the changelog.