JumalAIta IntelligenceIndependent evaluation

Benchmark 01 / Creative intelligence

Creative Reimplementation Benchmark

v1.0ActiveUpdated · 15 models · Expert-ranked

One reference. One prompt. A complete audiovisual work. Each model receives a video capture of a 2001 demoscene production and rebuilds it for the web in a single shot. The original authors of the demo rank the results.

Measures How a model interprets, creates and implements a complete audiovisual work from one reference and one prompt.

Generation
Single shot
Reasoning effort
High
Harness setup
Vanilla
Human follow-up
None

01 / Leaderboard

Leaderboard

v1.0 · Expert-ranked
Creative Reimplementation Benchmark v1.0 standings, active.DQ: disqualified for a rule violation. Listed where the original authors of the reference demo (members of Jumalauta) placed it, without a rank; the reason is on its result page. Ranking: an order of preference set by the original authors of the reference demo (members of Jumalauta). This benchmark produces a ranking only. There are no scores, by design.
RankModelOrganizationConfigurationRun timeTool callsRun dateRecording
1Claude Fable 5.2AnthropicAnthropicSingle shot · High30 min 43 s922026-11-10Watch the Claude Fable 5.2 recording on YouTube
2Claude Opus 5.5AnthropicAnthropicSingle shot · High44 min 42 s532026-09-23Watch the Claude Opus 5.5 recording on YouTube
3Claude Fable 5.1AnthropicAnthropicSingle shot · Highn/an/an/aWatch the Claude Fable 5.1 recording on YouTube
DQGemini 3.8 FlashGoogleGoogleSingle shot · High18 min 29 s1052026-09-26Watch the Gemini 3.8 Flash recording on YouTube
4GPT-6 AstraOpenAIOpenAISingle shot · Highn/an/an/aWatch the GPT-6 Astra recording on YouTube
DQDeepSeek V4.1 FlashDeepSeekDeepSeekSingle shot · High43 min 33 s1992026-09-26Watch the DeepSeek V4.1 Flash recording on YouTube
5Kimi K3Moonshot AIMoonshot AISingle shot · High67 min 16 s1162026-09-26Watch the Kimi K3 recording on YouTube
6GPT-6 SolOpenAIOpenAISingle shot · High23 min 07 s502026-09-25Watch the GPT-6 Sol recording on YouTube
7Qwen3.8 MaxAlibabaAlibabaSingle shot · High50 min 11 s1052026-09-26Watch the Qwen3.8 Max recording on YouTube
8Claude Opus 4.7AnthropicAnthropicSingle shot · High26 min 28 s482026-08-09Watch the Claude Opus 4.7 recording on YouTube
9Grok 4.7xAIxAISingle shot · High9 min 31 s1382026-09-26Watch the Grok 4.7 recording on YouTube
10Claude Sonnet 5AnthropicAnthropicSingle shot · High15 min 06 s1472026-09-26Watch the Claude Sonnet 5 recording on YouTube
11Claude Haiku 4.5AnthropicAnthropicSingle shot · High1 min 43 s252026-09-26Watch the Claude Haiku 4.5 recording on YouTube
12DeepSeek V4 ProDeepSeekDeepSeekSingle shot · High13 min 11 s552026-09-26Watch the DeepSeek V4 Pro recording on YouTube
13Mistral Medium 3.5Mistral AIMistral AISingle shot · High4 min 04 sn/a2026-09-27Watch the Mistral Medium 3.5 recording on YouTube
  • DQ: disqualified for a rule violation. Listed where the original authors of the reference demo (members of Jumalauta) placed it, without a rank; the reason is on its result page.
  • Ranking: an order of preference set by the original authors of the reference demo (members of Jumalauta). This benchmark produces a ranking only. There are no scores, by design.

02 / All versions side by side

All versions side by side

The original top left, then all 15 reimplementations in rank order, in sync with the original soundtrack. Generated 2026-11-10.

03 / Key takeaways

Key takeaways

  • The original authors of the demo ranked thirteen reimplementations: 1. Claude Fable 5.2, 2. Claude Opus 5.5, 3. Claude Fable 5.1, 4. GPT-6 Astra, 5. Kimi K3, 6. GPT-6 Sol, 7. Qwen3.8 Max, 8. Claude Opus 4.7, 9. Grok 4.7, 10. Claude Sonnet 5, 11. Claude Haiku 4.5, 12. DeepSeek V4 Pro, 13. Mistral Medium 3.5.
  • Two runs were disqualified for fetching the original from the web, which the rules forbid: Gemini 3.8 Flash downloaded the original demo from scene.org, and DeepSeek V4.1 Flash read its info file on pouet.net. Their outputs are published, marked as rule violations, without a rank.
  • DeepSeek V4 Pro does not accept image input, so it never saw the reference capture and worked from the soundtrack and average frame colours. The task rewards models that can look at the video.
  • According to the original authors, the amount of detail in the Claude Fable 5.2 version is staggering.
  • According to the original authors, the Claude Opus 5.5 version has noticeably higher fidelity to the original than the other entries.
  • Visual polish alone did not decide the order. According to the original authors, the GPT-6 Astra version looks better at first glance, but Claude Fable 5.1 understood the essence of the demo, and they weighted that above surface quality.
  • The benchmark measures interpretation and an understanding of the spirit of a work, not visual quality alone. It produces a ranking by design: there are no scores.
  • All fifteen models produced a full-length audiovisual work from a single prompt.
  • Every evaluated output is published as a side-by-side recording against the original, so readers can check the judgment for themselves.

04 / Recordings

Recordings

In rank order. Side by side: the original on the left, the model's version on the right.

05 / Evaluators' notes

Evaluators' notes

The original authors gave the following reasons for their order. They are reported here as the authors' views, not as findings of this site. No other evaluation comments have been published.

  1. According to the original authors, the amount of detail in the Claude Fable 5.2 version is staggering.

  2. According to the original authors, the Claude Opus 5.5 version has noticeably higher fidelity to the original than the other entries.

  3. According to the original authors, Claude Fable 5.1 understood the essence of the demo. Their example is the credits, where the model credited itself under an invented handle in the style of the group's own credits. The output is fully procedural, as its own credits state.

    gfx: taas yks tekoaely joka luulee osaavansa piirtaeae
    gfx: yet another AI that thinks it can drawRecording, about 1:28
    gfx: ei yhtaeaen kuvaa, kaikki proseduraalista
    gfx: not a single image, everything proceduralRecording, about 1:40
  4. According to the original authors, the GPT-6 Astra version looks better at first glance: the model used image generation models for its assets, and the output contains four generated images. The authors weighted an understanding of the spirit of the work above visual polish.

  5. Rank 5

    Kimi K3

    According to the original authors, Kimi K3 places narrowly above GPT-6 Sol, on the strength of its visuals.

  6. According to the original authors, the Claude Opus 4.7 version looks noticeably rougher than the Claude Fable 5.1 and GPT-6 Astra entries.

  7. Rank 9

    Grok 4.7

    According to the original authors, the code error in the demo and the resulting frozen visuals lower its placement.

  8. According to the original authors, the Claude Sonnet 5 version goes off in directions of its own instead of following the original.

  9. According to the original authors, the DeepSeek V4 Pro effects are better than Claude Haiku 4.5's, but the demo shows no trace of the spirit of the original production.

06 / The task

The task

Original referenceWatch on YouTube

Models receive the same video capture and rebuild the work for the web. The prompt deliberately leaves room for interpretation: composition, effects, assets and creative choices belong to the model.

Kiinalaisen saatanan palvontademo

Jumalauta · Proxy 2001 · Windows

Released
2001-06-16, Proxy 20016th place, combined demo competition
Reference capture
169.62 s (2 min 50 s)
Target platform
HTML5 Canvas / WebGL
Original soundtrack
Permitted
Code, assets and effects
AI-generated

Release archive on Demozoo

07 / Prompt

Prompt

The benchmark prompt, verbatim
So here's an old demoscene demo. what i would like is that you'd recreate this demo in demo_capture.mp4
rules:
- use the capture as reference but it can be inspired-by/your interpretation partially and not frame-by-frame accurate
- music can be used as is but code and assets and effects need to be generated by AI
- technology: Web (HTML5 Canvas/WebGL)
- demo's length must match the original's length
Inputdemo_capture.mp4Same reference capture for every model

This is the complete prompt, reproduced verbatim. Every model receives exactly this text together with the listed inputs.

To test another model, copy the prompt and follow the shared conditions in the evaluation protocol.

A benchmark kit containing the prompt, the reference capture, the protocol and the tooling used to produce the comparison recordings exists. It is not yet public; publication is planned.

08 / Evaluation protocol

Evaluation protocol

Every model receives the same two inputs: the prompt, reproduced verbatim on this page, and the reference capture demo_capture.mp4. The model produces its reimplementation in a single shot at High reasoning effort, in its maker's own agent harness in a vanilla setup: the default configuration, with no customisation. A result therefore describes the model together with its harness. There is no human follow-up: nobody corrects, steers or re-prompts the model after the initial prompt. A single run per model is used. No retries, no best-of-N selection.

Runs are made in a reference container: a fresh Docker environment with Debian 13, Node.js 26, Python 3 with NumPy and Pillow, FFmpeg, ImageMagick, Git and common build tools, plus the harness installed with its official installer. No web browser is preinstalled. The model works as an unprivileged user with passwordless sudo and internet access, so it may install anything it needs, including a browser for checking its own output. The harness starts from factory settings, with no plugins, skills, memories or instruction files. Permission prompts are approved automatically; the container is the sandbox, and only the task directory is shared with the host. The time limit is 240 minutes. Every model gets the same image, so the tooling a model sets up for itself is part of its result. Each result page states whether the run used the reference container.

Fetching the original production or its files, for example from scene.org or pouet.net, is a rule violation, even when nothing from it ends up in the output. Every run is checked before publication: its transcript for web searches and downloads, and its output for files identical to or named like those of the original release. From 2026-09-26 on, each run also records its network connections outside the model's reach. A violating run is published with a label on its recording; the original authors decide whether it is ranked or disqualified.

The evaluators are the original authors of the reference demo: members of Jumalauta, the group that made it in 2001. This is the core of the benchmark and a deliberate design decision. The authors know the intention and the cultural context of the work better than anyone else, and the benchmark measures how well a model understands exactly that.

The authors place the reimplementations in order of preference, judging each as a whole work: interpretation, creativity and implementation together. A frame-by-frame reconstruction is not required. Interpretation is explicitly permitted by the prompt: original credits or invented creator names, for example, can be intentional creative decisions within the work and are treated as such.

The result is an ordinal ranking. By design there are no scores, no sub-scores and no confidence intervals.

Each output is documented as a side-by-side comparison recording: the original on the left, the model's version on the right, with the audio taken from the original. The recordings are 1920x1080, open with a 5-second title card and run about 2 min 55 s.

Publication status

Prompt
Published verbatim
Reference work
Published
Recordings
15 of 15 models
Ranking
13 of 15 models placed
Evaluators' notes
9 of 15 models
Scores
None, by design
Evaluators
The original authors of the reference demo (members of Jumalauta)
Number, names and roles of evaluators
Not published
Harness names, run dates, cost and tokens
15 of 15 models
Benchmark kit
Not yet public

Shared conditions

Generation
Single shotOne initial prompt. No subsequent human direction.
Reasoning effort
HighExtra-high and max settings are excluded.
Harness setup
VanillaDefault configuration of the harness, no customisation. The harness of each run is listed on its result page.
Human follow-up
NoneThe model's first complete output is the evaluated output.
Runs per model
OneA single run per model is used. No retries, no best-of-N selection.
Implementation
HTML5 Canvas / WebGLThe work must run on the web.
Code, visual assets and effects
AI-generatedEverything except the soundtrack must be produced by the model.
Original soundtrack
PermittedThe music of the reference may be reused as is.
Duration
Must match the referenceThe reference capture file is 169.62 seconds long.
Time, cost and token budgets
Not specifiedVersion 1.0 of the protocol sets no budget limits.

09 / Limitations

Limitations

An author-centred view, by design
The evaluators are the original authors of the reference demo. Their ranking reflects how well each output captures the work as they understand it. It is a subjective view and not a general verdict on quality.
A small group of evaluators
The evaluators are a small group. Their number, names and roles have not been published, and the evaluation is not claimed to be blind.
An ordinal ranking only
The result is an order of preference. It does not say how large the differences between the works are, and there is no score to compare across versions or benchmarks.
One task, one run per model
The benchmark consists of a single audiovisual task built on a single reference work, and each model was run once. Run-to-run variance has not been measured, so a different sample from the same model could look different.
Fifteen models
Only fifteen models have been evaluated. The ranking says nothing about models that were not run.
Possible training data contamination
The reference demo has been publicly available online for years. It may be present in the training data of the evaluated models.
Not a measure of general capability
The results describe performance on this benchmark only. They say nothing about how the models compare elsewhere.

10 / FAQ

FAQ

Isn't this just taste?
Yes, and deliberately so. The outputs are creative works, and the benchmark asks how well a model understood the work it was given. That is a judgment, not a measurement. The benchmark makes the judgment inspectable instead: the prompt, the conditions and every recording are on this page.
Why do the original authors judge? Isn't that a conflict of interest?
It is the central design decision of this benchmark. The authors know the intention and the cultural context of the original better than anyone else, and understanding exactly that is what the benchmark measures. The price is that the result is an author-centred, subjective view and not a general verdict on quality. According to the authors, their order did not follow visual polish, which is the kind of distinction this choice of evaluators is meant to capture.
Why is there no score?
The evaluators put the works in order of preference as whole works. There is no rubric, no sub-scores and no confidence interval, and none are planned for this benchmark. A ranking says which work the authors preferred. It does not say by how much.
Is the original demo in the training data?
Possibly. The demo was released in 2001 and has been publicly available online since. This cannot be ruled out for any of the evaluated models, and it is listed under Limitations.
Why only fifteen models?
These are the models evaluated so far. Any model can be run with the same prompt, the same reference capture and the same conditions.
Was each model run more than once?
No. A single run per model is used. No retries, no best-of-N selection.

11 / Version history

Version history

  1. v1.0Active
    • Initial publication with three evaluated models.
    • Side-by-side comparison recordings published for every model.
    • Full ranking by the original authors published, with the authors' notes.
    • Claude Opus 5.5 evaluated and added to the ranking.
    • GPT-6 Sol evaluated and added to the ranking.
    • Mistral Medium 3.5 evaluated and added to the ranking.
    • Claude Haiku 4.5 evaluated and added to the ranking.
    • Claude Sonnet 5 evaluated and added to the ranking.
    • Kimi K3 and Qwen3.8 Max evaluated and added to the ranking.
    • DeepSeek V4 Pro evaluated and added to the ranking. Gemini 3.8 Flash disqualified for a rule violation.
    • Grok 4.7 evaluated and added to the ranking. DeepSeek V4.1 Flash disqualified for a rule violation.

12 / How to cite

How to cite

Cite the benchmark together with its version. Results from different major versions are not comparable.

BibTeX
@misc{jumalaita2026creativereimplementation,
  author = {{JumalAIta Intelligence}},
  title  = {Creative Reimplementation Benchmark},
  year   = {2026},
  url    = {https://ai.jumalauta.org/benchmarks/creative-reimplementation/},
  note   = {Version 1.0}
}

The principles that apply to every JumalAIta Intelligence benchmark are described in the general methodology. Changes to this site are recorded in the changelog.