Creative Reimplementation Benchmark

• The original authors of the demo ranked the three reimplementations: 1. Claude Fable 5.1, 2. GPT-6 Astra, 3. Claude Opus 4.7.
• The order did not follow visual polish. According to the original authors, the GPT-6 Astra version looks better at first glance, but Claude Fable 5.1 understood the essence of the demo, and they weighted that above surface quality.
• The benchmark measures interpretation and an understanding of the spirit of a work, not visual quality alone. It produces a ranking by design: there are no scores.
• All three models produced a full-length, working audiovisual work from a single prompt.
• Every evaluated output is published as a side-by-side recording against the original, so readers can check the judgment for themselves.

The original authors gave the following reasons for their order. They are reported here as the authors' views, not as findings of this site. No other evaluation comments have been published.

1. Rank 1

   Claude Fable 5.1

   According to the original authors, Claude Fable 5.1 understood the essence of the demo. Their example is the credits, where the model credited itself under an invented handle in the style of the group's own credits. The output is fully procedural, as its own credits state.

   gfx: taas yks tekoaely joka luulee osaavansa piirtaeae

   gfx: ei yhtaeaen kuvaa, kaikki proseduraalista

2. Rank 2

   GPT-6 Astra

   According to the original authors, the GPT-6 Astra version looks better at first glance: the model used image generation models for its assets, and the output contains four generated images. The authors weighted an understanding of the spirit of the work above visual polish.

3. Rank 3

   Claude Opus 4.7

   According to the original authors, the Claude Opus 4.7 version looks noticeably rougher than the two newer entries.

Shared conditions

Generation

Single shotOne initial prompt. No subsequent human direction.

Reasoning effort

HighExtra-high and max settings are excluded.

Harness setup

VanillaDefault configuration of the harness, no customisation. Harness names and versions have not been published.

Human follow-up

NoneThe model's first complete output is the evaluated output.

Runs per model

OneA single run per model is used. No retries, no best-of-N selection.

Implementation

HTML5 Canvas / WebGLThe work must run on the web.

Code, visual assets and effects

AI-generatedEverything except the soundtrack must be produced by the model.

Original soundtrack

PermittedThe music of the reference may be reused as is.

Duration

Must match the referenceThe reference capture file is 169.62 seconds long.

Time, cost and token budgets

Not specifiedVersion 1.0 of the protocol sets no budget limits.

An author-centred view, by design

The evaluators are the original authors of the reference demo. Their ranking reflects how well each output captures the work as they understand it. It is a subjective view and not a general verdict on quality.

A small group of evaluators

The evaluators are a small group. Their number, names and roles have not been published, and the evaluation is not claimed to be blind.

An ordinal ranking only

The result is an order of preference. It does not say how large the differences between the works are, and there is no score to compare across versions or benchmarks.

One task, one run per model

The benchmark consists of a single audiovisual task built on a single reference work, and each model was run once. Run-to-run variance has not been measured, so a different sample from the same model could look different.

Three models

Only three models have been evaluated. The ranking says nothing about models that were not run.

Possible training data contamination

The reference demo has been publicly available online for years. It may be present in the training data of the evaluated models.

Not a measure of general capability

The results describe performance on this benchmark only. They say nothing about how the models compare elsewhere.

Isn't this just taste?

Yes, and deliberately so. The outputs are creative works, and the benchmark asks how well a model understood the work it was given. That is a judgment, not a measurement. The benchmark makes the judgment inspectable instead: the prompt, the conditions and every recording are on this page.

Why do the original authors judge? Isn't that a conflict of interest?

It is the central design decision of this benchmark. The authors know the intention and the cultural context of the original better than anyone else, and understanding exactly that is what the benchmark measures. The price is that the result is an author-centred, subjective view and not a general verdict on quality. According to the authors, their order did not follow visual polish, which is the kind of distinction this choice of evaluators is meant to capture.

Why is there no score?

The evaluators put the works in order of preference as whole works. There is no rubric, no sub-scores and no confidence interval, and none are planned for this benchmark. A ranking says which work the authors preferred. It does not say by how much.

Is the original demo in the training data?

Possibly. The demo was released in 2001 and has been publicly available online since. This cannot be ruled out for any of the evaluated models, and it is listed under Limitations.

Why only three models?

These are the models evaluated so far. Any model can be run with the same prompt, the same reference capture and the same conditions.

Was each model run more than once?

No. A single run per model is used. No retries, no best-of-N selection.

1. v1.02026-09-21Active
   • Initial publication with three evaluated models.
   • Side-by-side comparison recordings published for every model.
   • Full ranking by the original authors published, with the authors' notes.

@misc{jumalaita2026creativereimplementation,
  author = {{JumalAIta Intelligence}},
  title  = {Creative Reimplementation Benchmark},
  year   = {2026},
  url    = {https://ai.jumalauta.org/benchmarks/creative-reimplementation/},
  note   = {Version 1.0}
}

The principles that apply to every JumalAIta Intelligence benchmark are described in the general methodology. Changes to this site are recorded in the changelog.