Skip to content
mager-bench1.3

A quick test of useful code.

Two algorithmic problems and three everyday programs. Each model gets one attempt at its lowest reasoning effort. We check whether the code works.

Explore the five programs

Five tasks, 60 checks

No v1.3 model results yet

Ready to benchmark

The graph and optimization tasks pass reference checks. Model calibration is still pending.

View attempts

What we ask it to build

12 checks per program

Plan a dependency build

Resolve a dependency graph, detect cycles, and find the critical path without enumerating every path.

Expected result

Finish in 9 time units. Critical path: lint → compile.

Input

{
  "jobs": [
    {
      "id": "compile",
      "duration": 4,
      "deps": [
        "fetch",
        "lint"
      ]
    },
    {
      "id": "fetch",
      "duration": 3,
      "deps": []
    },
    {
      "id": "lint",
      "duration": 5,
      "deps": []
    }
  ],
  "targets": [
    "compile"
  ]
}

Exact expected output

{
  "order": [
    "fetch",
    "lint",
    "compile"
  ],
  "finish": 9,
  "criticalPath": [
    "lint",
    "compile"
  ]
}

These are the published test cases, not a model’s answers. Saved attempts show what the model actually returned.

Read the exact prompt
Write a synchronous JavaScript function solve(input). Return only JavaScript, without Markdown or exports. Standard ECMAScript only: no filesystem, network, packages, Node globals, timers, or async. Inputs are valid under the contract. Return a JSON-serializable value. Do not mutate built-ins. Each case runs in a fresh QuickJS context with 16 MiB memory and a 100 ms CPU limit. All arithmetic inputs and results fit safe integers. We compare JSON structurally, ignoring object key order but preserving array order. Implement the full contract, not just an example.

Input: {jobs: {id: string, duration: nonnegative integer, deps: string[]}[], targets: string[]}. There are at most 240 jobs and 1000 dependency entries. IDs are unique strings of 1 to 24 ASCII letters, digits, or underscores. Every dependency and target names an existing job; repeated dependencies or targets have no extra effect. Ignore jobs not needed by a target, including any cycles in that ignored region. Required jobs are the targets and all their transitive dependencies. If the required graph has a cycle, return {error:"cycle"}. Otherwise return {order:string[], finish:integer, criticalPath:string[]}. order is the lexicographically smallest valid topological ordering of all required IDs: repeatedly take the smallest currently ready ID, not a whole batch. Jobs may execute with unlimited parallel workers; each starts as soon as all its dependencies finish. finish is the earliest time all targets are finished. criticalPath is a dependency chain starting at a dependency-free job and ending at any target with total duration equal to finish; choose the lexicographically smallest ID sequence among tied chains. Compare sequences element-by-element using ASCII order, with a proper prefix smaller than its extension. Empty targets returns {order:[],finish:0,criticalPath:[]}. Do not enumerate all paths: the case set includes a layered graph with exponentially many dependency paths. Times fit safe integers.

Where shortcuts fail

A dependency graph with 40 layers has over a trillion possible paths. The program must reuse computed results. Enumerating every path exceeds the time limit.

In the scheduling task, one job pays 10, but two compatible jobs pay 12 together. The full problem also has a spending budget and cooldown periods. Picking the biggest job first gives the wrong answer.

The 80-job case makes exhaustive subset search impractical. These cases test algorithm choice as well as edge cases; how models perform is still unmeasured.

Built to finish quickly

Five short answers
One JavaScript program per task. No retries or model judges.
Lowest effort
The runner selects the lowest supported setting. For GPT-6 Sol, that’s low.
60 exact checks
A check passes when the returned value matches its expected output.
Timing, scoring, and limits

Each generation call has a 90-second deadline and a prompted target of 3,072 output tokens. Five calls share the same total budget, for at most 270 seconds of generation waiting. We stop at the first provider failure.

Programs run in fresh QuickJS contexts with no filesystem or network bindings, 16 MiB memory, and a 100 ms CPU limit per case. Incorrect values and code errors fail checks. Missing answers and provider failures are unscored.

These 60 public cases are a small, inspectable sample of JavaScript correctness. A single attempt does not establish a model ranking. Timings include the local Codex harness and are not pure model latency.

Full protocol on GitHub