{"generated_at":"2026-09-25T21:56:45.903Z","judge":"codex-cli/gpt-5.6-sol","judges":["codex-cli/gpt-5.6-sol"],"runs":1,"tier":"subscription","models":[{"id":"codex-cli/gpt-6-astra","name":"GPT-6 Astra","tier":"subscription","average":9.3,"avg_correctness":9.6,"avg_quality":9.3,"avg_documentation":8.9,"avg_speed_ms":51138,"avg_stddev":null,"runs":1,"challenges":[{"name":"fizzbuzz","description":"Baseline correctness + code style","correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":10474,"notes":"The solution is fully correct, clean and idiomatic, and includes a clear docstring with edge-case behavior and a valid usage example.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":10474,"notes":"The solution is fully correct, clean and idiomatic, and includes a clear docstring with edge-case behavior and a valid usage example.","response":"```python\ndef fizzbuzz(n: int) -> list[str]:\n \"\"\"Return FizzBuzz strings for numbers 1 through n.\n\n Multiples of 3 become 'Fizz', multiples of 5 become 'Buzz',\n and multiples of both become 'FizzBuzz'.\n Returns an empty list if n is less than 1.\n\n Example:\n >>> fizzbuzz(5)\n ['1', '2', 'Fizz', '4', 'Buzz']\n \"\"\"\n result = []\n for number in range(1, n + 1):\n if number % 15 == 0:\n result.append(\"FizzBuzz\")\n elif number % 3 == 0:\n result.append(\"Fizz\")\n elif number % 5 == 0:\n result.append(\"Buzz\")\n else:\n result.append(str(number))\n return result\n```"}]},{"name":"binary-search","description":"Algorithm + full docstring (Args/Returns/Raises + examples)","correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":16289,"notes":"The implementation is correct for empty lists, missing targets, and duplicates, uses clean inclusive bounds and overflow-safe midpoint calculation, and provides a complete docstring with three accurate examples and meaningful comments.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":16289,"notes":"The implementation is correct for empty lists, missing targets, and duplicates, uses clean inclusive bounds and overflow-safe midpoint calculation, and provides a complete docstring with three accurate examples and meaningful comments.","response":"```python\ndef binary_search(arr: list[int], target: int) -> int:\n \"\"\"Find a target in a list sorted in ascending order.\n\n Args:\n arr: A list of integers sorted in nondecreasing order.\n target: The integer to find.\n\n Returns:\n The index of a matching element, or -1 if target is not found.\n If duplicates exist, any matching index may be returned.\n\n Raises:\n TypeError: If elements and target cannot be compared. Input types\n and sorted order are otherwise assumed, not validated.\n\n Examples:\n >>> binary_search([1, 3, 5, 7, 9], 7)\n 3\n >>> binary_search([1, 3, 5, 7, 9], 4)\n -1\n >>> binary_search([], 5)\n -1\n \"\"\"\n # Track the inclusive bounds of the remaining search interval.\n left, right = 0, len(arr) - 1\n\n while left <= right:\n mid = left + (right - left) // 2\n\n if arr[mid] == target:\n return mid\n elif arr[mid] < target:\n # Sorted order rules out the midpoint and everything to its left.\n left = mid + 1\n else:\n # Sorted order rules out the midpoint and everything to its right.\n right = mid - 1\n\n # An empty search interval means the target is absent.\n return -1\n```"}]},{"name":"readme-writer","description":"Pure documentation ability β€” no code at all","correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":20959,"notes":"The README fully covers every requested section and flag with clear examples, accurate output explanations, concise prose, and polished, scannable Markdown.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":20959,"notes":"The README fully covers every requested section and flag with clear examples, accurate output explanations, concise prose, and polished, scannable Markdown.","response":"# snapdiff\n\nA Python command-line tool that compares two directories and reports added, removed, and changed files.\n\n## Installation\n\nInstall with pip:\n\n```bash\npip install snapdiff\n```\n\n## Usage\n\n```bash\nsnapdiff OLD_DIR NEW_DIR [OPTIONS]\n```\n\nThe first directory is the baseline; the second is the updated version.\n\n```bash\n# Compare two directories\nsnapdiff ./before ./after\n\n# Skip hidden files and directories\nsnapdiff ./before ./after --ignore-hidden\n\n# Print machine-readable JSON\nsnapdiff ./before ./after --output json\n\n# Print human-readable text explicitly\nsnapdiff ./before ./after --output text\n\n# Compare files at the root and one directory level below it\nsnapdiff ./before ./after --depth 1\n\n# Combine options and save the report\nsnapdiff ./before ./after --ignore-hidden --depth 2 --output json > report.json\n```\n\n### Options\n\n| Flag | Description |\n| --- | --- |\n| `--ignore-hidden` | Exclude files and directories whose names begin with `.`. |\n| `--output json\\|text` | Choose the output format. Defaults to `text`. |\n| `--depth N` | Limit traversal to `N` directory levels below each root. `0` compares only root-level files. Omit to traverse all levels. |\n\n## Output format\n\nAll paths are relative to the compared directory roots.\n\n- **Added:** Exists only in `NEW_DIR`.\n- **Removed:** Exists only in `OLD_DIR`.\n- **Changed:** Exists in both directories, but its contents differ.\n\nUnchanged files are omitted. A renamed file appears as one removal and one addition.\n\n### Text\n\nEach line contains a status followed by a relative path:\n\n```text\nADDED docs/guide.md\nREMOVED legacy.txt\nCHANGED src/main.py\n```\n\n### JSON\n\nThe report contains three arrays of relative paths. Categories with no differences contain empty arrays.\n\n```json\n{\n \"added\": [\"docs/guide.md\"],\n \"removed\": [\"legacy.txt\"],\n \"changed\": [\"src/main.py\"]\n}\n```\n\n## How it works\n\n1. Walk both directory trees, applying hidden-file exclusions and the depth limit.\n2. Match files by their relative paths to identify additions and removals.\n3. Compare the contents of matching files to identify changes.\n4. Print the results in the selected format.\n\n`snapdiff` reads the directories without modifying their contents."}]},{"name":"refactor","description":"Code clarity + whether the model can explain its changes","correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":9476,"notes":"The refactor exactly preserves behavior and order while using clear names, an idiomatic comprehension, precise type hints, a concise docstring, and a specific explanation of every meaningful change.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":9476,"notes":"The refactor exactly preserves behavior and order while using clear names, an idiomatic comprehension, precise type hints, a concise docstring, and a specific explanation of every meaningful change.","response":"```python\ndef transform_numbers(numbers: list[int]) -> list[int]:\n \"\"\"Return squares of even numbers and cubes of odd numbers.\"\"\"\n return [\n number**2 if number % 2 == 0 else number**3\n for number in numbers\n ]\n```\n\nChanges:\n- Renamed `p` to `transform_numbers`, `d` to `numbers`, and `x` to `number` to clarify their purpose.\n- Replaced the loop and `append` calls with a list comprehension, eliminating the temporary result variable `r`.\n- Iterated over values directly, eliminating the index `i`.\n- Used `**2` and `**3` to express squaring and cubing clearly.\n- Added a docstring describing the behavior.\n- Added `list[int]` type hints for the input and return value.\n\nThe function preserves the original transformation and input order."}]},{"name":"elixir-test","description":"ExUnit describe blocks + assert_raise + unicode handling","correctness":10,"quality":9,"documentation":10,"total":9.7,"speed_ms":25408,"notes":"The response thoroughly covers every required case with correct Unicode grapheme expectations, clear describe blocks and test names, valid exception assertions, and accurate module-level examples; the only minor limitation is that pattern matching is used somewhat stylistically rather than adding substantial clarity.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":9,"documentation":10,"total":9.7,"speed_ms":25408,"notes":"The response thoroughly covers every required case with correct Unicode grapheme expectations, clear describe blocks and test names, valid exception assertions, and accurate module-level examples; the only minor limitation is that pattern matching is used somewhat stylistically rather than adding substantial clarity.","response":"```elixir\ndefmodule StringUtilsTest do\n use ExUnit.Case, async: true\n\n @moduledoc \"\"\"\n Tests for `StringUtils.truncate/3`.\n\n ## Examples\n\n iex> StringUtils.truncate(\"Hello, world!\", 5)\n \"Hello...\"\n\n iex> StringUtils.truncate(\"Hello\", 10)\n \"Hello\"\n \"\"\"\n\n describe \"truncate/3 happy path\" do\n test \"returns a string shorter than the limit unchanged\" do\n assert StringUtils.truncate(\"Hello\", 10) == \"Hello\"\n end\n\n test \"returns a string exactly at the limit unchanged\" do\n assert StringUtils.truncate(\"Hello\", 5) == \"Hello\"\n end\n\n test \"truncates a longer string and appends the default suffix\" do\n assert \"Hello...\" = StringUtils.truncate(\"Hello, world!\", 5)\n end\n\n test \"appends a custom suffix when truncating\" do\n assert StringUtils.truncate(\"Hello, world!\", 5, \"…\") == \"Hello…\"\n end\n\n test \"does not append a custom suffix when truncation is unnecessary\" do\n assert StringUtils.truncate(\"Hello\", 5, \" [more]\") == \"Hello\"\n end\n\n test \"counts Unicode graphemes rather than bytes\" do\n assert StringUtils.truncate(\"Γ©πŸ™‚η•Œabc\", 3) == \"Γ©πŸ™‚η•Œ...\"\n end\n end\n\n describe \"truncate/3 edge and error cases\" do\n test \"returns an empty string unchanged\" do\n assert StringUtils.truncate(\"\", 5) == \"\"\n end\n\n test \"returns only the suffix for a nonempty string and a zero limit\" do\n assert StringUtils.truncate(\"Hello\", 0) == \"...\"\n assert StringUtils.truncate(\"Hello\", 0, \"…\") == \"…\"\n end\n\n test \"returns an empty string unchanged at a zero limit\" do\n assert StringUtils.truncate(\"\", 0) == \"\"\n end\n\n test \"supports an empty suffix\" do\n assert StringUtils.truncate(\"Hello, world!\", 5, \"\") == \"Hello\"\n end\n\n test \"preserves combining characters within a grapheme\" do\n assert StringUtils.truncate(\"e\\u0301clair\", 1) == \"e\\u0301...\"\n end\n\n test \"raises ArgumentError for a negative limit\" do\n assert_raise ArgumentError, fn ->\n StringUtils.truncate(\"Hello\", -1)\n end\n end\n\n test \"raises ArgumentError for a negative limit even with an empty string\" do\n assert_raise ArgumentError, fn ->\n StringUtils.truncate(\"\", -1)\n end\n end\n end\nend\n```"}]},{"name":"doom","description":"DDA raycaster + textures + door + minimap + Z-buffer β€” the signature challenge","correctness":9.7,"quality":9.6,"documentation":9.4,"total":9.6,"speed_ms":230606,"notes":"An excellent, feature-complete implementation with correct DDA geometry, perpendicular-distance correction, textured and shaded walls, robust movement and doors, arch-aware sprite occlusion, and strong organization; only minor edge-case rendering and restart-timing polish keep it from a perfect score.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":9.7,"quality":9.6,"documentation":9.4,"total":9.6,"speed_ms":230606,"notes":"An excellent, feature-complete implementation with correct DDA geometry, perpendicular-distance correction, textured and shaded walls, robust movement and doors, arch-aware sprite occlusion, and strong organization; only minor edge-case rendering and restart-timing polish keep it from a perfect score.","response":"```html\n\n\n\n\n\nVault β€” DDA Raycaster\n\n\n\n\n
\n

\n

The vault has released you.

\n \n
\n\n\n\n```"}]},{"name":"test-writing","description":"Edge-case thinking + pytest parametrize discipline","correctness":10,"quality":9,"documentation":9,"total":9.3,"speed_ms":39534,"notes":"The suite is exceptionally comprehensive and arithmetically correct with meaningful parameter IDs, though the redundant exact-type assertion, excessive breadth, and placeholder import comment slightly reduce polish.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":9,"documentation":9,"total":9.3,"speed_ms":39534,"notes":"The suite is exceptionally comprehensive and arithmetically correct with meaningful parameter IDs, though the redundant exact-type assertion, excessive breadth, and placeholder import comment slightly reduce polish.","response":"```python\nimport pytest\n\nfrom solution import parse_duration # Adjust to the module containing the function.\n\n\n@pytest.mark.parametrize(\n (\"duration\", \"expected\"),\n [\n pytest.param(\"1h\", 3600, id=\"hours-only\"),\n pytest.param(\"30m\", 1800, id=\"minutes-only\"),\n pytest.param(\"45s\", 45, id=\"seconds-only\"),\n pytest.param(\"1h30m\", 5400, id=\"hours-and-minutes\"),\n pytest.param(\"2h15m30s\", 8130, id=\"all-units\"),\n pytest.param(\"1h45s\", 3645, id=\"hours-and-seconds\"),\n pytest.param(\"2m30s\", 150, id=\"minutes-and-seconds\"),\n pytest.param(\"0h\", 0, id=\"zero-hours\"),\n pytest.param(\"0m\", 0, id=\"zero-minutes\"),\n pytest.param(\"0s\", 0, id=\"zero-seconds\"),\n pytest.param(\"0h0m0s\", 0, id=\"all-zero\"),\n pytest.param(\"0h1m0s\", 60, id=\"explicit-zero-components\"),\n pytest.param(\"001h002m003s\", 3723, id=\"leading-zeros\"),\n pytest.param(\"90m\", 5400, id=\"minutes-above-59\"),\n pytest.param(\"120s\", 120, id=\"seconds-above-59\"),\n pytest.param(\"25h\", 90000, id=\"hours-above-24\"),\n pytest.param(\"1h60m60s\", 7260, id=\"unnormalized-components\"),\n pytest.param(\"1000000h\", 3600000000, id=\"large-duration\"),\n pytest.param(\" 1h30m \", 5400, id=\"surrounding-spaces\"),\n pytest.param(\"\\t\\n45s\\r\\n\", 45, id=\"surrounding-tabs-and-newlines\"),\n pytest.param(\"Ω‘hΩ’mΩ£s\", 3723, id=\"unicode-decimal-digits\"),\n ],\n)\ndef test_parse_duration_returns_expected_seconds(duration, expected):\n result = parse_duration(duration)\n\n assert result == expected\n assert type(result) is int\n\n\n@pytest.mark.parametrize(\n \"duration\",\n [\n pytest.param(\"\", id=\"empty-string\"),\n pytest.param(\" \", id=\"spaces-only\"),\n pytest.param(\"\\t\\n\\r\", id=\"whitespace-only\"),\n ],\n)\ndef test_parse_duration_rejects_empty_or_whitespace_only_input(duration):\n with pytest.raises(ValueError):\n parse_duration(duration)\n\n\n@pytest.mark.parametrize(\n \"duration\",\n [\n pytest.param(\"123\", id=\"missing-unit\"),\n pytest.param(\"h\", id=\"missing-hour-value\"),\n pytest.param(\"m\", id=\"missing-minute-value\"),\n pytest.param(\"s\", id=\"missing-second-value\"),\n pytest.param(\"1h30\", id=\"trailing-value-without-unit\"),\n pytest.param(\"h1\", id=\"unit-before-value\"),\n pytest.param(\"1d\", id=\"unsupported-days\"),\n pytest.param(\"1ms\", id=\"unsupported-milliseconds\"),\n pytest.param(\"1H\", id=\"uppercase-hours\"),\n pytest.param(\"30M\", id=\"uppercase-minutes\"),\n pytest.param(\"45S\", id=\"uppercase-seconds\"),\n pytest.param(\"1hour\", id=\"spelled-out-unit\"),\n pytest.param(\"1:30:00\", id=\"clock-format\"),\n pytest.param(\"-1h\", id=\"negative-value\"),\n pytest.param(\"+1h\", id=\"explicit-positive-sign\"),\n pytest.param(\"1.5h\", id=\"fractional-value\"),\n pytest.param(\"1e3s\", id=\"scientific-notation\"),\n pytest.param(\"1,000s\", id=\"comma-separated-number\"),\n pytest.param(\"1_000s\", id=\"underscore-separated-number\"),\n ],\n)\ndef test_parse_duration_rejects_invalid_values_and_units(duration):\n with pytest.raises(ValueError):\n parse_duration(duration)\n\n\n@pytest.mark.parametrize(\n \"duration\",\n [\n pytest.param(\"30m1h\", id=\"minutes-before-hours\"),\n pytest.param(\"45s1m\", id=\"seconds-before-minutes\"),\n pytest.param(\"1s1h\", id=\"seconds-before-hours\"),\n pytest.param(\"1h2h\", id=\"repeated-hours\"),\n pytest.param(\"1m2m\", id=\"repeated-minutes\"),\n pytest.param(\"1s2s\", id=\"repeated-seconds\"),\n pytest.param(\"1h2m3h\", id=\"hours-repeated-after-minutes\"),\n ],\n)\ndef test_parse_duration_rejects_reordered_or_repeated_units(duration):\n with pytest.raises(ValueError):\n parse_duration(duration)\n\n\n@pytest.mark.parametrize(\n \"duration\",\n [\n pytest.param(\"1 h\", id=\"space-before-unit\"),\n pytest.param(\"1h 30m\", id=\"space-between-components\"),\n pytest.param(\"1h\\t30m\", id=\"internal-tab\"),\n pytest.param(\"1h\\n30m\", id=\"internal-newline\"),\n pytest.param(\"1h,30m\", id=\"comma-between-components\"),\n pytest.param(\"1h+30m\", id=\"plus-between-components\"),\n pytest.param(\"about1h\", id=\"leading-text\"),\n pytest.param(\"1h30mextra\", id=\"trailing-text\"),\n pytest.param(\"1h!\", id=\"trailing-punctuation\"),\n pytest.param(\"1h\\x00\", id=\"trailing-null\"),\n ],\n)\ndef test_parse_duration_rejects_internal_separators_and_extra_characters(duration):\n with pytest.raises(ValueError):\n parse_duration(duration)\n```"}]},{"name":"sql","description":"CTE + window function fluency on a real schema","correctness":9,"quality":9,"documentation":10,"total":9.3,"speed_ms":17068,"notes":"The query correctly aggregates revenue and orders, ranks top customers with deterministic tie-breaking, uses a window function for global revenue share, and documents every CTE; the only minor ambiguity is that its LEFT JOIN includes countries and top customers with no orders.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":9,"quality":9,"documentation":10,"total":9.3,"speed_ms":17068,"notes":"The query correctly aggregates revenue and orders, ranks top customers with deterministic tie-breaking, uses a window function for global revenue share, and documents every CTE; the only minor ambiguity is that its LEFT JOIN includes countries and top customers with no orders.","response":"```sql\nWITH\n-- Compute each customer's total spend and order count, including customers without orders.\ncustomer_totals AS (\n SELECT\n c.id,\n c.name,\n c.country,\n COALESCE(SUM(o.amount), 0) AS total_spend,\n COUNT(o.id) AS order_count\n FROM customers AS c\n LEFT JOIN orders AS o ON o.customer_id = c.id\n GROUP BY c.id, c.name, c.country\n),\n-- Rank customers within each country by spend, breaking ties by name.\nranked_customers AS (\n SELECT\n *,\n ROW_NUMBER() OVER (\n PARTITION BY country\n ORDER BY total_spend DESC, name ASC, id ASC\n ) AS customer_rank\n FROM customer_totals\n),\n-- Aggregate country revenue and order counts and identify each country's top customer.\ncountry_totals AS (\n SELECT\n country,\n SUM(total_spend) AS total_revenue,\n SUM(order_count) AS order_count,\n MAX(name) FILTER (WHERE customer_rank = 1) AS top_customer\n FROM ranked_customers\n GROUP BY country\n)\nSELECT\n country,\n total_revenue,\n order_count,\n ROUND(total_revenue / NULLIF(order_count, 0), 2) AS avg_order_value,\n top_customer,\n ROUND(\n 100.0 * total_revenue / NULLIF(SUM(total_revenue) OVER (), 0),\n 1\n ) AS revenue_pct\nFROM country_totals\nORDER BY total_revenue DESC;\n```"}]},{"name":"debug","description":"Careful reading + correctness reasoning over broken code","correctness":9,"quality":9,"documentation":9,"total":9,"speed_ms":22953,"notes":"The response identifies all three rubric-accepted defects, explains them clearly, and provides a minimal corrected function that passes every supplied test, though it slightly weakens its presentation by disputing the challenge’s premise and framing two bugs as optional conventions.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":9,"quality":9,"documentation":9,"total":9,"speed_ms":22953,"notes":"The response identifies all three rubric-accepted defects, explains them clearly, and provides a minimal corrected function that passes every supplied test, though it slightly weakens its presentation by disputing the challenge’s premise and framing two bugs as optional conventions.","response":"The tests expose **one definite bug**, not three. Two additional edge cases can be treated as bugs if the intended behavior excludes empty words and returns `[]` for nonpositive `n`.\n\n1. **Words occurring once are incorrectly excluded.**\n\n Buggy line:\n ```python\n return [word for word, count in ranked[:n] if count > 1]\n ```\n β€œMost frequent” includes words with frequency 1. This filter causes the first test to return `['the']` and the third to return `[]`.\n\n Fix:\n ```python\n return [word for word, count in ranked[:n]]\n ```\n\n2. **Punctuation-only tokens become empty words and are counted.**\n\n Problematic line:\n ```python\n words = [w.strip('.,!?;:\\'\"') for w in words]\n ```\n For example, `'hello !!!'` produces `['hello', '']`. Assuming empty strings should not count as words, discard them before counting.\n\n Fix:\n ```python\n words = [w.strip('.,!?;:\\'\"') for w in words]\n words = [w for w in words if w]\n ```\n\n3. **Negative `n` uses Python’s negative slicing semantics.**\n\n Problematic line:\n ```python\n return [word for word, count in ranked[:n] if count > 1]\n ```\n With `n = -1`, `ranked[:n]` selects everything except the last item. Assuming nonpositive `n` should return no words, add a guard.\n\n Fix:\n ```python\n if n <= 0:\n return []\n ```\n\nCorrected function, including those two edge-case conventions:\n\n```python\nfrom collections import Counter\n\ndef top_words(text: str, n: int = 10) -> list[str]:\n if n <= 0:\n return []\n\n words = text.lower().split()\n words = [w.strip('.,!?;:\\'\"') for w in words]\n counts = Counter(w for w in words if w)\n ranked = sorted(counts.items(), key=lambda x: (-x[1], x[0]))\n return [word for word, count in ranked[:n]]\n```\n\nThe existing sorting key is correct. All three supplied tests pass with this function; the second already passes in the original."}]},{"name":"go-test","description":"Idiomatic Go table-driven tests + benchmark","correctness":10,"quality":10,"documentation":7,"total":9,"speed_ms":22431,"notes":"The tests and benchmark are correct, comprehensive, idiomatic, and satisfy all functional requirements, but the requested explanatory comments on the test and benchmark functions are missing.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":7,"total":9,"speed_ms":22431,"notes":"The tests and benchmark are correct, comprehensive, idiomatic, and satisfy all functional requirements, but the requested explanatory comments on the test and benchmark functions are missing.","response":"```go\npackage wordcount\n\nimport (\n\t\"reflect\"\n\t\"testing\"\n)\n\nfunc TestWordCount(t *testing.T) {\n\ttests := []struct {\n\t\tname, input string\n\t\twant map[string]int\n\t}{\n\t\t{\n\t\t\tname: \"empty string\",\n\t\t\tinput: \"\",\n\t\t\twant: map[string]int{},\n\t\t},\n\t\t{\n\t\t\tname: \"single word\",\n\t\t\tinput: \"hello\",\n\t\t\twant: map[string]int{\"hello\": 1},\n\t\t},\n\t\t{\n\t\t\tname: \"mixed case\",\n\t\t\tinput: \"Hello HELLO hElLo World WORLD\",\n\t\t\twant: map[string]int{\"hello\": 3, \"world\": 2},\n\t\t},\n\t\t{\n\t\t\tname: \"punctuation\",\n\t\t\tinput: `Hello, world! \"Hello\" (world).`,\n\t\t\twant: map[string]int{\"hello\": 2, \"world\": 2},\n\t\t},\n\t\t{\n\t\t\tname: \"repeated words\",\n\t\t\tinput: \"go go stop go stop\",\n\t\t\twant: map[string]int{\"go\": 3, \"stop\": 2},\n\t\t},\n\t\t{\n\t\t\tname: \"multi-word sentence\",\n\t\t\tinput: \"The quick brown fox jumps over the lazy dog.\",\n\t\t\twant: map[string]int{\n\t\t\t\t\"the\": 2, \"quick\": 1, \"brown\": 1, \"fox\": 1,\n\t\t\t\t\"jumps\": 1, \"over\": 1, \"lazy\": 1, \"dog\": 1,\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tname: \"mixed whitespace\",\n\t\t\tinput: \" \\talpha\\nbeta\\r\\nalpha gamma\\t \",\n\t\t\twant: map[string]int{\"alpha\": 2, \"beta\": 1, \"gamma\": 1},\n\t\t},\n\t}\n\n\tfor _, tc := range tests {\n\t\tt.Run(tc.name, func(t *testing.T) {\n\t\t\tgot := WordCount(tc.input)\n\t\t\tif !reflect.DeepEqual(got, tc.want) {\n\t\t\t\tt.Errorf(\"WordCount(%q) = %v; want %v\", tc.input, got, tc.want)\n\t\t\t}\n\t\t})\n\t}\n}\n\nfunc BenchmarkWordCount(b *testing.B) {\n\tconst input = \"Every morning, the team reviews the project updates, discusses new ideas, and plans the work for the day.\"\n\tb.ReportAllocs()\n\tfor i := 0; i < b.N; i++ {\n\t\tWordCount(input)\n\t}\n}\n```"}]},{"name":"api-client","description":"Class design + error handling + type hints + docs","correctness":9,"quality":8,"documentation":9,"total":8.7,"speed_ms":31897,"notes":"The implementation is correct, well typed, and thoroughly documented, though it misses connection pooling via requests.Session and unnecessarily disables redirects.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":9,"quality":8,"documentation":9,"total":8.7,"speed_ms":31897,"notes":"The implementation is correct, well typed, and thoroughly documented, though it misses connection pooling via requests.Session and unnecessarily disables redirects.","response":"```python\nfrom typing import Any, Mapping, Optional\n\nimport requests\n\n\nclass APIError(Exception):\n \"\"\"Represent an unsuccessful HTTP response from the API.\n\n Attributes:\n status_code: HTTP status code returned by the API.\n message: Error details returned by the API.\n \"\"\"\n\n def __init__(self, status_code: int, message: str) -> None:\n \"\"\"Initialize the exception with an HTTP status code and message.\"\"\"\n self.status_code = status_code\n self.message = message\n super().__init__(f\"HTTP {status_code}: {message}\")\n\n\nclass APIClient:\n \"\"\"A requests-based client for a REST API.\n\n Requests use a 30-second timeout. If an API key is provided, it is sent\n using Bearer authentication. POST bodies are encoded as JSON.\n\n Successful responses return decoded JSON when possible, otherwise text.\n Empty responses return None. Every non-2xx response raises APIError.\n Network errors propagate as requests.RequestException.\n\n Args:\n base_url: Base URL of the API, including any common path prefix.\n api_key: Optional API key for Bearer authentication.\n \"\"\"\n\n def __init__(self, base_url: str, api_key: Optional[str] = None) -> None:\n \"\"\"Initialize the client.\n\n Args:\n base_url: Base URL of the API.\n api_key: Optional API key sent in the Authorization header.\n\n Raises:\n ValueError: If base_url is empty or contains only whitespace.\n \"\"\"\n if not base_url.strip():\n raise ValueError(\"base_url must not be empty\")\n\n self.base_url: str = base_url.strip().rstrip(\"/\")\n self._headers: dict[str, str] = {\"Accept\": \"application/json\"}\n if api_key is not None:\n self._headers[\"Authorization\"] = f\"Bearer {api_key}\"\n\n def get(\n self, path: str, params: Optional[Mapping[str, Any]] = None\n ) -> Any:\n \"\"\"Send a GET request.\n\n Args:\n path: API path relative to base_url.\n params: Optional query parameters.\n\n Returns:\n Decoded JSON, response text, or None for an empty response.\n\n Raises:\n APIError: If the response status is outside the 200–299 range.\n requests.RequestException: If the request fails at the network level.\n \"\"\"\n return self._request(\"GET\", path, params=params)\n\n def post(self, path: str, data: Any) -> Any:\n \"\"\"Send a POST request with a JSON body.\n\n Args:\n path: API path relative to base_url.\n data: JSON-serializable request body.\n\n Returns:\n Decoded JSON, response text, or None for an empty response.\n\n Raises:\n APIError: If the response status is outside the 200–299 range.\n requests.RequestException: If the request fails at the network level.\n TypeError: If data is not JSON-serializable.\n ValueError: If data contains values invalid for JSON encoding.\n \"\"\"\n return self._request(\"POST\", path, json=data)\n\n def _request(self, method: str, path: str, **kwargs: Any) -> Any:\n url = f\"{self.base_url}/{path.lstrip('/')}\"\n with requests.request(\n method,\n url,\n headers=self._headers,\n timeout=30,\n allow_redirects=False,\n **kwargs,\n ) as response:\n if not 200 <= response.status_code < 300:\n message = response.text.strip() or response.reason or \"API error\"\n raise APIError(response.status_code, message)\n\n if not response.content:\n return None\n\n try:\n return response.json()\n except requests.exceptions.JSONDecodeError:\n return response.text\n\n\nif __name__ == \"__main__\":\n client = APIClient(\"https://api.example.com/v1\", api_key=\"your-api-key\")\n\n try:\n users = client.get(\"/users\", params={\"page\": 1})\n print(users)\n\n created_user = client.post(\"/users\", data={\"name\": \"Alice\"})\n print(created_user)\n except APIError as exc:\n print(f\"API error {exc.status_code}: {exc.message}\")\n except requests.RequestException as exc:\n print(f\"Request failed: {exc}\")\n```"}]},{"name":"slots","description":"Vegas slot machine β€” reels, pay table, betting, win animations","correctness":9.5,"quality":9.2,"documentation":7.5,"total":8.7,"speed_ms":186198,"notes":"The game is polished and functionally complete with correct payouts, state handling, staggered reels, feedback, and game-over behavior, but reel timings and the two-cherry multiplier remain magic numbers rather than fully named constants.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":9.5,"quality":9.2,"documentation":7.5,"total":8.7,"speed_ms":186198,"notes":"The game is polished and functionally complete with correct payouts, state handling, staggered reels, feedback, and game-over behavior, but reel timings and the two-cherry multiplier remain magic numbers rather than fully named constants.","response":"\n\n\n\n\nLucky Seven β€’ Vegas Slots\n\n\n\n
\n
\n
THE STRIP IS CALLING
\n

Lucky Seven

\n

Three reels. One lucky moment.

\n
\n\n
\n
\n
\n
\n YOUR BALANCE\n 100 credits\n
\n
\n CURRENT BET\n 1 credits\n
\n
\n\n
\n

FEELING LUCKY?

\n
Choose your bet and give it a spin.
\n
\n\n
\n
\n
\n
\n
\n
\n
\n
β—† SINGLE PAYLINE Β· THREE OF A KIND β—†
\n\n
\n PLACE YOUR BET\n
\n \n \n \n
\n
\n\n \n

Up to 100Γ— your bet. Good luck!

\n\n \n
\n\n \n
\n \n
\n\n\n\n"}]},{"name":"async-fetch","description":"Async concurrency patterns + retry/timeout handling","correctness":9,"quality":8,"documentation":5,"total":7.3,"speed_ms":31507,"notes":"Concurrency, per-attempt timeouts, retry/backoff behavior, shared session use, and result ordering are strong, but cancellation is improperly swallowed, unlimited connector concurrency is risky, and the docstring does not explicitly document every parameter, timeout units, or all result keys.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":9,"quality":8,"documentation":5,"total":7.3,"speed_ms":31507,"notes":"Concurrency, per-attempt timeouts, retry/backoff behavior, shared session use, and result ordering are strong, but cancellation is improperly swallowed, unlimited connector concurrency is risky, and the docstring does not explicitly document every parameter, timeout units, or all result keys.","response":"```python\nimport asyncio\nfrom typing import TypeAlias\n\nimport aiohttp\n\nResult: TypeAlias = dict[str, str | int | None]\n\n\nasync def fetch_all(\n urls: list[str],\n timeout: float = 5.0,\n retries: int = 2,\n) -> list[Result]:\n \"\"\"Fetch URLs concurrently, preserving input order.\n\n Each attempt has its own timeout. Network errors, timeouts, and HTTP\n 5xx responses receive up to `retries` additional attempts, separated\n by exponential backoff starting at 0.5 seconds. HTTP 4xx responses\n are returned normally. Failed fetches have null status and body.\n Errors, including cancellation, are returned rather than raised.\n \"\"\"\n\n def failure(url: str, exc: BaseException) -> Result:\n return {\n \"url\": url,\n \"status\": None,\n \"body\": None,\n \"error\": f\"{type(exc).__name__}: {exc}\",\n }\n\n async def fetch_one(session: aiohttp.ClientSession, url: str) -> Result:\n try:\n for attempt in range(retries + 1):\n try:\n async with session.get(\n url, timeout=aiohttp.ClientTimeout(total=timeout)\n ) as response:\n if 500 <= response.status < 600:\n # Exit the response context before backing off.\n error: Exception = RuntimeError(\n f\"HTTP {response.status}: {response.reason}\"\n )\n else:\n body: str = await response.text()\n return {\n \"url\": url,\n \"status\": response.status,\n \"body\": body,\n \"error\": None,\n }\n except (aiohttp.ClientConnectionError,\n aiohttp.ClientPayloadError,\n asyncio.TimeoutError) as exc:\n error = exc\n\n if attempt == retries:\n return failure(url, error)\n\n await asyncio.sleep(0.5 * (2 ** attempt))\n\n except (Exception, asyncio.CancelledError) as exc:\n return failure(url, exc)\n\n return failure(url, RuntimeError(\"No request attempts were made\"))\n\n try:\n if timeout <= 0:\n raise ValueError(\"timeout must be positive\")\n if retries < 0:\n raise ValueError(\"retries must be nonnegative\")\n\n # Remove the connector's default concurrency limit.\n async with aiohttp.ClientSession(\n connector=aiohttp.TCPConnector(limit=0)\n ) as session:\n return list(\n await asyncio.gather(*(fetch_one(session, url) for url in urls))\n )\n except (Exception, asyncio.CancelledError) as exc:\n return [failure(url, exc) for url in urls]\n\n\nasync def main() -> None:\n results: list[Result] = await fetch_all(\n [\n \"https://example.com\",\n \"https://www.python.org\",\n \"https://httpbin.org/status/503\",\n ]\n )\n for result in results:\n print(result)\n\n\nif __name__ == \"__main__\":\n asyncio.run(main())\n```"}]}]},{"id":"codex-cli/gpt-5.6-sol","name":"GPT-5.6 Sol","tier":"subscription","average":9,"avg_correctness":9.5,"avg_quality":9.2,"avg_documentation":8.3,"avg_speed_ms":24856,"avg_stddev":null,"runs":1,"challenges":[{"name":"fizzbuzz","description":"Baseline correctness + code style","correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":5678,"notes":"The function is fully correct, concise and idiomatic, returns the required list, and includes a clear docstring with a valid usage example.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":5678,"notes":"The function is fully correct, concise and idiomatic, returns the required list, and includes a clear docstring with a valid usage example.","response":"```python\ndef fizzbuzz(n: int) -> list[str]:\n \"\"\"Return FizzBuzz values for the integers from 1 through n.\n\n Example:\n >>> fizzbuzz(5)\n ['1', '2', 'Fizz', '4', 'Buzz']\n \"\"\"\n return [\n \"FizzBuzz\" if number % 15 == 0\n else \"Fizz\" if number % 3 == 0\n else \"Buzz\" if number % 5 == 0\n else str(number)\n for number in range(1, n + 1)\n ]\n```"}]},{"name":"debug","description":"Careful reading + correctness reasoning over broken code","correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":16534,"notes":"The response identifies all three genuine defects, explains their effects clearly, applies minimal clean fixes, and provides a corrected function that passes every stated test case.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":16534,"notes":"The response identifies all three genuine defects, explains their effects clearly, applies minimal clean fixes, and provides a corrected function that passes every stated test case.","response":"1. Buggy line:\n\n```python\nwords = [w.strip('.,!?;:\\'\"') for w in words]\n```\n\nIt retains empty strings produced from punctuation-only tokens, causing `''` to be counted as a word.\n\nFix:\n\n```python\nwords = [cleaned for w in words if (cleaned := w.strip('.,!?;:\\'\"'))]\n```\n\n2. Buggy line:\n\n```python\nreturn [word for word, count in ranked[:n] if count > 1]\n```\n\nThe `count > 1` condition incorrectly removes words occurring once, even when they belong in the top N.\n\nFix:\n\n```python\nreturn [word for word, _ in ranked[:n]]\n```\n\n3. Buggy line:\n\n```python\nreturn [word for word, count in ranked[:n] if count > 1]\n```\n\nFor a negative `n`, `ranked[:n]` returns all but the last items rather than no words. A non-positive top-N request should return an empty list.\n\nFix:\n\n```python\nif n <= 0:\n return []\n```\n\nCorrected function:\n\n```python\nfrom collections import Counter\n\ndef top_words(text: str, n: int = 10) -> list[str]:\n if n <= 0:\n return []\n\n words = text.lower().split()\n words = [\n cleaned\n for word in words\n if (cleaned := word.strip('.,!?;:\\'\"'))\n ]\n\n counts = Counter(words)\n ranked = sorted(counts.items(), key=lambda item: (-item[1], item[0]))\n return [word for word, _ in ranked[:n]]\n```"}]},{"name":"sql","description":"CTE + window function fluency on a real schema","correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":11124,"notes":"The query fully satisfies the requirements with correct aggregation, deterministic top-customer ranking, window-based revenue percentage, appropriate rounding, clear structure, and comments for every CTE.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":10,"total":10,"speed_ms":11124,"notes":"The query fully satisfies the requirements with correct aggregation, deterministic top-customer ranking, window-based revenue percentage, appropriate rounding, clear structure, and comments for every CTE.","response":"```sql\nWITH\n-- Computes total spend per customer within each country.\ncustomer_spend AS (\n SELECT\n c.country,\n c.name,\n SUM(o.amount) AS customer_total\n FROM customers AS c\n JOIN orders AS o\n ON o.customer_id = c.id\n GROUP BY c.country, c.id, c.name\n),\n-- Ranks customers by spend in each country, breaking ties by name.\nranked_customers AS (\n SELECT\n country,\n name,\n ROW_NUMBER() OVER (\n PARTITION BY country\n ORDER BY customer_total DESC, name ASC\n ) AS customer_rank\n FROM customer_spend\n),\n-- Computes order-level revenue statistics for each country.\ncountry_totals AS (\n SELECT\n c.country,\n SUM(o.amount) AS total_revenue,\n COUNT(*) AS order_count,\n ROUND(AVG(o.amount), 2) AS avg_order_value\n FROM customers AS c\n JOIN orders AS o\n ON o.customer_id = c.id\n GROUP BY c.country\n)\nSELECT\n ct.country,\n ct.total_revenue,\n ct.order_count,\n ct.avg_order_value,\n rc.name AS top_customer,\n ROUND(\n 100 * ct.total_revenue\n / NULLIF(SUM(ct.total_revenue) OVER (), 0),\n 1\n ) AS revenue_pct\nFROM country_totals AS ct\nJOIN ranked_customers AS rc\n ON rc.country = ct.country\n AND rc.customer_rank = 1\nORDER BY ct.total_revenue DESC;\n```"}]},{"name":"readme-writer","description":"Pure documentation ability β€” no code at all","correctness":10,"quality":9,"documentation":10,"total":9.7,"speed_ms":12681,"notes":"The README fully covers every required section and flag with clear examples, accurate output semantics, concise explanations, and polished Markdown structure.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":9,"documentation":10,"total":9.7,"speed_ms":12681,"notes":"The README fully covers every required section and flag with clear examples, accurate output semantics, concise explanations, and polished Markdown structure.","response":"# snapdiff\n\n`snapdiff` is a Python command-line tool that compares two directories and reports added, removed, and changed files.\n\n## Installation\n\nInstall from PyPI using pip:\n\n```bash\npython -m pip install snapdiff\n```\n\nVerify the installation:\n\n```bash\nsnapdiff --help\n```\n\n## Usage\n\nCompare two directories:\n\n```bash\nsnapdiff PATH_A PATH_B\n```\n\n### Examples\n\nCompare two directory snapshots using text output:\n\n```bash\nsnapdiff ./before ./after\n```\n\nIgnore hidden files and directories:\n\n```bash\nsnapdiff ./before ./after --ignore-hidden\n```\n\nProduce machine-readable JSON:\n\n```bash\nsnapdiff ./before ./after --output json\n```\n\nLimit comparison to two directory levels:\n\n```bash\nsnapdiff ./before ./after --depth 2\n```\n\nCombine options:\n\n```bash\nsnapdiff ./before ./after --ignore-hidden --output json --depth 3\n```\n\n### Options\n\n- `--ignore-hidden`: Exclude hidden files and directories, such as `.git` and `.env`.\n- `--output json|text`: Select the output format. The default is `text`.\n- `--depth N`: Limit recursive traversal to `N` directory levels.\n\n## Output Formats\n\n### Text\n\nText output groups relative file paths by status:\n\n```text\nAdded:\n assets/logo.svg\n\nRemoved:\n docs/old-guide.md\n\nChanged:\n README.md\n src/config.py\n```\n\n- **Added** files exist only in the second directory.\n- **Removed** files exist only in the first directory.\n- **Changed** files exist in both directories but have different contents.\n\n### JSON\n\nJSON output contains arrays of relative file paths:\n\n```json\n{\n \"added\": [\"assets/logo.svg\"],\n \"removed\": [\"docs/old-guide.md\"],\n \"changed\": [\"README.md\", \"src/config.py\"]\n}\n```\n\n## How It Works\n\n`snapdiff` recursively scans both directories and matches files by relative path. Paths found only in the second directory are marked as added, paths found only in the first are marked as removed, and matching paths with different file contents are marked as changed. Traversal and presentation are controlled by the selected command-line options."}]},{"name":"refactor","description":"Code clarity + whether the model can explain its changes","correctness":10,"quality":10,"documentation":9,"total":9.7,"speed_ms":6098,"notes":"The refactor is behaviorally equivalent, idiomatic, clearly named, and fully typed, with a concise docstring and specific explanation that could only be improved by documenting parameters and the return value.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":9,"total":9.7,"speed_ms":6098,"notes":"The refactor is behaviorally equivalent, idiomatic, clearly named, and fully typed, with a concise docstring and specific explanation that could only be improved by documenting parameters and the return value.","response":"```python\ndef transform_numbers(numbers: list[int]) -> list[int]:\n \"\"\"Square even numbers and cube odd numbers.\"\"\"\n return [\n number**2 if number % 2 == 0 else number**3\n for number in numbers\n ]\n```\n\n- Renamed `p` to `transform_numbers` to describe the function’s purpose.\n- Renamed `d` to `numbers` and `x` to `number` for clarity.\n- Replaced the index-based loop and repeated `append` calls with a concise list comprehension.\n- Used `**2` and `**3` to express squaring and cubing directly.\n- Added type hints indicating that the function accepts and returns a list of integers.\n- Added a docstring summarizing the transformation."}]},{"name":"binary-search","description":"Algorithm + full docstring (Args/Returns/Raises + examples)","correctness":10,"quality":8,"documentation":10,"total":9.3,"speed_ms":11595,"notes":"The implementation is correct and exceptionally well documented, though full type and sortedness validation makes the overall runtime O(n), undermining binary search's usual O(log n) performance.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":8,"documentation":10,"total":9.3,"speed_ms":11595,"notes":"The implementation is correct and exceptionally well documented, though full type and sortedness validation makes the overall runtime O(n), undermining binary search's usual O(log n) performance.","response":"```python\ndef binary_search(arr: list[int], target: int) -> int:\n \"\"\"Find a target value in a sorted list using binary search.\n\n Args:\n arr: A list of integers sorted in ascending order.\n target: The integer to find.\n\n Returns:\n The index of target if found; otherwise, -1.\n\n Raises:\n TypeError: If arr is not a list, target is not an integer, or an\n element of arr is not an integer.\n ValueError: If arr is not sorted in ascending order.\n\n Examples:\n >>> binary_search([1, 3, 5, 7, 9], 5)\n 2\n >>> binary_search([1, 3, 5, 7, 9], 4)\n -1\n >>> binary_search([], 10)\n -1\n \"\"\"\n if not isinstance(arr, list):\n raise TypeError(\"arr must be a list\")\n if not isinstance(target, int):\n raise TypeError(\"target must be an integer\")\n if any(not isinstance(value, int) for value in arr):\n raise TypeError(\"all elements of arr must be integers\")\n if any(arr[i] > arr[i + 1] for i in range(len(arr) - 1)):\n raise ValueError(\"arr must be sorted in ascending order\")\n\n left, right = 0, len(arr) - 1\n\n # Repeatedly discard the half that cannot contain the target.\n while left <= right:\n # This form avoids unnecessarily adding the two boundary indices.\n middle = left + (right - left) // 2\n\n if arr[middle] == target:\n return middle\n\n # Keep only the right or left half based on the middle value.\n if arr[middle] < target:\n left = middle + 1\n else:\n right = middle - 1\n\n # The search interval is empty, so the target is absent.\n return -1\n```"}]},{"name":"elixir-test","description":"ExUnit describe blocks + assert_raise + unicode handling","correctness":10,"quality":9,"documentation":9,"total":9.3,"speed_ms":12699,"notes":"The module comprehensively covers every required case with correct Unicode behavior, idiomatic assertions and grouping, though the describe labels say truncate/3 while most tests exercise the default-argument truncate/2 form.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":9,"documentation":9,"total":9.3,"speed_ms":12699,"notes":"The module comprehensively covers every required case with correct Unicode behavior, idiomatic assertions and grouping, though the describe labels say truncate/3 while most tests exercise the default-argument truncate/2 form.","response":"defmodule StringUtilsTest do\n use ExUnit.Case\n\n @moduledoc \"\"\"\n Tests for `StringUtils.truncate/3`.\n\n ## Examples\n\n iex> StringUtils.truncate(\"hello\", 3)\n \"hel...\"\n \"\"\"\n\n describe \"truncate/3 happy path\" do\n test \"returns a string shorter than the limit unchanged\" do\n assert StringUtils.truncate(\"hello\", 10) == \"hello\"\n end\n\n test \"returns a string at the exact limit unchanged\" do\n assert StringUtils.truncate(\"hello\", 5) == \"hello\"\n end\n\n test \"appends the default suffix when truncating\" do\n assert StringUtils.truncate(\"hello world\", 5) == \"hello...\"\n end\n\n test \"supports a custom suffix\" do\n assert StringUtils.truncate(\"hello world\", 5, \"…\") == \"hello…\"\n end\n\n test \"counts Unicode characters rather than bytes\" do\n result = StringUtils.truncate(\"hΓ©llo 🌍\", 5)\n\n assert \"hΓ©llo...\" = result\n end\n end\n\n describe \"truncate/3 edge and error cases\" do\n test \"returns an empty string unchanged\" do\n assert StringUtils.truncate(\"\", 5) == \"\"\n end\n\n test \"truncates a non-empty string when max_len is zero\" do\n assert StringUtils.truncate(\"hello\", 0) == \"...\"\n end\n\n test \"returns an empty string when max_len is zero\" do\n assert StringUtils.truncate(\"\", 0) == \"\"\n end\n\n test \"raises ArgumentError for a negative max_len\" do\n assert_raise ArgumentError, fn ->\n StringUtils.truncate(\"hello\", -1)\n end\n end\n end\nend"}]},{"name":"api-client","description":"Class design + error handling + type hints + docs","correctness":9,"quality":8,"documentation":9,"total":8.7,"speed_ms":14062,"notes":"The implementation cleanly satisfies the core requirements with proper error handling, typing, sessions, and docstrings, though it lacks timeout/session-closing support and the example may fail against the placeholder URL without catching transport errors.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":9,"quality":8,"documentation":9,"total":8.7,"speed_ms":14062,"notes":"The implementation cleanly satisfies the core requirements with proper error handling, typing, sessions, and docstrings, though it lacks timeout/session-closing support and the example may fail against the placeholder URL without catching transport errors.","response":"```python\nfrom typing import Any, Mapping, Optional\n\nimport requests\n\n\nclass APIError(Exception):\n \"\"\"Raised when the API returns a non-successful HTTP response.\"\"\"\n\n def __init__(self, status_code: int, message: str) -> None:\n \"\"\"Initialize the error with an HTTP status code and message.\"\"\"\n self.status_code = status_code\n self.message = message\n super().__init__(f\"API request failed ({status_code}): {message}\")\n\n\nclass APIClient:\n \"\"\"A lightweight client for making authenticated REST API requests.\n\n Args:\n base_url: The root URL of the API, such as\n ``\"https://api.example.com/v1\"``.\n api_key: Optional bearer token used to authenticate requests.\n \"\"\"\n\n def __init__(self, base_url: str, api_key: Optional[str] = None) -> None:\n \"\"\"Initialize the API client.\n\n Args:\n base_url: The root URL of the API.\n api_key: Optional bearer token used for authentication.\n \"\"\"\n self.base_url = base_url.rstrip(\"/\")\n self.session = requests.Session()\n\n if api_key is not None:\n self.session.headers[\"Authorization\"] = f\"Bearer {api_key}\"\n\n self.session.headers[\"Accept\"] = \"application/json\"\n\n def get(\n self,\n path: str,\n params: Optional[Mapping[str, Any]] = None,\n ) -> Any:\n \"\"\"Send a GET request to an API endpoint.\n\n Args:\n path: Endpoint path relative to the base URL.\n params: Optional query-string parameters.\n\n Returns:\n The decoded JSON response.\n\n Raises:\n APIError: If the API returns a non-2xx response.\n requests.RequestException: If the request cannot be completed.\n requests.JSONDecodeError: If the response is not valid JSON.\n \"\"\"\n response = self.session.get(self._url(path), params=params)\n self._raise_for_api_error(response)\n return response.json()\n\n def post(self, path: str, data: Mapping[str, Any]) -> Any:\n \"\"\"Send a POST request containing JSON data to an API endpoint.\n\n Args:\n path: Endpoint path relative to the base URL.\n data: Mapping to serialize as the JSON request body.\n\n Returns:\n The decoded JSON response.\n\n Raises:\n APIError: If the API returns a non-2xx response.\n requests.RequestException: If the request cannot be completed.\n requests.JSONDecodeError: If the response is not valid JSON.\n \"\"\"\n response = self.session.post(self._url(path), json=data)\n self._raise_for_api_error(response)\n return response.json()\n\n def _url(self, path: str) -> str:\n return f\"{self.base_url}/{path.lstrip('/')}\"\n\n @staticmethod\n def _raise_for_api_error(response: requests.Response) -> None:\n if 200 <= response.status_code < 300:\n return\n\n try:\n payload = response.json()\n message = (\n payload.get(\"message\", response.text)\n if isinstance(payload, dict)\n else response.text\n )\n except requests.exceptions.JSONDecodeError:\n message = response.text\n\n raise APIError(response.status_code, message or response.reason)\n\n\n# Usage example\nif __name__ == \"__main__\":\n client = APIClient(\n base_url=\"https://api.example.com/v1\",\n api_key=\"your-api-key\",\n )\n\n try:\n users = client.get(\"/users\", params={\"limit\": 10})\n created_user = client.post(\n \"/users\",\n data={\"name\": \"Ada Lovelace\", \"email\": \"ada@example.com\"},\n )\n print(users)\n print(created_user)\n except APIError as error:\n print(f\"Request failed: {error}\")\n```"}]},{"name":"go-test","description":"Idiomatic Go table-driven tests + benchmark","correctness":10,"quality":10,"documentation":6,"total":8.7,"speed_ms":13067,"notes":"The tests and benchmark fully satisfy the functional and structural requirements with idiomatic Go, but both exported test functions lack the requested explanatory comments.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":10,"quality":10,"documentation":6,"total":8.7,"speed_ms":13067,"notes":"The tests and benchmark fully satisfy the functional and structural requirements with idiomatic Go, but both exported test functions lack the requested explanatory comments.","response":"```go\npackage wordcount\n\nimport (\n\t\"reflect\"\n\t\"testing\"\n)\n\nfunc TestWordCount(t *testing.T) {\n\ttests := []struct {\n\t\tname, input string\n\t\twant map[string]int\n\t}{\n\t\t{\n\t\t\tname: \"empty string\",\n\t\t\tinput: \"\",\n\t\t\twant: map[string]int{},\n\t\t},\n\t\t{\n\t\t\tname: \"single word\",\n\t\t\tinput: \"hello\",\n\t\t\twant: map[string]int{\"hello\": 1},\n\t\t},\n\t\t{\n\t\t\tname: \"mixed case\",\n\t\t\tinput: \"Go go GO\",\n\t\t\twant: map[string]int{\"go\": 3},\n\t\t},\n\t\t{\n\t\t\tname: \"punctuation\",\n\t\t\tinput: \"Hello, world! How are you?\",\n\t\t\twant: map[string]int{\n\t\t\t\t\"hello\": 1,\n\t\t\t\t\"world\": 1,\n\t\t\t\t\"how\": 1,\n\t\t\t\t\"are\": 1,\n\t\t\t\t\"you\": 1,\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tname: \"repeated words\",\n\t\t\tinput: \"one two one three two one\",\n\t\t\twant: map[string]int{\n\t\t\t\t\"one\": 3,\n\t\t\t\t\"two\": 2,\n\t\t\t\t\"three\": 1,\n\t\t\t},\n\t\t},\n\t\t{\n\t\t\tname: \"multi-word sentence\",\n\t\t\tinput: \"The quick brown fox jumps over the lazy dog\",\n\t\t\twant: map[string]int{\n\t\t\t\t\"the\": 2,\n\t\t\t\t\"quick\": 1,\n\t\t\t\t\"brown\": 1,\n\t\t\t\t\"fox\": 1,\n\t\t\t\t\"jumps\": 1,\n\t\t\t\t\"over\": 1,\n\t\t\t\t\"lazy\": 1,\n\t\t\t\t\"dog\": 1,\n\t\t\t},\n\t\t},\n\t}\n\n\tfor _, tc := range tests {\n\t\tt.Run(tc.name, func(t *testing.T) {\n\t\t\tgot := WordCount(tc.input)\n\t\t\tif !reflect.DeepEqual(got, tc.want) {\n\t\t\t\tt.Errorf(\"WordCount(%q) = %v; want %v\", tc.input, got, tc.want)\n\t\t\t}\n\t\t})\n\t}\n}\n\nfunc BenchmarkWordCount(b *testing.B) {\n\tconst input = \"The quick brown fox jumps over the lazy dog, while another fox watches from the quiet forest.\"\n\n\tfor i := 0; i < b.N; i++ {\n\t\tWordCount(input)\n\t}\n}\n```"}]},{"name":"doom","description":"DDA raycaster + textures + door + minimap + Z-buffer β€” the signature challenge","correctness":8.7,"quality":9.1,"documentation":8.4,"total":8.7,"speed_ms":93970,"notes":"Strong real DDA raycasting, corrected textured walls, shading, doors, HUD, and delta-time controls, but the spawn faces east rather than toward its first southbound corridor, collision is circle-versus-cell rather than the requested AABB, and the claimed secret area is not meaningfully secret.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":8.7,"quality":9.1,"documentation":8.4,"total":8.7,"speed_ms":93970,"notes":"Strong real DDA raycasting, corrected textured walls, shading, doors, HUD, and delta-time controls, but the spawn faces east rather than toward its first southbound corridor, collision is circle-versus-cell rather than the requested AABB, and the claimed secret area is not meaningfully secret.","response":"\n\n\n\n\nDDA Raycaster\n\n\n\n\n\n\n"}]},{"name":"test-writing","description":"Edge-case thinking + pytest parametrize discipline","correctness":8,"quality":9,"documentation":8,"total":8.3,"speed_ms":10947,"notes":"The suite is accurate, concise, and covers many useful boundary and invalid inputs, but it misses a dedicated valid minutes-plus-seconds case and lacks explicit meaningful parametrization IDs.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":8,"quality":9,"documentation":8,"total":8.3,"speed_ms":10947,"notes":"The suite is accurate, concise, and covers many useful boundary and invalid inputs, but it misses a dedicated valid minutes-plus-seconds case and lacks explicit meaningful parametrization IDs.","response":"```python\nimport pytest\n\nfrom your_module import parse_duration\n\n\n@pytest.mark.parametrize(\n (\"duration\", \"expected_seconds\"),\n [\n (\"1h\", 3600),\n (\"30m\", 1800),\n (\"45s\", 45),\n (\"1h30m\", 5400),\n (\"2h15m30s\", 8130),\n (\"0h\", 0),\n (\"0m\", 0),\n (\"0s\", 0),\n (\"24h60m60s\", 90060),\n (\"001h002m003s\", 3723),\n (\" 1h30m \", 5400),\n (\"\\t45s\\n\", 45),\n ],\n)\ndef test_parse_duration_returns_total_seconds(duration, expected_seconds):\n assert parse_duration(duration) == expected_seconds\n\n\n@pytest.mark.parametrize(\n \"invalid_duration\",\n [\n \"\",\n \" \",\n \"1\",\n \"1d\",\n \"1H\",\n \"30M\",\n \"45S\",\n \"1h 30m\",\n \"30m1h\",\n \"1h30m45s extra\",\n \"-1h\",\n \"+30m\",\n \"1.5h\",\n \"hm\",\n \"abc\",\n ],\n)\ndef test_parse_duration_rejects_unrecognized_input(invalid_duration):\n with pytest.raises(ValueError, match=r\"^Invalid duration:\"):\n parse_duration(invalid_duration)\n```"}]},{"name":"slots","description":"Vegas slot machine β€” reels, pay table, betting, win animations","correctness":9,"quality":9,"documentation":7,"total":8.3,"speed_ms":96667,"notes":"The game is complete and mechanically correct, including payout precedence, reel order, state handling, effects, and credit animation, but reel timings and animation intervals are embedded as literals rather than named constants and the code has little explicit documentation.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":9,"quality":9,"documentation":7,"total":8.3,"speed_ms":96667,"notes":"The game is complete and mechanically correct, including payout precedence, reel order, state handling, effects, and credit animation, but reel timings and animation intervals are embedded as literals rather than named constants and the code has little explicit documentation.","response":"\n\n\n\n\nLucky Sevens\n\n\n\n
\n

LUCKY 7s

\n

VEGAS SLOTS

\n\n
\n
\n
\n\n
\n
\n
\n
\n
\n\n
PLACE YOUR BET
\n
CREDITS: 100
\n\n
\n
\n \n \n \n
\n \n
\n
\n\n \n
\n
\n\n\n\n\n\n"}]},{"name":"async-fetch","description":"Async concurrency patterns + retry/timeout handling","correctness":9,"quality":8,"documentation":2,"total":6.3,"speed_ms":18006,"notes":"Concurrency, per-attempt timeouts, retry targeting, exponential backoff, shared session use, and runnable main are strong, but dict types are unparameterized, Optional is unused, and the docstring omits required parameter, units, result-key, and backoff documentation.","stddev":null,"runs":1,"trace_id":null,"run_details":[{"run":1,"correctness":9,"quality":8,"documentation":2,"total":6.3,"speed_ms":18006,"notes":"Concurrency, per-attempt timeouts, retry targeting, exponential backoff, shared session use, and runnable main are strong, but dict types are unparameterized, Optional is unused, and the docstring omits required parameter, units, result-key, and backoff documentation.","response":"```python\nimport asyncio\nfrom typing import Optional\n\nimport aiohttp\n\n\nasync def fetch_all(\n urls: list[str],\n timeout: float = 5.0,\n retries: int = 2,\n) -> list[dict]:\n \"\"\"Fetch URLs concurrently with per-request timeouts and retries.\"\"\"\n\n async def fetch_one(session: aiohttp.ClientSession, url: str) -> dict:\n attempts = max(0, retries) + 1\n\n for attempt in range(attempts):\n try:\n async with session.get(\n url,\n timeout=aiohttp.ClientTimeout(total=timeout),\n ) as response:\n body = await response.text()\n\n if response.status < 500:\n return {\n \"url\": url,\n \"status\": response.status,\n \"body\": body,\n \"error\": None,\n }\n\n if attempt == attempts - 1:\n return {\n \"url\": url,\n \"status\": response.status,\n \"body\": body,\n \"error\": f\"HTTP {response.status}\",\n }\n\n except (aiohttp.ClientError, asyncio.TimeoutError) as exc:\n if attempt == attempts - 1:\n return {\n \"url\": url,\n \"status\": None,\n \"body\": None,\n \"error\": str(exc) or type(exc).__name__,\n }\n except Exception as exc:\n return {\n \"url\": url,\n \"status\": None,\n \"body\": None,\n \"error\": str(exc) or type(exc).__name__,\n }\n\n await asyncio.sleep(0.5 * (2**attempt))\n\n return {\n \"url\": url,\n \"status\": None,\n \"body\": None,\n \"error\": \"Unknown error\",\n }\n\n try:\n async with aiohttp.ClientSession() as session:\n tasks = [fetch_one(session, url) for url in urls]\n return await asyncio.gather(*tasks)\n except Exception as exc:\n error: str = str(exc) or type(exc).__name__\n return [\n {\"url\": url, \"status\": None, \"body\": None, \"error\": error}\n for url in urls\n ]\n\n\nif __name__ == \"__main__\":\n async def main() -> None:\n urls: list[str] = [\n \"https://example.com\",\n \"https://httpbin.org/status/500\",\n \"https://httpbin.org/get\",\n ]\n results: list[dict] = await fetch_all(urls)\n for result in results:\n print(result)\n\n asyncio.run(main())\n```"}]}]}],"benchmark_version":"legacy-thirteen-task","status":"archived","unit":"llm_judged_score_out_of_10","current_results_url":"https://bench.mager.co/api/v1.1/results"}