Skip to content
mager-bench1.3
← historical challenges

readme-writer

Write a README for a CLI tool — tests documentation ability directly

the prompt

$ cat readme-writer.prompt
Write a README.md for a command-line tool called `snapdiff` that compares two directories and reports added, removed, and changed files. The tool is written in Python, installable via pip. Include: a one-line description, installation instructions, usage examples with flags (--ignore-hidden, --output json|text, --depth N), output format explanation, and a short 'How it works' section. Use proper Markdown formatting.

rubric

correctness

Does the README cover all the required sections? Are the flags documented correctly?

quality

Is the README well-structured and scannable? Would a new user understand the tool from this alone?

documentation

Is the Markdown formatting correct? Are code blocks used for commands? Is the writing clear and concise?

results

GPT-6 Astra

10.0
correctness 10.0quality 10.0documentation 10.020959ms

# The README fully covers every requested section and flag with clear examples, accurate output explanations, concise prose, and polished, scannable Markdown.

inspect full trace →

GPT-5.6 Sol

9.7
correctness 10.0quality 9.0documentation 10.012681ms

# The README fully covers every required section and flag with clear examples, accurate output semantics, concise explanations, and polished Markdown structure.

inspect full trace →