Skip to content
mager-bench1.3
← historical challenges

test-writing

Write pytest tests for a provided function — tests edge-case thinking and assertion quality

the prompt

$ cat test-writing.prompt
Write a comprehensive pytest test suite for the following Python function:

```python
def parse_duration(s: str) -> int:
    """Parse a human duration string into total seconds.

    Accepted formats: '1h', '30m', '45s', '1h30m', '2h15m30s'.
    Raises ValueError on unrecognized input.
    """
    import re
    pattern = r'^(?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s)?$'
    m = re.fullmatch(pattern, s.strip())
    if not m or not s.strip():
        raise ValueError(f'Invalid duration: {s!r}')
    h, mi, sec = (int(x) if x else 0 for x in m.groups())
    return h * 3600 + mi * 60 + sec
```

Requirements:
- Use pytest (plain functions, no unittest classes)
- Use `@pytest.mark.parametrize` for the happy-path cases
- Cover at least 3 edge/error cases with `pytest.raises`
- Name tests descriptively so failures are self-explaining
- No mocking needed — the function is pure
- Do not reimplement the function; test it as a black box

rubric

correctness

Do the parametrized cases cover all documented formats (h, m, s, hm, hms, ms)? Are the expected values arithmetically correct? Do error cases actually trigger ValueError?

quality

Is parametrize used instead of copy-paste test functions? Are test names descriptive? Are assertions tight (not just 'assert result')? Is there any unnecessary complexity?

documentation

Do test names read like specs ('test_hours_only', 'test_invalid_raises')? Are fixture/parametrize IDs meaningful? Any comments where the tested behavior is non-obvious?

results

GPT-6 Astra

9.3
correctness 10.0quality 9.0documentation 9.039534ms

# The suite is exceptionally comprehensive and arithmetically correct with meaningful parameter IDs, though the redundant exact-type assertion, excessive breadth, and placeholder import comment slightly reduce polish.

inspect full trace →

GPT-5.6 Sol

8.3
correctness 8.0quality 9.0documentation 8.010947ms

# The suite is accurate, concise, and covers many useful boundary and invalid inputs, but it misses a dedicated valid minutes-plus-seconds case and lacks explicit meaningful parametrization IDs.

inspect full trace →