test-writing

Write pytest tests for a provided function — tests edge-case thinking and assertion quality

the prompt

$ cat test-writing.prompt

Write a comprehensive pytest test suite for the following Python function:

```python
def parse_duration(s: str) -> int:
    """Parse a human duration string into total seconds.

    Accepted formats: '1h', '30m', '45s', '1h30m', '2h15m30s'.
    Raises ValueError on unrecognized input.
    """
    import re
    pattern = r'^(?:(\d+)h)?(?:(\d+)m)?(?:(\d+)s)?$'
    m = re.fullmatch(pattern, s.strip())
    if not m or not s.strip():
        raise ValueError(f'Invalid duration: {s!r}')
    h, mi, sec = (int(x) if x else 0 for x in m.groups())
    return h * 3600 + mi * 60 + sec
```

Requirements:
- Use pytest (plain functions, no unittest classes)
- Use `@pytest.mark.parametrize` for the happy-path cases
- Cover at least 3 edge/error cases with `pytest.raises`
- Name tests descriptively so failures are self-explaining
- No mocking needed — the function is pure
- Do not reimplement the function; test it as a black box

rubric

correctness

Do the parametrized cases cover all documented formats (h, m, s, hm, hms, ms)? Are the expected values arithmetically correct? Do error cases actually trigger ValueError?

quality

Is parametrize used instead of copy-paste test functions? Are test names descriptive? Are assertions tight (not just 'assert result')? Is there any unnecessary complexity?

documentation

Do test names read like specs ('test_hours_only', 'test_invalid_raises')? Are fixture/parametrize IDs meaningful? Any comments where the tested behavior is non-obvious?

results

GPT-OSS 120B

6.3

correctness 5.0quality 7.0documentation 7.02854ms

# Happy-path parametrization is thorough and arithmetically correct, and most error cases correctly raise ValueError, but the added test_parse_duration_non_string_input_raises_type_error is factually wrong—passing an int would raise AttributeError (since s.strip() is called before any type check), not TypeError—so this test would actually fail when run, undermining the suite's correctness despite otherwise good structure and naming.

inspect full trace →

Llama 3.3 70B

6.3

correctness 7.0quality 6.0documentation 6.01103ms

# Covers h, m, s, hm, hms and a zero-case, but omits the standalone 'ms' format explicitly required by the rubric; error cases are adequate (empty, invalid char, garbage string) though lack tests for malformed ordering; whitespace test is a nice touch but generic test names and lack of parametrize ids weaken documentation quality.

inspect full trace →

Claude Sonnet 4.6

5.3

correctness 3.0quality 5.0documentation 8.028558ms

# Well-designed, thorough parametrized happy-path and error tests with accurate arithmetic and clear naming, but the response is truncated mid-function ('with pyt'), producing a syntax error that would prevent the test suite from even being collected.

inspect full trace →

Claude Haiku 4.5

5.0

correctness 4.0quality 5.0documentation 6.011766ms

# Good format coverage and error cases, but the response is cut off mid-docstring leaving an unterminated string/incomplete class, making the file a syntax error and thus non-runnable, and it also disregards the 'plain functions, no classes' requirement by wrapping everything in three TestX classes.

inspect full trace →

Gemini 2.5 Flash

4.3

correctness 3.0quality 4.0documentation 6.030096ms

# Good coverage of formats and edge cases with meaningful parametrize IDs and docstrings, but the response is cut off mid-function with an unterminated docstring, leaving invalid Python that would fail to even import/run.

inspect full trace →