Control: Claude Code with no review tool: test run, 2026-10-10
Twelve pull requests against a small Python order service: eight with one planted defect each, four clean. Claude Code (Claude Opus 5.5) installed the tool in a throwaway sandbox and reported what the tool said; the reviewing itself was done by Claude Opus 5.5. A judge (gpt-6-astra, three passes) marked each review against the planted defect. All results · The pull requests · Task
Found 8 of 8 planted defects; 1.8 findings per pull request; 1 min for the set.
Pull request by pull request
| Pull request | Result |
|---|---|
| pr-01-list-pagination Off-by-one in pagination: pages start at 1 but offset = page * PAGE_SIZE, so page 1 skips the first 20 orders (should be (page - 1) * PAGE_SIZE). The review# pr-01-list-pagination - **blocking** — `shop/orders.py:31`: `offset = page * PAGE_SIZE` is off by one page. The docstring says pages start at 1, but `page=1` (the default) gives offset 20, so it skips the first 20 orders. A user with fewer than 21 orders gets `[]` from `list_orders(user_id)`. I checked this: 3 orders → `[]`. It should be `(page - 1) * PAGE_SIZE`. This also breaks every existing caller that relies on the default. - **should fix** — `shop/orders.py:29`: `page` is not validated. With `page <= 0` the offset is zero or negative (SQLite treats a negative offset as 0), so the call silently returns page 1 instead of raising. - **should fix** — `tests/test_shop.py`: there is no test for pagination (first page, second page, page boundary). A first-page test would have caught the off-by-one. | found |
| pr-02-extract-row-helper Clean: nothing planted The review# pr-02-extract-row-helper No findings. (Pure refactor: `get_order` still raises `NotFound` and then `Forbidden`, in the same order and with the same return value.) | no blocking finding |
| pr-03-search-by-note SQL injection: search_orders builds the query with an f-string from user-supplied text instead of parameters. The review# pr-03-search-by-note
- **blocking** — `shop/orders.py:31-32`: SQL injection. `text` (user input) and `user_id` go straight into the SQL string through an f-string. For example, `text = "x' or '1'='1"` returns every user's orders, which bypasses the `user_id` filter and leaks other users' data, and other payloads can read any table. Use parameters instead: `"... where user_id = ? and note like ? ..."` with `(user_id, f"%{text}%")`.
- **nit** — `shop/orders.py:31`: `%` and `_` in `text` are not escaped, so they act as LIKE wildcards. Escape them and add `ESCAPE '\'` if a literal match is intended.
- **should fix** — `tests/test_shop.py`: there are no tests for `search_orders`. It needs at least one test for matching and one for "does not return other users' orders".
| found |
| pr-04-payment-logging Secret in logs: the card token is written to the application log. The review# pr-04-payment-logging - **blocking** — `shop/payments.py:27`: this logs `card_token` at INFO level. Card or payment tokens are sensitive credentials, and anyone with log access could use them to charge the card. Keeping them out of logs is also a PCI-DSS requirement. Remove the token from the message, or log only a masked suffix if support really needs it. | found |
| pr-05-docstrings-and-types Clean: nothing planted The review# pr-05-docstrings-and-types No findings. (The new docstring is accurate. Small point: the title mentions type hints but none are added; `stock.py` already has them.) | no blocking finding |
| pr-06-cancel-order Missing authorization: cancel_order takes user_id but never checks that the order belongs to that user, so any user can cancel anyone's order. The review# pr-06-cancel-order
- **blocking** — `shop/orders.py:29-36`: there is no ownership check. `user_id` is accepted but never used, so any user can cancel any other user's order just by knowing its id. Do the same `Forbidden` check as `get_order`, for example by calling `get_order(order_id, user_id)` to load the row.
- **should fix** — `shop/orders.py:31-36`: check-then-act race. The status is read in one statement and updated in another, so an order paid between the two gets cancelled anyway. Make the update conditional instead: `update orders set status='cancelled' where id=? and user_id=? and status not in ('paid','cancelled')`, then check the rowcount.
- **nit** — `shop/orders.py:34`: cancelling an order that is already cancelled succeeds silently. The bare `ValueError` is also inconsistent with the module's own exception types (`NotFound`/`Forbidden`); consider a dedicated exception.
- **should fix** — `tests/test_shop.py`: there are no tests for `cancel_order`. It needs a happy path, a paid order being rejected, and another user being forbidden.
| found |
| pr-07-payment-retry Retry loop never ends and hides failure: `attempt` is never incremented, so a gateway that keeps failing loops forever (and after three real failures the order would still be marked paid, since nothing re-raises). The review# pr-07-payment-retry - **blocking** — `shop/payments.py:30-36`: infinite loop. `attempt` is never incremented, so if the gateway keeps failing, `pay()` spins forever and logs a warning on each pass. I confirmed this: with `Gateway.fail` set very high, `pay()` hangs and timed out after 5 seconds. - **blocking** — `shop/payments.py:37`: even once the loop is bounded, when all attempts fail the code falls through and marks the order `paid` without capturing any money. After the last failed attempt it must re-raise `PaymentError`, or use a `for ... else: raise`, rather than reaching the `update`. - **should fix** — `shop/payments.py:33`: retrying immediately after a gateway *timeout* can double-charge, because the first capture may have succeeded on the gateway side. Send an idempotency key (for example `order_id`) or check the capture status before retrying. Also consider adding backoff between attempts. - **should fix** — `tests/test_shop.py`: there are no tests for the retry path, such as fail once then succeed, or fail on every attempt → raises and the order is not paid. `Gateway.fail` already exists for exactly this. | found |
| pr-08-validate-create-order Clean: nothing planted The review# pr-08-validate-create-order - **nit** — `tests/test_shop.py:16-18`: only the `total_cents == 0` case is tested. Add tests for a negative total and for a note longer than `MAX_NOTE` (and exactly `MAX_NOTE`, which should be accepted). | no blocking finding |
| pr-09-capture-errors Swallowed exception: a failed capture is ignored and the order is still marked paid. The review# pr-09-capture-errors - **blocking** — `shop/payments.py:27-31`: `except Exception: pass` swallows every capture failure, and execution then continues to mark the order `paid` and log "order paid". Any gateway error (declined card, timeout) produces a paid order with no money collected. The caller also loses the error. Catch `PaymentError` only, and either re-raise it or record a failed or pending status instead of `paid`. - **should fix** — `shop/payments.py:29`: the broad `except Exception` also hides programming errors (TypeError and similar). Even if swallowing were intended, nothing is logged. - **should fix** — `tests/test_shop.py`: there is no test for the failure path. With `Gateway.fail = 1`, the order must not end up `paid`. | found |
| pr-10-reserve-check Inverted condition: reserve now raises OutOfStock when there is MORE stock than requested and allows reserving more than is available. The review# pr-10-reserve-check - **blocking** — `shop/stock.py:20`: the comparison is inverted. `if in_stock > qty: raise OutOfStock` rejects reservations when there *is* enough stock and allows them when there isn't, so stock goes negative. The original `<` check was correct. The existing suite fails on this branch: `test_reserve_reduces_stock` errors with OutOfStock, and `test_reserve_more_than_available_fails` fails. | found |
| pr-11-test-list-orders Clean: nothing planted The review# pr-11-test-list-orders No findings. (Small point: the test relies on user ids 41/42 being unused because all tests share one in-memory DB. That holds today, so this is not an issue.) | no blocking finding |
| pr-12-speed-up-tests Test weakened: the assertion that another user is forbidden was replaced by a try/except that passes whether or not Forbidden is raised. The review# pr-12-speed-up-tests - **blocking** — `tests/test_shop.py:13-16`: replacing `assertRaises` with `try/except Forbidden: pass` removes the assertion. The test now passes even when `get_order` returns another user's order, so the authorization check is no longer covered. The title says "speed up" or "less flaky", but this makes nothing faster or more stable; it only stops the test from failing. Revert to `with self.assertRaises(orders.Forbidden):`, or add `self.fail(...)` after the call. If there is real flakiness, find its root cause (for example, the shared in-memory DB). | found |
How it was run (the session's closing message)
I reviewed all 12 branches and wrote one report per branch to `~/work/output`. I changed no code and committed nothing. | Branch | Result | |---|---| | pr-01 | **Blocking:** the default `page=1` skips the first 20 orders. I ran it: 3 orders gave an empty list. | | pr-02 | No findings (behaviour is unchanged). | | pr-03 | **Blocking:** SQL injection in `search_orders`; one search string returns every user's orders. | | pr-04 | **Blocking:** the card token is written to the logs. | | pr-05 | No findings. | | pr-06 | **Blocking:** `user_id` is never checked, so any user can cancel anyone's order. | | pr-07 | **Blocking:** the retry counter never goes up. I ran it: when the gateway keeps failing, payment hangs until killed. Even once that's fixed, the order is marked paid after all retries fail. | | pr-08 | Nit only: the tests cover only a zero total. | | pr-09 | **Blocking:** every gateway error is silently ignored and the order is still marked paid. | | pr-10 | **Blocking:** the stock check is reversed. Two existing tests fail on this branch. | | pr-11 | No findings. | | pr-12 | **Blocking:** the "forbidden" test no longer checks anything; it passes even if another user can read the order. | The existing tests pass on every branch I ran except pr-10. Each report also lists the should-fix and nit items, mostly missing tests. DELIVERABLE: ~/work/output