awesome-skills/code-review-skill: test run, 2026-10-10
Twelve pull requests against a small Python order service: eight with one planted defect each, four clean. Claude Code (Claude Opus 5.5) installed the tool in a throwaway sandbox and reported what the tool said; the reviewing itself was done by Claude Opus 5.5. A judge (gpt-6-astra, three passes) marked each review against the planted defect. Repo commit 4850184dfd77. All results · The pull requests · Task
Found 8 of 8 planted defects; 1.8 findings per pull request; 1 min for the set.
Pull request by pull request
| Pull request | Result |
|---|---|
| pr-01-list-pagination Off-by-one in pagination: pages start at 1 but offset = page * PAGE_SIZE, so page 1 skips the first 20 orders (should be (page - 1) * PAGE_SIZE). The review# pr-01-list-pagination — Add pagination to list_orders Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 5 passed. Size: XS (+5/-2). ## Findings 1. 🔴 **[blocking]** Off-by-one in offset — `shop/orders.py:31` Docstring says pages start at 1, but `offset = page * PAGE_SIZE` makes page 1 skip the first 20 orders; the user's first page can never be returned. Should be `(page - 1) * PAGE_SIZE`. 2. 🟡 **[important]** No validation of `page` — `shop/orders.py:29` `page <= 0` yields a zero/negative offset with no error; validate `page >= 1`. 3. 🟡 **[important]** No tests for pagination — `tests/test_shop.py` No test covers page 1 / page boundaries, which would have caught the off-by-one. **Verdict:** 🔄 Request Changes | found |
| pr-02-extract-row-helper Clean: nothing planted The review# pr-02-extract-row-helper — Extract the order row lookup into a helper Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 5 passed. Size: XS (+8/-3). No findings. (Behaviour-preserving refactor: NotFound and Forbidden checks are kept in the same order in `get_order`, `shop/orders.py:27-31`.) **Verdict:** ✅ Approve | no blocking finding |
| pr-03-search-by-note SQL injection: search_orders builds the query with an f-string from user-supplied text instead of parameters. The review# pr-03-search-by-note — Search a user's orders by note text
Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion).
Tests on branch: 5 passed. Size: XS (+6/-0).
## Findings
1. 🔴 **[blocking]** SQL injection — `shop/orders.py:31`
`user_id` and `text` are interpolated into the SQL with an f-string. A `text` such as `%' or '1'='1` returns every user's orders (breaks the per-user restriction). Use placeholders: `... where user_id = ? and note like ? ...` with `(user_id, f"%{text}%")`.
2. 🟢 **[nit]** LIKE wildcards not escaped — `shop/orders.py:31`
`%` and `_` in the search text act as wildcards; escape them (with `ESCAPE`) if literal matching is intended.
3. 🟡 **[important]** No tests for `search_orders` — `tests/test_shop.py`
Add tests for matching, the user-scoping, and a malicious input.
**Verdict:** 🔄 Request Changes
| found |
| pr-04-payment-logging Secret in logs: the card token is written to the application log. The review# pr-04-payment-logging — Log payment attempts for support Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 5 passed. Size: XS (+1/-0). ## Findings 1. 🔴 **[blocking]** Sensitive data written to logs — `shop/payments.py:27` The card token is logged in plain text at INFO level. Card tokens are payment credentials and must not appear in logs (PCI). Drop it or log only a masked suffix. **Verdict:** 🔄 Request Changes | found |
| pr-05-docstrings-and-types Clean: nothing planted The review# pr-05-docstrings-and-types — Add docstrings and type hints to stock Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 5 passed. Size: XS (+1/-0). ## Findings 1. 🟢 **[nit]** PR title does not match the change — `shop/stock.py:10` Title says "docstrings and type hints", but the diff only adds one docstring (to `set_stock`); no type hints are added and `available`/`reserve` get no docstrings. Update the title or finish the change. The docstring itself is accurate. **Verdict:** ✅ Approve (non-blocking nit) | no blocking finding |
| pr-06-cancel-order Missing authorization: cancel_order takes user_id but never checks that the order belongs to that user, so any user can cancel anyone's order. The review# pr-06-cancel-order — Let users cancel an order Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 5 passed. Size: XS (+10/-0). ## Findings 1. 🔴 **[blocking]** Missing authorization check (IDOR) — `shop/orders.py:29-36` `user_id` is accepted but never used: any user can cancel any other user's order by ID. Check ownership the way `get_order` does (raise `Forbidden`), e.g. reuse `get_order(order_id, user_id)`. 2. 🟡 **[important]** Check-then-update race (TOCTOU) — `shop/orders.py:31-36` Status is read and then updated in a separate statement; an order paid in between gets cancelled. Do it atomically: `update orders set status='cancelled' where id=? and user_id=? and status != 'paid'` and check the affected row count. 3. 🟡 **[important]** No tests for `cancel_order` — `tests/test_shop.py` Cover: cancel own order, cancel paid order fails, cancel another user's order is forbidden, unknown order raises NotFound. **Verdict:** 🔄 Request Changes | found |
| pr-07-payment-retry Retry loop never ends and hides failure: `attempt` is never incremented, so a gateway that keeps failing loops forever (and after three real failures the order would still be marked paid, since nothing re-raises). The review# pr-07-payment-retry — Retry a failed capture up to three times Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 5 passed (no test exercises a failing gateway). Size: XS (+10/-1). ## Findings 1. 🔴 **[blocking]** Infinite loop: `attempt` is never incremented — `shop/payments.py:30-36` On a `PaymentError` the loop repeats forever (a permanently failing gateway hangs checkout). Use `for attempt in range(1, MAX_ATTEMPTS + 1):`. 2. 🔴 **[blocking]** Order marked paid even when all attempts fail — `shop/payments.py:37` Once the loop is bounded, falling out of it after the last failure still runs the `update ... status = 'paid'`. Re-raise `PaymentError` (e.g. `for ... else: raise`) after the final attempt. 3. 🟡 **[important]** No tests for retry behaviour — `tests/test_shop.py` `Gateway.fail` exists for this; add tests for success after N failures and failure after MAX_ATTEMPTS (order must stay unpaid). 4. 💡 **[suggestion]** No backoff between retries — `shop/payments.py:31` Immediate retries against a timing-out gateway; consider a small backoff. **Verdict:** 🔄 Request Changes | found |
| pr-08-validate-create-order Clean: nothing planted The review# pr-08-validate-create-order — Validate input when creating an order Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 6 passed. Size: XS (+11/-0). ## Findings 1. 🟢 **[nit]** Note-length validation is untested — `tests/test_shop.py:16` Only the non-positive total is tested; add a test for a note over `MAX_NOTE` (and one exactly at the limit). 🎉 **[praise]** Fail-fast validation with clear messages and a named constant instead of a magic number. **Verdict:** ✅ Approve | no blocking finding |
| pr-09-capture-errors Swallowed exception: a failed capture is ignored and the order is still marked paid. The review# pr-09-capture-errors — Do not crash checkout when the gateway errors Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 5 passed. Size: XS (+4/-1). ## Findings 1. 🔴 **[blocking]** Swallowed exception marks unpaid orders as paid — `shop/payments.py:27-31` `except Exception: pass` hides every capture failure, then the order is set to `paid` and "order paid" is logged, though no money was captured. Catch `PaymentError` specifically, log it, and leave the order unpaid / surface the error to the caller. 2. 🟡 **[important]** Overly broad `except Exception` — `shop/payments.py:29` Also hides programming errors (TypeError etc.). Catch only the expected exception type. 3. 🟡 **[important]** No test for the gateway-error path — `tests/test_shop.py` Add a test with `Gateway.fail = 1` asserting the order is not marked paid. **Verdict:** 🔄 Request Changes | found |
| pr-10-reserve-check Inverted condition: reserve now raises OutOfStock when there is MORE stock than requested and allows reserving more than is available. The review# pr-10-reserve-check — Tidy the stock reservation check Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: **FAILED** (failures=1, errors=1). Size: XS (+2/-1). ## Findings 1. 🔴 **[blocking]** Inverted comparison — `shop/stock.py:20` `if in_stock > qty: raise OutOfStock` rejects reservations when there is enough stock and allows them when there isn't (stock can go negative). The original was `available(sku) < qty`; restore `if in_stock < qty:`. Both stock tests fail on this branch. **Verdict:** 🔄 Request Changes | found |
| pr-11-test-list-orders Clean: nothing planted The review# pr-11-test-list-orders — Add a test for list_orders Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 6 passed. Size: XS (+5/-0). No findings. (The test uses user IDs 41/42 that no other test touches, so it is safe against the shared in-memory DB.) **Verdict:** ✅ Approve | no blocking finding |
| pr-12-speed-up-tests Test weakened: the assertion that another user is forbidden was replaced by a try/except that passes whether or not Forbidden is raised. The review# pr-12-speed-up-tests — Make the forbidden-order test less flaky Reviewed with code-review-skill (manual review workflow: context → high-level → line-by-line → verdict; severity labels 🔴 blocking / 🟡 important / 🟢 nit / 💡 suggestion). Tests on branch: 5 passed. Size: XS (+3/-1). ## Findings 1. 🔴 **[blocking]** Test no longer asserts anything — `tests/test_shop.py:13-16` Replacing `assertRaises(orders.Forbidden)` with `try/except Forbidden: pass` makes the test pass even when `get_order` returns another user's order, so it no longer guards the authorization check. Restore `with self.assertRaises(orders.Forbidden):` (or add `self.fail(...)` after the call). The test was not flaky; if flakiness was seen, find its real cause. **Verdict:** 🔄 Request Changes | found |
How it was run (the session's closing message)
I reviewed all 12 branches and wrote one report per branch to `~/work/output/<branch>.md`. The installed tool is `code-review-skill`. It doesn't scan code itself; it gives a review method with severity labels (🔴 blocking, 🟡 important, 🟢 nit, 💡 suggestion). So the findings below are from my review done by that method, not output from an automated scanner. As part of it, I ran the skill's `pr-analyzer.py` (it rated every diff "XS") and each branch's unit tests. | Branch | Verdict | Main findings | |---|---|---| | pr-01 | Request changes | 🔴 Page 1 skips the first 20 orders (`shop/orders.py:31`); 🟡 `page` isn't validated; 🟡 no tests | | pr-02 | Approve | No findings | | pr-03 | Request changes | 🔴 SQL injection via f-string (`shop/orders.py:31`); 🟢 LIKE wildcards not escaped; 🟡 no tests | | pr-04 | Request changes | 🔴 Card token written to the logs (`shop/payments.py:27`) | | pr-05 | Approve | 🟢 Title promises type hints, but the diff only adds one docstring | | pr-06 | Request changes | 🔴 `user_id` is ignored, so any user can cancel anyone's order (`shop/orders.py:29-36`); 🟡 status check and update aren't atomic; 🟡 no tests | | pr-07 | Request changes | 🔴 `attempt` is never incremented, so a failing gateway loops forever; 🔴 order is marked paid even after every attempt fails; 🟡 no tests; 💡 no backoff between retries | | pr-08 | Approve | 🟢 Note-length check is untested | | pr-09 | Request changes | 🔴 `except Exception: pass` marks unpaid orders as paid; 🟡 the catch is too broad; 🟡 no test for the error path | | pr-10 | Request changes | 🔴 Stock comparison is reversed (`shop/stock.py:20`); this is the only branch whose tests fail | | pr-11 | Approve | No findings | | pr-12 | Request changes | 🔴 The forbidden-order test no longer asserts anything (`tests/test_shop.py:13-16`) | I didn't change or commit anything in the repo. To run the tests I extracted each branch into temporary folders, and I deleted those afterwards. DELIVERABLE: ~/work/output