nathankim0/clean-architecture-skills: test run, 2026-10-10
Twelve pull requests against a small Python order service: eight with one planted defect each, four clean. Claude Code (Claude Opus 5.5) installed the tool in a throwaway sandbox and reported what the tool said; the reviewing itself was done by Claude Opus 5.5. A judge (gpt-6-astra, three passes) marked each review against the planted defect. Repo commit 725cbeeb12e3. All results · The pull requests · Task
Found 8 of 8 planted defects; 2.7 findings per pull request; 2 min for the set.
Pull request by pull request
| Pull request | Result |
|---|---|
| pr-01-list-pagination Off-by-one in pagination: pages start at 1 but offset = page * PAGE_SIZE, so page 1 skips the first 20 orders (should be (page - 1) * PAGE_SIZE). The review# pr-01-list-pagination Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-01-list-pagination`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | High | kent-beck-style (Rule 1: code must work correctly) | shop/orders.py:31 | `offset = page * PAGE_SIZE` skips the first page. Pages are documented as starting at 1, so `page=1` (the default) returns rows 20–39. Any existing caller of `list_orders(user_id)` with 20 or fewer orders now gets an empty list. Use `(page - 1) * PAGE_SIZE`. | | 2 | Medium | kent-beck-style (Rule 1: passes the tests) | shop/orders.py:29 | The new pagination behaviour has no test. A test that lists page 1 would have caught finding 1. | | 3 | Long-term | clean-architecture (2-2: what data crosses boundaries) | shop/orders.py:32 | `list_orders` returns raw database row tuples straight from `db.query`. Database rows are crossing the boundary instead of a DTO. This was already true before the PR and is now extended to paged results. | ## Kent Beck Style Code Review Report - Overall Code Health: Needs Improvement - Positive: it reuses the existing `PAGE_SIZE` constant (no magic number) and keeps the query parameterized. ## Clean Architecture Review Report - Overall Assessment: Good. The PR adds no new dependency-rule violations. The only item is finding 3, which is long-term. | found |
| pr-02-extract-row-helper Clean: nothing planted The review# pr-02-extract-row-helper Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-02-extract-row-helper`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | Low | kent-beck-style (Rename Method / intention-revealing names) | shop/orders.py:20 | `_row` does not say what it does. Rename it to something like `_fetch_order_row`. | | 2 | Low | kent-beck-style (Primitive Obsession / magic index) | shop/orders.py:29 | `row[1] != user_id` uses a positional tuple index to mean "owner". Name the field, for example with a namedtuple/dataclass or a `OWNER_COLUMN` constant. | ## Kent Beck Style Code Review Report - Overall Code Health: Good - Positive: a clean Extract Method that preserves behaviour (NotFound/Forbidden semantics unchanged; tests pass). ## Clean Architecture Review Report - Overall Assessment: Good. No findings. | no blocking finding |
| pr-03-search-by-note SQL injection: search_orders builds the query with an f-string from user-supplied text instead of parameters. The review# pr-03-search-by-note
Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-03-search-by-note`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term).
## Findings
| # | Priority | Skill | Location | Finding |
|---|---|---|---|---|
| 1 | High | kent-beck-style (Rule 1: code must work correctly) | shop/orders.py:31 | The SQL is built with an f-string that interpolates `user_id` and `text`. Any note search containing `'` breaks the query, and a crafted `text` (e.g. `%' or 1=1 --`) returns other users' orders (SQL injection). Use `?` placeholders as the rest of the module does: `... where user_id = ? and note like ? ...`, with `(user_id, f"%{text}%")`. |
| 2 | Medium | kent-beck-style (Rule 1: passes the tests) | shop/orders.py:29 | The new `search_orders` has no test. |
| 3 | Low | kent-beck-style (Rule 1) | shop/orders.py:31 | The `%` and `_` characters in `text` are not escaped, so they act as LIKE wildcards. |
| 4 | Low | kent-beck-style (Duplicate Code) | shop/orders.py:31 | `select id, status, total_cents from orders where user_id = ... order by id` duplicates the `list_orders` query (line 36). Consider a shared column list/helper. |
| 5 | Medium | clean-architecture (Dependency Rule / DIP) | shop/orders.py:31-32 | The order use-case module now hand-builds raw SQL strings and bypasses the `db` helper's parameter mechanism, which pushes persistence details further into business logic. Long-term, put order queries behind a repository function. |
## Kent Beck Style Code Review Report
- Overall Code Health: Critical (finding 1)
## Clean Architecture Review Report
- Overall Assessment: Needs Improvement (finding 5)
| found |
| pr-04-payment-logging Secret in logs: the card token is written to the application log. The review# pr-04-payment-logging Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-04-payment-logging`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | High | kent-beck-style (Rule 1: code must work correctly) | shop/payments.py:27 | The new log line writes `card_token` to the application log in plain text. Card tokens are payment credentials and must not end up in support logs. Log only the order id and amount, or a masked token. | ## Kent Beck Style Code Review Report - Overall Code Health: Needs Improvement - Positive: it uses lazy `%s` logging arguments. ## Clean Architecture Review Report - Overall Assessment: Good. No findings. | found |
| pr-05-docstrings-and-types Clean: nothing planted The review# pr-05-docstrings-and-types Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-05-docstrings-and-types`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings No findings. (kent-beck-style: the docstring added at shop/stock.py:10 states intent, including the "replacing any earlier value" semantics, so it is not a "comment as smell". clean-architecture: no structural change.) | no blocking finding |
| pr-06-cancel-order Missing authorization: cancel_order takes user_id but never checks that the order belongs to that user, so any user can cancel anyone's order. The review# pr-06-cancel-order Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-06-cancel-order`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | High | kent-beck-style (Rule 1: code must work correctly) | shop/orders.py:29-36 | The `user_id` parameter is never used, so any user can cancel any other user's order. `get_order` enforces ownership (`Forbidden`), but `cancel_order` does not. | | 2 | Medium | kent-beck-style (Duplicate Code) | shop/orders.py:31-33 | It reimplements the order lookup and `NotFound` handling from `get_order`. Reusing `get_order(order_id, user_id)` would remove the duplication and also add the ownership check. | | 3 | Medium | kent-beck-style (Rule 1: passes the tests) | shop/orders.py:29 | There are no tests for cancellation: the happy path, paid orders, and another user's order. | | 4 | Low | kent-beck-style (Primitive Obsession / magic literals) | shop/orders.py:34,36 | The status values `"paid"` and `"cancelled"` are bare string literals. Use named constants or an enum. Only `paid` is guarded, so an already-cancelled order can be "cancelled" again. | | 5 | Low | kent-beck-style (intention-revealing code) | shop/orders.py:35 | The generic `ValueError` is used for a domain rule. A dedicated exception (like `NotFound`/`Forbidden`) would reveal intent. | | 6 | Long-term | clean-architecture (Layer separation) | shop/orders.py:29-36 | The business rule "paid orders cannot be cancelled" is mixed with SQL in one function. Long-term, put the rule on an Order entity and keep persistence in a repository. | ## Kent Beck Style Code Review Report - Overall Code Health: Critical (finding 1) ## Clean Architecture Review Report - Overall Assessment: Needs Improvement (finding 6, long-term) | found |
| pr-07-payment-retry Retry loop never ends and hides failure: `attempt` is never incremented, so a gateway that keeps failing loops forever (and after three real failures the order would still be marked paid, since nothing re-raises). The review# pr-07-payment-retry Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-07-payment-retry`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | High | kent-beck-style (Rule 1: code must work correctly) | shop/payments.py:30-36 | `attempt` is never incremented, so `while attempt <= MAX_ATTEMPTS` never terminates when the gateway keeps failing: an infinite retry loop that hammers the gateway. | | 2 | High | kent-beck-style (Rule 1: code must work correctly) | shop/payments.py:37 | There is no failure path after the retries. Once the loop exits (after fixing finding 1), the order is still marked `paid` even though every capture failed. Re-raise `PaymentError` when attempts are exhausted, e.g. `for attempt in range(1, MAX_ATTEMPTS + 1): ... else: raise`. | | 3 | Medium | kent-beck-style (Rule 1: passes the tests) | shop/payments.py:29 | The retry behaviour is untested: transient failure then success, and permanent failure. `Gateway.fail` already supports both. | | 4 | Low | kent-beck-style (KISS) | shop/payments.py:30-36 | A manual counter plus `while`/`break` is harder to read than a `for ... range` loop with `else`. | | 5 | Long-term | clean-architecture (Layer separation) | shop/payments.py:30-36 | The retry policy is a gateway/infrastructure concern embedded in the payment use case. Consider moving it into the gateway adapter. | ## Kent Beck Style Code Review Report - Overall Code Health: Critical - Positive: `MAX_ATTEMPTS` is a named constant, not a magic number. ## Clean Architecture Review Report - Overall Assessment: Good, except for the long-term item (finding 5). | found |
| pr-08-validate-create-order Clean: nothing planted The review# pr-08-validate-create-order Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-08-validate-create-order`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | Medium | kent-beck-style (Rule 1: passes the tests) | shop/orders.py:21-22 | The note-length validation has no test. Only the non-positive total is tested (tests/test_shop.py:16-18). | | 2 | Low | kent-beck-style (Naming) | shop/orders.py:15 | `MAX_NOTE` does not say what is limited. Rename it to `MAX_NOTE_LENGTH`. | | 3 | Low | kent-beck-style (Primitive Obsession) | shop/orders.py:18-20 | Money is a bare `int` (`total_cents`) validated inline. Consider a small value object (Replace Primitive with Object) so the "positive amount" rule lives in one place. | | 4 | Long-term | clean-architecture (Entities) | shop/orders.py:19-22 | The order invariants (positive total, note length) are enforced in the use-case function next to the SQL insert. Long-term, they belong on an Order entity. | ## Kent Beck Style Code Review Report - Overall Code Health: Good - Positive: guard clauses, a named constant, and a test added with the change. ## Clean Architecture Review Report - Overall Assessment: Good (finding 4 is long-term only) | no blocking finding |
| pr-09-capture-errors Swallowed exception: a failed capture is ignored and the order is still marked paid. The review# pr-09-capture-errors Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-09-capture-errors`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | High | kent-beck-style (Rule 1: code must work correctly) | shop/payments.py:27-31 | `except Exception: pass` swallows every capture failure. Execution then falls through to mark the order `paid` and log "order paid" although no money was captured. It also hides programming errors. Catch `PaymentError` and propagate it, or record a failed status, instead of marking the order paid. | | 2 | Medium | kent-beck-style (Rule 1: passes the tests) | shop/payments.py:26 | There is no test for a failing gateway (`Gateway.fail = 1`). Such a test would show the order wrongly becoming `paid`. | ## Kent Beck Style Code Review Report - Overall Code Health: Critical ## Clean Architecture Review Report - Overall Assessment: Good. No findings. | found |
| pr-10-reserve-check Inverted condition: reserve now raises OutOfStock when there is MORE stock than requested and allows reserving more than is available. The review# pr-10-reserve-check Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-10-reserve-check`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | High | kent-beck-style (Rule 1: passes the tests) | shop/stock.py:20 | The comparison is inverted: `if in_stock > qty: raise OutOfStock`. Reservations now fail whenever there is enough stock and succeed when there is not, which drives `available` negative. The existing suite fails on this branch: `python -m unittest` gives 1 failure and 1 error (test_reserve_reduces_stock, test_reserve_more_than_available_fails). It should be `if in_stock < qty`. A "tidy" refactor must preserve behaviour. | ## Kent Beck Style Code Review Report - Overall Code Health: Critical - Positive: introducing the explaining variable `in_stock` is fine in itself. ## Clean Architecture Review Report - Overall Assessment: Good. No findings. | found |
| pr-11-test-list-orders Clean: nothing planted The review# pr-11-test-list-orders Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-11-test-list-orders`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | Low | kent-beck-style (Magic numbers / intention-revealing code) | tests/test_shop.py:17-18 | User ids `41`/`42` are magic numbers chosen to avoid colliding with the other tests in the shared module-level in-memory DB. Name them, or isolate DB state per test, so the test does not depend on other tests' data. | | 2 | Low | kent-beck-style (Magic index) | tests/test_shop.py:19 | `row[0]` relies on tuple position to mean the order id. | ## Kent Beck Style Code Review Report - Overall Code Health: Good - Positive: a focused test with an intention-revealing name that covers ownership filtering. ## Clean Architecture Review Report - Overall Assessment: Good. No findings. | no blocking finding |
| pr-12-speed-up-tests Test weakened: the assertion that another user is forbidden was replaced by a try/except that passes whether or not Forbidden is raised. The review# pr-12-speed-up-tests Tool: clean-architecture-skills (~/src) — the `kent-beck-style` and `clean-architecture` review skills, applied to `git diff main...pr-12-speed-up-tests`. Priorities are the ones in each skill's report format (High / Medium / Low; Clean Architecture: High / Medium / Long-term). ## Findings | # | Priority | Skill | Location | Finding | |---|---|---|---|---| | 1 | High | kent-beck-style (Rule 1: passes the tests — tests must be meaningful) | tests/test_shop.py:13-16 | Replacing `assertRaises(orders.Forbidden)` with `try/except Forbidden: pass` makes the test pass even if `get_order` returns another user's order. The ownership check is no longer tested. This does not fix flakiness; it removes the assertion. Restore `assertRaises`. | ## Kent Beck Style Code Review Report - Overall Code Health: Critical ## Clean Architecture Review Report - Overall Assessment: Good. No findings. | found |
How it was run (the session's closing message)
I wrote one review report for each of the 12 branches in ~/work/output. Eleven have findings; pr-05 has none. **What the tool is.** The repo in ~/src is a set of two review skills, `kent-beck-style` and `clean-architecture`, both already installed in ~/.claude/skills. There is no program, no MCP server and no LLM call: each skill is a set of written instructions that tells the agent how to review. So I ran both skills and followed them on each `git diff main...<branch>`. The findings are what that review produced, using each skill's report format and priority levels (High / Medium / Low, and Long-term for clean-architecture). As part of the Kent Beck "passes the tests" rule, I ran `python -m unittest` on a copy of each branch. **Judgement calls you should know about:** - **Security findings:** neither skill has a checklist item for security. I filed pr-03 (SQL built with an f-string, allowing injection) and pr-04 (card token written to the logs) under the Kent Beck rule "code must work correctly". - **Diff scope:** both skills are written for reviewing a whole codebase, not a PR. I limited the findings to code each PR adds or changes. | Branch | Highest priority | Main finding | |---|---|---| | pr-01 | High | The page offset is off by one: page 1, the default, skips the first 20 orders. | | pr-02 | Low | The helper name `_row` doesn't say what it does, and `row[1]` is a bare position index. | | pr-03 | High | SQL injection, because the query is built with an f-string. | | pr-04 | High | `card_token` is logged in plain text. | | pr-05 | — | No findings. | | pr-06 | High | `user_id` is never used, so any user can cancel any order. | | pr-07 | High | `attempt` is never incremented, so the retry loop never ends; the order is also marked paid after the retries run out. | | pr-08 | Medium | The note-length check has no test. | | pr-09 | High | `except Exception: pass` lets the order be marked paid when the capture failed. | | pr-10 | High | The stock comparison is reversed; this is the only branch whose tests fail (1 failure, 1 error). | | pr-11 | Low | Magic user ids 41/42 rely on the database shared between tests. | | pr-12 | High | The forbidden-order test no longer asserts anything. | Each file lists every finding with its priority, the skill that raised it, and file:line. I didn't change or commit any code in the shop repo, and main is still checked out with a clean working tree. DELIVERABLE: ~/work/output