getsentry/warden: test run, 2026-10-10
Twelve pull requests against a small Python order service: eight with one planted defect each, four clean. Claude Code (Claude Opus 5.5) installed the tool in a throwaway sandbox and reported what the tool said; the reviewing itself was done by gpt-6-astra. A judge (gpt-6-astra, three passes) marked each review against the planted defect. Repo commit 7ccb81a001a9. All results · The pull requests · Task
Found 4 of 8 planted defects; 0.5 findings per pull request; 25 min for the set.
Its verification pass threw out candidate findings on the pull requests it missed, the SQL injection among them.
Pull request by pull request
| Pull request | Result |
|---|---|
| pr-01-list-pagination Off-by-one in pagination: pages start at 1 but offset = page * PAGE_SIZE, so page 1 skips the first 20 orders (should be (page - 1) * PAGE_SIZE). The review# Warden review: pr-01-list-pagination Tool: Warden v0.51.1 (`warden main..pr-01-list-pagination`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). 1 finding(s). ## 1. [medium] Page 1 skips the first 20 orders - Skill: code-review - Severity: medium - Confidence: high - Location: `shop/orders.py:31` The offset is calculated as `page * PAGE_SIZE`, so the documented first page starts at row 21 and every page is shifted by one page. Verification: - `list_orders` documents that pages start at 1, but `page=1` computes `offset=20`. - SQLite therefore returns rows 21–40 for the default call instead of rows 1–20; `page=2` similarly skips the first 40 rows. - The query orders by `id`, so this deterministically omits the user's earliest orders from every requested page. | found |
| pr-02-extract-row-helper Clean: nothing planted The review# Warden review: pr-02-extract-row-helper Tool: Warden v0.51.1 (`warden main..pr-02-extract-row-helper`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). No findings. | no blocking finding |
| pr-03-search-by-note SQL injection: search_orders builds the query with an f-string from user-supplied text instead of parameters. The review# Warden review: pr-03-search-by-note Tool: Warden v0.51.1 (`warden main..pr-03-search-by-note`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). No findings. Note: Warden's verification pass rejected 2 candidate finding(s) for this branch; Warden does not report them, so they are not listed here. | missed |
| pr-04-payment-logging Secret in logs: the card token is written to the application log. The review# Warden review: pr-04-payment-logging Tool: Warden v0.51.1 (`warden main..pr-04-payment-logging`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). No findings. Note: Warden's verification pass rejected 1 candidate finding(s) for this branch; Warden does not report them, so they are not listed here. | missed |
| pr-05-docstrings-and-types Clean: nothing planted The review# Warden review: pr-05-docstrings-and-types Tool: Warden v0.51.1 (`warden main..pr-05-docstrings-and-types`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). No findings. | no blocking finding |
| pr-06-cancel-order Missing authorization: cancel_order takes user_id but never checks that the order belongs to that user, so any user can cancel anyone's order. The review# Warden review: pr-06-cancel-order Tool: Warden v0.51.1 (`warden main..pr-06-cancel-order`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). 2 finding(s). ## 1. [medium] Cancellation overwrites a payment completed after the status check - Skill: code-review - Severity: medium - Confidence: high - Location: `shop/orders.py:37` If payment completes between cancellation's status query and update, cancellation overwrites `paid` with `cancelled` despite the successful charge. Enforce the unpaid condition atomically in the cancellation update and handle a failed transition. Verification: - The reported `cancel_order()` change rejects a status of `paid`, then updates the order by ID alone; the update does not recheck its status. - `payments.pay()` calls `Gateway.capture()` and then writes `paid` through `db.execute()`, which commits the change. - If cancellation reads `new`, payment captures and commits `paid`, and cancellation then executes its update, the captured order ends in `cancelled`, contradicting the explicit rejection of paid orders. - `shop/db.py` enables cross-thread connection use and provides no lock spanning the status query and update. The inspected checkout contains the base orders implementation; the cancellation statements are supplied by the candidate's change excerpt. ## 2. [medium] Missing ownership check allows cancelling another user's order - Skill: security-review - Severity: medium - Confidence: high - Location: `shop/orders.py:32-37` A caller can supply another user's unpaid order ID and cancel it because `cancel_order` ignores `user_id`; enforce ownership before updating the order. Verification: - The added `cancel_order(order_id, user_id)` service function accepts a caller-controlled order ID but never uses `user_id`. - Its lookup checks only existence and whether the status is `paid`; an order belonging to another user with status `new` passes both checks. - The update selects solely by order ID, and `shop/db.py::execute` executes and commits the SQL without an authorization guard. - `get_order` explicitly compares the stored owner with `user_id` and raises `Forbidden` on mismatch, but the new cancellation path does not invoke that check. | found |
| pr-07-payment-retry Retry loop never ends and hides failure: `attempt` is never incremented, so a gateway that keeps failing loops forever (and after three real failures the order would still be marked paid, since nothing re-raises). The review# Warden review: pr-07-payment-retry Tool: Warden v0.51.1 (`warden main..pr-07-payment-retry`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). 1 finding(s). ## 1. [high] Retry loop never advances and hangs on persistent capture failures - Skill: code-review - Severity: high - Confidence: high - Location: `shop/payments.py:30-35` `attempt` is never incremented after a failed capture, so a gateway that continues raising `PaymentError` loops forever instead of stopping after `MAX_ATTEMPTS`; the order remains unpaid and `pay` never returns. Verification: - `pay()` initializes `attempt = 1` and enters `while attempt <= MAX_ATTEMPTS`. - The `except PaymentError` branch only logs; it does not change `attempt`, so the loop condition remains true on every failure. - `Gateway.capture()` raises `PaymentError` while `fail > 0`, and a gateway that persistently raises is a reachable capture failure path. - The database update and success log occur only after the loop, so persistent failures hang before the order can be marked paid or the error can be surfaced. Note: Warden's verification pass rejected 1 candidate finding(s) for this branch; Warden does not report them, so they are not listed here. | found |
| pr-08-validate-create-order Clean: nothing planted The review# Warden review: pr-08-validate-create-order Tool: Warden v0.51.1 (`warden main..pr-08-validate-create-order`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). 1 finding(s). ## 1. [medium] New validation test fails because create_order accepts zero - Skill: code-review - Severity: medium - Confidence: high - Location: `tests/test_shop.py:16-18` The new test fails with `AssertionError: ValueError not raised`; add the non-positive-total validation to `create_order` alongside this test. Verification: - The added test calls `orders.create_order(1, 0)` inside `assertRaises(ValueError)`. - `shop/orders.py::create_order` unconditionally passes the total to `db.execute` without validation. - `shop/db.py` defines `total_cents` as an integer with no CHECK constraint; `execute` inserts and commits zero successfully, then returns the row ID. - README.md documents `python -m unittest` as the test entry point, which discovers this test and fails its exception assertion. | no blocking finding |
| pr-09-capture-errors Swallowed exception: a failed capture is ignored and the order is still marked paid. The review# Warden review: pr-09-capture-errors Tool: Warden v0.51.1 (`warden main..pr-09-capture-errors`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). 1 finding(s). ## 1. [high] Gateway failures still mark the order as paid - Skill: code-review - Severity: high - Confidence: high - Location: `shop/payments.py:30-31` When `gateway.capture()` raises, this handler suppresses the failure and execution continues to update the order to `paid`, even though no capture occurred. Verification: - `Gateway.capture()` raises `PaymentError` when `fail > 0` and does not append the order to `captured`. - The new broad handler catches that exception and does not return or propagate an explicit failure state. - Control then reaches the unconditional `db.execute(... status = 'paid' ...)`, so a failed capture produces a paid order without a payment. Note: Warden's verification pass rejected 1 candidate finding(s) for this branch; Warden does not report them, so they are not listed here. | found |
| pr-10-reserve-check Inverted condition: reserve now raises OutOfStock when there is MORE stock than requested and allows reserving more than is available. The review# Warden review: pr-10-reserve-check Tool: Warden v0.51.1 (`warden main..pr-10-reserve-check`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). No findings. Note: Warden's verification pass rejected 1 candidate finding(s) for this branch; Warden does not report them, so they are not listed here. | missed |
| pr-11-test-list-orders Clean: nothing planted The review# Warden review: pr-11-test-list-orders Tool: Warden v0.51.1 (`warden main..pr-11-test-list-orders`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). No findings. | no blocking finding |
| pr-12-speed-up-tests Test weakened: the assertion that another user is forbidden was replaced by a try/except that passes whether or not Forbidden is raised. The review# Warden review: pr-12-speed-up-tests Tool: Warden v0.51.1 (`warden main..pr-12-speed-up-tests`), skills: code-review, security-review; model astra/gpt-6-astra (chat-completions endpoint). No findings. Note: Warden's verification pass rejected 1 candidate finding(s) for this branch; Warden does not report them, so they are not listed here. | missed |
How it was run (the session's closing message)
I ran Warden v0.51.1, the review tool in `~/src`, on all 12 branches and wrote one report per branch to `~/work/output/<branch>.md`. Warden reported 7 findings across 6 branches; the other 6 branches got "No findings." **How I ran it** - **Command:** for each branch, `warden main..<branch>` with the `code-review` and `security-review` skills. These are the two skills Warden ships for pull requests, and its git mode diffs `main...<branch>`. - **Config:** the shop repo has no `warden.toml`, and with no config Warden runs nothing. So I wrote one outside the repo (`~/work/warden-config/warden.toml`) that just turns those two skills on. - **Model:** your endpoint only accepts `chat/completions`, but Warden's built-in OpenAI provider calls `/v1/responses` and got a 404. I added a custom provider in `~/work/pi-agent/models.json` that points `gpt-6-astra` at your `OPENAI_BASE_URL` over chat-completions. Warden's main, verification and synthesis steps all use that model. - **Retry:** pr-04's code-review failed once with an upstream HTTP/2 error, so I re-ran that branch and it completed cleanly. - **Source of the reports:** each file is built from Warden's own JSON output and matches its terminal summary for that branch. | Branch | Findings | |---|---| | pr-01-list-pagination | **medium**: `shop/orders.py:31`, page 1 skips the first 20 orders | | pr-02-extract-row-helper | No findings | | pr-03-search-by-note | No findings | | pr-04-payment-logging | No findings | | pr-05-docstrings-and-types | No findings | | pr-06-cancel-order | **medium**: `shop/orders.py:37`, cancelling overwrites a payment that completed after the status check<br>**medium**: `shop/orders.py:32-37`, no ownership check, so a user can cancel another user's order | | pr-07-payment-retry | **high**: `shop/payments.py:30-35`, the retry loop never advances and hangs if capture keeps failing | | pr-08-validate-create-order | **medium**: `tests/test_shop.py:16-18`, the new validation test fails because `create_order` accepts zero | | pr-09-capture-errors | **high**: `shop/payments.py:30-31`, the order is still marked paid when the payment gateway fails | | pr-10-reserve-check | No findings | | pr-11-test-list-orders | No findings | | pr-12-speed-up-tests | No findings | Warden's own verification step rejected some candidate findings: 2 on pr-03 and 1 each on pr-04, pr-07, pr-09, pr-10 and pr-12. Warden doesn't report these, so they aren't listed as findings; each affected report has a one-line note giving the count. I didn't change any code or commit anything. The shop repo is clean and on `main`; I deleted the `.warden/` log directory Warden had written into it. Warden's raw output for each run is in `~/work/logs/`. DELIVERABLE: ~/work/output