imbue-ai/vet: test run, 2026-10-10
Twelve pull requests against a small Python order service: eight with one planted defect each, four clean. Claude Code (Claude Opus 5.5) installed the tool in a throwaway sandbox and reported what the tool said; the reviewing itself was done by gpt-6-astra. A judge (gpt-6-astra, three passes) marked each review against the planted defect. Repo commit 59757aec70b3. All results · The pull requests · Task
Found 7 of 8 planted defects; 1.1 findings per pull request; 18 min for the set.
Pull request by pull request
| Pull request | Result |
|---|---|
| pr-01-list-pagination Off-by-one in pagination: pages start at 1 but offset = page * PAGE_SIZE, so page 1 skips the first 20 orders (should be (page - 1) * PAGE_SIZE). The review# Vet review: pr-01-list-pagination Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-01-list-pagination`, default confidence threshold 0.8. ## 1. logic_error — severity 4, confidence 1 - Location: `shop/orders.py:30` - The offset is off by one page for the documented 1-based page numbering. With PAGE_SIZE = 20, the default page skips the first 20 orders and returns an empty list for users with 20 or fewer orders; those first orders are inaccessible through valid page numbers. Calculate offset as (page - 1) * PAGE_SIZE. ## 2. test_coverage — severity 3, confidence 1 - Location: `shop/orders.py:28` - Pagination introduces new behavior, but the diff adds no tests despite the existing unittest suite. Add tests for the default first page, subsequent pages, a partially filled final page, and pages beyond the available results, checking ordering and user isolation. | found |
| pr-02-extract-row-helper Clean: nothing planted The review# Vet review: pr-02-extract-row-helper Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-02-extract-row-helper`, default confidence threshold 0.8. No findings. | no blocking finding |
| pr-03-search-by-note SQL injection: search_orders builds the query with an f-string from user-supplied text instead of parameters. The review# Vet review: pr-03-search-by-note Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-03-search-by-note`, default confidence threshold 0.8. ## 1. test_coverage — severity 3, confidence 0.97 - Location: `(no file)` - The diff adds a new search API but does not add tests for matching notes, restricting results to the requested user, or handling search text containing SQL-special characters. Existing tests do not exercise `search_orders`. | missed |
| pr-04-payment-logging Secret in logs: the card token is written to the application log. The review# Vet review: pr-04-payment-logging Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-04-payment-logging`, default confidence threshold 0.8. ## 1. insecure_code — severity 4, confidence 0.99 - Location: `shop/payments.py:26` - The payment attempt log includes the full card token. Payment tokens are sensitive credentials and should not be written to logs, where they may be retained or exposed to support and monitoring systems. Log only a redacted token or a non-sensitive payment reference instead. | found |
| pr-05-docstrings-and-types Clean: nothing planted The review# Vet review: pr-05-docstrings-and-types Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-05-docstrings-and-types`, default confidence threshold 0.8. ## 1. commit_message_mismatch — severity 2, confidence 0.98 - Location: `shop/stock.py:18` - The request covers docstrings and type hints for the stock module, but the diff only documents `set_stock`. The public `available` function lacks a docstring, and `reserve` lacks both a docstring and type annotations; these functions should be updated as well. | no blocking finding |
| pr-06-cancel-order Missing authorization: cancel_order takes user_id but never checks that the order belongs to that user, so any user can cancel anyone's order. The review# Vet review: pr-06-cancel-order Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-06-cancel-order`, default confidence threshold 0.8. ## 1. logic_error — severity 4, confidence 1 - Location: `shop/orders.py:30` - cancel_order ignores user_id, allowing one user to cancel another user's unpaid order simply by supplying its ID. This contradicts the ownership enforcement in get_order. Retrieve the order through get_order(order_id, user_id) before checking its status and updating it, so unauthorized cancellation raises Forbidden without changing the order. ## 2. test_coverage — severity 3, confidence 1 - Location: `shop/orders.py:28` - The diff introduces cancellation without adding tests to the existing unittest suite. Add tests for successful owner cancellation, rejection of another user without modifying the order, missing orders raising NotFound, and paid orders being rejected without changing their status. | found |
| pr-07-payment-retry Retry loop never ends and hides failure: `attempt` is never incremented, so a gateway that keeps failing loops forever (and after three real failures the order would still be marked paid, since nothing re-raises). The review# Vet review: pr-07-payment-retry Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-07-payment-retry`, default confidence threshold 0.8. ## 1. logic_error — severity 5, confidence 1 - Location: `shop/payments.py:29` - The retry loop never increments `attempt`, so a gateway that continues to fail will be retried indefinitely rather than at most three times. The failure path should increment the counter and, after the final failed attempt, re-raise the payment error instead of proceeding to mark the order as paid. ## 2. test_coverage — severity 3, confidence 0.98 - Location: `tests/test_shop.py:25` - The diff adds retry behavior but does not add tests verifying that failed captures are retried no more than three times or that an order is not marked paid when all attempts fail. Existing tests only cover an immediately successful capture. | found |
| pr-08-validate-create-order Clean: nothing planted The review# Vet review: pr-08-validate-create-order Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-08-validate-create-order`, default confidence threshold 0.8. No findings. | no blocking finding |
| pr-09-capture-errors Swallowed exception: a failed capture is ignored and the order is still marked paid. The review# Vet review: pr-09-capture-errors Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-09-capture-errors`, default confidence threshold 0.8. ## 1. logic_error — severity 5, confidence 1 - Location: `shop/payments.py:28` - The gateway exception handler catches every exception, suppresses it without logging or reporting, and then marks the order as paid even though capture failed. The code should catch the expected payment error, make the failure observable, and leave the order unpaid or assign an explicit payment-failed status. ## 2. test_coverage — severity 3, confidence 0.98 - Location: `(no file)` - The change alters the gateway-error path but adds no regression test. A test should configure Gateway.fail, call pay, verify checkout does not raise, and assert that the order is not marked paid and the failure is observable according to the chosen error-handling contract. | found |
| pr-10-reserve-check Inverted condition: reserve now raises OutOfStock when there is MORE stock than requested and allows reserving more than is available. The review# Vet review: pr-10-reserve-check Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-10-reserve-check`, default confidence threshold 0.8. ## 1. logic_error — severity 5, confidence 1 - Location: `shop/stock.py:18` - The reservation check is inverted. It raises OutOfStock when the available quantity is greater than the requested quantity, preventing valid reservations, while allowing reservations that exceed available stock and driving the inventory negative. The comparison should be `if in_stock < qty:`. | found |
| pr-11-test-list-orders Clean: nothing planted The review# Vet review: pr-11-test-list-orders Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-11-test-list-orders`, default confidence threshold 0.8. No findings. | no blocking finding |
| pr-12-speed-up-tests Test weakened: the assertion that another user is forbidden was replaced by a try/except that passes whether or not Forbidden is raised. The review# Vet review: pr-12-speed-up-tests Tool: vet (verify-everything), model gpt-6-astra, diff `main...pr-12-speed-up-tests`, default confidence threshold 0.8. ## 1. logic_error — severity 4, confidence 1 - Location: `tests/test_shop.py:12` - The test now passes even if get_order returns another user's order without raising Forbidden, because the try/except has no failure path when the exception is absent. This removes the authorization assertion rather than fixing flakiness and allows access-control regressions to go undetected. Preserve assertRaises(orders.Forbidden), or add an else branch calling self.fail(), and address any actual source of nondeterminism separately. | found |
How it was run (the session's closing message)
I reviewed all 12 branches with the installed tool, **vet** (`verify-everything`), and wrote one report per branch to `~/work/output/<branch>.md`. Each file lists every finding vet gave, with its issue type, severity, confidence, and file:line. Three branches came back with "No findings." **How I ran it:** I installed vet from `~/src` into a separate virtual environment and pointed it at your OpenAI-compatible endpoint with model `gpt-6-astra`, set up in `~/.config/vet/models.json`. On each branch I ran `vet "<commit subject>" --base-commit $(git merge-base main HEAD) --output-format json`, so it reviewed exactly `main...<branch>`. I used the default confidence threshold of 0.8 and no session history. The raw JSON output is kept in `~/work/vetjson/`. I didn't change or commit any code, and the repo is back on `main` with a clean working tree. Two things to know: - **Results can vary between runs.** I ran pr-01 once as a test before the full run, and it found the same two issues with slightly different wording. The reports contain the full-run output. - **pr-03 (search by note):** vet only reported missing tests. Its note mentions "SQL-special characters," but it did not raise a separate issue about the query itself, so the report doesn't include one. Findings by branch (severity is vet's own score; higher is more severe): | Branch | Findings | |---|---| | pr-01 | logic_error, sev 4, `shop/orders.py:30`: page offset is off by one page. test_coverage, sev 3, `shop/orders.py:28` | | pr-02 | No findings | | pr-03 | test_coverage, sev 3, no file given | | pr-04 | insecure_code, sev 4, `shop/payments.py:26`: the full card token is written to the log | | pr-05 | commit_message_mismatch, sev 2, `shop/stock.py:18`: only `set_stock` got docstrings and type hints | | pr-06 | logic_error, sev 4, `shop/orders.py:30`: `cancel_order` ignores `user_id`, so one user can cancel another's order. test_coverage, sev 3, `shop/orders.py:28` | | pr-07 | logic_error, sev 5, `shop/payments.py:29`: the retry counter never increases, so it retries forever. test_coverage, sev 3, `tests/test_shop.py:25` | | pr-08 | No findings | | pr-09 | logic_error, sev 5, `shop/payments.py:28`: every gateway error is swallowed and the order is still marked paid. test_coverage, sev 3, no file given | | pr-10 | logic_error, sev 5, `shop/stock.py:18`: the stock check is the wrong way round | | pr-11 | No findings | | pr-12 | logic_error, sev 4, `tests/test_shop.py:12`: the test no longer fails when the `Forbidden` error isn't raised | DELIVERABLE: ~/work/output