alibaba/open-code-review: test run, 2026-10-10
Twelve pull requests against a small Python order service: eight with one planted defect each, four clean. Claude Code (Claude Opus 5.5) installed the tool in a throwaway sandbox and reported what the tool said; the reviewing itself was done by gpt-6-astra. A judge (gpt-6-astra, three passes) marked each review against the planted defect. Repo commit 357c3f09a264. All results · The pull requests · Task
Found 7 of 8 planted defects; 0.6 findings per pull request; 7 min for the set.
It skipped the two pull requests that only change tests, so the weakened test went unreviewed.
Pull request by pull request
| Pull request | Result |
|---|---|
| pr-01-list-pagination Off-by-one in pagination: pages start at 1 but offset = page * PAGE_SIZE, so page 1 skips the first 20 orders (should be (page - 1) * PAGE_SIZE). The review# pr-01-list-pagination
Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-01-list-pagination`, model gpt-6-astra
Status: complete — files reviewed: 1, failed: 0
## 1. [high] [bug] `shop/orders.py:31-33`
`page` is documented as 1-based, but this computes page 1's offset as 20, so the first page skips the first 20 orders and page 2 starts at 40. Subtract one before multiplying (and consider validating that `page >= 1`).
Existing code:
```
offset = page * PAGE_SIZE
return db.query("select id, status, total_cents from orders where user_id = ? order by id limit ? offset ?",
(user_id, PAGE_SIZE, offset))
```
Suggested code:
```
offset = (page - 1) * PAGE_SIZE
return db.query("select id, status, total_cents from orders where user_id = ? order by id limit ? offset ?",
(user_id, PAGE_SIZE, offset))
```
| found |
| pr-02-extract-row-helper Clean: nothing planted The review# pr-02-extract-row-helper Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-02-extract-row-helper`, model gpt-6-astra Status: complete — files reviewed: 1, failed: 0 No findings. | no blocking finding |
| pr-03-search-by-note SQL injection: search_orders builds the query with an f-string from user-supplied text instead of parameters. The review# pr-03-search-by-note
Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-03-search-by-note`, model gpt-6-astra
Status: complete — files reviewed: 1, failed: 0
## 1. [high] [security] `shop/orders.py:31-32`
This interpolates both `user_id` and the caller-provided `text` directly into SQL. A note such as `%' OR 1=1 --` can bypass the user filter and expose other users' orders (and other SQL syntax can cause execution errors or worse). Use placeholders for both values and pass the wildcard pattern as a parameter, e.g. `where user_id = ? and note like ?`, `(user_id, f"%{text}%")`.
Existing code:
```
sql = f"select id, status, total_cents from orders where user_id = {user_id} and note like '%{text}%' order by id"
return db.query(sql)
```
| found |
| pr-04-payment-logging Secret in logs: the card token is written to the application log. The review# pr-04-payment-logging
Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-04-payment-logging`, model gpt-6-astra
Status: complete — files reviewed: 1, failed: 0
## 1. [high] [security] `shop/payments.py:27`
This logs the full `card_token`, which is a payment credential and may allow replay or otherwise expose sensitive customer data in application logs. Do not log the token; if support needs correlation, log a non-sensitive request/payment ID or a carefully masked/irreversible representation instead.
Existing code:
```
log.info("capturing order %s amount %s with card token %s", order_id, amount_cents, card_token)
```
| found |
| pr-05-docstrings-and-types Clean: nothing planted The review# pr-05-docstrings-and-types Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-05-docstrings-and-types`, model gpt-6-astra Status: complete — files reviewed: 1, failed: 0 No findings. | no blocking finding |
| pr-06-cancel-order Missing authorization: cancel_order takes user_id but never checks that the order belongs to that user, so any user can cancel anyone's order. The review# pr-06-cancel-order
Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-06-cancel-order`, model gpt-6-astra
Status: complete — files reviewed: 1, failed: 0
## 1. [high] [security] `shop/orders.py:31-36`
The `user_id` argument is never used, so any caller who knows an order ID can cancel another user's unpaid order. This should enforce ownership (for example, load via `get_order(order_id, user_id)` or include `user_id` in the lookup/update) before changing the status.
Existing code:
```
rows = db.query("select status from orders where id = ?", (order_id,))
if not rows:
raise NotFound(order_id)
if rows[0][0] == "paid":
raise ValueError("paid orders cannot be cancelled")
db.execute("update orders set status = 'cancelled' where id = ?", (order_id,))
```
| found |
| pr-07-payment-retry Retry loop never ends and hides failure: `attempt` is never incremented, so a gateway that keeps failing loops forever (and after three real failures the order would still be marked paid, since nothing re-raises). The review# pr-07-payment-retry
Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-07-payment-retry`, model gpt-6-astra
Status: complete — files reviewed: 1, failed: 0
## 1. [high] [bug] `shop/payments.py:35-36`
`attempt` is never incremented, so any `PaymentError` causes an infinite retry loop and the order is never marked paid. Increment it after a failed attempt (or use a bounded `for` loop) so capture is attempted at most three times; after the final failure, re-raise/return instead of executing the paid update.
Existing code:
```
except PaymentError:
log.warning("capture failed for order %s, attempt %s", order_id, attempt)
```
| found |
| pr-08-validate-create-order Clean: nothing planted The review# pr-08-validate-create-order Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-08-validate-create-order`, model gpt-6-astra Status: complete — files reviewed: 1, failed: 0 No findings. | no blocking finding |
| pr-09-capture-errors Swallowed exception: a failed capture is ignored and the order is still marked paid. The review# pr-09-capture-errors
Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-09-capture-errors`, model gpt-6-astra
Status: complete — files reviewed: 1, failed: 0
## 1. [high] [bug] `shop/payments.py:27-31`
A capture failure is swallowed, but execution continues to mark the order as `paid` and logs successful payment. This can leave orders paid without a gateway charge (and silently loses the failure); handle the error by recording a non-paid/failed or retryable state and logging it, while allowing checkout to continue.
Existing code:
```
try:
gateway.capture(order_id, amount_cents, card_token)
except Exception:
pass
db.execute("update orders set status = 'paid' where id = ?", (order_id,))
```
| found |
| pr-10-reserve-check Inverted condition: reserve now raises OutOfStock when there is MORE stock than requested and allows reserving more than is available. The review# pr-10-reserve-check
Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-10-reserve-check`, model gpt-6-astra
Status: complete — files reviewed: 1, failed: 0
## 1. [high] [bug] `shop/stock.py:19-21`
The comparison is inverted: this raises `OutOfStock` when inventory exceeds the requested quantity, while allowing reservations that exceed available stock. Please retain the insufficient-stock check (`in_stock < qty`) so valid reservations succeed and over-reservations are rejected.
Existing code:
```
in_stock = available(sku)
if in_stock > qty:
raise OutOfStock(sku)
```
Suggested code:
```
in_stock = available(sku)
if in_stock < qty:
raise OutOfStock(sku)
```
| found |
| pr-11-test-list-orders Clean: nothing planted The review# pr-11-test-list-orders Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-11-test-list-orders`, model gpt-6-astra Status: skipped — "Review skipped: no items were selected." No findings. Note: the only changed file in this branch, `tests/test_shop.py`, is excluded from review by ocr's built-in default path rules (`ocr review --preview` reports "Excluded from review: [M] tests/test_shop.py (default_path)"). So the tool never sent this diff to the model. "No findings" here means it was not reviewed, not that it was reviewed and found clean. | no blocking finding |
| pr-12-speed-up-tests Test weakened: the assertion that another user is forbidden was replaced by a try/except that passes whether or not Forbidden is raised. The review# pr-12-speed-up-tests Tool: open-code-review (ocr) v1.12.13, `ocr review --from main --to pr-12-speed-up-tests`, model gpt-6-astra Status: skipped — "Review skipped: no items were selected." No findings. Note: the only changed file in this branch, `tests/test_shop.py`, is excluded from review by ocr's built-in default path rules (`ocr review --preview` reports "Excluded from review: [M] tests/test_shop.py (default_path)"). So the tool never sent this diff to the model. "No findings" here means it was not reviewed, not that it was reviewed and found clean. | missed |
How it was run (the session's closing message)
I reviewed all 12 branches with open-code-review (`ocr` v1.12.13) and wrote one file per branch to `~/work/output/<branch>.md`. Two of the "No findings" results come from branches the tool never actually reviewed (pr-11 and pr-12, explained below). **Setup:** `ocr` wasn't on the PATH, and there's no Go toolchain to build it from `~/src`. So I installed the npm package into `~/.local`, which is the skill's documented install route. I pointed it at your OpenAI-compatible endpoint with model gpt-6-astra, and the connection test passed. Each branch was run as `ocr review --audience agent --from main --to <branch> -b "<context>"` with JSON output. Every run exited with code 0. I included every finding the tool returned. The skill says to drop low-severity items, but you asked for all of them; in practice every finding came back as high, so nothing would have been dropped anyway. | Branch | Finding | |---|---| | pr-01-list-pagination | high/bug, `shop/orders.py:31-33`: the page offset is off by one, so page 1 skips the first 20 orders | | pr-02-extract-row-helper | No findings | | pr-03-search-by-note | high/security, `shop/orders.py:31-32`: SQL injection, because `user_id` and the note text are put straight into the query | | pr-04-payment-logging | high/security, `shop/payments.py:27`: the full `card_token` is written to the log | | pr-05-docstrings-and-types | No findings | | pr-06-cancel-order | high/security, `shop/orders.py:31-36`: `user_id` is never used, so any user can cancel someone else's order | | pr-07-payment-retry | high/bug, `shop/payments.py:35-36`: `attempt` is never incremented, so a failed capture retries forever | | pr-08-validate-create-order | No findings | | pr-09-capture-errors | high/bug, `shop/payments.py:27-31`: a capture error is swallowed and the order is still marked paid | | pr-10-reserve-check | high/bug, `shop/stock.py:19-21`: the stock check is inverted | | pr-11-test-list-orders | No findings (not reviewed) | | pr-12-speed-up-tests | No findings (not reviewed) | **pr-11 and pr-12:** both branches only change `tests/test_shop.py`, and `ocr` skips test files by default. It reported "Review skipped: no items were selected", so neither diff was sent to the model. Since you said to take the tool's defaults, I didn't override that. Both files say "No findings." with a note explaining it. I didn't change or commit anything in the repo; it's still on `main` with a clean working tree. DELIVERABLE: ~/work/output