Control: Claude Code with plain file tools: test run, 2026-10-10

15 questions about a 560-note test vault, three for each of LongMemEval's five abilities. Claude Code (Claude Opus 5.5) installed the tool in a throwaway sandbox and answered through it; a judge (gpt-6-astra, three passes) marked the answers. All results · The vault and questions · Task

15 of 15 right. 0 calls through the tool, 9 through plain file tools; 32,109 tokens; 0.7 min.

Question by question

Question	Mark
1. What budget was approved for the Harbor project? extraction · expected: 48,000 euros.	right
2. Which warehouse did the team choose for Lantern? extraction · expected: ClickHouse.	right
3. How does Mira Okafor prefer to receive feedback? extraction · expected: As comments in Figma, not in chat.	right
4. Who is the manager of the engineer who was assigned to migrate the reports API for Harbor? multi-note · expected: Priya Raman (Jonas Field migrates the reports API; his manager is Priya Raman).	right
5. Lantern is paused until another project ships. Who owns that other project? multi-note · expected: Priya Raman (Lantern is paused until Harbor ships; Harbor's owner is Priya Raman).	right
6. The registrar account is under one person's email. What is that person's role? multi-note · expected: Customer support lead (Elena Voss).	right
7. Which happened first: the Lantern warehouse decision or the Harbor kickoff? temporal · expected: The Lantern warehouse decision (14 January 2026), before the Harbor kickoff (2 March 2026).	right
8. How many days passed between the Harbor kickoff and the Harbor review? temporal · expected: 49 days (2 March to 20 April 2026).	right
9. By how many open tickets did the support backlog fall between 2 April and 6 May 2026? temporal · expected: By 148 tickets (212 on 2 April to 64 on 6 May 2026).	right
10. When is Harbor's launch date? update · expected: 9 June 2026 (moved from 15 May at the 20 April review).	right
11. What does the team currently pay per SMS? update · expected: 0.031 euros per message (switched to Corvid on 18 March 2026; the older provider's 0.04 and 0.055 no longer apply).	right
12. Where is the team offsite? update · expected: Braga, 21 to 23 May (moved from Porto after the venue cancelled).	right
13. What is Tomasz Lind's phone number? abstain · expected: Not in the vault; the correct answer is that the notes do not say.	right
14. How much did the Harbor project actually spend? abstain · expected: Not in the vault; only the approved budget (48,000 euros) is recorded, not the spend.	right
15. Which CI system does the team use? abstain · expected: Not in the vault; the notes do not say.	right

The answers as written

1. Harbor's approved budget is 48,000 euros. The first ask was 60,000 euros, but finance cut it.
2. They chose ClickHouse over BigQuery (decided 2026-01-14).
3. Mira Okafor prefers feedback as comments in Figma, not in chat.
4. Jonas Field was assigned the reports API migration; Jonas Field's manager is Priya Raman.
5. Lantern is paused until Harbor ships, and Harbor's owner is Priya Raman.
6. The registrar account is under Elena Voss's work email; Elena Voss is the customer support lead.
7. The Lantern warehouse decision came first (2026-01-14), before the Harbor kickoff (2026-03-02).
8. 49 days passed, from 2 March 2026 to 20 April 2026.
9. It fell by 148 open tickets, from 212 to 64.
10. Harbor's launch is 9 June 2026. It was moved from 15 May at the Harbor review on 2026-04-20.
11. The team pays 0.031 euros per SMS, after switching to Corvid on 2026-03-18.
12. The 2026 offsite is in Braga, 21 to 23 May. It was moved from Porto after that venue cancelled.
13. The notes do not say; only Tomasz Lind's email (tomasz@example.invalid) is recorded.
14. The notes do not say; actual spend is tracked in the finance system, not in the vault.
15. The notes do not say.

How it was run (the session's closing message)

I answered 12 of the 15 questions from the vault. For the other 3, the answer file says the notes do not say. I did not change the vault.

- **No answer in the notes:**
  - **Q13:** No phone number is recorded for Tomasz Lind, only an email.
  - **Q14:** Actual Harbor spend isn't in the vault. The budget notes say it is tracked in the finance system.
  - **Q15:** Nothing anywhere mentions a CI system.
- **Where I took the later note:**
  - **Launch date:** The 2026-04-20 Harbor review moved it from 15 May to 9 June 2026.
  - **SMS price:** The 2026-03-18 switch to Corvid set it at 0.031 euros per message.
  - **Offsite:** The 6 May note moved it from Porto to Braga, same dates (21 to 23 May).
- **Answers I worked out:**
  - **Q8:** 49 days passed between the Harbor kickoff (2 March) and the review (20 April).
  - **Q9:** The support backlog fell by 148 tickets (212 to 64). I left out a separate engineering backlog of 97 issues, because that note says it is not the support queue.

DELIVERABLE: ~/work/output/answers.md