bitbonsai/mcpvault: test run, 2026-10-10
15 questions about a 560-note test vault, three for each of LongMemEval's five abilities. Claude Code (Claude Opus 5.5) installed the tool in a throwaway sandbox and answered through it; a judge (gpt-6-astra, three passes) marked the answers. All results · The vault and questions · Task
15 of 15 right. 22 calls through the tool, 2 through plain file tools; 67,784 tokens; 1.1 min.
Reads the vault folder directly, no Obsidian app needed; nearly every step went through its own tools.
Question by question
| Question | Mark |
|---|---|
| 1. What budget was approved for the Harbor project? extraction · expected: 48,000 euros. | right |
| 2. Which warehouse did the team choose for Lantern? extraction · expected: ClickHouse. | right |
| 3. How does Mira Okafor prefer to receive feedback? extraction · expected: As comments in Figma, not in chat. | right |
| 4. Who is the manager of the engineer who was assigned to migrate the reports API for Harbor? multi-note · expected: Priya Raman (Jonas Field migrates the reports API; his manager is Priya Raman). | right |
| 5. Lantern is paused until another project ships. Who owns that other project? multi-note · expected: Priya Raman (Lantern is paused until Harbor ships; Harbor's owner is Priya Raman). | right |
| 6. The registrar account is under one person's email. What is that person's role? multi-note · expected: Customer support lead (Elena Voss). | right |
| 7. Which happened first: the Lantern warehouse decision or the Harbor kickoff? temporal · expected: The Lantern warehouse decision (14 January 2026), before the Harbor kickoff (2 March 2026). | right |
| 8. How many days passed between the Harbor kickoff and the Harbor review? temporal · expected: 49 days (2 March to 20 April 2026). | right |
| 9. By how many open tickets did the support backlog fall between 2 April and 6 May 2026? temporal · expected: By 148 tickets (212 on 2 April to 64 on 6 May 2026). | right |
| 10. When is Harbor's launch date? update · expected: 9 June 2026 (moved from 15 May at the 20 April review). | right |
| 11. What does the team currently pay per SMS? update · expected: 0.031 euros per message (switched to Corvid on 18 March 2026; the older provider's 0.04 and 0.055 no longer apply). | right |
| 12. Where is the team offsite? update · expected: Braga, 21 to 23 May (moved from Porto after the venue cancelled). | right |
| 13. What is Tomasz Lind's phone number? abstain · expected: Not in the vault; the correct answer is that the notes do not say. | right |
| 14. How much did the Harbor project actually spend? abstain · expected: Not in the vault; only the approved budget (48,000 euros) is recorded, not the spend. | right |
| 15. Which CI system does the team use? abstain · expected: Not in the vault; the notes do not say. | right |
The answers as written
1. The approved budget for Harbor is 48,000 euros. The first ask was 60,000 euros, but finance cut it. 2. The team chose ClickHouse for Lantern over BigQuery and Postgres (decision meeting on 2026-01-14). 3. Mira Okafor prefers feedback as comments in Figma, not in chat. 4. Jonas Field was assigned the reports API migration, and his manager is Priya Raman. 5. Lantern is paused until Harbor ships, and Harbor is owned by Priya Raman. 6. The registrar account is under Elena Voss's work email, and she is the customer support lead. 7. The Lantern warehouse decision came first (2026-01-14), before the Harbor kickoff (2026-03-02). 8. 49 days passed between the Harbor kickoff (2026-03-02) and the Harbor review (2026-04-20). 9. The support backlog fell by 148 open tickets, from 212 on 2 April to 64 on 6 May 2026. 10. Harbor's launch date is 9 June 2026. The Harbor review on 2026-04-20 moved it from the original 15 May. 11. The team pays 0.031 euros per SMS with Corvid, the provider they switched to on 2026-03-18 after the old provider's price rise. 12. The 2026 offsite is in Braga, 21 to 23 May. It moved there from Porto after the Porto venue cancelled. 13. The notes do not say; Tomasz Lind's note gives only an email address (tomasz@example.invalid), not a phone number. 14. The notes do not say; the Harbor budget notes say spend is tracked in the finance system, not in the vault. 15. The notes do not say which CI system the team uses.
How it was run (the session's closing message)
I answered all 15 questions using the Obsidian tool's search and read functions. I didn't change anything in the vault. - **Answered from the notes (12):** where the notes contradict each other, I used the latest entry: - **Q10:** the launch was moved from 15 May to 9 June 2026. - **Q11:** the team switched SMS provider, so the current price is 0.031 euros per message. - **Q12:** the offsite moved from Porto to Braga. - **Calculated (2):** - **Q8:** 2 March to 20 April 2026 is 49 days. - **Q9:** the support backlog fell from 212 to 64 open tickets, a drop of 148. - **Not in the notes (3):** - **Q13:** Tomasz Lind's note has an email address but no phone number. - **Q14:** the notes say Harbor's actual spend is tracked in the finance system, not in the vault. - **Q15:** searches turned up nothing about which CI system the team uses. DELIVERABLE: ~/work/output/answers.md