AgriciDaniel/claude-obsidian: test run, 2026-10-10
15 questions about a 560-note test vault, three for each of LongMemEval's five abilities. Claude Code (Claude Opus 5.5) installed the tool in a throwaway sandbox and answered through it; a judge (gpt-6-astra, three passes) marked the answers. All results · The vault and questions · Task
15 of 15 right. 1 calls through the tool, 7 through plain file tools; 53,840 tokens; 1.5 min.
The skill was loaded once; the searching and reading were done with Claude Code's own file tools.
Question by question
| Question | Mark |
|---|---|
| 1. What budget was approved for the Harbor project? extraction · expected: 48,000 euros. | right |
| 2. Which warehouse did the team choose for Lantern? extraction · expected: ClickHouse. | right |
| 3. How does Mira Okafor prefer to receive feedback? extraction · expected: As comments in Figma, not in chat. | right |
| 4. Who is the manager of the engineer who was assigned to migrate the reports API for Harbor? multi-note · expected: Priya Raman (Jonas Field migrates the reports API; his manager is Priya Raman). | right |
| 5. Lantern is paused until another project ships. Who owns that other project? multi-note · expected: Priya Raman (Lantern is paused until Harbor ships; Harbor's owner is Priya Raman). | right |
| 6. The registrar account is under one person's email. What is that person's role? multi-note · expected: Customer support lead (Elena Voss). | right |
| 7. Which happened first: the Lantern warehouse decision or the Harbor kickoff? temporal · expected: The Lantern warehouse decision (14 January 2026), before the Harbor kickoff (2 March 2026). | right |
| 8. How many days passed between the Harbor kickoff and the Harbor review? temporal · expected: 49 days (2 March to 20 April 2026). | right |
| 9. By how many open tickets did the support backlog fall between 2 April and 6 May 2026? temporal · expected: By 148 tickets (212 on 2 April to 64 on 6 May 2026). | right |
| 10. When is Harbor's launch date? update · expected: 9 June 2026 (moved from 15 May at the 20 April review). | right |
| 11. What does the team currently pay per SMS? update · expected: 0.031 euros per message (switched to Corvid on 18 March 2026; the older provider's 0.04 and 0.055 no longer apply). | right |
| 12. Where is the team offsite? update · expected: Braga, 21 to 23 May (moved from Porto after the venue cancelled). | right |
| 13. What is Tomasz Lind's phone number? abstain · expected: Not in the vault; the correct answer is that the notes do not say. | right |
| 14. How much did the Harbor project actually spend? abstain · expected: Not in the vault; only the approved budget (48,000 euros) is recorded, not the spend. | right |
| 15. Which CI system does the team use? abstain · expected: Not in the vault; the notes do not say. | right |
The answers as written
1. Harbor's approved budget is 48,000 euros. The first ask was 60,000 euros, but finance cut it ([[Harbor]], [[Harbor budget notes]]). 2. The team chose ClickHouse for Lantern, over BigQuery and Postgres ([[2026-01-14 Lantern warehouse decision]]). 3. Mira Okafor wants feedback as comments in Figma, not in chat ([[Mira Okafor]]). 4. Priya Raman is the manager. Jonas Field was assigned to migrate the reports API, and his manager is Priya Raman ([[2026-03-02 Harbor kickoff]], [[Jonas Field]]). 5. Lantern is paused until Harbor ships, and Priya Raman owns Harbor ([[Lantern]], [[Harbor]]). 6. The registrar account is under Elena Voss's work email, and she is the customer support lead (Daily 2026-02-03, [[Elena Voss]]). 7. The Lantern warehouse decision came first, on 14 January 2026. The Harbor kickoff was on 2 March 2026. 8. 49 days passed between the Harbor kickoff (2 March 2026) and the Harbor review (20 April 2026). 9. The support backlog fell by 148 open tickets, from 212 on 2 April 2026 to 64 on 6 May 2026. The 97 engineering issues on 10 April are a separate queue. 10. Harbor's launch date is 9 June 2026. It was first set for 15 May 2026 but moved after the reports API migration slipped ([[2026-04-20 Harbor review]]). 11. The team pays 0.031 euros per SMS, after switching to Corvid on 18 March 2026 (Daily 2026-03-18). 12. The 2026 offsite is in Braga on 21 to 23 May. It was first booked in Porto, but that venue cancelled (Daily 2026-05-06). 13. The notes do not give Tomasz Lind's phone number; his page lists only an email address (tomasz@example.invalid). 14. The notes do not say how much Harbor actually spent. Spending is tracked in the finance system, and the notes only say the budget is unchanged. 15. The notes do not say which CI system the team uses.
How it was run (the session's closing message)
All 15 answers are in `~/work/output/answers.md`. Questions 13, 14 and 15 say the notes don't hold the answer: there's no phone number for Tomasz Lind, no figure for what Harbor actually spent, and no mention of a CI system. **How the tool was used:** the claude-obsidian `wiki-query` skill was the tool installed (no MCP server was configured). Its search index hasn't been built for this vault. The skill doesn't allow building one during a query, and this vault also lacks the `wiki/hot.md` and `wiki/index.md` pages it starts from. So I followed the skill's own fallback for that case: read-only text search and reading notes, which I did with shell `grep` and `cat`. Nothing in the vault was changed. **Answers that depended on reading past a first, wrong value:** - **Launch date (Q10):** Harbor's launch moved from 15 May to 9 June 2026. - **SMS price (Q11):** the team now pays 0.031 euros per message after switching to a new provider, Corvid. The 0.055 euros mentioned earlier was the old provider's planned price rise. - **Offsite (Q12):** it was first booked in Porto, then moved to Braga on the same dates (21 to 23 May). - **Support backlog (Q9):** it fell by 148 tickets (212 down to 64). I left out the 97 engineering issues, which the notes say are a separate queue. DELIVERABLE: ~/work/output/answers.md