knoxgraeme/skillfish: test run, 2026-10-09
Claude Code (Claude Opus 5.5) drove this tool in a throwaway sandbox: install 20 test skills (local folders, or two public fixture repositories for tools that install only from GitHub), install one more that ships a
curl … | sh setup script (the URL is under .invalid; it was never run), prune to five, share them with Codex. Repo commit b34c0688bc55.
All results · Task · The risky script
What it did
| Step | Command | Note |
|---|---|---|
| Install the tool | npm i -g --prefix ~/.npm-global skillfish (README says `npm i -g skillfish`; plain form failed with EACCES on /usr/local, so a user prefix was used; skillfish 1.0.39) | |
| 1. Install 20 local skills | skillfish add zhuyansen/skill-manager-test-fixtures --all --agent "Claude Code" | Non-interactive (stdin not a TTY), so it used the global (~/) default. All 20 went to ~/.claude/skills. measure: 21137 -> 21482 context tokens. |
| 3. List, then remove all but five | skillfish list; then skillfish remove <name> for each of the 16 unwanted skills (accessibility-check ... unit-test-scaffold, pdf-tools-pro included) | skillfish list now shows exactly csv-cleaner, git-commit-message, pr-description, readme-polisher, sql-explainer. measure: 5 skills, 21277 context tokens. |
| 4. Give the five to Codex | skillfish add zhuyansen/skill-manager-test-fixtures csv-cleaner --agent Codex | Codex is a supported agent (~/.codex/skills), but skillfish only installs to detected agents (it looks for ~/.codex or ./.codex). Here it failed with 'Agent not found: "Codex". Detected agents: Claude Code'. I did not create ~/.codex to fake detection, and did not run measure.sh synced. |
2. Asked to install the skill with a curl | sh script: what the tool printed
│ ◇ Found 1 skill │ ● Pdf Tools Pro - Merge, split and compress PDF files. Run scripts/setup.sh o… Installing to 1 agent(s): Claude Code │ ◇ Installing pdf-tools-pro │ ◇ Installed ✓ Claude Code → ~/.claude/skills/pdf-tools-pro │ └ Done! Installed 1 skill
Installed without any check. The only mention of the script is the skill's own description text, cut off ('Run scripts/setup.sh o…'). The source code has an interactive-only confirmInstallBatch: a generic 'Skills can instruct AI agents...' warning, the source line and a confirm prompt. Non-interactive mode skips it, and it never inspects or lists scripts. README: 'skillfish does not vet third-party skills.' setup.sh was not run.
Measurements
| Checkpoint | Tokens Claude Code loads before you type | Skills on disk |
|---|---|---|
| baseline | 21,137 | .claude/skills: 0 |
| installed-20 | 21,482 | .claude/skills: 20 |
| risky | 21,485 | .claude/skills: 21 |
| pruned | 21,277 | .claude/skills: 5 |
On loading skills only when needed, the tool's docs say: README says skills give agents expertise '... loaded on demand to extend capabilities'. That describes how agents use skills; skillfish itself does nothing to control loading.