Eval: High-performance LLM evaluation framework with parallel API calls — up to 17× faster than sequential tools

by ai-twinkle · LLM Plugin · ★ 119

Last updated: · Indexed by AgentSkillsHub · Auto-synced every 8h

How to install Eval

pip install twinkle-eval
pip install twinkle-eval[math]

Taken from the project's README on GitHub.

Security audit verdict: SAFE · quality 65/100

None of the 44 rules matched — no credential harvesting, no data exfiltration, no curl-pipe-shell installer. Scanned against 44 published rules built on the SlowMist agent-security taxonomy, refreshed every 8 hours.

The quality score combines nine heuristic checks of the repo and its README: completeness, clarity, specificity, examples, structure, agent readiness, step-by-step content, instruction quality and security notes. It rates how well the project explains itself to an agent, not the code.

evalevaluationllm

Quick Facts

Stars119
Forks17
LanguagePython
CategoryLLM Plugin
LicenseMIT
Quality Score65/100
Open Issues31
Last Updated2026-09-15
Created2025-03-31
Platformspython
Est. Tokens~16k

About Eval

Twinkle Eval -- High-Performance LLM Evaluation Framework 🇺🇸 English | 🇹🇼 繁體中文 Twinkle Eval 是一個以並行 API 請求為核心的 LLM 評測框架,支援選擇題、數學推理、指令遵循、函式呼叫、長文本理解、RAG、Text-to-SQL 等多類型評測。透過 OpenAI 相容 API 呼叫已部署的模型端點,單機即可完成完整評測流程。 目錄 為什麼選擇 Twinkle Eval 支援的評測資料集 評測方法一覽 安裝 快速開始 CLI 參考 設定檔 輸出格式 排行榜 使用 Coding Agent 貢獻 貢獻者 授權條款 引用 致謝 為什麼選擇 Twinkle Eval 2025 年推理模型(reasoning model)大量出現,每次 API 回應時間大幅增加。傳統評測框架逐題同步呼叫,一個...

Compatible Skills

These tools work well together with Eval for enhanced workflows:

  • mcp-interviewer — semantic(0.17)+complementary+rare_topics+same_lang+similar_pop+shared_platform (56%)
  • eval-view — semantic(0.17)+complementary+rare_topics+same_lang+similar_pop+shared_platform (56%)
  • just-eval — semantic(0.58)+rare_topics+same_lang+similar_pop+shared_platform (55%)
  • toolbench — semantic(0.22)+complementary+same_lang+similar_pop+shared_platform (53%)

Eval alternative? Top 6 similar tools

Looking for a Eval alternative? If you're comparing Eval with other llm plugin tools, these 6 projects are the closest alternatives on Agent Skills Hub — ranked by topic overlap, star count, and community traction.

  • litefuse by litefuse · ⭐ 113

    Litefuse - Agent Observability and Evaluation Platform

  • skill-optimizer by fastxyz · ⭐ 85

    Benchmark, evaluate, and optimize skills to ensure reliable performance across all LLMs

  • arag by Ayanami0730 · ⭐ 365

    A-RAG: Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces. State-of-the-art RAG fram

  • bocoel by rentruewang · ⭐ 288

    Bayesian Optimization as a Coverage Tool for Evaluating LLMs. Accurate evaluation (benchmarking) that's 10 tim

  • Hegelion by Hmbown · ⭐ 176

    Dialectical reasoning architecture for LLMs (Thesis → Antithesis → Synthesis)

  • mcp-interviewer by microsoft · ⭐ 158

    Catch MCP server issues before your agents do.

More LLM Plugin Tools

Explore other popular llm plugin tools:

View all LLM Plugin tools →

Popular Python Agent Tools

How this security grade is produced

Grades come from a rule-based scan built on the SlowMist agent-security taxonomy: 44 published rules, covering credential harvesting, data exfiltration, curl | sh installers and more. It is a first-layer scan, not a manual audit — we say so rather than overstate it.

The scale of the problem is documented independently: Liu et al. (2026), in a study of 31,132 agent skills, report that 26.1% contain security vulnerabilities. Our own full-catalog census is published as a citable open dataset.

Sources & who's responsible:

View on GitHub → Browse LLM Plugin tools