bigcodebench: [ICLR'25] BigCodeBench: Benchmarking Code Generation Towards AGI

by bigcode-project · Agent Tool · ★ 519

Last updated: · Indexed by AgentSkillsHub · Auto-synced every 8h

How to install bigcodebench

pip install bigcodebench --upgrade
pip install "git+https://github.com/bigcode-project/bigcodebench.git" --upgrade

Taken from the project's README on GitHub. Check the security audit before running it.

Security audit verdict: SAFE · quality 65/100

None of the 42 rules matched — no credential harvesting, no data exfiltration, no curl-pipe-shell installer. Scanned against 42 published rules built on the SlowMist agent-security taxonomy, refreshed every 8 hours. Full audit →

The quality score combines nine heuristic checks of the repo and its README: completeness, clarity, specificity, examples, structure, agent readiness, step-by-step content, instruction quality and security notes. It rates how well the project explains itself to an agent, not the code.

agentagentsbenchmarkchatgptclaude-3code-generationdeepseekfunction-callinggeminigpt-4

Quick Facts

Stars519
Forks74
LanguagePython
CategoryAgent Tool
LicenseApache-2.0
Quality Score65/100
Open Issues30
Last Updated2026-01-03
Created2024-04-29
Platformsclaude-code, gemini, python
Est. Tokens~443k

About bigcodebench

BigCodeBench 💥 Impact • 📰 News • 🔥 Quick Start • 🚀

Compatible Skills

These tools work well together with bigcodebench for enhanced workflows:

  • gptme — semantic(0.18)+complementary+rare_topics+same_lang+similar_pop+shared_platform (56%)
  • cursor-agent — semantic(0.16)+complementary+same_lang+similar_pop+shared_platform (56%)
  • agent-studio — semantic(0.16)+complementary+rare_topics+same_lang+similar_pop+shared_platform (55%)
  • PairCoder — semantic(0.38)+rare_topics+same_lang+similar_pop+shared_platform (52%)
  • mcp-gemini-search — semantic(0.17)+complementary+same_lang+similar_pop+shared_platform (51%)

bigcodebench alternative? Top 6 similar tools

Looking for a bigcodebench alternative? If you're comparing bigcodebench with other agent tool tools, these 6 projects are the closest alternatives on Agent Skills Hub — ranked by topic overlap, star count, and community traction.

  • DemoGPT by melih-unsal · ⭐ 1.9k

    🤖 Everything you need to create an LLM Agent—tools, prompts, frameworks, and models—all in one place.

  • kani by zhudotexe · ⭐ 611

    kani (カニ) is a highly hackable microframework for tool-calling language models. (NLP-OSS @ EMNLP 2023)

  • LLM-Tool-Survey by quchangle1 · ⭐ 489

    This is the repository for the Tool Learning survey.

  • awesome-llm-powered-agent by hyp1231 · ⭐ 2.3k

    Awesome things about LLM-powered agents. Papers / Repos / Blogs / ...

  • py-gpt by szczyglis-dev · ⭐ 2.0k

    Desktop AI Assistant powered by GPT-6, GPT-5, Gemini, Claude, Grok, Ollama, DeepSeek, Perplexity, and more - c

  • awesome-gpt-prompt-engineering by snwfdhmp · ⭐ 1.6k

    A curated list of awesome resources, tools, and other shiny things for LLM prompt engineering.

More Agent Tool Tools

Explore other popular agent tool tools:

View all Agent Tool tools →

Popular Python Agent Tools

How this security grade is produced

Grades come from a rule-based scan built on the SlowMist agent-security taxonomy: 42 published rules, covering credential harvesting, data exfiltration, curl | sh installers and more. It is a first-layer scan, not a manual audit — we say so rather than overstate it.

The scale of the problem is documented independently: Liu et al. (2026), in a study of 31,132 agent skills, report that 26.1% contain security vulnerabilities. Our own full-catalog census is published as a citable open dataset.

Sources & who's responsible:

View on GitHub → Browse Agent Tool tools