No red flags found in any of the 11 categories — no credential harvesting, no data exfiltration, no curl-pipe-shell installer. Scanned against the SlowMist agent-security taxonomy, refreshed every 8 hours. Full audit →
by microsoft · Agent Tool · ★ 57
Last updated: · Indexed by AgentSkillsHub · Auto-synced every 8h
🔒 Is a11y-llm-eval safe to install? View the security audit →
A11y LLM Evaluation Harness and Dataset This is a research project to evaluate how well various LLM models generate accessible HTML content. Problem LLMs currently generate code with accessibility bugs, resulting in blockers for people with disabilities and costly re-work and fixes downstream. Goal Create a public test suite which can be used to benchmark how well various LLMs generates accessible HTML code. Eventually, it could also be used to help train models to generate more accessible code by default. Methdology Each test case contains a prompt to generate an HTML page to demonstrate a specific pattern or component. This page is rendered in a real browser using Playwright (Chromium). Tests are executed against this rendered page. The HTML is evaluated against axe-core, one of the most popular automated accessibility testing engines. The HTML is also evaluated against a manually defined set of assertions, customized for the specific test case. This allows for more robust testing than just using axe-core. Tests only pass if zero axe-core failures are found AND all requirement assertions pass. Best Practice (BP) assertion failures do not fail the test but are tracked separately.
| Stars | 57 |
| Forks | 8 |
| Language | Python |
| Category | Agent Tool |
| License | MIT |
| Quality Score | 67.0067676345031/100 |
| Open Issues | 3 |
| Last Updated | 2026-05-07 |
| Created | 2025-09-24 |
| Platforms | python |
| Est. Tokens | ~70k |
Explore other popular agent tool tools:
a11y-llm-eval is An eval tool to benchmark how well LLMs generate accessible HTML. It is categorized as a Agent Tool with 57 GitHub stars.
a11y-llm-eval is primarily written in Python.
You can find installation instructions and usage details in the a11y-llm-eval GitHub repository at github.com/microsoft/a11y-llm-eval. The project has 57 stars and 8 forks, indicating an active community.
a11y-llm-eval is released under the MIT license, making it free to use and modify according to the license terms.
Grades come from a rule-based scan built on the SlowMist agent-security taxonomy, covering 11 red-flag categories including credential harvesting, data exfiltration, and curl | sh installers. It is a first-layer scan, not a manual audit — we say so rather than overstate it.
The scale of the problem is documented independently: Liu et al. (2026), in a study of 31,132 agent skills, report that 26.1% contain security vulnerabilities. Our own full-catalog census is published as a citable open dataset.
Sources & who's responsible: