by microsoft · Agent Tool · ★ 89
Last updated: · Indexed by AgentSkillsHub · Auto-synced every 8h
🔒 Is ToolTalk safe to install? View the security audit →
:wrench: ToolTalk :speechballoon: :pagefacingup: Paper | :mailbox: Contact Introducing ToolTalk a benchmark for evaluating Tool LLMs in a conversational setting. Details ToolTalk is designed to evaluate tool-augmented LLMs as a chatbot, an increasingly popular paradigm for everyday users to harness the power of LLMs. ToolTalk contains a handcrafted dataset of 28 easy conversations and 50 hard conversations. We annotate these conversations to contain a ground truth usage of 28 unique tools belonging to 7 themed "plugins". Evaluation consists of prompting an LLM to predict the correct sequence of tools after every user utterance in a conversation. Thus, evaluating on a single conversation requires an LLM to correctly predict multiple sub-tasks. Predictions are compared against the ground truth to determine success for a single conversation. We evaluate two chatbots on ToolTalk powered by gpt-3.5-turbo-0613 and gpt-4-0613 implemented by using the chat completions API from OpenAI. | GPT-4
| Stars | 89 |
| Forks | 15 |
| Language | Python |
| Category | Agent Tool |
| License | MIT |
| Quality Score | 65.9751484846764/100 |
| Open Issues | 4 |
| Last Updated | 2024-05-31 |
| Created | 2023-10-10 |
| Platforms | python |
| Est. Tokens | ~19k |
Looking for a ToolTalk alternative? If you're comparing ToolTalk with other agent tool tools, these 3 projects are the closest alternatives on Agent Skills Hub — ranked by topic overlap, star count, and community traction.
A Claude Code skill that turns PDFs, docs, and codebases into Obsidian study vaults
Power rename/refactor tool (now with agent skill support!)
A desktop companion app for AI coding tools. Browse Claude Code chat history, manage configurations, commands,
Explore other popular agent tool tools:
ToolTalk is Evaluating tool-augmented LLMs in conversation settings. It is categorized as a Agent Tool with 89 GitHub stars.
ToolTalk is primarily written in Python.
You can find installation instructions and usage details in the ToolTalk GitHub repository at github.com/microsoft/ToolTalk. The project has 89 stars and 15 forks, indicating an active community.
ToolTalk is released under the MIT license, making it free to use and modify according to the license terms.
The top alternatives to ToolTalk on Agent Skills Hub include tutor-skills, repren, lovcode. Each offers a different approach to the same problem space — compare them side-by-side by stars, quality score, and community activity.