by abacusai · Agent Tool · ★ 600
Last updated: · Indexed by AgentSkillsHub · Auto-synced every 8h
Extending LLM Context Length The choice of how to encode positional information for transformers has been one of the key components of LLM architectures. An area that has been interesting to us and others in the community recently is whether LLMs can be extended to longer contexts. We have conducted a range of experiments with different schemes for extending context length capabilities of Llama, which has been pretrained on 2048 context length with the RoPE (Rotary Position Embedding) encoding. Here we share some of the results as well as the training and evaluation scripts in the hope that it will be useful to the community. For our best performing models - linear scaling with IFT at scales 4 and 16 - we are also sharing the weights in case others wish to use them, or to conduct their own tests. We believe the scale 16 model should perform well on real world tasks up to 16k context lengths, and potentially even up to about 20-24k context lengths. Scale 16 model Technical Paper Overview We conducted a wide variety of experiments to try to extend the context length of the models. First, we tried simply using the base Llama model zero-shot.
| Stars | 600 |
| Forks | 46 |
| Language | Python |
| Category | Agent Tool |
| License | Apache-2.0 |
| Quality Score | 52.9599012604374/100 |
| Open Issues | 7 |
| Last Updated | 2023-11-17 |
| Created | 2023-07-27 |
| Platforms | python |
| Est. Tokens | ~36k |
These tools work well together with Long-Context for enhanced workflows:
Looking for a Long-Context alternative? If you're comparing Long-Context with other agent tool tools, these 6 projects are the closest alternatives on Agent Skills Hub — ranked by topic overlap, star count, and community traction.
This repository contains a collection of Agent Skills developed by GudaStudio, enabling seamless collaboration
Supercharge Claude Code with 11 AI agents, 36 commands & 15 skills — the claude-code plugin framework inspired
Skill to give Claude Code (and any coding agent) the ability to generate beautiful and practical Excalidraw di
A collection of Agent skills and Claude Code plugins for HashiCorp products.
A collection of standardized Agent Skills to teach GitHub Copilot, Claude, Gemini and Cursor about modern Andr
Claude Code Skill Factory — A powerful open-source toolkit for building and deploying production-ready Claude
Explore other popular agent tool tools:
Long-Context is This repository contains code and tooling for the Abacus.AI LLM Context Expansion project. Also included are evaluation scripts and benchmark tasks that evaluate a model’s information retrieval capabi. It is categorized as a Agent Tool with 600 GitHub stars.
Long-Context is primarily written in Python.
You can find installation instructions and usage details in the Long-Context GitHub repository at github.com/abacusai/Long-Context. The project has 600 stars and 46 forks, indicating an active community.
Long-Context is released under the Apache-2.0 license, making it free to use and modify according to the license terms.
The top alternatives to Long-Context on Agent Skills Hub include skills, claude-forge, excalidraw-diagram-skill. Each offers a different approach to the same problem space — compare them side-by-side by stars, quality score, and community activity.