Speculators is a unified library for creating, representing, and storing speculative decoding algorithms for large language model (LLM) serving, such as in vLLM. It provides a standard format and tools to facilitate the productization of decoding algorithms for inference servers.
- Speculative Decoding: Simplify the creation and representation of speculative decoding algorithms for LLMs.
- Standardized Format: Ensure compatibility and ease of use with a standardized format for decoding algorithms.
- Integration Ready: Designed to integrate seamlessly with LLM inference servers like vLLM.
- Productization Tools: Streamline the process of deploying decoding algorithms in production environments.
Before installing, ensure you have the following prerequisites:
- OS: Linux or MacOS
- Python: 3.9 or higher
The latest Speculators release can be installed using pip:
pip install speculatorsOr from source code using pip:
pip install git+https://github.com/neuralmagic/speculators.gitComing soon
Coming soon
Visit our GitHub Releases page and review the release notes to stay updated with the latest releases.
Speculators is licensed under the Apache License 2.0.
If you find Speculators helpful in your research or projects, please consider citing it:
@misc{speculators2025,
title={Speculators: A Unified Library for Speculative Decoding Algorithms in LLM Serving},
author={Red Hat},
year={2025},
howpublished={\url{https://github.com/neuralmagic/speculators}},
}