Warning
The API is still under beta version, major breaking changes may occur.
OpenGateLLM is an open-source, production-ready API gateway, optimized for self-hosted models. It's designed to centralize, secure, and manage Generative AI access in a sovereign and cost-effective way.
OpenGateLLM addresses three critical challenges for organizations:
- Cost control - Reduce expenses of commercial APIs and GPU infrastructure by using self-hosted models and build a mutualized infrastructure with your peers.
- Data sovereignty - Keep sensitive data under your control
- Privacy & security - No chat history storage, robust access control
Core principles
- Open source and free forever - All features available without commercial licensing
- High code quality - Built with maintainability and reliability in mind
- Lightweight architecture - Focused feature set for optimal performance
- High compatibility - Seamlessly integrates with GenAI ecosystem frameworks by OpenAI-compatible API
- Production-ready - Engineered to handle high loads with advanced QoS features
OpenGateLLM is an alternative to...
| Key features | OpenGateLLM | LiteLLM | TensorZero |
|---|---|---|---|
| 🔌 OpenAI Compatibility | ✅ | ✅ | ✅ |
| 📖 [Open-source](https://github.com/etalab-ia/OpenGateLLM) | ✅ | ✅ | ✅ |
| 💻 [Self-hostable](./getting-started/quickstart.md) | ✅ | ✅ | ✅ |
| 💸 Free (all features) | ✅ | ❌ | ✅ |
| 🏭 Support commercial and self-hosted models | ✅ | ✅ | ✅ |
| Account management | |||
| 🎲 Playground UI | ✅ | ✅ | ✅ |
| 👤 User management (API keys, budget...) | ✅ | ✅ | ✅ |
| 🏢 Organization management | 🚧 | ✅ | ❌ |
| ✏️ Project management | 🔜 | ✅ | ❌ |
| 🔑 SSO support | 🚧 | ✅ | ❌ |
| High load features | |||
| ⌛ Rate limiting | ✅ | ❌ | ❌ |
| ⚡ Requests prioritization | ✅ | ✅ | ❌ |
| 📈 Quality of service thresholds | ✅ | ✅ | ❌ |
| 🔀 Model load balancing | ✅ | ✅ | ❌ |
| 🔄 Model fallback | 🔜 | ✅ | ❌ |
| Monitoring & analytics | |||
| 📊 Usage tracking | ✅ | ❌ | ❌ |
| 📈 Carbon footprint | ✅ | ❌ | ❌ |
| 🔗 Prometheus integration | ✅ | ❌ | ❌ |
| Privacy & security | |||
| 🚫 No chat history storage | ✅ | ❌ | ❌ |
| 🔒 Role-based access control | ✅ | ❌ | ❌ |
Legend: ✅ supported — ❌ not supported — 🚧 work in progress — 🔜 in roadmap
Deploy and start using OpenGateLLM in minutes with our quickstart guide here.
This project exists thanks to all the people who contribute. OpenGateLLM thrives on open-source contributions. Join our community!
Check out our Contribution Guide to get started.
OpenGateLLM is still under beta version, major breaking changes may occur. Check our current roadmap here to see what we are working on.
