Aether Gateway
Self-hosted LLM gateway — routing, failover, semantic caching, quota enforcement, and cost accounting. Cut cache-hit latency to 8.9ms p95.
- Status
- Active
- Started
- Built with
- java, kotlin, llm, self-hosted, and kubernetes
Point an existing OpenAI-compatible SDK at Aether instead of at a provider directly. It handles routing, failover, semantic caching, quota enforcement, cost accounting, and observability transparently.
The engineering value isn’t in calling models — it’s in everything around the call:
- Routing and failover — Circuit breakers, retries, and bulkheads across multiple providers (Ollama, Groq, Gemini, or any OpenAI-compatible endpoint)
- Semantic caching — Near-duplicate prompts served from a vector cache (Postgres + pgvector) instead of hitting a provider again
- Quota enforcement — Per-API-key rate limits, concurrency caps, and monthly token budgets, checked atomically before dispatch
- Cost accounting — Every request’s real cost, exposed in response headers and Grafana dashboard
- Kubernetes-proven — Graceful SSE drain under rolling updates, in-flight-stream-based autoscaling
All numbers are reproducible via make bench against a real running stack — not estimates.