# FreeLLM

Case study by Devansh Tiwari (https://www.devanshtiwari.com) · https://www.devanshtiwari.com/work/freellm

OpenAI-compatible gateway that pools 6 free LLM tiers into one endpoint. Multi-key rotation, circuit breakers, response cache.

## Key facts

- Role: Solo Builder
- Client: Side Project
- Year: 2026
- Status: Shipped
- Live: https://freellms.vercel.app
- Source code: https://github.com/Devansh-365/freellm
- Tech: TypeScript, Express 5, Node.js 22, React, Vite, Tailwind CSS, Astro Starlight, Zod, Docker

## At a glance

- Problem: Free LLM tiers hit rate limits within an hour, and each one has its own SDK.
- What I built: One OpenAI-compatible endpoint over the free tiers of Groq, Gemini, Mistral and more.
- Result: Apps keep answering when one free tier hits its limit.

## Numbers

| Metric | Value |
| --- | --- |
| Cost | $0 |
| Free requests a minute | ~360 |
| Providers pooled | 6 |
| Repeat prompts | <1ms |
| To deploy on Vercel | 2 min |

## The problem

Every free LLM tier rate-limits you within an hour of real use. Groq gives 30 RPM, Gemini gives 15, Mistral gives 1. To ship anything you end up juggling 4 SDKs, 4 sets of keys, and 4 different error shapes. Switch providers and your client code changes. Hit a 429 and your app stops. Nobody wants to write that plumbing again.

## What I built

An Express gateway that speaks the OpenAI Chat Completions API and routes requests across Groq, Gemini, Mistral, Cerebras, OpenRouter, and Ollama. Three virtual models (free-fast, free-smart, free) pick the right backend automatically. Per-provider circuit breakers with CLOSED/OPEN/HALF-OPEN states absorb 429s and 5xx without surfacing them. Multi-key rotation per provider multiplies free quota linearly. SHA-256 response cache cuts repeat-prompt cost to zero. Token usage tracked per provider on a rolling 24h window and surfaced in a React dashboard. Ships as a pnpm monorepo with the gateway, the dashboard, and an Astro Starlight docs site. MIT licensed, deploys to Vercel in 2 minutes.

### OpenAI-Compatible Gateway

Speaks the full Chat Completions API spec: streaming, function calling, JSON mode, system prompts, and multi-turn. Existing OpenAI clients work with a one-line baseURL change. No SDK swap needed.

### Virtual Model Routing

Three virtual models (free-fast, free-smart, free) abstract away provider choice. free-fast routes to Groq first for sub-100ms latency. free-smart routes to Gemini for reasoning tasks. free picks the cheapest available provider. Routing logic is a priority queue with fallback chains.

### Circuit Breaker Resilience

Each provider has an independent circuit breaker with CLOSED/OPEN/HALF-OPEN states. After 5 consecutive errors, the circuit opens for 30 seconds. A background health checker probes the HALF-OPEN state every 10 seconds. Failed requests never surface to the client.

### Multi-Key Rotation

Each provider accepts up to 10 API keys. The router round-robins across keys per request, multiplying free quota linearly. Groq at 30 RPM with 3 keys becomes 90 RPM. Key health is tracked independently so a dead key is skipped automatically.

### SHA-256 Response Cache

Identical prompts hash to the same SHA-256 key and return the cached response in under 1ms. Cache TTL is configurable per model. Streaming responses are cached as chunked arrays and replayed identically. Cuts repeat-prompt cost to zero for deterministic workloads.

## Further reading

- [FreeLLM on the blog](https://www.devanshtiwari.com/blog/freellm-one-endpoint-five-providers) (Markdown: https://www.devanshtiwari.com/blog/freellm-one-endpoint-five-providers.md)

## Work with Devansh

Devansh Tiwari is an AI product engineer based in India, available for freelance and consulting work: custom data feeds from sites without APIs, AI features (agents, chat, RAG, MCP servers) built on that data, and running them after launch.

- Book a 15 minute call: https://cal.com/devansh0718/15min
- Email: devanshtiwari365@gmail.com
- All case studies: https://www.devanshtiwari.com/work
- Site summary for AI tools: https://www.devanshtiwari.com/llms.txt
