A public AI demo is an open invitation to spend your money. Every agent demo on this site — all 73 of them — accepts free-text input from anonymous visitors and forwards it to an LLM API. Without a limiter, a single scraping bot or one good day on Hacker News can burn a month's free-tier quota in an hour. This post describes the rate-limiting design I actually run: @vercel/kv-backed counters, token-aware budgets, and a degradation path that tells visitors the truth instead of returning a 500.
Why an AI demo is a different rate-limiting problem than a normal API
Classic rate limiting assumes requests cost roughly the same, so counting requests per minute is a fair proxy for load. LLM endpoints break that assumption. One request might produce a 50-token reply; the next, a 4,000-token essay with retrieval calls behind it. Two users making the same number of requests can differ in cost by two orders of magnitude. Request-count limiting alone protects your server, but it does not protect your budget — and for a free-tier or fixed-ceiling deployment, the budget is the thing that dies first.
The second difference is who shows up. A normal API has authenticated clients you can suspend. A portfolio demo has anonymous traffic by design — the whole point is that a recruiter or prospect can try it without signing up. That means you cannot lean on accounts for identity; you have to build identity out of weaker signals and accept their weaknesses openly.
Choosing a rate-limit store for serverless
In-memory counters are the first instinct and they simply do not work on serverless. Each invocation may land on a fresh instance, so a counter held in module scope resets on every cold start and is never shared across concurrent instances. An abuser hitting your function from ten parallel connections sees ten independent, mostly-empty counters.
The store has to live outside the function. I use @vercel/kv because it is Redis-compatible, reachable from edge and Node runtimes alike, and adds low single-digit milliseconds to the request path — cheap enough to check before every LLM call. The limiter itself is a sliding-window counter: increment a key scoped to the identity and window, set an expiry, reject when the count crosses the threshold. Atomic increments matter here; read-then-write counters have a race that concurrent abuse will find.
Designing the limit: per-IP, per-session, or both
Per-IP limiting is the workhorse, and it is honest to admit its flaws. Corporate offices, universities, and mobile carriers put many real users behind one IP, so a strict per-IP cap punishes exactly the audience a portfolio wants — several people at one company trying your demo the same afternoon. VPNs cut the other way: a motivated abuser rotates IPs faster than you can block them.
ONE TACTIC A WEEK
Per-session limiting (a cookie or signed session ID) has the opposite failure: it is trivially rotated by clearing cookies, but it distinguishes individuals behind a shared IP. So I blend them — a generous per-IP ceiling that only heavy shared-IP use would touch, plus a tighter per-session allowance that governs a normal visit. Neither is unbeatable alone; layered, each covers the other's blind spot. Fingerprinting adds a third signal but I keep it minimal, because aggressive fingerprinting is a privacy smell on a site that argues for trustworthy AI.
Token-aware limiting, not just request counts
The layer most guides skip: cap spend, not just calls. Before each request I estimate cost — prompt length is known, output is bounded by max_tokens — and debit it from a daily token budget in KV. After the response, I reconcile with the actual usage the API reports. A visitor who asks three long, retrieval-heavy questions consumes more budget than one who asks ten short ones, and the limiter sees that. Above everything sits a hard daily cap across all visitors: a circuit breaker that closes the demos entirely before the provider's quota does it for me. This site runs against a hard $0 ceiling, so that breaker is not theoretical — it is the last line that keeps the ceiling intact.
The graceful degradation UX
What happens at the limit matters as much as the limit. A generic error page reads as a broken product — the worst outcome for a demo meant to prove engineering quality. When a visitor trips a limit on this site, the UI says so plainly: this is a free public demo, it is rate-limited to keep it free, and here is when you can continue, with a visible cooldown timer. When the global daily breaker trips, the message changes to say the demo has hit its daily budget and will reset, and points to recorded outputs and case studies as the fallback path. Honest copy converts a failure state into a credibility signal; a stack trace converts it into a bounce.
Abuse patterns vs. legitimate heavy use
Bots and humans hit limits differently. Scrapers arrive with no session history, fire requests at machine cadence, and often replay identical payloads; a real enthusiast ramps up gradually inside one session with varied inputs. I treat the first pattern more harshly — near-identical repeated payloads from a fresh identity get a short ban rather than a polite cooldown. CAPTCHAs I use reluctantly and late: as a challenge after suspicious behaviour, never as a gate in front of first contact. A recruiter with thirty seconds of curiosity should never meet a puzzle before the demo.
Monitoring the limiter itself
A limiter you cannot see is a limiter you find out about from an overage email. The counters already live in KV, so the dashboard is nearly free: daily token spend against budget, limit-hit counts by identity type, and top consumers. The alert that matters fires at a percentage of the daily budget, early enough in the day to act — because the useful signal is "you will run out by tonight at this pace," not "you ran out." Watching hit rates over time also tunes the thresholds: constant limit-hits from plausibly-human sessions means the limits are too tight and are costing you the exact audience the demos exist for. This monitoring habit is the same discipline I apply inside the agents themselves, which is its own post — tracing every step before it costs you.