Skip to content
Hammad Yousuf.

AI AGENTS

2 min read · updated 2026-09-19

Rate limiting public AI demos: current limits and production gaps

TL;DR

The current demo limiter counts requests in memory: 12 per minute per instance, 20 per network in a rolling 24-hour window, and 10 per session in that window. These are not shared deployment-wide limits and they do not enforce a token or money budget. A production implementation needs durable shared counters, cost controls and multi-instance failure tests before those stronger guarantees can be claimed.

This walkthrough describes the local website code reviewed on 19 September 2026, not a load test or verification of the deployed service. The relevant files are src/lib/agent-engine/rateLimit.ts, src/app/api/agents/run/route.ts and src/components/agent-grid/RunPanel.tsx. An earlier version described shared KV counters and token-budget enforcement as implemented; that description did not match these files.

What the current limiter actually counts

The limiter stores request timestamps in module-level arrays and Maps. It checks a 12-request rolling minute window shared by requests reaching that instance, a 20-request rolling 24-hour window for an IP address, and a 10-request rolling 24-hour window for a session. The word global in the code means global to one process, not all deployed instances. These numbers are implementation settings, not a production service-level commitment.

Editorial flow: read network and session identifiers, check in-memory counters, then allow the request or return HTTP 429.
Editorial diagram — a reading aid, not a screenshot or measured result.

Where the check happens

The run route reads the first forwarded IP value and an existing wh_agent_session cookie, or creates a session ID. It checks the limiter before parsing the request body and selecting the agent. Consequently, a request admitted by the limiter can consume a slot even if its body later fails validation. Successful admission reports the remaining session count; it does not reserve money or tokens for a provider call.

What a visitor sees when a limit is reached

The endpoint returns HTTP 429 with a reason and a throttled flag. The run panel handles that separately from other failures and offers replay behavior. MDN documents 429 as the rate-limiting status and Retry-After as an optional wait indication. The current route does not return a Retry-After header, so this article does not promise an exact cooldown timer. A replay is an example output, not a fresh result for the submitted input.

Why this is not a deployment-wide budget control

The state exists only in the running process. A new instance starts with new counters; other instances do not share this Map. Restarting the process loses its history. Request counting also cannot establish a spending ceiling: provider input and output sizes can differ, and the limiter has no cost reservation or usage reconciliation. Clearing a session cookie changes one identity signal; multiple visitors may share a network address. Neither signal alone reliably identifies a person.

The 24-hour limit is a rolling window, not a midnight reset. Voice admission has separate accounting and is outside this text-run walkthrough. No shared daily token breaker, automated abuse ban, CAPTCHA challenge or quota-alert dashboard is established by this limiter code. Provider quotas may impose additional limits, but they are not a substitute for an application budget guarantee.

Production acceptance checks still needed

Before using this design as a paid customer control, define the identity and budget policy, implement atomic admission against shared durable state, and test concurrent requests across multiple instances. Decide what happens when the shared store is unavailable. Add bounded provider requests and explicit cost accounting if a money or token ceiling is part of the promise. These are required design and verification tasks, not features claimed to be finished here.

The test plan should include exact boundary requests, rolling-window expiry, process restarts, missing or rotated cookies, shared networks and provider failure. Verify that a rejected request does not reach the provider, that accounting matches the intended policy, and that replay output is visibly distinguished from live output. Use isolated test infrastructure and mocked providers first; do not stress a public demo or incur customer API charges to prove a point.

AI Marketing Automation Engineer · Dubai, UAE

Changelog
  • · 2026-09-19: Corrected KV, token-budget, cooldown and abuse-detection claims against the local implementation. Production upgrades are now clearly separated from implemented behavior.

FAQ

Common questions

Does this demo use a shared KV rate limiter?

The reviewed local implementation uses in-memory arrays and Maps. Counters are per instance and reset when that process is replaced; shared KV storage is not implemented by this limiter.

Does a request limit guarantee a spending limit?

No. The current limiter counts requests, not provider token usage or money. A budget guarantee requires separate cost accounting, enforcement and failure testing.

When do the demo limits reset?

The code uses rolling time windows: one minute for the instance-wide gate and 24 hours for network/session counts. It is not a guaranteed midnight reset, and process restarts discard the stored history.