An AI product that calls a large language model spends real money on every request. Each call consumes tokens, the units of text the model processes, and the provider bills you per token. A per-user quota is the limit you set on how much of that cost and compute a single user can use. It protects your margins and stops one heavy user from degrading the service for everyone else. It also enforces the difference between free and paid accounts.
Before you write code, make three decisions: what to count, over what time window, and how to record usage so the counts stay correct when clients retry requests or send several at once. The third decision is where implementations fail. If you count retries and concurrent requests naively, you either bill a user twice for one generation or let that user slip past the limit.
You will end with a data model you can adapt, the decision rules for request counts versus token budgets, and the enforcement bugs to test for before you trust the counters.
What a per-user quota actually measures
Rate limiting controls how fast a user or service can access a resource. When requests exceed the threshold, the server throttles or blocks them. Rate limits stop denial-of-service traffic from starving resources and heavy usage from overloading servers. They also reduce cost overruns when someone overuses a resource.
Most rate limiters share three concepts. The limit is a ceiling on requests within a time span, and that time span is the window. The identifier is the attribute that tells callers apart, such as a user ID or IP address. For a per-user quota, the identifier is the user account, and the limit can vary by account level, so your enforcement code reads the limit for the user’s tier rather than a single global number.
For AI products, two kinds of limits matter. A request count limits how many API calls a user can make per minute or per day. A token budget limits how many tokens those calls consume. They protect different things. Request counts protect your infrastructure from call volume; token budgets protect your wallet, because a single request with a huge prompt or a long completion costs many times more than a short one.
Request counts, token budgets, or both
OpenAI defines its limits in five forms: requests per minute, tokens per minute, requests per day, tokens per day, and images per minute. Your per-user quota should include at least one request limit and one token limit, because they catch different abuse patterns.
One user sends many tiny requests in an hour and trips a request limit. Another sends a few requests with large context windows and trips a token limit. If your quota counts only requests, the second user quietly burns your most expensive resource. If it counts only tokens, the first user hammers your API endpoints and crowds out everyone else.
A token is the unit of text the model reads or writes. A rough rule of thumb is that one token is about three-quarters of an English word, but the exact count depends on the tokenizer, the software that splits text into tokens. Providers report token usage in the API response, so you do not need to estimate it yourself. You do need to decide whether the quota counts the prompt tokens, the completion tokens, or both; count the total, since that is what the provider bills. Once you know what to count, the next decision is the time span over which the counters reset.
Choosing the time window
The window sets how quickly a user can consume quota. The simplest implementation is the fixed window counter: divide time into fixed-size windows, keep a counter per window, increment it on each request, and block requests once the counter reaches the threshold until the next window starts. With a 1-second window and a limit of 3 requests per second, the counter resets every second. The counter does not have to increment by one; for token budgets it can increment by the token cost of the request.
The weakness of the fixed window is the boundary burst. A user sends 3 requests just before the window ends and another 3 just after the next window starts. Both pass the per-window check, and the user has made 6 requests in a fraction of a second. For LLM calls, which are expensive and slow, that burst pattern is exactly what you want to prevent, because a user who scripts concurrent generations clusters them at window edges.
A sliding window closes the gap. Instead of counting inside a fixed bucket, the server looks back over the last N seconds and counts every request in that span. A sliding window log stores the timestamp of each request and counts how many fall inside the window; it is precise but keeps one row per request. A sliding window counter keeps a coarser approximation by combining the current partial window with the previous full window; it uses less storage but is less exact.
For token budgets, prefer the sliding window. Token spending is sensitive to bursts in a way that request counts are not, and the boundary burst doubles a user’s effective allowance at every window edge. The cost is storage: the longer the window, the more timestamp rows the server must keep, and the longer it must keep them durable across restarts. When the server blocks a request, it returns 429 Too Many Requests, and the client should back off and retry later.
Prepaid quota accounting
Prepaid quota changes the model from "how fast can the user go" to "how much does the user have left." In a credit-based system, users buy credits upfront and consume them over time, with each action costing a set number of credits. This gives users predictable spend and lets you collect revenue before usage happens, and it fits AI tools and image generation well because the cost of an action is known in advance.
The core of prepaid accounting is the balance and the ledger. The balance is a number on the user’s account. The ledger is a list that only grows; it records every debit and credit and never changes past entries. Your code inserts a ledger entry and updates the balance in the same transaction; it never edits the balance on its own. That history settles disputes and gives you the audit trail that metering systems need.
The decrement must be atomic and conditional. In SQL, one statement does both the check and the subtraction:
UPDATE accounts SET balance = balance - ? WHERE id = ? AND balance >= ?;
The database updates the row only when the balance covers the cost, and the affected-row count tells you whether the user had enough credit. If zero rows changed, reject the request. If you read the balance in one query and write a new balance in another, two concurrent requests both pass the read and both decrement, leaving a negative balance.
A refund is a ledger entry with the opposite sign, never a deletion of the original debit. Deleting rows that other entries reference corrupts the history.
Usage visibility is part of the design. Users who can see their remaining balance in real time make fewer support requests, and alerts at 50%, 80%, and 100% of plan limits give heavy users a chance to slow down before they hit a hard stop.
Recording usage so retries and concurrent requests cannot double-count
When a client retries a request, the server cannot tell the retry from a new request by looking at the payload alone. The HTTP method matters. Under RFC 9110, the HTTP semantics specification, PUT, DELETE, and safe methods are idempotent, meaning repeating them has the same intended effect as sending them once. POST, which most AI APIs use for generation, is not. A retried POST produces a second generation and a second quota charge unless the server recognizes it.
Idempotency keys close that gap. The client generates a unique key, attaches it to the operation, and sends the same key on every retry. The server uses the key to recognize that it has seen the request and returns the original result instead of doing the work again. A random UUID generated once and stored with the pending operation works well, as does a key derived from business data such as an order ID plus an operation name. Do not use sensitive data like an email address as the key.
The server-side flow has four steps. When a request arrives with a key, the server tries to insert a row for that key with a hash of the request parameters (a fixed-size fingerprint of the exact request) and a status of in_progress. The key column is the primary key, so the database rejects the insert if the key already exists. If the insert succeeds, the server processes the request, then updates the row with the response and a status of completed, in the same transaction as the business change. If the insert fails, the server reads the existing row. A different parameter hash means the client reused the key for a different request, so the server rejects it. A completed status means the server returns the stored response. An in_progress status means the server returns a conflict response asking the client to retry later.
The in_progress state makes concurrent retries safe. If two requests with the same key arrive at the same moment, only one insert succeeds. The other request finds the row still in_progress and waits, so it does not start a second generation. Without that state, both requests pass the check and both do the work.
If the server crashes mid-processing, the row stays in_progress. The row should expire after a timeout so a later retry can proceed, or the server should resolve it by checking whether the business change actually happened before retrying.
The same rule applies when a request is rejected for exceeding the quota. If the client retries after the window rolls over, it must send the same key, or the retry becomes a new request and the user is charged for work they never received.
For prepaid quota, the same key protects the money. The server records the debit against the user’s balance with the same idempotency key as the generation request, in the same transaction as the in_progress row. When the client retries, the server replays the original response instead of charging twice.
With token budgets, you do not know the exact token count until the response arrives. Reserve an upper bound, such as the max_tokens parameter, before calling the model, and reconcile the actual usage when the response returns. Your code decrements the balance by the estimated maximum at reservation time and refunds the difference after the response arrives. If you skip the reservation and only record actual usage after the response, two concurrent requests can both pass the quota check before either records usage.
Metering in general needs idempotency and high throughput. It also has to aggregate usage in real time and provide data retention for disputes and audits. An idempotency key stops the same event from being counted twice when a retry delivers it again.
Example data model
This Postgres-flavored schema combines the pieces into one place. The idempotency table follows the standard pattern: the key is the primary key, a hash of the request parameters sits next to it, and the status moves from in_progress to completed.
CREATE TABLE users (
id UUID PRIMARY KEY,
tier TEXT NOT NULL, -- selects the limit set
balance BIGINT NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE fixed_window_usage (
user_id UUID NOT NULL REFERENCES users(id),
window_start TIMESTAMPTZ NOT NULL,
request_count BIGINT NOT NULL DEFAULT 0,
token_count BIGINT NOT NULL DEFAULT 0,
PRIMARY KEY (user_id, window_start)
);
CREATE TABLE request_events (
id BIGSERIAL PRIMARY KEY,
user_id UUID NOT NULL REFERENCES users(id),
occurred_at TIMESTAMPTZ NOT NULL,
tokens BIGINT NOT NULL,
idempotency_key TEXT NOT NULL UNIQUE
);
CREATE INDEX request_events_user_time_idx
ON request_events (user_id, occurred_at);
CREATE TABLE ledger_entries (
id BIGSERIAL PRIMARY KEY,
user_id UUID NOT NULL REFERENCES users(id),
amount BIGINT NOT NULL, -- negative for debit, positive for credit
idempotency_key TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
);
CREATE TABLE idempotency_keys (
idem_key TEXT PRIMARY KEY,
request_hash CHAR(64),
status TEXT NOT NULL, -- in_progress or completed
response_code INTEGER,
response_body TEXT,
created_at TIMESTAMPTZ NOT NULL
);
The request_events and ledger_entries tables each carry a unique idempotency_key. A second insert with the same key fails, so the database cannot double-count usage. The idempotency_keys table stores the original response so the server can answer a retry without re-running the model.
Two transaction rules hold the design together. First, the debit against the balance, the usage row, and the status update to completed belong in one transaction. Second, increment the fixed-window counter with an upsert (an INSERT that updates the existing row on conflict) that adds to the current value, never with a read-then-write sequence. Store balances and token counts as integers in the smallest unit you care about so floating-point rounding never creeps into the ledger.
Common enforcement bugs
Checking the limit and recording usage as separate steps. If the server reads the current usage, compares it to the limit, and then writes a new usage row, two concurrent requests can both read a count under the limit and both proceed. The check and the record must be one atomic operation: a conditional UPDATE, or an INSERT that fails on a unique constraint.
Fixed-window boundary bursts. A per-window counter lets a user cluster requests at the edges of two windows and effectively double the allowance. With a 1-minute window, the user sends the full allowance just before the boundary and again just after. A sliding window closes the gap at the cost of storing more timestamps.
Idempotency key mistakes. The common mistakes are generating a new key on each retry, storing the key only in memory so a restart loses it, and reusing a key without comparing request parameters. All three mistakes corrupt the usage numbers. The first two double-charge the user, and the third attributes one request’s usage to a different request. The key must be generated once, stored durably with the pending operation before the first attempt, and checked against a hash of the parameters on every replay.
Splitting the business change and the key update across transactions. If the server commits the generation and then crashes before updating the idempotency row, the retry sees no completed row and runs the generation again. The response, the usage record, and the idempotency update belong in one transaction. A related failure is a row stuck in in_progress forever; it needs a timeout so a later retry can proceed.
Expiring keys too soon. Keys need an expiration date, but a short retention reopens the duplicate risk: a queued retry that arrives after the key expired is treated as a new request. Set the retention longer than the longest realistic retry window, including retries that waited out an outage, and document the period so clients know how long a key stays valid.
Never testing the duplicate path. The standard test set is: send the same request twice with the same key; send two requests with the same key at the same moment; reuse a key with different parameters; and simulate a crash between the business change and the key update. Each should produce exactly one business record and exactly one quota charge.
