Handle AI API rate limits without duplicate work
Diagnose 429 errors, bound retry budgets, preserve request identity, and distinguish a lost response from a confirmed failed AI request.
AI API rate limits and retries belong in the same request policy. Read the error code, respect an applicable waiting period, and decide whether repeating the operation is safe before sending it again. A timeout can leave the result uncertain. It does not establish that the model did no work or that the wallet was never charged.
Decide what to do before retrying
| What you know | What to do next |
|---|---|
| HTTP 429 identifies a temporary rate limit | Reduce concurrency and honor a valid Retry-After. Retry within your deadline only when the operation can be repeated safely. |
| The task timed out in your own queue before any HTTP submission | Requeue the same logical task if it is still needed and the deadline permits. |
| The request may have been sent, the stream broke, or billing is pending | Preserve the request reference and inspect Usage or the original job. Use the same key and payload only when the endpoint documents that recovery path. Keep an unresolved charge separate from finalized spend. |
For a first Meldport integration, start with one non-streaming request and a small output limit. Record its request identity and inspect usage before adding automatic retries. The API documentation describes the endpoints, while the free allowance page explains shared monthly tokens and wallet fallback. This guide addresses request recovery; it does not promise that every failure is refundable or replayable.
What does an AI API 429 response mean?
A 429 response means the server has applied a rate limit. It does not identify the limiting dimension by itself. RFC 6585, section 4 defines the status and permits a Retry-After header. Read the accompanying error details to distinguish a request-rate limit from another restriction exposed through the same status family.
In an application, separate requests per time window, concurrent work, output-token bounds, remaining free allowance, key permissions, and available wallet funds. Waiting can help a temporary rate window. It cannot repair an invalid key or enable a disabled fallback setting. Store the machine-readable error code alongside the HTTP status so your application can choose the correct action without matching English error messages.
A useful incident record contains the attempt time, endpoint, model ID, response status, error code, request ID, and whether any response data arrived. Avoid logging authorization headers or full prompts. If a user reports repeated failures, those limited fields help distinguish a client loop from an upstream problem while keeping private request content out of ordinary logs.
Put a queue before a retry loop
A small queue with bounded concurrency can prevent a burst before it becomes a stream of rejected requests. Give each queued operation an expiry time. If the user has already left or a scheduled job is no longer useful, discard the operation before dispatch rather than spending money to complete stale work.
Consider an illustrative application that receives twenty independent tasks at once and allows two active requests. Eighteen tasks wait locally. This is a scheduling example, not a published Meldport concurrency limit. The queue reduces simultaneous dispatch, but its waiting time still counts against your own user-facing deadline. Expose that state honestly instead of showing a spinner that suggests the model is already responding.
Keep the queue small enough that a provider outage cannot accumulate hours of hidden demand. Record separate durations for local waiting, connection setup, first response, and completed response. A slow queue and a slow model require different fixes. Reducing prompt size will not solve a worker that never becomes available, and adding more workers can worsen an upstream rate limit.
How should you use Retry-After?
When an applicable Retry-After header is present, use it as the server's waiting instruction. RFC 9110, section 10.2.3 allows either a number of seconds or an HTTP date. Parse both forms and reject malformed values. A future date also depends on clock accuracy, so keep the client's clock synchronized.
Your application still owns its deadline. If the server asks you to wait longer than the remaining operation budget, stop and explain when the user can try again. Do not shorten that instruction merely to keep an interface moving. An invalid or missing header should follow a documented local policy rather than a zero-delay retry.
For example, assume an internal task has a twenty-second remaining deadline and receives a valid instruction to wait thirty seconds. The task cannot complete within its budget, even before another model call. Return a deferred or failed application result with the recorded reason. A scheduled worker may reschedule the same logical task, but that is a deliberate later action with its own identity and accounting.
Bound exponential backoff and add jitter
Where another attempt is safe, increase the delay between attempts and vary that delay across clients. The Amazon Builders' Library discussion of backoff and jitter explains how synchronized retries can amplify load. Set a maximum attempt count and a maximum elapsed time; neither should be left implicit in a library default.
As a hypothetical starting policy for a confirmed pre-dispatch transient rejection, allow the initial attempt plus two retries, with a two-second base and an eight-second cap. Those values are examples, not service limits or a recommendation for every workload. A valid longer server waiting instruction takes priority. Measure your own completion and delay distribution before adopting any policy broadly.
Assign retry ownership to one layer. If a job runner retries three times and an SDK also makes three attempts, one logical job can cause up to nine submissions. A reverse proxy or agent loop may add another multiplier. During evaluation, inspect each layer's settings and count actual outbound requests, including unsuccessful ones. A setting named “two retries” commonly means three attempts including the original.
Preserve identity when repeating a supported operation
An idempotency key identifies one logical operation within a service's stated contract. Keep the key stable for a supported retry with the same normalized request. Generate a new key when the user intentionally requests a new result. Changing the key after an uncertain dispatch can create new work instead of recovering the original operation.
Meldport's Chat Completions handler validates the key and request fingerprint. A conflicting payload is rejected. An operation already in progress is reported separately. A completed non-streaming response may be replayed while its stored replay data remains available. Older or unavailable replay data can produce an explicit replay-unavailable error; key reuse is not a permanent archive of every response.
The scope and retention of this behavior matter. Preserve the same API key and request body during your first recovery check, and consult the current endpoint documentation before building a wider guarantee. Do not infer identical behavior for another vendor, another endpoint, or a different authentication context. A local task identifier is useful even when the remote operation has no replay facility.
A broken stream needs a separate recovery path
A streaming response can deliver partial text before the connection closes. Repeating the prompt is then a second generation unless the service explicitly provides a continuation or replay contract. Keep the partial content and request ID available to the user. Do not silently append a restarted answer to an earlier fragment as if it were one uninterrupted response.
Meldport does not replay saved Server-Sent Events frames for a completed Chat Completions stream. Its duplicate and in-flight protections do not turn streaming into an exactly-once transcript service. A later retry can fail closed when no replay body is available. Start integration testing with stream disabled, then test streaming and disconnect behavior as separate cases.
For a coding agent, there is an additional boundary: the model may already have proposed a tool call. Your application must know whether the tool ran before asking for more work. A repeated file edit or external API action can have effects beyond inference cost. Test with a read-only task and manual tool approval before enabling autonomous changes in a real repository.
Diagnose free allowance and wallet errors directly
Meldport's monthly free allowance is one shared account pool across eligible :free models. It is not a new pool for each alias or API key. The current allowance is published on the free models page. A request can require more reserved tokens than remain in the pool, so the fallback decision may arise before the displayed balance reaches literal zero.
Wallet fallback requires both the account preference and the relevant API key's fallback setting, plus sufficient usable wallet funds and the other request permissions. If a required setting is disabled, repeating the request will not enable it. Explain the policy to the user and let them change the setting or choose a suitable paid model deliberately.
An application should distinguish exhausted free tokens from insufficient wallet credit in its own interface. The same user may be allowed to use a small free request while lacking funds for a larger paid request. Conversely, a funded wallet does not override a key that prohibits fallback. Verify the actual account and key configuration instead of treating every quota-related error as a temporary service outage.
Treat media submission and polling as different actions
A media job is an asynchronous operation. Create it once using the endpoint's required idempotency identity, retain the returned job ID, and query that job's state. Do not create another job merely because the first polling response is slow. The media quickstart explains quoting and the generation lifecycle.
Meldport checks media request fingerprints, including the configuration, prompt, options, and input assets. A reused key with a different request conflicts. A successful replay can return the existing job instead of dispatching another one. That identity is different from an LLM conversation ID and should remain attached to the application's original media task.
A job in reconciliation needs investigation of the existing operation. An unresolved provider response can leave the final cost uncertain. Neither a missing output URL nor a client-side timeout proves that a reservation should be released immediately. Show the known state, keep the job reference, and avoid promising a refund until the financial record establishes the outcome.
Test recovery before enabling it for everyone
Use a fixture or controlled test service to exercise a temporary rejection, an invalid key, a conflicting idempotency payload, a lost response after dispatch, and a partial stream. Count provider submissions as well as client attempts. A test that returns the expected JSON without checking duplicate dispatch can miss the most expensive failure.
Next run a small, explicitly bounded request against the intended environment and model. Record client version, endpoint, model ID, output cap, status, usage, and final wallet or allowance outcome. A fixture pass proves your recovery logic against a known contract. It does not prove that every live provider supports the same stream shape, tools, latency, or billing evidence.
Keep a recovery budget beside the token budget. If each of three illustrative attempts costs $0.01, a completed task can cost $0.03 before other services, even when the displayed model rate is unchanged. Use the completed-task cost guide and workload calculator to compare planned demand with recorded usage. Limit repeat work before increasing traffic.