Handling broker API rate limits and connection errors requires more than adding a retry loop. A reliable trading application must control request volume, distinguish rejected requests from uncertain outcomes, and recover without duplicating orders or trading on stale data.
The central rule is simple: retrying a data request and retrying an order are different decisions. A delayed quote request is an availability problem. An order whose acknowledgment disappeared is an exposure problem. Treat both as “try again” and the recovery process can become more dangerous than the original fault.
Within a trading API integration, build error handling around three questions: what failed, could the request have taken effect, and what evidence is required before trading resumes?
Separate Rate Limits From Other API Failures
HTTP 429 means the client has sent too many requests within a given period. The response may include a Retry-After header, but that header is not mandatory. The HTTP standard defining 429 Too Many Requests also leaves request counting and client identification to the server. A universal “requests per second” assumption is therefore a poor starting point.
Read the response body and the broker’s documented application error codes alongside the HTTP status. Build the following distinctions into the application’s decision process rather than sending every exception through one retry handler.
| Failure category | First action | Retry decision |
|---|---|---|
| Rate limit response | Pause affected traffic and update the shared request budget. | Wait for the permitted time; recheck whether the request is still useful. |
| Authentication or permission rejection | Inspect credentials, permissions, signing and session state. | Correct the cause rather than repeat unchanged requests. |
| Invalid order or request | Check parameters against endpoint requirements. | Do not resubmit the same invalid payload. |
| Server or gateway error | Record the error and assess whether processing could have occurred. | Use bounded retries for safe reads; reconcile uncertain orders. |
| Timeout or lost connection | Identify the failure stage and preserve request context. | Do not equate a missing acknowledgment with a rejected order. |
Keep transport status and trading status separate in the application. “Request failed locally” should not automatically become “order rejected.” Use an explicit unknown state when the evidence does not establish what happened.
Build a Request Budget Before Adding Retries
For each API, record the applicable limit, its scope, the counting window and any endpoint weights. Check whether the allowance is shared across an account, credentials, an IP address or an endpoint group. Also check simultaneous request limits, connection limits and subscription limits. Do not assume that opening another connection creates more permitted capacity.
Manage shared allowances through a shared limiter. If several strategy workers, a dashboard and a reconciliation service use the same budget, independent local counters are not enough. Route them through a common scheduler or coordinate consumption through shared state. Make budget allocation atomic so two workers cannot spend the same remaining allowance.
Calculate Demand by Workload
Consider a hypothetical allowance of 120 equally weighted requests per minute. Polling 30 instruments every 10 seconds consumes 180 requests per minute, before account checks or orders. No retry setting fixes that arithmetic. The workload must change.
Reduce duplicate requests, cache suitable reference data and combine compatible reads where the API supports batching. If streaming subscriptions meet the data requirement, use them to reduce repeated polling. The choice between REST and WebSocket APIs should reflect both delivery needs and recovery requirements, not just the number of open connections.
Reserve part of the permitted capacity for reconciliation and risk related actions. Give these requests priority over historical downloads and dashboard refreshes, where they share an allowance. A local reservation does not override the broker’s limit, so verify which actions consume which budget.
Bound the queue as well as the sending rate. Attach an expiry time to trading intentions and discard expired work before transmission. An entry signal that waited through a connection incident should not become an order simply because capacity is available again.
Use Backoff With Jitter and a Retry Deadline
For eligible transient failures, increase the delay between attempts rather than retrying immediately. Add jitter, meaning a random variation in the wait. This spreads retries instead of encouraging every worker to return at the same instant. These are the overload controls covered in Amazon’s guidance on timeouts, retries and backoff with jitter.
An illustrative policy for a safe read could use growing delay ceilings of one, two, four and eight seconds, selecting a random wait within each interval. Those values are examples, not broker requirements. Choose the schedule from the endpoint’s documented behavior, measured response times and the application’s tolerance for delay.
When a valid Retry-After instruction is present, do not retry before it permits. Support both its date and seconds formats. If the required wait exceeds the operation’s deadline, abandon or escalate the operation instead of shortening the server’s wait instruction.
Apply both an attempt cap and an elapsed time budget. Put retry ownership in one layer, or explicitly coordinate it across layers. Check whether the broker SDK or HTTP client already retries requests before adding application retries on top. Every attempt should pass through the same rate limiter as an ordinary request.
Set connection and response deadlines deliberately, then verify what the client library includes in each timeout. Cover the full operation, including connection establishment, rather than assuming one setting covers every network stage. For orders, a deadline determines when the client stops waiting; it does not determine whether the broker accepted the request.
Resolve Unknown Order Outcomes Before Resubmitting
An order submission can reach the broker even when its response never reaches the client. Automatically repeating a request that may create another order is unsafe without a documented deduplication mechanism or evidence that the first attempt was not applied. The HTTP rules for idempotency and automatic retries make this distinction explicit: non-idempotent requests should not be retried automatically without grounds for doing so safely.
Before sending an order, durably record the trading intention, account, instrument, quantity, direction and client identifier where supported. Keep enough information to investigate the outcome after a process restart. Preserve the original request’s identity throughout reconciliation.
Suppose an application submits an order for 100 shares and then times out. It should not immediately submit another 100 shares. Place the first request in an unknown state, query the documented order lookup facilities and inspect relevant execution updates. Check fills as well as open orders: an order that filled immediately may no longer appear in an open order list.
Do not assume that a client order ID is an idempotency guarantee. Verify whether the API treats it as a searchable reference, a uniqueness constraint or a deduplication token. Check its scope, retention period and behavior when the same identifier arrives with different order details.
Where documented idempotent submission is available, follow its exact retry rules. Otherwise, use broker supported reconciliation and escalate unresolved outcomes. One immediate “not found” response should not become proof of nonacceptance unless the API contract supports that interpretation.
Apply the same discipline to cancellations and replacements. A cancellation timeout leaves the order’s status uncertain. An acknowledgment that a cancellation request was received may also differ from confirmation that the order is canceled. Keep possible fills and remaining quantity in the risk calculation until authoritative updates resolve the state.
Reconnect WebSockets Without Declaring Premature Recovery
Reconnect with delay and backoff rather than a tight loop. The WebSocket protocol’s abnormal closure recovery guidance recommends a randomized initial delay and increasingly longer waits after repeated failures. Immediate reconnect attempts from many clients can obstruct service recovery.
Use one reconnect owner per logical session. Prevent competing callbacks from opening replacement connections, and record the close reason or transport error when available. Reset the backoff after a meaningful period of healthy operation, not immediately after the socket opens.
Treat transport recovery and trading readiness as separate states. A suggested recovery sequence is:
- Reconnect and authenticate using the documented session procedure.
- Restore required subscriptions and verify acknowledgments where provided.
- Reconcile orders, fills, positions and relevant account balances.
- Restore market data using the feed’s documented snapshot and recovery process.
- Resume trading only after freshness and consistency checks pass.
Track connection liveness, application messages and market data freshness separately. A healthy heartbeat should not, by itself, authorize a strategy to use an old quote. Equally, a quiet instrument should not automatically trigger a reconnect. Set freshness rules around the feed’s behavior, market hours and the strategy’s actual data requirements.
If a feed supplies sequence numbers or recovery cursors, follow its stated semantics. Do not assume they are consecutive, global or persistent across sessions. When the application cannot establish continuity, mark the affected state invalid and rebuild it through the supported recovery procedure.
Stop Repeated Failures From Becoming a Trading Incident
Use a circuit breaker to stop repeated calls to a persistently failing dependency. In the Microsoft circuit breaker pattern, normal requests pass while the circuit is closed, failures can open it, and a small number of test requests assess recovery in a half-open state.
For a trading application, define which operations each breaker controls. A failure in historical data retrieval should not automatically disable an otherwise healthy order cancellation route. Conversely, successful market data requests should not prove that order submission has recovered. Separate failure groups where the API and architecture allow it.
Use safe reads or documented health checks for recovery probes. Do not place live test orders just to see whether connectivity has returned. Before reopening strategy execution, require fresh data, reconciled exposure and resolution of outstanding unknown requests.
A circuit breaker is not a trading kill switch. Define separately when new entries stop, whether risk reducing actions remain permitted and who can authorize a restart. Stopping the local strategy does not cancel orders already held by the broker.
Log Decisions, Not Credentials
Record the endpoint category, request identifier, response status, application error code, attempt count, latency, queue delay and selected retry delay. For order operations, connect those records to the persisted intention and subsequent broker identifiers. Record why the handler retried, reconciled, paused or escalated each failure.
Exclude API secrets, authorization headers, session tokens and sensitive payload fields from logs. Prefer explicit allowlists for diagnostic fields, with redaction before data reaches monitoring services. Keep useful order correlation without turning incident logs into a second credential store.
Monitor throttling frequency, oldest queued request, connection churn, quote age, reconciliation duration and unresolved order outcomes. Alert on trading readiness, not simply whether the process is running. A green process indicator beside an unreconciled account is not much comfort.
Test the Recovery Path Before Live Use
Use a simulator or controlled fault injection to test delayed responses, throttling, disconnects, missed updates and service restarts. Include the awkward case where an order is accepted but its acknowledgment is lost. Restart the application during that uncertainty and verify that persisted records support recovery without a duplicate submission.
Test several workers exhausting one shared allowance, a reconnect followed by reconciliation traffic, and an expired signal waiting in the queue. Check that cancellation requests do not sit behind optional downloads and that retry exhaustion produces an actionable alert rather than disappearing into a log.
Make these checks part of ongoing trading bot monitoring and failure detection. Successful recovery means more than restored connectivity: the application knows its exposure, trusts its data and has no unresolved request that could change the next trading decision.