API Integration Best Practices: Retries and Webhooks

A reliable API integration assumes that things will fail. Requests time out, the same webhook arrives twice, the other system is down for an hour, and a field you thought was always present turns up empty. The core practices are to make operations idempotent, retry only what is safe to retry with increasing delays, verify and acknowledge webhooks quickly, respect rate limits, and log enough to explain what happened. Do those well and most of the horror stories about "the sync created 400 duplicates" never happen.

Start by deciding who owns the data

Before any code, decide which system is the source of truth for each piece of data. A CRM may own contact details while an accounting system owns invoice status. When two systems both write the same field, you get ping-pong updates and arguments about whose value is right. Write the ownership down, per field, and make the integration obey it. This planning step is part of any CRM implementation plan.

Idempotency: make repeats harmless

Imagine your integration sends a request to create an invoice, the connection drops before the response arrives, and your code retries. Did the first request succeed? If the API is not idempotent for that call, you may now have two invoices.

Ways to achieve it

• Idempotency keys. Many APIs let you send a unique key with a request. If the same key is seen again, the server returns the original result instead of repeating the action. Check the provider documentation for support and for how long keys are remembered.
• Natural unique identifiers. Use something stable and unique, such as an order number from your system, and have the receiving side upsert (update or insert) on it rather than blindly create.
• Check before you create. Search for an existing record by the identifier first. This is slower and has a race condition under concurrency, so prefer the first two options when available.
• Store what you have processed. When receiving events, record each event ID once processed and skip any you have already seen.

As a rule of thumb, reads and updates that set a value (such as "set status to paid") are naturally safe to repeat, while "create" and "add to balance" operations need protection.

Retries: bounded, delayed and selective

Retrying blindly can make an outage worse and can double-apply actions. A sensible policy has four elements.

1. Retry only transient failures. Timeouts, connection errors, rate-limit responses and server-side errors (HTTP 5xx) may succeed later. Client errors such as 400 (bad request), 401 or 404 will not be fixed by trying again.
2. Increase the wait between attempts. This is usually called exponential backoff. Add a small random variation (jitter) so many clients do not retry at the same instant.
3. Cap the attempts. A bounded number of retries, then stop.
4. Park the failures. Move anything that still fails to a dead-letter queue or an error table that a person reviews, rather than dropping it silently.

attempt = 1
while attempt <= MAX_ATTEMPTS:
    response = send(request, idempotency_key)
    if response.ok: return response
    if not is_transient(response): break
    wait(base_delay * 2 ** (attempt - 1) + random_jitter())
    attempt += 1
send_to_review_queue(request, response)

If the API returns a Retry-After header with a rate-limit response, honour it instead of guessing.

Webhooks: receive fast, verify, process later

Webhooks let a source system call you when something happens. They are efficient, but receiving them well takes care.

Webhook receiving checklist

• Verify the signature. Compute the signature over the raw request body using the shared secret and compare it using a constant-time comparison. Reject mismatches.
• Acknowledge quickly. Return a success response promptly, then do the real work asynchronously from a queue. Providers often retry if you are slow, which creates duplicates.
• Expect duplicates and disorder. The same event can arrive more than once, and events can arrive out of order. De-duplicate on event ID and use timestamps or version numbers to ignore stale updates.
• Do not trust the payload blindly. For important changes, fetch the current record from the API using the ID in the event, instead of acting on possibly outdated data.
• Serve over HTTPS and keep the endpoint path unguessable and the secret out of source control.
• Plan for missed events. Add a periodic reconciliation job that compares recent records on both sides and repairs gaps.

Polling versus webhooks

Approach	Strengths	Weaknesses	Use when
Webhooks	Timely, efficient	Need a public endpoint, duplicates and gaps possible	You need prompt updates and the provider supports them
Polling	Simple, you control timing	Delay, wasted calls, rate-limit pressure	No webhooks exist, or changes are infrequent
Both	Fast with a safety net	More to build	Data accuracy matters and you can afford the extra work

Authentication and secrets

• Use the authentication method the provider recommends, typically OAuth 2.0 for user-delegated access or API keys for server-to-server calls.
• Grant the narrowest permissions that do the job.
• Keep keys and tokens in a secrets manager or environment configuration, never in code or shared documents.
• Plan for rotation: know how to replace a key without downtime.
• Handle token expiry and refresh explicitly.

Rate limits, pagination and batching

Every serious API limits how fast you can call it. Read the limits in the provider documentation and design around them rather than discovering them in production.

• Back off when you receive a rate-limit response and honour any wait time supplied.
• Paginate through large result sets using the cursor or page tokens the API provides, and do not assume everything arrives in one response.
• Use bulk or batch endpoints where they exist, instead of thousands of single calls.
• Fetch changes since your last sync rather than the whole dataset each time, if the API supports a modified-since filter.

Validation, mapping and versioning

Treat incoming data as untrusted. Validate required fields, types and formats before writing them into your system, and decide what to do with records that fail. Keep field mappings in one clearly documented place. When the provider releases a new API version, schedule an upgrade deliberately, and read their deprecation notices so a retired endpoint does not surprise you.

Logging and monitoring

When an integration misbehaves, you need to answer "what happened to this record?" quickly.

• Log a correlation ID with each request, event and outcome, so one record can be traced end to end.
• Record request status, duration and retry count. Avoid logging secrets or unnecessary personal data.
• Alert on sustained failure rates, a growing error queue and a long gap since the last successful sync.
• Give someone responsibility for reviewing failures weekly.

Common mistakes

• Retrying everything immediately. This hammers a struggling service and can duplicate actions.
• Treating webhooks as exactly-once. Design for at-least-once delivery.
• Doing slow work inside the webhook request, causing timeouts and repeat deliveries.
• No reconciliation. Silent gaps accumulate until someone notices the numbers do not match.
• Hard-coded credentials and shared admin keys.
• Assuming field formats are stable. Dates, currencies and null values vary between systems.
• No owner for the integration once the project that built it ends.

When a no-code tool is enough

Not every connection needs custom code. For low-volume, low-risk flows, a tool such as Zapier or Make may be perfectly adequate, and we compare the options in workflow automation: what to automate first. Move to custom integration when you need strict error handling, high volume, complex transformations or audit requirements.

Where to go from here

Pick your most important integration and test it against this list: can a repeated request do harm, what happens after an hour of downtime, and how would you find one missing record? Gaps in those answers are your priority. If you want help designing or hardening integrations between your CRM and other systems, CloudWiss provides CRM and integration services, including third-party API integrations and custom API work.