X

How to Set Up Secure Webhooks: Signatures, Retries, and Troubleshooting

A webhook lets a service send your application an HTTP request when an event happens, such as a payment succeeding or a repository changing. It saves you from repeatedly asking an API whether anything has changed, but it also gives an outside service a route into your application.

For a small product, a reliable webhook endpoint needs to do four things: authenticate each request, accept it quickly, avoid processing the same event twice, and make failures visible. The exact signature format and retry schedule depend on the provider, so use its current documentation rather than treating every webhook as interchangeable.

1. Create a dedicated HTTPS endpoint

Give each provider and environment a specific route, such as /webhooks/github or /webhooks/stripe. Configure the provider to send only the event types your application needs. Use HTTPS in production, and keep development, staging, and production secrets separate.

Treat the URL as a public endpoint, not as authentication. Anyone who discovers it can send requests to it. A hard-to-guess path may reduce noise, but it does not prove that a request came from the service you expect.

2. Verify the signature against the original request body

Providers sign requests in different ways. GitHub, for example, sends an HMAC-SHA256 digest in the X-Hub-Signature-256 header. Its delivery validation guide says to calculate a digest with the configured secret and compare it with the header using a constant-time comparison.

Here is a small Node.js helper for GitHub's signature format:

import { createHmac, timingSafeEqual } from 'node:crypto';

export function verifyGitHubSignature(rawBody, signatureHeader, secret) {
  if (!Buffer.isBuffer(rawBody)) {
    throw new TypeError('Webhook body must be the original request bytes.');
  }
  if (typeof secret !== 'string' || secret.length === 0) {
    throw new Error('GitHub webhook secret is not configured.');
  }

  const match = /^sha256=([a-f0-9]{64})$/i.exec(signatureHeader ?? '');
  if (!match) return false;

  const expected = createHmac('sha256', secret).update(rawBody).digest();
  const received = Buffer.from(match[1], 'hex');
  return received.length === expected.length && timingSafeEqual(expected, received);
}

This checks only GitHub's HMAC header; it is not a universal webhook verifier. For other providers, follow their signing format or use their maintained SDK. For example, Stripe's signature guidance requires the original request body, the Stripe-Signature header, and the secret for that endpoint.

Do not parse and re-serialize JSON before verification. Even harmless changes to whitespace or character encoding change the bytes being signed. In Express, use the raw body parser for the webhook route before any JSON-parsing middleware, and set a reasonable body-size limit. Parse the JSON only after the signature has passed. Keep the secret in an environment variable or secret manager, never in source control or a client-side bundle.

3. Accept the event before doing slow work

Avoid running a long task—such as sending email, generating a report, or calling several other services—while the provider waits for your HTTP response. A slow endpoint can time out and cause another delivery even if some of the work already happened.

A safer flow is:

  1. Read the request with a size limit and verify its signature.
  2. Check that the event type is one your application handles.
  3. Persist the delivery identifier and event, or enqueue them durably.
  4. Return a successful 2xx response after that acceptance is complete.
  5. Process the queued event separately and record its outcome.

The acknowledgment should mean the event is safely recorded, not merely that a server process started. If the database or queue is unavailable, return an error so the provider or your recovery process can retry. Stripe's webhook guidance likewise recommends responding before doing slower downstream work.

4. Make event handling idempotent

Assume that an event can arrive more than once. A provider may retry after a timeout even when your application completed the work but its response never reached the provider. Stripe documents duplicate event delivery and recommends recording processed event IDs; it also notes that events are not guaranteed to arrive in creation order.

Store a unique key for each accepted delivery, such as the provider name plus GitHub's X-GitHub-Delivery value or Stripe's event ID. Enforce uniqueness in the database, not just in application memory. When the same key appears again, acknowledge it without repeating the side effect. For operations that can be triggered by two distinct events, make the business operation itself idempotent too—for example, enforce one fulfillment per order ID.

Do not assume that event order reflects the current state of a resource. When order matters, retrieve the latest state from the provider's API or compare version/timestamp fields according to that provider's documentation.

5. Plan for provider-specific retries and recovery

Retry behavior differs. As its current documentation describes, Stripe retries live-mode deliveries for up to three days with exponential backoff, while sandbox deliveries have a shorter retry schedule. GitHub does not automatically redeliver failed webhook deliveries; you need to inspect and redeliver them using GitHub's tools or recover them another way.

Build a recovery path around the provider you use:

  • Check its delivery dashboard for response codes, attempts, and timestamps.
  • Alert on repeated failures or a growing queue rather than relying on manual checks.
  • Retain enough event IDs and processing status to identify events that need replay.
  • Make replay safe by using the same idempotency checks as normal delivery.
  • Confirm the provider's retry window and manual-redelivery behavior before relying on it as your only recovery mechanism.

Keep a small log record with the provider, delivery ID, event type, receipt time, processing status, and error category. Avoid logging signing secrets, signature headers, authorization data, or complete payloads that may contain personal or payment information.

6. Test the real failure paths

Use the provider's documented test events or command-line tools, plus a development endpoint or secure tunnel when local testing requires one. Use a test secret and test data; never forward production events into a development environment by default.

Test more than the happy path:

  • A valid signature is accepted, while a missing or altered signature is rejected.
  • A repeated delivery does not repeat the business operation.
  • A slow or temporarily failing worker can recover from a queued event.
  • An unknown event type is safely ignored or recorded without triggering an unintended action.
  • A malformed or oversized body is rejected without exhausting server resources.
  • A provider outage or deployment interruption has a documented replay or reconciliation path.

For debugging, compare the endpoint's delivery ID, event type, timestamp, and response status with the provider's delivery log. Do not disable signature checks to make a test pass; a mismatch usually means the wrong secret, a different environment, or a body parser modified the bytes before verification.

Common webhook problems

Symptom What to check
Every request fails signature verification Confirm the endpoint secret and environment, verify the unmodified raw body, and check that JSON middleware has not run first.
The provider keeps sending the same event Check whether the endpoint timed out or returned a non-2xx response. Look up the delivery ID before repeating side effects.
An event seems to be missing Check that the provider is subscribed to that event type and inspect its delivery history; do not assume every provider retries automatically.
Events appear out of order Make the handler idempotent and retrieve the current resource state when event order is significant.
Local testing succeeds but production fails Check the deployed route, HTTPS and reverse-proxy configuration, body-size limits, and the production endpoint's own signing secret.

Webhooks are one part of an integration, not a substitute for understanding the service you connect to. If you are also choosing an API to call from your application, see our guide to API directories and discovery tools.

Official references

Categories: Development
Tags: API services
Related Post