Production Kafka for Order and Payment Events

Kafka in a demo is easy: produce a JSON blob, consume it, log "success." Kafka in a food-delivery pipeline — payments, orders, driver soft-offers, POS — fails in boring, expensive ways: silent loss, duplicate side effects, and untraceable hops.

This post summarizes the hardening patterns we applied after auditing a live multi-service Kafka surface.

The failure modes that matter

Failure	Naive behavior	Better behavior
Handler throws / DB timeout	Catch, log, auto-commit → message lost	Retry with backoff → DLT → alert + replay
Poison JSON	Parse fails, swallow	`ErrorHandlingDeserializer` → DLT immediately
Kafka down after DB commit	Produce logged and discarded	Transactional outbox + poller
Redelivery of `PAYMENT_SUCCESS`	Re-send email/WhatsApp	Idempotent consume (`eventId` + status guards)
Cross-service debug	New trace id every hop	Propagate `traceId` HTTP → MDC → Kafka headers

If your consumers catch Exception and never rethrow while enable-auto-commit=true, you do not have retries — you have best-effort logging.

Inventory before you "improve"

Map every producer and @KafkaListener: topics, serializers, ack mode, and whether anything actually publishes to orphaned topics. In our audit, a POS consumer listened on a topic nothing produced. Dead code looks like coverage until you need it on a Friday night.

Shared messaging library

We extracted a small shared module used by producers/consumers:

Modern Jackson JSON serializers (no deprecated Spring Kafka JSON helpers)
DefaultErrorHandler + dead-letter topics
ErrorHandlingDeserializer for poison payloads
Trace context helpers (headers in, MDC out)
Structured logging with redaction hooks
Idempotency store (processed eventIds)
Outbox writer/publisher for dual-write-sensitive paths

Centralizing this stops each service inventing a slightly wrong retry story.

Consumer stop-loss checklist

For every listener:

Do not swallow — let the error handler own retries
Prefer manual ack with clear success boundaries
Configure DLT topics and know who watches them
Keep application Kafka logs visible at ERROR (silencing everything to save CloudWatch dollars hides outages)
Add idempotency: eventId + durable "already processed" record + status guards for business transitions

Producer and outbox

Uneven producer configs are common: some services set acks=all and retries; others fire-and-forget. Lock the baseline:

Explicit idempotent producer settings where supported
No ignored CompletableFuture / send callbacks on critical paths
Outbox for payment and order publishes that must not diverge from DB commits

The classic dual-write (write Dynamo/SQL, then produce) will eventually lose events. Outbox turns that into an operatable lag problem instead of silent inconsistency.

Observability that operators can use

Propagate correlation/trace ids across HTTP and Kafka
Micrometer counters for produce/consume/DLT/retry
OpenTelemetry sampling on ECS tasks (start conservative)
Runbook: bootstrap servers per env, broker upgrade notes, DLT replay, lag alerts

Broker version lag (old broker, newer clients) deserves an explicit upgrade path — do not discover it during a deserializer CVE weekend.

Closing

Industry-standard Kafka is not "we use Kafka." It is retry, quarantine, replay, idempotency, and traces. Harden the consumers that move money and drivers first; orphaned topics and fire-and-forget producers are the audit findings that prevent the next silent incident.