2024 / Backend / data pipeline

Async Reconciliation Engine

A settlement reconciliation pipeline for T+0, T+1, and T+2 cycles using CDC, Logstash, and Kafka while isolating heavy workloads from live payout traffic.

KafkaCDCLogstashPaymentsObservability

Context

Built around payout and settlement workflows that process high transaction volume.

Problem

Reconciliation workloads were heavy and time-sensitive, but running them too close to live traffic could affect payout system reliability.

Approach

  1. Used CDC through Maxwell to capture transaction changes without coupling to the hot path.
  2. Moved reconciliation processing through Logstash and Kafka for asynchronous handling.
  3. Separated T+0, T+1, and T+2 flows so each settlement cycle could be tracked clearly.

Impact

  • Improved reconciliation SRT by roughly 20-30%.
  • Isolated heavy workloads from live payout traffic.
  • Reduced manual reconciliation and operational effort.

Settlement timeline

A sample timeline can visualize transaction capture, Kafka processing, settlement grouping, mismatch detection, and final reconciliation status.

01 Capture CDC events02 Stream to Kafka03 Group by cycle04 Detect mismatches05 Produce reconciliation report
Back to Projects