Title :
DUAL-WRITE RECONCILIATION BETWEEN REAL-TIME SERVING STORES AND OPEN LAKEHOUSE TABLES: A COMPARATIVE STUDY OF THREE STRATEGIES
Muralikrishna Dudaka
Abstract : Modern enterprise streaming platforms increasingly adopt a hybrid architecture in which a real-time OLAP store serves sub-second dashboards and a parallel open lakehouse table serves long-horizon analytics, audit, and machine-learning training. Both tiers receive the same source events through a dual-write fan-out and must remain mutually consistent under continuous load, partial failures, and out-of-order arrivals. The reconciliation discipline that keeps them aligned determines whether dashboards diverge from compliance reports, whether trained models reflect the current production state, and whether downstream consumers receive contradictory inputs. This article compares three reconciliation strategies—idempotent-key upsert, compaction-merge, and stream-replay—on the dimensions of 99th-percentile staleness, recall under failure, and compute and storage cost. The central falsifiable claim is that under a uniform-key workload, the compaction-merge strategy holds p99 staleness at or below one compaction interval T for write rates up to a derived inflection point R = W / L, where W is compaction-window capacity and L is mean key lifetime; above R, p99 staleness grows as O(T • (rate/R)²). A Python discrete-event simulator validates the inflection behavior across four swept write-rate configurations. A strategy-fit decision matrix for production deployment is provided.
Keywords : dual-write, reconciliation, streaming, lakehouse, Apache Iceberg, Delta Lake, change data capture, p99 staleness, compaction, LSM-tree