Event-Driven Microservices and Streaming Data Platforms: Applying Kafka + Confluent in Real-Time Microservice Mesh on AWS & Azure

Read full text
Authors: Girish Rameshbabu
Cited by: 0

Abstract

As enterprise ecosystems transition from monolithic architectures to distributed microservices, traditional synchronous communication patterns (REST/gRPC) increasingly lead to temporal coupling, operational fragility, and data silos. These challenges are exacerbated in multi-cloud strategies where data must reside and synchronize across Amazon Web Services (AWS) and Microsoft Azure. This paper proposes a solution through the implementation of a decentralized "Event Mesh" architecture using Apache Kafka and the Confluent Ecosystem as the streaming backbone. We investigate the transition from request-response orchestration to asynchronous event choreography, addressing the "Dual-Write" problem via Change Data Capture (CDC) and ensuring data governance through the Confluent Schema Registry. A primary focus is placed on the technical mechanics of Confluent Cluster Linking to bridge the cloud divide without the overhead of traditional mirroring tools. Furthermore, we address critical implementation challenges, including Exactly-Once Semantics (EOS) for cross-cloud transactions, security via mTLS, and performance optimization across high-latency inter-cloud links. The resulting architecture provides a persistent, replayable, and scalable communication mesh that ensures real-time data consistency and fault tolerance across a heterogeneous cloud landscape.

Fields of Study & Subjects

Software System Performance and Reliability Advanced Database Systems and Queries Distributed systems and fault tolerance

Publication details

Journal / source
International Journal on Science and Technology
Volume
17
Issue
1
Publication date
9 January 2026
Language
EN