Big Data Pipeline Architecture: Building the Reliability of Real-Time Messaging System on Apache Kafka

In big data era, increasing real-time requirements and reliability requirements drive organizations to make time-critical decisions.Reliable messaging systems are essential for innovative infrastructure and business application.Apache Kafka is a widely used distributed real-time messaging platform but ensuring reliable message delivery remains challenging due to message loss risks during server failures.The paper examines the default checkpoint interval for controlling the message loss problems and increases recovery time.The paper emphasizes the checkpoint interval-based message logging approach for the message recovery.The CIMLA (Checkpoint Interval Message Logging Algorithm) enhances the reliable message delivery by defining the optimal checkpoint interval and achieves the message logging to implement the recoverability requirements of Apache Kafka.The paper investigates various comparative studies on different data sizes, server failure cases, and performs the reliability enhancement of Apache Kafka by contributing to better message delivery and faster recovery time of the modified system.