Data Engineering Challenges in Visualizing Streaming and IoT Data

Abstract The proliferation of Internet of Things (IoT) devices and real-time streaming platforms has significantly increased the complexity of data engineering workflows. Continuous data streams generated by IoT systems require efficient processing, storage, and real-time visualization to support time-sensitive analytics and decision-making. However, the visualization of streaming and IoT data introduces several technical challenges, including high data velocity and volume, low-latency processing requirements, data heterogeneity, data quality management, and system scalability and reliability. This paper investigates the key data engineering challenges associated with real-time visualization of streaming and IoT data. It analyzes existing data processing architectures and stream-based technologies and proposes a structured data engineering methodology to address these challenges. Additionally, the study identifies relevant datasets and tools suitable for implementing and evaluating real-time streaming data visualization systems.