Construction of highly reliable distributed computing model in multi-source heterogeneous data environment

View at publisher Summarise with VIRAT
Authors: Huanhuan Guo, Guikun Cao, Lei Wang
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

In order to meet the requirements of high reliability and low latency in multi-source heterogeneous data scenarios, this paper proposes a highly reliable distributed computing model based on "data-computing-fault tolerance". Firstly, the heterogeneous data adaptation layer is designed, and the lossless fusion of cross-domain data is realized by dynamic format conversion and ontology alignment mechanism, which reduces the preprocessing delay to 90 ms and improves the throughput to 5.8 GB/s; Secondly, an adaptive fault-tolerant framework based on reinforcement learning (RL) is constructed. Through LSTM fault prediction and Q-learning task migration, the task success rate is still 96% under the 15% node failure rate, and the average recovery time is only 2.1s; Finally, the game theory hybrid resource scheduler is introduced to balance the heterogeneous CPU/GPU/FPGA load with the node utility function, and the Pareto optimization of throughput and delay is realized. The 32-node cluster experiment shows that the end-to-end delay of the model is 4.5× lower than that of Spark, and the delay fluctuation under fault impact is less than 8%, which can support millisecond-level key tasks such as remote surgery and industrial control.

Fields of Study & Subjects

Cloud Computing and Resource Management Distributed and Parallel Computing Systems Distributed systems and fault tolerance

Publication details

Journal / source
IET conference proceedings.
Volume
2025
Issue
47
Pages
695–700
Publication date
1 March 2026
Language
EN