FAULT-TOLERANT ARCHITECTURE FOR STREAM PROCESSING FOR REAL-TIME DATA ANALYSIS
DOI:
https://doi.org/10.17721/3041-2323.2024.161-171Keywords:
fault tolerance, stream data, distributed systems, data replication, real-time data analyticsAbstract
This article addresses the increasing demand for processing continuous data streams with low latency in the field of real-time data analytics. The paper explores existing approaches to fault tolerance in distributed stream processing systems and proposes a new hybrid architecture of asynchronous mutual replication. This architecture is designed to resolve issues related to latency and network interruptions while minimally impacting data consistency and availability at nodes. The article discusses fault tolerance mechanisms, data transmission stability, and system architecture to enhance the performance of distributed stream processing systems.
References
Bilokon, A., Borysov, S., Usatenko, M., & Fedorchenko, V. (2024). Analysis of the operation of distributed data processing and storage systems. Systems of Control, Navigation and Communications: Proceedings of the Scientific Papers, 84-88 [in Ukrainian]. https://doi.org/10.26906/SUNZ.2024.3.084.
Brewer, E. A. (2000). Towards robust distributed systems. Proceedings of the 19th Annual ACM Symposium on Principles of Distributed Computing, 7–10. Carbone, P. et al. (2015). Apache Flink™: Stream and batch processing in a single engine. IEEE Data Engineering Bulletin, 38(4), 28-38.
Corbett, J. C. (2012). Spanner: Google's globally distributed database. Proceedings of the 10th USENIX Symposium on Operating Systems Design and Implementation (OSDI ‘12), 251–264. DeCandia, G. (2007). Dynamo: Amazon's highly available key-value store. In Proceedings of the Twenty-First ACM SIGOPS Symposium on Operating Systems Principles (pp. 205–220).
Distributed Data Processing Technologies. (n. d.). StudFiles. Retrieved September 28, 2024, from https://studfile.net/preview/5118185/page:39.
Gilbert, S., & Lynch, N. (2002). Brewer's conjecture and the feasibility of consistent, available, partition-tolerant web services. ACM SIGACT News, 33(2), 51–59.
Globa, L. S. (2013). Development of Information Resources and Systems. Kyiv. [in Ukrainian].
Hunt, P., Konar, M., Junqueira, F. P., & Reed, B. (2010). ZooKeeper: Wait-free coordination for internet-scale systems. In Proceedings of the 2010 USENIX Annual Technical Conference (pp. 145–158).
Kreps, J., Narkhede, N., & Rao, J. (2011). Kafka: A distributed messaging system for log processing. Proceedings of the NetDB, 1–7.
Lamport, L. (2001). Paxos made simple. ACM SIGACT News, 32(4), 18–25.
Lakshman, A., & Malik, P. (2010). Cassandra: A decentralized structured storage system. ACM SIGOPS Operating Systems Review, 44(2), 35–40.
Lin, Q. (2017). STREAM: A scalable fault-tolerant real-time stream processing system. In Proceedings of the 2017 ACM International Conference on Management of Data (pp. 265–278).
Tanenbaum, A. S., & Van Steen, M. (2007). Distributed Systems: Principles and
Paradigms. Prentice-Hall.
Van Renesse, R., & Altinbuken, D. (2015). Paxos made moderately complex. ACM Computing Surveys, 47(3), Article 42.
Zhuravel, C., Dumych, S., & Shpur, O. (2021). Research on methods of data collection and processing in distributed information systems. Information and communication technologies electronic engineering [in Ukrainian]. https://science.lpnu.ua/sites/default/files/journalpaper/2021/dec/25668/stattya3czhuravelsdumichoshpur.pdf
Downloads
Published
Issue
Section
License
Copyright (c) 2024 Applied information systems and technologies in the digital society
This work is licensed under a Creative Commons Attribution 4.0 International License.
