BUILDING AND MANAGING INDEX OF DISTRIBUTED SEARCH ENGINE BASED ON *BFT CONSENSUS ALGORITHM

In this article, we consider an algorithm for building a search index of a distributed search engine that uses the consensus algorithm of the *BFT (Byzantine Fault Tolerance) family, as well as its reorganization caused by the addition or removal of nodes from the search cluster. The article describes the structure of the cluster, addition of data to the index, cluster reorganization algorithm, and also discusses possible related problems and their solutions. At the same time, the limitations imposed by the very nature of distributed computing and the need to comply with BFT requirements are considered. The value of the task is due to the increasing need to use distributed systems to solve different practical tasks, including the organization of search, as well as the need to adapt existing solutions to the conditions of distributed systems and to take into account the limitations imposed by them, as evidenced by the existence of active research in this area and the current in the market situation. The methods and approaches used in this work, with some modifications, can be transferred to similar tasks arising in distributed systems of other types. The results and materials of the article can be used to solve applied problems in the implementation of distributed computing systems, distributed search systems, distributed data storage systems.

1. Benet J. IPFS - Content Addressed, Versioned, P2P File System / J. Benet – RAFT 3 – 2014. – 11 p. https://arxiv.org/pdf/1407.3561.pdf

2. Ongaro D., Ousterhout J. In Search of an Understandable Consensus Algorithm / D. Ongaro, J. Ousterhout – Stanford University – 2013. – 18 p.

3. Castro M., Liskov B. Practical Byzantine Fault Tolerance / Miguel Castro, Barbara Liskov – Laboratory for Computer Science, Massachusetts Institute of Technology – 1999. – 14 p.

4. Page L., Brin S., Motwani R., Winograd T. The PageRank Citation Ranking: Bringing Order to the Web. / Lawrence Page, Sergey Brin, Rajeev Motwani, Terry – Winograd Stanford InfoLab – 1999. – 17 p. http://ilpubs.stanford.edu:8090/422/

5. Back A. Hashcash - A Denial of Service Counter-Measure / A. Back – 2002. – 10 p. http://www.hashcash.org/papers/hashcash.pdf

6. Nakamoto S. Bitcoin: A Peer-to-Peer Electronic Cash System. – 2008. – 9 p. https://bitcoin.org/bitcoin.pdf

7. The Community-Powered Search Engine. – 2017. – 39 p. https://www.presearch.io/uploads/WhitePaper.pdf