Integrated data storage system for geological laboratory experiments

The article examines the development of a new approach to storing and organizing the results of laboratory experiments with consideration to the features of their subsequent processing. To solve this problem, laboratory experiments are considered as structured data with unstructured parts. During the development of the system, the features of storing and processing laboratory test data were analyzed, after which the basic requirements for the system were formulated. The main data models were defined as well as the database entities. A standard relational data model has been chosen for storing structured data, and the storage of unstructured information such as experiment results or experiment parameters is implemented through the BJSON field. To solve the problem of providing secure access and creating an API for the system, the asynchronous FastAPI framework was chosen. The implementation of storing additional experiment files, which are located in the object storage and are associated with the experiment in the relational model through an additional entity, is also considered. The presented approach is notable for its flexibility to the structure of stored laboratory experiments, takes into account the features of geological laboratory experiments and also provides opportunities for complex meta-analysis of large volume of data. The system was tested and implemented into the technological process of the geotechnical laboratory at JSC MOSTDORGEOTREST.

1. Krasilnikov P.A., Hronusov V.V., Barskii M.G. Principles of creation and modelling engineering geology database. Geologiya i poleznye iskopaemye Zapadnogo Urala. 2018;(18):252–257. (In Russ.).

2. Dos Santos J.V., Thiesen S., dos Reis Higashi R.A. Geological-Geotechnical database from standard penetration test investigations using geographic information systems. Management of Information Systems. 2018:245–256. DOI: 10.5772/intechopen.74208.

3. Anishhik B.B. et al. Geological databases, as the basis for operation in geological direction. PRONEFT. Professionalno o nefti = PROneft. Professionally about Oil. 2022;(3):76–79. (In Russ.).

4. Krasilnikov P.A. Principles of creation and modelling engineering geology database of mineral deposits. Uchenye zapiski Krymskogo federalnogo universiteta imeni V.I. Vernadskogo. Geografiya. Geologiya. 2019;5(3):345–357. (In Russ.).

5. Levshina M.I., Tokarevskii P.A. Avtomatizirovannoe khranenie i obrabotka dannyh inzhenerno-geologicheskikh izyskanii. Perspektivy razvitiya inzhenernykh izyskanii v stroitelstve v Rossiiskoi Federatsii. 2016:44–47. (In Russ.).

6. Krasilnikov P.A. Review of modern software products used in engineering geology. Geologija i poleznye iskopaemye Zapadnogo Urala. 2020;(3):51–60. (In Russ.).

7. Boldyrev G.G., Diveev A.A. K voprosu ispolzovaniya informatsionnykh sistem pri izyskaniyah i proektirovanii osnovanii fundamentov zdanii i sooruzhenii. Nezavisimy elektronniy zhurnal "GeoInfo". 2020;(2). URL: https://geoinfo.ru/product/boldyrev-gennadij-grigorevich/k-voprosu-ispolzovaniya-informacionnyh-sistem-pri-izyskaniyah-i-proektirovanii-osnovanij-fundamentov-zdanij-i-sooruzhenij-42530.shtml (accessed on 25.12.2023). (In Russ.).

8. Strokova L.A. Opredelenie parametrov dlya chislennogo modelirovaniya povedeniya gruntov. Izvestiya Tomskogo politekhnicheskogo universiteta. 2008;313(1:):69–74. URL: https://cyberleninka.ru/article/n/opredelenie-parametrov-dlya-chislennogo-modelirovaniya-povedeniya-gruntov (accessed on 25.12.2023). (In Russ.).

9. Strokova L.A. Nauchno-metodicheskie aspekty sozdaniya raschetnykh modelei gruntovykh osnovanii. Izvestiya Tomskogo politekhnicheskogo universiteta. 2010;316(1):151–156. URL: https://cyberleninka.ru/article/n/nauchno-metodicheskie-aspekty-sozdaniya-raschetnyh-modeley-gruntovyh-osnovaniy (accessed on 25.12.2023). (In Russ.).

10. Lindvall J., Sturesson A. A comparison of latency for MongoDB and PostgreSQL with a focus on analysis of source code. Dissertation; 2021.

11. Peralta J.H. Microservice APIs: Using Python, Flask, FastAPI, OpenAPI and More. New York, Simon and Schuster; 2023. 411 p.