Features of deep learning application for license plate detection in images and their subsequent classification using computer vision methods

The article presents a technique for recognizing Russian car license plates using modern technologies of deep learning, computer vision and optical character recognition. The relevance of the study is due to the growing need for automated license plate recognition systems to improve road safety, optimize traffic flows and implement intelligent transport systems. The study consists of two stages. At the first stage, a neural network was trained to detect license plates in the image using the appropriate dataset of license plate. At the second stage, based on the received detections, image processing is carried out using computer vision methods, the selection of individual characters by segmentation, as well as their subsequent classification using an optical character recognition system with an adapted alphabet. The results obtained demonstrate the effectiveness of the proposed approach and the possibility of its application in real conditions. The materials of the article are of practical value for specialists involved in the development of automatic license plate recognition systems and can be used in the areas of access control, transport monitoring and road safety.

1. Zou Z., Chen K., Shi Z., Guo Y., Ye J. Object Detection in 20 Years: A Survey. Proceedings of the IEEE. 2023;111(3):257–276. https://doi.org/10.1109/JPROC.2023.3238524

2. Redmon J., Divvala S., Girshick R., Farhadi A. You Only Look Once: Unified, Real-Time Object Detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 27–30 June 2016, Las Vegas, NV, USA. IEEE; 2016. pp. 779–788. https://doi.org/10.1109/CVPR.2016.91

3. Sabu A.M., Das A.S. A Survey on Various Optical Character Recognition Techniques. In: 2018 Conference on Emerging Devices and Smart Systems (ICEDSS), 02–03 March 2018, Tiruchengode, India. IEEE; 2018. pp. 152–155. https://doi.org/10.1109/ICEDSS.2018.8544323

4. Anagnostopoulos C.N.E., Anagnostopoulos I.E., Loumos V., Kayafas E. A License Plate-Recognition Algorithm for Intelligent Transportation System Applications. IEEE Transactions on Intelligent Transportation Systems. 2006;7(3):377–392. https://doi.org/10.1109/TITS.2006.880641

5. Deb K., Le M.H., Woo B.-S., Jo K.-H. Automatic Vehicle Identification by Plate Recognition for Intelligent Transportation System Applications. In: Modern Approaches in Applied Intelligence: 24th International Conference on Industrial Engineering and Other Applications of Applied Intelligent Systems, IEA/AIE 2011: Proceedings, Part II, 28 June – 01 July 2011, Syracuse, NY, USA. Berlin, Heidelberg: Springer; 2011. pp. 163–172. https://doi.org/10.1007/978-3-642-21827-9_17

6. Ren S., He K., Girshick R., Sun J. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017;39(6):1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031

7. Liu W., Anguelov D., Erhan D., Szegedy C., Reed S., Fu C.-Y., Berg A.C. SSD: Single Shot MultiBox Detector. In: Computer Vision – ECCV 2016: 14th European Conference: Proceedings, Part I, 08–16 October 2016, Amsterdam, The Netherlands. Cham: Springer; 2016. pp. 21–37. https://doi.org/10.1007/978-3-319-46448-0_2

8. Smith R. An Overview of the Tesseract OCR Engine. In: Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), 23–26 September 2007, Curitiba, Brazil. IEEE; 2007. pp. 629–633. https://doi.org/10.1109/ICDAR.2007.4376991

9. Shi B., Bai X., Yao C. An End-to-End Trainable Neural Network for Image-Based Sequence Recognition and Its Application to Scene Text Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2017;39(11):2298–2304. https://doi.org/10.1109/TPAMI.2016.2646371

10. Baek Y., Lee B., Han D., Yun S., Lee H. Character Region Awareness for Text Detection. In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 16–20 June 2019, Long Beach, CA, USA. IEEE; 2019. pp. 9357–9366. https://doi.org/10.1109/CVPR.2019.00959

11. Lin T.-Y., Maire M., Belongie S., Hays J., Perona P., Ramanan D., Dollár P., Zitnick C.L. Microsoft COCO: Common Objects in Context. In: Computer Vision – ECCV 2014: 13th European Conference: Proceedings, Part V, 06–12 September 2014, Zurich, Switzerland. Cham: Springer; 2014. pp. 740–755. https://doi.org/10.1007/978-3-319-10602-1_48

12. Wang C.-Y., Yeh I.-H., Liao H.-Y.M. YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information. arXiv. URL: https://arxiv.org/abs/2402.13616 [Accessed 30th September 2024].

13. Weiss K., Khoshgoftaar T.M., Wang D. A Survey of Transfer Learning. Journal of Big Data. 2016;3(1). https://doi.org/10.1186/s40537-016-0043-6

14. Shorten C., Khoshgoftaar T.M. A survey on Image Data Augmentation for Deep Learning. Journal of Big Data. 2019;6(1). https://doi.org/10.1186/s40537-019-0197-0

15. Zheng Z., Wang P., Liu W., Li J., Ye R., Ren D. Distance-IoU Loss: Faster and Better Learning for Bounding Box Regression. In: Proceedings of the AAAI Conference on Artificial Intelligence: Volume 34, No. 07: AAAI-20 Technical Tracks 7, 07–12 February 2020, New York, NY, USA. Palo Alto: AAAI Press; 2020. pp. 12993–13000. https://doi.org/10.1609/aaai.v34i07.6999

16. Chollet F. Deep Learning with Python. Second Edition. Saint Petersburg: Piter; 2023. 576 p. (In Russ.).

17. Kaehler A., Bradski G. Learning OpenCV 3: Computer Vision in C++ with the OpenCV Library. Sebastopol: O'Reilly Media; 2016. 1024 p.

18. Neubeck A., Van Gool L. Efficient Non-Maximum Suppression. In: 18th International Conference on Pattern Recognition (ICPR'06), 20–24 August 2006, Hong Kong, China. IEEE; 2006. pp. 850–855. https://doi.org/10.1109/ICPR.2006.479

19. Gonzalez R.C., Woods R.E. Digital Image Processing. New York: Pearson Education Limited; 2018. 1168 p.

20. Onim M.S.H., Akash M.I., Haque M., Hafiz R.I. Traffic Surveillance using Vehicle License Plate Detection and Recognition in Bangladesh. In: 2020 11th International Conference on Electrical and Computer Engineering (ICECE), 17–19 December 2020, Dhaka, Bangladesh. IEEE; 2020. pp. 121–124. https://doi.org/10.1109/ICECE51571.2020.9393109

21. Shambharkar Y., Salagrama S., Sharma K., Mishra O., Parashar D. An Automatic Framework for Number Plate Detection using OCR and Deep Learning Approach. International Journal of Advanced Computer Science and Applications. 2023;14(4). http://doi.org/10.14569/IJACSA.2023.0140402

22. Abramov M.V., Yeltsov D.A. Neural networks with lstm and gru in application to the task of multiclass classification of text posts of social network users. Proceedings of Voronezh State University. Series: System Analysis and Information Technologies. 2023;(3):95–106. (In Russ.). https://doi.org/10.17308/sait/1995-5499/2023/3/95-106