Influence of the TensorFlow library’s version on the quality of code generation from an image

This study compares the efficiency of training models that implement two different approaches: complicating the original neural network architecture, or maintaining the architecture while improving the tools used in the training framework. Attempts to complicate the architecture of the solution for generating source code based on an image lead to solutions that might be difficult to support in the future. At the same time, such improvements do not use more modern tools and libraries that such systems are built upon. The relevance of the study is due to the lack of attempts to use more modern and relevant libraries. In this regard, during the experiment to compare the indicators of models of three versions of image-based source code generation systems: the original pix2code system, its complex version, and the version with a modern version of the TensorFlow library - in the process of their training, it was revealed that approaches with a complex architecture and the current TensorFlow have the same indicators, higher quality than the original pix2code. Based on the experiment, we can conclude that updating the TensorFlow library can provide an additional increase in the quality of results that the image-based source code generation system can predict.

1. Beltramelli T. pix2code: Generating Code from a Graphical User Interface Screenshot. In: EICS '18: Proceedings of the ACM SIGCHI Symposium on Engineering Interactive Computing Systems, 19–22 June 2018, Paris, France. New York: Association for Computing Machinery; 2018. https://doi.org/10.1145/3220134.3220135

2. Zou D., Wu G. Automatic Code Generation for Android Applications Based on Improved Pix2code. Journal of Artificial Intelligence and Technology. 2024;4(4):325–331. https://doi.org/10.37965/jait.2024.0515

3. Balog M., Gaunt A.L., Brockschmidt M., Nowozin S., Tarlow D. DeepCoder: Learning to Write Programs. In: 5th International Conference on Learning Representations, ICLR 2017: Proceedings, 24–26 April 2017, Toulon, France. 2017. https://doi.org/10.48550/arXiv.1611.01989

4. Liu Y., Hu Q., Shu K. Improving pix2code based BI-directional LSTM. In: 2018 IEEE International Conference on Automation, Electronics and Electrical Engineering (AUTEEE), 16–18 November 2018, Shenyang, China. IEEE; 2019. pp. 220–223. https://doi.org/10.1109/AUTEEE.2018.8720784

5. Chen W.-Y., Podstreleny P., Cheng W.-H., Chen Y.-Y., Hua K.-L. Code generation from a graphical user interface via attention-based encoder-decoder model. Multimedia Systems. 2022;28(1):121–130. https://doi.org/10.1007/s00530-021-00804-7

6. Zhu Z., Xue Z., Yuan Z. Automatic Graphics Program Generation Using Attention-Based Hierarchical Decoder. In: Computer Vision – ACCV 2018: 14th Asian Conference on Computer Vision: Revised Selected Papers: Part VI, 02–06 December 2018, Perth, Australia. Cham: Springer; 2019. pp. 181–196. https://doi.org/10.1007/978-3-030-20876-9_12

7. Yao X., Yap M.H., Zhang Y. Towards a Deep Learning Approach for Automatic GUI Layout Generation. In: Proceedings of International Conference on Computing and Communication Networks, ICCCN 2021, 19–20 November 2021, Manchester, United Kingdom. Singapore: Springer; 2022. pp. 19–27. https://doi.org/10.1007/978-981-19-0604-6_2

8. Nguyen T.A., Csallne C. Reverse Engineering Mobile Application User Interfaces with REMAUI (T). In: 2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE), 09–13 November 2015, Lincoln, USA. IEEE; 2016. pp. 248–259. https://doi.org/10.1109/ASE.2015.32

9. Lin T.-Y., Goyal P., Girshick R., He K., Dollár P. Focal Loss for Dense Object Detection. IEEE Transactions on Pattern Analysis and Machine Intelligence. 2020;42(2):318–327. https://doi.org/10.1109/TPAMI.2018.2858826

10. Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I. Attention Is All You Need. In: NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems, 04–09 December 2017, Long Beach, USA. New York: Curran Associates Inc.; 2017. pp. 6000–6010.