| 2024 | |
| [1] | , "Continuous Software Engineering Practices in AI/ML Development Past the Narrow Lens of MLOps: Adoption Challenges", In e-Informatica Software Engineering Journal, vol. 18, no. 1, pp. 240102, 2024.
DOI: 10.37190/e-Inf240102. Download article (PDF)Get article BibTeX file |
Authors
Sini Vänskä, Kai-Kristian Kemell, Tommi Mikkonen, Pekka Abrahamsson
Abstract
Background: Continuous software engineering practices are currently considered state of the art in Software Engineering (SE). Recently, this interest in continuous SE has extended to ML system development as well, primarily through MLOps. However, little is known about continuous SE in ML development outside the specific continuous practices present in MLOps.
Aim: In this paper, we explored continuous SE in ML development more generally, outside the specific scope of MLOps. We sought to understand what challenges organizations face in adopting all the 13 continuous SE practices identified in existing literature.
Method: We conducted a multiple case study of organizations developing ML systems. Data from the cases was collected through thematic interviews. The interview instrument focused on different aspects of continuous SE, as well as the use of relevant tools and methods.
Results: We interviewed 8 ML experts from different organizations. Based on the data, we identified various challenges associated with the adoption of continuous SE practices in ML development. Our results are summarized through 7 key findings.
Conclusion: The largest challenges we identified seem to stem from communication issues. ML experts seem to continue to work in silos, detached from both the rest of the project and the customers.
Keywords
Artificial Intelligence, Machine Learning, Continuous Software Engineering, Continuous Star, Multiple Case Study
References
1. C. Parnin, E. Helms, C. Atlee, H. Boughton, M. Ghattas et al., “The top 10 adages in continuous deployment,” IEEE Software, Vol. 34, No. 3, 2017, pp. 86–95, https://doi.org/10.1109/MS.2017.86.
2. L. Leite, C. Rocha, F. Kon, D. Milojicic, and P. Meirelles, “A survey of DevOps concepts and challenges,” ACM Computing Surveys (CSUR), Vol. 52, No. 6, 2019, pp. 1–35, https://doi.org/10.1145/3359981.
3. S. Moreschini, F. Lomio, D. Hästbacka, and D. Taibi, “MLOps for evolvable AI intensive software systems,” in 2022 IEEE International Conference on Software Analysis, Evolution and Reengineering (SANER), 2022, pp. 1293–1294, https://doi.org/10.1109/SANER53432.2022.00155.
4. G. Giray, “A software engineering perspective on engineering machine learning systems: State of the art and challenges,” Journal of Systems and Software, Vol. 180, 2021, p. 111031, https://doi.org/10.1016/j.jss.2021.111031. [Online]. https://www.sciencedirect.com/science/article/pii/S016412122100128X
5. S. Martínez-Fernández, J. Bogner, X. Franch, M. Oriol, J. Siebert et al., “Software engineering for AI-Based systems: A survey,” ACM Trans. Softw. Eng. Methodol., Vol. 31, No. 2, apr 2022, https://doi.org/10.1145/3487043. [Online]. https://doi.org/10.1145/3487043
6. M.M. John, H.H. Olsson, and J. Bosch, “Towards MLOps: A framework and maturity model,” in 2021 47th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2021, pp. 1–8, https://doi.org/10.1109/SEAA53835.2021.00050.
7. B. Fitzgerald and K.J. Stol, “Continuous software engineering: A roadmap and agenda,” Journal of Systems and Software, Vol. 123, 2017, pp. 176–189, https://doi.org/10.1016/j.jss.2015.06.063. [Online]. https://www.sciencedirect.com/science/article/pii/S0164121215001430
8. I. Karamitsos, S. Albarhami, and C. Apostolopoulos, “Applying DevOps practices of continuous automation for machine learning,” Information, Vol. 11, No. 7, 2020, https://doi.org/10.3390/info11070363. [Online]. https://www.mdpi.com/2078-2489/11/7/363
9. R.V. O’Connor, P. Elger, and P.M. Clarke, “Continuous software engineering — a microservices architecture perspective,” Journal of Software: Evolution and Process, Vol. 29, No. 11, 2017, p. e1866, https://doi.org/10.1002/smr.1866. [Online]. https://onlinelibrary.wiley.com/doi/abs/10.1002/smr.1866
10. S. Mäkinen, H. Skogström, E. Laaksonen, and T. Mikkonen, “Who needs MLOps: What data scientists seek to accomplish and how can MLOps help?” in 2021 IEEE/ACM 1st Workshop on AI Engineering – Software Engineering for AI (WAIN), 2021, pp. 109–112, https://doi.org/10.1109/WAIN52551.2021.00024.
11. D. Zhang and J.J. Tsai, “Machine learning and software engineering,” Software Quality Journal, Vol. 11, 2003, pp. 87–119, https://doi.org/10.1023/A:1023760326768.
12. T. Mikkonen, J.K. Nurminen, M. Raatikainen, I. Fronza, N. Mäkitalo et al., “Is machine learning software just software: A maintainability view,” in Software Quality: Future Perspectives on Software Engineering Quality, D. Winkler, S. Biffl, D. Mendez, M. Wimmer, and J. Bergsmann, Eds. Cham: Springer International Publishing, 2021, pp. 94–105, https://doi.org/10.1007/978-3-030-65854-0_8.
13. K. Srinivasan and D. Fisher, “Machine learning approaches to estimating software development effort,” IEEE Transactions on Software Engineering, Vol. 21, No. 2, 1995, pp. 126–137, https://doi.org/10.1109/32.345828.
14. A.B. Kolltveit and J. Li, “Operationalizing machine learning models: A systematic literature review,” in Proceedings of the 1st Workshop on Software Engineering for Responsible AI, SE4RAI ’22. New York, NY, USA: Association for Computing Machinery, 2023, p. 1–8, https://doi.org/10.1145/3526073.3527584.
15. L.E. Lwakatare, I. Crnkovic, and J. Bosch, “DevOps for AI – challenges in development of AI-enabled applications,” in 2020 International Conference on Software, Telecommunications and Computer Networks (SoftCOM), 2020, pp. 1–6, https://doi.org/10.23919/SoftCOM50211.2020.9238323.
16. G. Symeonidis, E. Nerantzis, A. Kazakis, and G.A. Papakostas, “MLOps – definitions, tools and challenges,” in 2022 IEEE 12th Annual Computing and Communication Workshop and Conference (CCWC), 2022, pp. 0453–0460, https://doi.org/10.1109/CCWC54503.2022.9720902.
17. T. Granlund, V. Stirbu, and T. Mikkonen, “Towards regulatory-compliant MLOps: Oravizio’s journey from a machine learning experiment to a deployed certified medical product,” SN Computer Science, Vol. 2, 2021, https://doi.org/10.1007/s42979-021-00726-1.
18. A. Lima, L. Monteiro, and A.P. Furtado, “MLOps: Practices, maturity models, roles, tools, and challenges-a systematic literature review.” ICEIS (1), 2022, pp. 308–320, https://doi.org/10.5220/0010997300003179.
19. A. Serban, K. van der Blom, H. Hoos, and J. Visser, “Adoption and effects of software engineering best practices in machine learning,” in Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), 2020, pp. 1–12, https://doi.org/10.1145/3382494.3410681.
20. E. de Souza Nascimento, I. Ahmed, E. Oliveira, M.P. Palheta, I. Steinmacher et al., “Understanding development process of machine learning systems: Challenges and solutions,” in 2019 acm/ieee international symposium on empirical software engineering and measurement (esem). IEEE, 2019, pp. 1–6, https://doi.org/10.1109/ESEM.2019.8870157.
21. N. Nahar, S. Zhou, G. Lewis, and C. Kästner, “Collaboration challenges in building ml-enabled systems: Communication, documentation, engineering, and process,” in Proceedings of the 44th International Conference on Software Engineering, 2022, pp. 413–425, https://doi.org/10.1145/3510003.3510209.
22. D. Piorkowski, S. Park, A.Y. Wang, D. Wang, M. Muller et al., “How AI developers overcome communication challenges in a multidisciplinary team: A case study,” Proc. ACM Hum.-Comput. Interact., Vol. 5, No. CSCW1, apr 2021, https://doi.org/10.1145/3449205.
23. M. Kim, T. Zimmermann, R. DeLine, and A. Begel, “Data scientists in software teams: State of the art and challenges,” IEEE Transactions on Software Engineering, Vol. 44, No. 11, 2017, pp. 1024–1038, https://doi.org/10.1109/TSE.2017.2754374.
24. L. Riungu-Kalliosaari, S. Mäkinen, L.E. Lwakatare, J. Tiihonen, and T. Männistö, “DevOps adoption benefits and challenges in practice: A case study,” in Product-Focused Software Process Improvement: 17th International Conference, PROFES 2016, Trondheim, Norway, November 22-24, 2016, Proceedings 17. Springer, 2016, pp. 590–597, https://doi.org/10.1007/978-3-319-49094-6_44.
25. M.S. Khan, A.W. Khan, F. Khan, M.A. Khan, and T.K. Whangbo, “Critical challenges to adopt devops culture in software organizations: A systematic review,” IEEE Access, Vol. 10, 2022, pp. 14 339–14 349, https://doi.org/10.1109/ACCESS.2022.3145970.
26. N. Tomas, J. Li, and H. Huang, “An empirical study on culture, automation, measurement, and sharing of devsecops,” in 2019 International Conference on Cyber Security and Protection of Digital Services (Cyber Security). IEEE, 2019, pp. 1–8, https://doi.org/10.1109/CyberSecPODS.2019.8884935.
27. S. Brandstädter and K. Sonntag, “Interdisciplinary collaboration: How to foster the dialogue across disciplinary borders?” in Advances in Ergonomic Design of Systems, Products and Processes: Proceedings of the Annual Meeting of GfA 2015. Springer, 2016, pp. 395–409, https://doi.org/10.1007/978-3-662-48661-0_26.
28. L. Baier, F. Jöhren, and S. Seebacher, “Challenges in the deployment and operation of machine learning in practice.” in ECIS, Vol. 1, 2019, https://aisel.aisnet.org/ecis2019_rp/163/.
29. K.M. Eisenhardt, “Building theories from case study research,” The Academy of Management Review, Vol. 14, No. 4, 1989, pp. 532–550, https://doi.org/10.5465/amr.1989.4308385.