| 2022 | |
| [1] | , "How good are my search strings? Reflections on using an existing review as a quasi-gold standard", In e-Informatica Software Engineering Journal, vol. 16, no. 1, pp. 220103, 2022.
DOI: 10.37190/e-Inf220103. Download article (PDF)Get article BibTeX file |
Authors
Huynh Khanh Vi Tran, Jürgen Börstler, Nauman bin Ali, Michael Unterkalmsteiner
Abstract
Background: Systematic literature studies (SLS) have become a core research methodology in Evidence-based Software Engineering (EBSE). Search completeness, i.e., finding all relevant papers on the topic of interest, has been recognized as one of the most commonly discussed validity issues of SLSs.
Aim: This study aims at raising awareness on the issues related to search string construction and on search validation using a quasi-gold standard (QGS). Furthermore, we aim at providing guidelines for search string validation.
Method: We use a recently completed tertiary study as a case and complement our findings with the observations from other researchers studying and advancing EBSE.
Results: We found that the issue of assessing QGS quality has not seen much attention in the literature, and the validation of automated searches in SLSs could be improved. Hence, we propose to extend the current search validation approach by the additional analysis step of the automated search validation results and provide recommendations for the QGS construction.
Conclusion: In this paper, we report on new issues which could affect search completeness in SLSs. Furthermore, the proposed guideline and recommendations could help researchers implement a more reliable search strategy in their SLSs.
Keywords
search string construction, automated search validation, quasi-gold standard, systematic literature review, systematic mapping study
References
[1] B.A. Kitchenham, D. Budgen, and P. Brereton, Evidence-Based Software Engineering and Systematic Reviews. Chapman and Hall/CRC, 2015.
[2] N.B. Ali and M. Usman, “A critical appraisal tool for systematic literature reviews in software engineering,” Information and Software Technology, Vol. 112, 2019, pp. 48–50.
[3] N.B. Ali and M. Usman, “Reliability of search in systematic reviews: Towards a quality assessment framework for the automated-search strategy,” Information and Software Technology, Vol. 99, Jul. 2018, pp. 133–147. [Online]. https://linkinghub.elsevier.com/retrieve/pii/S0950584917304263
[4] M. Usman, N.B. Ali, and C. Wohlin, “A quality assessment instrument for systematic literature reviews in software engineering,” CoRR, Vol. abs/2109.10134, 2021.
[5] H. Zhang, M. Babar, and P. Tell, “Identifying relevant studies in software engineering,” Information and Software Technology, Vol. 53, No. 6, 2011, pp. 625–637.
[6] O. Dieste and A.G. Padau, “Developing search strategies for detecting relevant experiments for systematic reviews,” in Proceedings of the 1st International Symposium on Empirical Software Engineering and Measurement, 2007, pp. 215–224.
[7] H.K.V. Tran, M. Unterkalmsteiner, J. Börstler, and N. bin Ali, “Assessing test artifact quality – A tertiary study,” Information and Software Technology, Vol. 139, 2021.
[8] V. Garousi and M. Mäntylä, “A systematic literature review of literature reviews in software testing,” Information and Software Technology, Vol. 80, 2016, pp. 195–216.
[9] K. Petersen, S. Vakkalanka, and L. Kuzniarz, “Guidelines for conducting systematic mapping studies in software engineering: An update,” Information and Software Technology, Vol. 64, 2015, pp. 1–18.
[10] A. Ampatzoglou, S. Bibi, P. Avgeriou, M. Verbeek, and A. Chatzigeorgiou, “Identifying, categorizing and mitigating threats to validity in software engineering secondary studies,” Information and Software Technology, Vol. 106, 2019, pp. 201–230.
[11] A. Ampatzoglou, S. Bibi, P. Avgeriou, and A. Chatzigeorgiou, “Guidelines for managing threats to validity of secondary studies in software engineering,” in Contemporary Empirical Methods in Software Engineering, M. Felderer and G.H. Travassos, Eds. Springer, 2020, pp. 415–441.
[12] J. Bailey, C. Zhang, D. Budgen, M. Turner, and S. Charters, “Search Engine Overlaps: Do they agree or disagree?” in Proceedings of the 2nd International Workshop on Realising Evidence-Based Software Engineering, May 2007, p. 2.
[13] S. Imtiaz, M. Bano, N. Ikram, and M. Niazi, “A tertiary study: experiences of conducting systematic literature reviews in software engineering,” in Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering, 2013, pp. 177–182.
[14] J. Krüger, C. Lausberger, I. von Nostitz-Wallwitz, G. Saake, and T. Leich, “Search. review. repeat? an empirical study of threats to replicating SLR searches,” Empirical Software Engineering, Vol. 25, No. 1, 2020, pp. 627–677.
[15] L. Chen, M.A. Babar, and H. Zhang, “Towards an evidence-based understanding of electronic data sources,” in Proceedings of the 14th International Conference on Evaluation and Assessment in Software Engineering. BCS, 2010, pp. 1–4.
[16] X. Zhou, Y. Jin, H. Zhang, S. Li, and X. Huang, “A map of threats to validity of systematic literature reviews in software engineering,” in Proceedings of the 23rd Asia-Pacific Software Engineering Conference, 2016, pp. 153–160.
[17] H. Heberle, G.V. Meirelles, F.R. da Silva, G.P. Telles, and R. Minghim, “Interactivenn: A web-based tool for the analysis of sets through venn diagrams,” BMC bioinformatics, Vol. 16, No. 1, 2015, pp. 1–7.
[18] J.R. Barbosa, A.M.R. Vincenzi, M.E. Delamaro, and J.C. Maldonado, “Software testing in critical embedded systems: A systematic review of adherence to the do-178b standard,” in Proceedings of the 3rd International Conference on Advances in System Testing and Validation Lifecycle, 2011, pp. 126–130.
[19] A. Sharma, T.D. Hellmann, and F. Maurer, “Testing of web services – A systematic mapping,” in Proceedings of the 8th IEEE World Congress on Services, 2012, pp. 346–352.
[20] T.K. Paul and M.F. Lau, “Redefinition of fault classes in logic expressions,” in Proceedings of the 12th International Conference on Quality Software, 2012, pp. 144–153.
[21] I.U. Munasinghe and T.D. Rupasinghe, “A supply chain network design optimization model from the perspective of a retail distribution supply chain,” in Proceedings of the Manufacturing and Industrial Engineering Symposium: Innovative Applications for Industry, 2016.
[22] J. Ahmad and S. Baharom, “A systematic literature review of the test case prioritization technique for sequence of events,” International Journal of Applied Engineering Research, Vol. 12, No. 7, 2017, pp. 1389–1395.
[23] S. Pradhan, M. Ray, and S. Patnaik, “Coverage criteria for state-based testing: A systematic review,” International Journal of Information Technology Project Management, Vol. 10, No. 1, 2019, pp. 1–20.
[24] P.K. Arora and R. Bhatia, “A systematic review of agent-based test case generation for regression testing,” Arabian Journal for Science and Engineering, Vol. 43, No. 2, 2018, pp. 447–470.
[25] B. Kitchenham, P. Brereton, Z. Li, D. Budgen, and A. Burn, “Repeatability of systematic literature reviews,” in Proceedings of the 15th Annual Conference on Evaluation and Assessment in Software Engineering, 2011, pp. 46–55.
[26] N.B. Ali, E. Engström, M. Taromirad, M.R. Mousavi, N.M. Minhas et al., “On the search for industry-relevant regression testing research,” Empir. Softw. Eng., Vol. 24, No. 4, 2019, pp. 2020–2055.
[27] C. Wohlin, P. Runeson, P. Da Mota Silveira Neto, E. Engström, I. Do Carmo Machado et al., “On the reliability of mapping studies in software engineering,” Journal of Systems and Software, Vol. 86, No. 10, 2013, pp. 2594–2610.
[28] C. Wohlin, “Guidelines for snowballing in systematic literature studies and a replication in software engineering,” in Proceedings of the 18th international conference on evaluation and assessment in software engineering, 2014, pp. 1–10.
[29] B. Kitchenham, O. Pearl Brereton, D. Budgen, M. Turner, J. Bailey et al., “Systematic literature reviews in software engineering – A systematic literature review,” Information and Software Technology, Vol. 51, No. 1, 2009, pp. 7–15.
[30] F. Da Silva, A. Santos, S. Soares, A. Frana, C. Monteiro et al., “Six years of systematic literature reviews in software engineering: An updated tertiary study,” Information and Software Technology, Vol. 53, No. 9, 2011, pp. 899–913.