| 2018 | |
| [1] | , "Tool Features to Support Systematic Reviews in Software Engineering – A Cross Domain Study", In e-Informatica Software Engineering Journal, vol. 12, no. 1, pp. 79–115, 2018.
DOI: 10.5277/e-Inf180104. Download article (PDF)Get article BibTeX file |
Authors
Pearl Brereton, Barbara Kitchenham, Chris Marshall
Abstract
Context: Previously, the authors had developed and evaluated a framework to evaluate systematic review (SR) lifecycle tools.
Goal: The goal of this study was to use the experiences of researchers in other domains to further evaluate and refine the evaluation framework.
Method: We investigate the opinions of researchers with experience of systematic reviews in the healthcare and social sciences domains. We used semi-structured interviews to elicit their experiences of systematic reviews and SR support tools.
Results: Study participants found broadly the same problems as software engineering (SE) researchers with the SR process. They agreed with the tool features we had included in our evaluation framework. Furthermore although there were some differences the majority of the importance assessments were very close.
Conclusions: In the context of SRs, experiences of researchers in other domains can be useful to software engineering researchers. The evaluation framework for SR lifecycle tools appeared quite robust.
Keywords
software engineering, systematic review tools, cross-domain survey, qualitative analysis
References
[1] C. Mulrow, “Rationale for systematic reviews,” British Medical Journal, Vol. 309, No. 6954, 1994, p. 597.
[2] D. Cook, C. Mulrow, and R. Hayes, “Systematic reviews: synthesis of best evidence for clinical decisions,” Annals of Internal Medicine, Vol. 126, No. 5, 1997, pp. 376–380.
[3] B. Kitchenham and S. Charters, “Guidelines for performing systematic literature reviews in software engineering,” Keele University and Durham University, Joint Report, 2007.
[4] D.S.W. Rosenberg, J. Gray, R. Hayes, and W. Richardson, “Evidence-based medicine: what it is and what it isn’t,” British Medical Journal, Vol. 312, No. 7023, 1996, p. 71.
[5] J. Higgins, Cochrane Handbook for Systematic Reviews of Interventions. Wiley-Blackwell, 2008.
[6] J.P. Ioannidis, “The mass production of redundant, misleading and conflicted systematic reviews and meta-analysis,” The Milbank Quarterly, Vol. 94, No. 3, 2016, pp. 485–514.
[7] D.S. Cruzes and T. Dybå, “Research synthesis in software engineering: A tertiary study,” Information and Software Technology, Vol. 53, No. 5, 2011, pp. 440 – 455.
[8] F.Q. da Silva, A.L. Santos, S. Soares, A.C.C. França, C.V. Monteiro, and F.F. Maciel, “Six years of systematic literature reviews in software engineering: An updated tertiary study,” Information and Software Technology, Vol. 53, No. 9, 2011, pp. 899–913.
[9] B. Kitchenham and P. Brereton, “A systematic review of systematic review process research in software engineering,” Information and Software Technology, Vol. 55, No. 12, 2013, pp. 2049 – 2075.
[10] P. Brereton, B.A. Kitchenham, D. Budgen, M. Turner, and M. Khalil, “Lessons from applying the systematic literature review process within the software engineering domain,” Journal of Systems and Software, Vol. 80, No. 4, 2007, pp. 571 – 583.
[11] M. Babar and H. Zhang, “Systematic literature reviews in software engineering: preliminary results from interview with researchers,” 2014, pp. 346–355.
[12] M. Riaz, M. Sulayman, N. Salled, and E. Mendes, “Experiences conducting systematic reviews from novices’ perspective.” in Proceedings of the 2010 International Conference on Evaluation and Assessment in Software Engineering, 2010.
[13] S. Imitiaz, M. Bano, N. Ikram, and M. Niazi, “A tertiary study: experiences of conducting systematic literature reviews in software engineering,” in In Proceedings of the 2013 International Conference on Evaluation and Assessment in Software Engineering, 2013, pp. 177–182.
[14] J. Carver, E. Hassler, E. Hernandes, and N. Kraft, “Identifying barriers to the systematic literature review process,” in Proceedings of the 13th International Symposium on Empirical Software Engineering and Measurement, 2013.
[15] M. Staples and M. Niazi, “Experience using systematic review guidelines,” Journal of Systems and Software, Vol. 80, No. 9, 2007, pp. 1425–1437.
[16] H. Ramampiaro, D. Cruzes, R. Conradi, and M. Mendona, “Supporting evidence-based software engineering with collaborative information retreival,” in Proceedings of the 2010 International Conference on Collective Computing: Networking Applications and WorkSharing, 2010, pp. 1–5.
[17] B.A. Kitchenham, D. Budgen, and P. Brereton, Evidence-Based Software Engineering and Systematic Reviews. CRC Press, 2015.
[18] C. Marshall, O.P. Brereton, and B.A. Kitchenham, “Tools to support systematic literature reviews in software engineering: A feature analysis,” in Proceedings of 18th International Conference on Evaluation and Assessment in Software Engineering (EASE’14). ACM Press, 2014, pp. 13:1–13:10.
[19] C. Marshall, O.P. Brereton, and B.A. Kitchenham, “Tools to support systematic literature reviews in software engineering: A cross-domain survey using structured interviews,” in Proceedings of 19th International Conference on Evaluation and Assessment in Software Engineering (EASE’15). ACM Press, 2015, pp. 26–31.
[20] D. Bowes, T. Hall, and S. Beecham, “SLuRp: a tools to help large complex systematic literature reviews,” in Proceedings 2012 International Workshop on Evidential Assessment of Software Technologies, 2012, pp. 33–36.
[21] C. Marshall and P. Brereton, “Tools to support systematic literature reviews in software engineering: A mapping study,” in Proceedings ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM). IEEE Computer Society Press, 2013, pp. 296–299.
[22] J. Molleri and F. Benitti, “Sear: a web-based automated tool to support the systematic literature review process,” in Proceedings of the 2015 International Conference on Evaluation and Assessment, 2015, pp. 24–33.
[23] B. Kitchenham, “Evaluating methods and tools part 1: The evaluation context and methods,” ACM SIGSOFT Notes, Vol. 21, No. 1, 1996, pp. 11–14.
[24] B. Kitchenham, T. Dybå, and M. Jørgensen, “Evidence-based software engineering,” in Proceedings of ICSE 2004. IEEE Computer Society Press, 2004, pp. 273–281.
[25] B. Kitchenham, “Procedures for undertaking systematic reviews,” Keele and Durham Universities, Joint Technical Report, 2004.
[26] M. Petticrew and H. Roberts, Systematic Reviews in the Social Sciences A Practical Guide. Blackwell Publishing, 2006.
[27] J.A.S. Torres, D.S. Cruzes, and L. Salvador, “Automatic results identification in software engineering papers. is it possible?” in Proceedings 12th International Conference on Computer Science and Its Applications, 2012.
[28] K.R. Felizardo, S. MacDonell, E. Mendes, and J. Maldonado, “A systematic mapping on the use of visual data mining to support the conduct of systematic literature reviews,” Journal of Systems and Software, Vol. 7, No. 2, 2012, pp. 450–461.
[29] C. Marshall and O.P. Brereton, “Systematic review toolbox: a catalogue of tools to support systematic review,” in Proceedings of 19th International Conference on Evaluation and Assessment in Software Engineering (EASE’15). ACM Press, 2015, pp. 26–31.
[30] E. Hassler, J.C. Carver, N.A. Kraft, and D. Hale, “Outcomes of a community workshop to identify and rank barriers to the systematic literatire review process,” in Proceedings 18th International Conference on Evaluation and Assessment in Software Engineering, 2014.
[31] E. Hassler, J.C. Carver, D. Hale, and A. Al-Zubidyb, “Identification of slr tool needs–results of a community workshop,” Information and Software Technology, Vol. 70, 2016, pp. 122–129.
[32] D. Remenyi, Grounded Theory A reader for Researchers, Student, Faculty and Others, 2nd ed. Academic Conferences and PublishingInternational Limited, 2014.
[33] M.B. Miles, A.M. Huberman, and J. Saldaa, Qualitative Data Analysis A Methods Sourcebook, 3rd ed. Sage Publications Inc., 2014.
[34] G. Tsafnat, P. Glasziou, M. Choong, A. Dunn, F. Galgani, and E. Coiera, “Systematic review automation technologies.” Systematic Reviews, Vol. 3, No. 1, 2014, p. 74.
[35] P. Clough and C. Nutbrown, A student’s guide to methodology, 3rd ed. SAGE, 2012.
[36] M. Denscombe, The good research guide: for small-scale social research. Open University Press, 2014.
[37] R. Gomm, Social Research methodology. Palgrave MacMillan, 2004.
[38] C. Marshall, “Tool support for systematic reviews in software engineering,” Ph.D. dissertation, School of Computer Science and Mathematics, Keele University, 2016.