| 2020 | |
| [1] | , "What Support do Systematic Reviews Provide for Evidence-informed Teaching about Software Engineering Practice?", In e-Informatica Software Engineering Journal, vol. 14, no. 1, pp. 7–60, 2020.
DOI: 10.37190/e-Inf200101. Download article (PDF)Get article BibTeX file |
Authors
David Budgen, Pearl Brereton, Nikki Williams, Sarah Drummond
Abstract
Background: The adoption of the evidence-based research paradigm by software engineering researchers has created a growing knowledge base provided by the outcomes from systematic reviews.
Aim: We set out to identify and catalogue a sample of the knowledge provided by systematic reviews, to determine what support they can provide for an evidence-informed approach to teaching about software engineering practice.
Method: We undertook a tertiary study (a mapping study of systematic reviews) covering the period to the end of 2015. We identified and catalogued those reviews that had findings or made recommendations that were considered relevant to teaching about industry practice.
Results: We examined a sample of 276 systematic reviews, selecting 49 for which we could clearly identify practice-oriented findings and recommendations that were supported by the data analysis provided in the review. We have classified these against established software engineering education knowledge categories and discuss the extent and forms of knowledge provided for each category.
Conclusion: While systematic reviews can provide knowledge that can inform teaching about practice, relatively few systematic reviews present the outcomes in a form suitable for this purpose. Using a suitable format for presenting a summary of outcomes could improve this. Additionally, the increasing number of published systematic reviews suggests that there is a need for greater coordination regarding the cataloguing of their findings and recommendations.
Keywords
Systematic review, education, provenance
References
1. P. Naur and B. Randell, Eds., Software Engineering: Report on a Conference Sponsored by the NATO Science Committee . NATO, 1968.
2. P. Bourque and R.E. Fairley, Eds., Guide to the Software Engineering Body of Knowledge (SWEBOK(R)): Version 3.0 , 3rd ed. IEEE Computer Society Press, 2014.
3. M. Ardis, D. Budgen, G.W. Hislop, J. Offutt, M. Sebern, and W. Visser, “SE2014: Curriculum Guidelines for undergraduate degree programs in software engineering,” IEEE Computer , November 2015, pp. 106–109.
4. B. Kitchenham, D. Budgen, P. Brereton, M. Turner, S. Charters, and S. Linkman, “Large-Scale Software Engineering Questions – expert opinion or empirical evidence?” IET Software , Vol. 1, No. 5, 2007, pp. 161–171.
5. P. Devanbu, T. Zimmermann, and C. Bird, “Belief and evidence in empirical software engineering,” in Proceedings of the 38th International Conference on Software Engineering (ICSE) . ACM Press, 2016, pp. 108–119.
6. E.M. Rogers, Diffusion of Innovations , 5th ed. Free Press, New York, 2003.
7. C. Theisen, M. Dunaiski, L. Williams, and W. Visser, “Software engineering research at the international conference on software engineering in 2016,” ACM Software Engineering Notes , Vol. 42, No. 4, 2017, pp. 1–10.
8. B. Kitchenham, T. Dybå, and M. Jørgensen, “Evidence-based software engineering,” in Proceedings of the 26th International Conference on Software Engineering (ICSE) . IEEE Computer Society Press, 2004, pp. 273–281.
9. D. Budgen, S. Drummond, P. Brereton, and N. Holland, “What scope is there for adopting evidence-informed teaching in software engineering?” in Proceedings of the 34th International Conference on Software Engineering (ICSE) . IEEE Computer Society Press, 2012, pp. 1205–1214.
10. D. Budgen, P. Brereton, S. Drummond, and N. Williams, “Reporting systematic reviews: Some lessons from a tertiary study,” Information and Software Technology , Vol. 95, 2018, pp. 62–74. [Online]. http://www.sciencedirect.com/science/article/pii/S0950584916303548
11. D. Budgen, P. Brereton, N. Williams, and S. Drummond, “The contribution that empirical studies performed in industry make to the findings of systematic reviews: A tertiary study,” Information and Software Technology , Vol. 94, 2018, pp. 234–244. [Online]. http://www.sciencedirect.com/science/article/pii/S0950584917303798
12. J. Gurevitch, J. Koricheva, S. Nakagawa, and G. Stewart, “Meta-analysis and the science of research synthesis,” Nature , Vol. 555, 2018, pp. 175–182.
13. E. Barends and D.M. Rousseau, Evidence-Based Management: How to use evidence to make better organizational decisions . Kogan Page, 2018.
14. M. Petticrew and H. Roberts, Systematic Reviews in the Social Sciences A Practical Guide . Blackwell Publishing, 2006.
15. A. Booth, D. Papaioannou, and A. Sutton, Systematic Approaches to a Successful Literature Review . Sage Publications, Ltd., 2012.
16. D.S. Cruzes and T. Dybå, “Research synthesis in software engineering: A tertiary study,” Information and Software Technology , Vol. 53, No. 5, 2011, pp. 440–455.
17. S. Martinez-Fernandez, P.S.M. dos Santos, G.P. Ayala, X. Franch, and G.H. Travassos, “Aggregating empirical evidence about the benefits and drawbacks of software reference architectures,” in Proceedings of 2015 the Conference on Empirical Software Engineering and Measurement , 2015, pp. 154–163.
18. D. Budgen, J. Bailey, M. Turner, B. Kitchenham, P. Brereton, and S. Charters, “Cross-domain investigation of empirical practices,” IET Software , Vol. 3, No. 5, 2009, pp. 410–421, eASE special section.
19. T.V. Ribeiro, J. Massollar, and G.H. Travassos, “Challenges and pitfalls on surveying evidence in the software engineering technical literature: an exploratory study with novices,” Empirical Software Engineering , Vol. 23, 2018, pp. 1594–1663.
20. U. Abelein and B. Paech, “Understanding the influence of user participation and involvement on system success – A systematic mapping study,” Empirical Software Engineering , Vol. 20, 2015, pp. 28–81.
21. D. Smite, C. Wohlin, T. Gorschek, and R. Feldt, “Empirical evidence in global software engineering: a systematic review,” Empirical Software Engineering , Vol. 15, 2010, pp. 91–118.
22. O. Dieste and N. Juristo, “Systematic review and aggregation of empirical studies on elicitation techniques,” IEEE Transactions on Software Engineering , Vol. 37, No. 2, 2011, pp. 283–304.
23. T. Dybå and T. Dingsøyr, “Strength of evidence in systematic reviews in software engineering,” in Proceedings of International Symposium on Empirical Software Engineering and Metrics (ESEM) , 2008, pp. 178–187.
24. M. Ivarsson and T. Gorschek, “A method for evaluating rigor and industrial relevance of technology evaluations,” Empirical Software Engineering , Vol. 16, 2011, pp. 365–395.
25. B. Kitchenham, P. Brereton, D. Budgen, M. Turner, J. Bailey, and S. Linkman, “Systematic literature reviews in software engineering – a systematic literature review,” Information and Software Technology , Vol. 51, No. 1, 2009, pp. 7–15.
26. B. Kitchenham, R. Pretorius, D. Budgen, P. Brereton, M. Turner, M. Niazi, and S. Linkman, “Systematic literature reviews in software engineering – a tertiary study,” Information and Software Technology , Vol. 52, 2010, pp. 792–805.
27. F.Q. da Silva, A.L. Santos, S. Soares, A.C.C. França, C.V. Monteiro, and F.F. Maciel, “Six years of systematic literature reviews in software engineering: An updated tertiary study,” Information and Software Technology , Vol. 53, No. 9, 2011, pp. 899–913.
28. M. Leitner and S. Rinderle-Ma, “A systematic review on security in process-aware information systems,” Information and Software Technology , Vol. 56, No. 3, 2014, pp. 273–293.
29. B.A. Kitchenham, D. Budgen, and P. Brereton, Evidence-Based Software Engineering and Systematic Reviews , Innovations in Software Engineering and Software Development. CRC Press, 2015.
30. T. Dybå, B. Kitchenham, and M. Jörgensen, “Evidence-based software engineering for practitioners,” IEEE Software , Vol. 22, No. 1, 2005, pp. 58–65.
31. B. Kitchenham, “Procedures for undertaking systematic reviews,” Joint Technical Report Keele and Durham Universities, Tech. Rep., 2004.
32. B. Kitchenham and S. Charters, “Guidelines for performing systematic literature reviews in software engineering,” Keele University and Durham University Joint Report, Tech. Rep., 2007.
33. P. Brereton, B.A. Kitchenham, D. Budgen, M. Turner, and M. Khalil, “Lessons from applying the systematic literature review process within the software engineering domain,” Journal of Systems and Software , Vol. 80, No. 4, 2007, pp. 571–583.
34. J. Biolchini, P. Mian, A. Natali, and G. Travassos, “Systematic review in software engineering,” COPPE/UFRJ, Tech. Rep. ES679/05, 2005.
35. J.L. Fleiss, “Measuring nominal scale agreement among many raters,” Psychological Bulletin , Vol. 76, 1971, pp. 378–382.
36. M. Banerjee, M. Capozzoli, L. McSweeney, and D. Sinha, “Beyond kappa: A review of interrater agreement measures,” Canadian Journal of Statistics , Vol. 27, No. 1, 1999, pp. 3–23.
37. M. Staples and M. Niazi, “Systematic review of organizational motivations for adopting CMM-based SPI,” Information and Software Technology , Vol. 50, 2008, pp. 605–620.
38. S. Beecham, N. Baddoo, T. Hall, H. Robinson, and H. Sharp, “Motivation in software engineering: A systematic literature review,” Information and Software Technology , Vol. 50, No. 9–10, 2008, pp. 860–878.
39. H. Sharp, N. Baddoo, S. Beecham, T. Hall, and H. Robinson, “Models of motivation in software engineering,” Information and Software Technology , Vol. 51, 2009, pp. 219–233.
40. H. Petersson, T. Thelin, P. Runeson, and C. Wohlin, “Capture-recapture in software inspections after 10 years research – theory, evaluation and application,” Journal of Systems and Software , Vol. 72, 2004, pp. 249–264.
41. W. Azfal, R. Torkar, and R. Feldt, “A systematic review of search-based testing for non-functional system properties,” Information and Software Technology , Vol. 51, 2009, pp. 957–976.
42. E. Engström, P. Runeson, and M. Skoglund, “A systematic review on regression test selection techniques,” Information and Software Technology , Vol. 52, 2010, pp. 14–30.
43. M. Jørgensen, “Forecasting of software development work effort: Evidence on expert judgement and formal models,” Int. Journal of Forecasting , Vol. 23, No. 3, 2007, pp. 449–462.
44. M. Jørgensen, “Evidence-based guidelines for assessment of software development cost uncertainty,” IEEE Transactions on Software Engineering , Vol. 31, No. 11, 2005, pp. 942–954.
45. P. Mohagheghi and R. Conradi, “Quality, productivity and economic benefits of software reuse: A review of industrial studies,” Empirical Software Engineering , Vol. 12, 2007, pp. 471–516.
46. F.J. Pino, F. Garcia, and M. Piattini, “Software process improvement in small and medium software enterprises: A systematic review,” Software Quality Journal , Vol. 16, 2008, pp. 237–261.
47. J. Hannay, T. Dybå, E. Arisholm, and D. Sjøberg, “The effectiveness of pair programming. A meta analysis,” Information and Software Technology , Vol. 51, No. 7, 2009, pp. 1110–1122.
48. J.S. Persson, L. Mathiassen, J. Boeg, T.S. Madsen, and F. Steinson, “Managing risks in distributed software projects: An integrative framework,” IEEE Transactions on Engineering Management , Vol. 56, No. 3, 2009, pp. 508–532.
49. A.H. Ghapanchi and A. Aurum, “Antecedents to IT personnel’s intentions to leave: A systematic literature review,” Journal of Systems and Software , Vol. 84, 2011, pp. 238–249.
50. I. Steinmacher, M.A.G. Silva, M.A. Gerosa, and D.F. Redmiles, “A systematic literature review on the barriers faced by newcomers to open source software projects,” Information and Software Technology , Vol. 59, No. 67-85, 2015.
51. Z. Li, H. Zhang, L. O’Brien, R. Cai, and S. Flint, “On evaluating commercial cloud services: A systematic review,” Journal of Systems and Software , Vol. 86, 2013, pp. 2371–2393.
52. D. Radjenović, M. Heričko, R. Torkar, and A. Živkovič, “Software fault prediction metrics: A systematic literature review,” Information and Software Technology , Vol. 55, 2013, pp. 1397–1418.
53. H. Munir, M. Moayyed, and K. Peterson, “Considering rigor and relevance when evaluating test driven development: A systematic review,” Information and Software Technology , Vol. 56, 2014, pp. 375–394.
54. E. Kupiainen, M.V. Mäntylä, and J. Itkonen, “Using metrics in agile and lean software development – a systematic literature review of industrial studies,” Information and Software Technology , Vol. 62, 2015, pp. 143–163.
55. B.J. Williams and J.C. Carver, “Characterizing software architecture changes: A systematic review,” Information and Software Technology , Vol. 52, No. 1, 2010, pp. 31–51.
56. M.S. Ali, M.A. Babar, L. Chen, and K.J. Stol, “A systematic review of comparative evidence of aspect-oriented programming,” Information and Software Technology , Vol. 52, No. 9, 2010, pp. 871–887.
57. C. Zhang and D. Budgen, “What do we know about the effectiveness of software design patterns?” IEEE Transactions on Software Engineering , Vol. 38, No. 5, 2012, pp. 1213–1231.
58. L.B. Lisboa, V.C. Garcia, D. Lucrédio, E.S. de Almeida, S.R. de Lemos Meira, and R.P. de Mattos Fortes, “A systematic review of domain analysis tools,” Information and Software Technology , Vol. 52, No. 1, 2010, pp. 1–13.
59. M. Turner, B. Kitchenham, P. Brereton, S. Charters, and D. Budgen, “Does the technology acceptance model predict actual use? A systematic literature review,” Information and Software Technology , Vol. 52, No. 5, 2010, pp. 463–479.
60. T.B.C. Arias, P. van der Spek, and P. Avgeriou, “A practice-driven systematic review of dependency analysis solutions,” Empirical Software Engineering , Vol. 16, 2011, pp. 544–586.
61. T. Hall, S. Beecham, D. Bowes, D. Gray, and S. Counsell, “A systematic literature review on fault prediction performance in software engineering,” IEEE Transactions on Software Engineering , Vol. 38, No. 6, 2012, pp. 1276–1304.
62. C. Pacheco and I. Garcia, “A systematic literature review of stakeholder identification methods in requirements elicitation,” Journal of Systems and Software , Vol. 85, 2012, pp. 2171–2181.
63. S. Tiwari and A. Gupta, “A systematic literature review of use case specifications research,” Information and Software Technology , Vol. 67, 2015, pp. 128–158.
64. R. Jabangwe, J. Borstler, D. Smite, and C. Wohlin, “Empirical evidence on the link between object-oriented measures and external quality attributes: a systematic literature review,” Empirical Software Engineering , Vol. 20, 2015, pp. 640–693.
65. K. Peterson, “Measuring and predicting software productivity: A systematic map and review,” Information and Software Technology , Vol. 53, 2011, pp. 317–343.
66. J. Díaz, J. Pérez, P.P. Alarcón, and J. Garbajosa, “Agile product line engineering – A systematic literature review,” Software – Practice and Experience , Vol. 41, 2011, pp. 921–941.
67. Y. Rafique and V. Misic, “The effects of test-driven development on external quality and productivity: A meta-analysis,” IEEE Transactions on Software Engineering , Vol. 39, No. 6, 2013.
68. A.M. Magdaleno, C.M.L. Werner, and R.M. de Araujo, “Reconciling software development models: A quasi-systematic review,” Journal of Systems and Software , Vol. 85, 2012, pp. 351–369.
69. N.B. Ali, K. Peterson, and C. Wohlin, “A systematic literature review on the industrial use of software process simulation,” Journal of Systems and Software , Vol. 97, 2014, pp. 65–85.
70. S.U. Khan, M. Niazi, and R. Ahmad, “Barriers in the selection of offshore software development oursourcing vendors: An exploratory study using a systematic literature review,” Information and Software Technology , Vol. 53, 2011, pp. 693–706.
71. R. Giuffrida and Y. Dittrich, “Empirical studies on the use of social software in global software development – A systematic mapping study,” Information and Software Technology , Vol. 55, 2013, pp. 1143–1164.
72. N. Paternoster, C. Giardino, M. Unterkalmsteiner, and T. Gorschek, “Software development in startup companies: A systematic mapping study,” Information and Software Technology , Vol. 56, 2014, pp. 1200–1218.
73. O. Al-Baik and J. Miller, “The Kanban approach between agility and leanness: a systematic review,” Empirical Software Engineering , Vol. 20, 2015, pp. 1861–1897.
74. M. Zarour, A. Abran, J.M. Desharnais, and A. Alarifi, “An investigation into the best practices for the successful design and implementation of lightweight software process assessment methods: A systematic literature review,” Journal of Systems and Software , Vol. 101, 2015, pp. 180–192.
75. A. Nguyen-Duc, D.S. Cruzes, and R. Conradi, “The impact of global dispersion on coordination, team performance and software quality – A systematic literature review,” Information and Software Technology , Vol. 57, 2015, pp. 277–294.
76. F.S. Silva, F.S.F. Soares, A.L. Peres, I.M. de Azevedo, A.P.L.F. Vasconcelos, F.K. Kamei, and S.R. de Lemos Meira, “Using CMMI together with agile software development: A systematic review,” Information and Software Technology , Vol. 58, No. 20-43, 2015.
77. M. Bano and D. Zowghi, “A systematic review on the relationship between user involvement and system success,” Information and Software Technology , Vol. 58, No. 148-169, 2015.
78. A. Idri, F.A. Amazal, and A. Abran, “Analogy-based software development effort estimation: A systematic mapping and review,” Information and Software Technology , Vol. 58, 2015, pp. 206–230.
79. M. Brhel, H. Meth, A. Maedche, and K. Werder, “Exploring principles of user-centered agile software development: A literature review,” Information and Software Technology , Vol. 61, 2015, pp. 163–181.
80. D. Heaton and J.C. Carver, “Claims about the use of software engineering practices in science: A systematic literature review,” Information and Software Technology , Vol. 67, 2015, pp. 207–219.
81. R. Rabiser, P. Grunbacher, and D. Dhungana, “Requirements for product derivation support: Results from a systematic literature review and an expert survey,” Information and Software Technology , Vol. 52, 2010, pp. 324–346.
82. E. Tüzün, B. Tekinerdogan, M.E. Kalender, and S. Bilgen, “Empirical evaluation of a decision support model for adopting software product line engineering,” Information and Software Technology , Vol. 60, 2015, pp. 77–101.
83. H.A. Simon, “The structure of ill-structured problems,” Artificial Intelligence , Vol. 4, 1973, pp. 181–201.
84. G.H. Guyatt, A.D. Oxman, G.E. Vist, R. Kunz, Y. Falck-Ytter, P. Alonso-Coello, and H.J. Schünemann, “GRADE: an emerging consensus on rating quality of evidence and strength of recommendations,” British Medical Journal , Vol. 336, 2008, pp. 924–926.
85. J. Nelson and C. O’Beirne, “Using evidence in the classroom: What works and why?” National Foundation for Educational Research (NFER), Tech. Rep., 2014.
86. S. Hopewell, A. Aisinga, and M. Clarke, “Better reporting of randomized trials in biomedical journal and conference abstracts,” Journal of Information Science , Vol. 34, No. 2, 2008, pp. 162–173.
87. S.E. Rosenbaum, C. Glenton, and A.D. Oxman, “Summary-of-findings tables in Cochrane reviews improved understanding and rapid retrieval of key information,” Journal of Clinical Epidemiology , Vol. 63, 2010, pp. 620–626.
88. S. Malick, K. Das, and K.S. Khan, “Tips for teaching evidence-based medicine in a clinical setting: Lessons from adult learning theory,” Journal of the Royal Society of Medicine , Vol. 101, No. 11, 2008, pp. 536–543.
89. M. Coldwell, T. Greany, S. Higgins, C. Brown, B. Maxwell, B. Stiell, L. Stoll, B. Willis, and H. Burns, “Evidence-informed teaching: an evaluation of progress in England,” Department for Education, Tech. Rep., 2017.
90. C.L. Goues, C. Jaspan, I. Ozkaya, M. Shaw, and K.T. Stolee, “Bridging the Gap: From research to practical advice,” IEEE Software , Vol. 35, No. 5, 2018, pp. 50–57.
91. J. Lavis, G. Permanand, A. Oxman, S. Lewin, and A. Fredheim, “SUPPORT tools for evidence-informed health policy-making (STP) 13: Preparing and using policy briefs to support evidence-informed policymaking,” Health Research Policy and Systems , Vol. 7, 2009, p. S13.
92. S. Oliver and K. Dickson, “Policy-relevant systematic reviews to strengthen health systems: models and mechanisms to support their production,” Evidence and Policy , Vol. 12, No. 2, 2016, pp. 235–259.
93. B. Cartaxo, G. Pinto, E. Vieira, and S. Soares, “Evidence Briefings: Towards a medium to transfer knowledge from systematic reviews to practitioners,” in Proceedings of the 2016 Conference on Empirical Software Engineering and Measurement (ESEM) , 2016, pp. 1–10.