Machine Learning or Information Retrieval Techniques for Bug Triaging: Which is better?

Authors

Anjali Goyal, Neetu Sardana

Abstract

Bugs are the inevitable part of a software system. Nowadays, large software development projects even release beta versions of their products to gather bug reports from users. The collected bug reports are then worked upon by various developers in order to resolve the defects and make the final software product more reliable. The high frequency of incoming bugs makes the bug handling a difficult and time consuming task. Bug assignment is an integral part of bug triaging that aims at the process of assigning a suitable developer for the reported bug who corrects the source code in order to resolve the bug. There are various semi and fully automated techniques to ease the task of bug assignment. This paper presents the current state of the art of various techniques used for bug report assignment. Through exhaustive research, the authors have observed that machine learning and information retrieval based bug assignment approaches are most popular in literature. A deeper investigation has shown that the trend of techniques is taking a shift from machine learning based approaches towards information retrieval based approaches. Therefore, the focus of this work is to find the reason behind the observed drift and thus a comparative analysis is conducted on the bug reports of the Mozilla, Eclipse, Gnome and Open Office projects in the Bugzilla repository. The results of the study show that the information retrieval based technique yields better efficiency in recommending the developers for bug reports.

Keywords

bug triaging, bug report assignment, developer recommendation, machine learning, information retrieval

References

1. J. Zhang, X. Wang, D. Hao, B. Xie, L. Zhang,and H. Mei, “A survey on bug-report analysis,”Science China Information Sciences, Vol. 58,No. 2, 2015, pp. 1–24.
2. T. Zhang, H. Jiang, X. Luo, and A.T. Chan,“A literature review of research in bug resolution:Tasks, challenges and future directions,”The Computer Journal, Vol. 59, No. 5, 2016, pp.741–773.
3. B. Kitchenham and S. Charters, “Guidelinesfor performing systematic literature reviewsin software engineering,” Software EngineeringGroup, School of Computer Science andMathematics, Keele University and Departmentof Computer Science, University ofDurham, Tech. Rep. EBSE 2007-001, 2007.[Online]. https://pdfs.semanticscholar.org/e62d/bbbbe70cabcde3335765009e94ed2b9883d5.pdf
4. D. Matter, A. Kuhn, and O. Nierstrasz, “Assigningbug reports using a vocabulary-basedexpertise model of developers,” in 6th IEEEInternational Working Conference on MiningSoftware Repositories. IEEE, 2009, pp. 131–140.
5. A.S.K. Singh, “Bug triaging: Profile oriented developerrecommendation,” International Journalof Innovative Research in Advanced Engineering,Vol. 2, 2014, pp. 36–42.
6. H. Naguib, N. Narayan, B. Brügge, and D. Helal,“Bug report assignee recommendation using activityprofiles,” in 10th IEEE Working Conferenceon Mining Software Repositories (MSR).IEEE, 2013, pp. 22–30.
7. I. Aljarah, S. Banitaan, S. Abufardeh, W. Jin,and S. Salem, “Selecting discriminating termsfor bug assignment: a formal analysis,” in Proceedingsof the 7th International Conference onPredictive Models in Software Engineering. ACM,2011, p. 12.
8. A. Sureka, H. kumar Singh, M. Bagewadi, A. Mitra,and R. Karanth, “A decision support platformfor guiding a bug triager for resolver recommendationusing textual and non-textual features,”in 3rd International Workshop on QuantitativeApproaches to Software Quality, 2015,p. 25.
9. R. Shokripour, J. Anvik, Z.M. Kasirun, andS. Zamani, “Why so complicated? simple termfiltering and weighting for location-based bugreport assignment recommendation,” in Proceedingsof the 10th Working Conference on MiningSoftware Repositories. IEEE Press, 2013, pp.2–11.
10. F. Servant and J.A. Jones, “WhoseFault: automaticdeveloper-to-fault assignment throughfault localization,” in Proceedings of the 34th InternationalConference on Software Engineering.IEEE Press, 2012, pp. 36–46.
11. G. Murphy and D. Cubranic, “Automatic bugtriage using text categorization,” in Proceedingsof the Sixteenth International Conference onSoftware Engineering & Knowledge Engineering.Citeseer, 2004.
12. J. Xuan, H. Jiang, Z. Ren, J. Yan, and Z. Luo,“Automatic bug triage using semi-supervised textclassification.” in SEKE, 2010, pp. 209–214.
13. J. Anvik, “Automating bug report assignment,”in Proceedings of the 28th international conferenceon Software engineering. ACM, 2006, pp.937–940.
14. J. Anvik, L. Hiew, and G.C. Murphy, “Copingwith an open bug repository,” in Proceedings ofthe 2005 OOPSLA workshop on Eclipse technologyeXchange. ACM, 2005, pp. 35–39.
15. J. Anvik and G.C. Murphy, “Reducing the effortof bug report triage: Recommenders fordevelopment-oriented decisions,” ACM Transactionson Software Engineering and Methodology(TOSEM), Vol. 20, No. 3, 2011, p. 10.
16. J.K. Anvik, “Assisting bug report triage throughrecommendation,” Ph.D. dissertation, Universityof British Columbia, 2007.
17. G.A. Di Lucca, M. Di Penta, and S. Gradara,“An approach to classify software maintenance requests,”in International Conference on SoftwareMaintenance. IEEE, 2002, pp. 93–102.
18. P. Bhattacharya, I. Neamtiu, and C.R. Shelton,“Automated, highly-accurate, bug assignment usingmachine learning and tossing graphs,” Journalof Systems and Software, Vol. 85, No. 10,2012, pp. 2275–2292.
19. H. Hu, H. Zhang, J. Xuan, and W. Sun, “Effectivebug triage based on historical bug-fixinformation,” in IEEE 25th International Symposiumon Software Reliability Engineering (ISSRE).IEEE, 2014, pp. 122–132.
20. J. Xuan, H. Jiang, Y. Hu, Z. Ren, W. Zou,Z. Luo, and X. Wu, “Towards effective bug triagewith software data reduction techniques,” IEEETransactions on Knowledge and Data Engineering,Vol. 27, No. 1, 2015, pp. 264–280.
21. X. Xia, D. Lo, X. Wang, and B. Zhou, “Accuratedeveloper recommendation for bug resolution,”in 20th working conference on Reverse engineering(WCRE). IEEE, 2013, pp. 72–81.
22. M.R. Karim, G. Ruhe, M. Rahman, V. Garousi,T. Zimmermann et al., “An empirical investigationof single-objective and multiobjective evolutionaryalgorithms for developer’s assignmentto bugs,” Journal of Software: Evolution andProcess, Vol. 28, No. 12, 2016, pp. 1025–1060.
23. A. Moin and G. Neumann, “Assisting bug triagein large open source projects using approximatestring matching,” in Seventh International Conferenceon Software Engineering Advances (ICSEA).Lisbon, Portugal: Citeseer, 2012.
24. T. Zhang and B. Lee, “A hybrid bug triage algorithmfor developer recommendation,” in Proceedings of the 28th annual ACM symposium onapplied computing. ACM, 2013, pp. 1088–1094.
25. S.N. Ahsan, J. Ferzund, and F. Wotawa, “Automaticsoftware bug triage system (BTS) basedon latent semantic indexing and support vectormachine,” in Fourth International Conferenceon Software Engineering Advances. IEEE, 2009,pp. 216–221.
26. G. Canfora and L. Cerulo, “How software repositoriescan help in resolving a new change request,”in IEEE International Workshop on SoftwareTechnology and Engineering Practice (STEP).Citeseer, 2005, pp. 99–103.
27. H. Kagdi, M. Gethers, D. Poshyvanyk, andM. Hammad, “Assigning change requests to softwaredevelopers,” Journal of Software: Evolutionand Process, Vol. 24, No. 1, 2012, pp. 3–33.
28. N.K. Nagwani and S. Verma, “Predicting expertdevelopers for newly reported bugs using frequentterms similarities of bug attributes,” in 9thInternational Conference on ICT and KnowledgeEngineering (ICT & Knowledge Engineering).IEEE, 2012, pp. 113–117.
29. N.K. Nagwani and S. Verma, “Predicting expertdevelopers for newly reported bugs using frequentterms similarities of bug attributes,” in 9thInternational Conference on ICT and KnowledgeEngineering (ICT & Knowledge Engineering).IEEE, 2012, pp. 113–117.
30. O. Baysal, M.W. Godfrey, and R. Cohen, “A bugyou like: A framework for automated assignmentof bugs,” in IEEE 17th International Conferenceon Program Comprehension. IEEE, 2009, pp.297–298.
31. K. Kevic, S.C. Müller, T. Fritz, and H.C. Gall,“Collaborative bug triaging using textual similaritiesand change set analysis,” in 6th InternationalWorkshop on Cooperative and Human Aspects ofSoftware Engineering (CHASE). IEEE, 2013, pp.17–24.
32. X. Xie, W. Zhang, Y. Yang, and Q. Wang,“Dretom: Developer recommendation based ontopic models for bug resolution,” in Proceedingsof the 8th international conference on predictivemodels in software engineering. ACM, 2012, pp.19–28.
33. M. Alenezi, K. Magel, and S. Banitaan, “Efficientbug triaging using text mining,” JSW, Vol. 8,No. 9, 2013, pp. 2185–2190.
34. G. Canfora and L. Cerulo, “Supporting changerequest assignment in open source development,”in Proceedings of the 2006 ACM symposium onApplied computing. ACM, 2006, pp. 1767–1772.
35. T. Zhang, G. Yang, B. Lee, and E.K. Lua, “Anovel developer ranking algorithm for automaticbug triage using topic model and developer relations,”in 21st Asia-Pacific Software EngineeringConference (APSEC), Vol. 1. IEEE, 2014, pp.223–230.
36. E. Linstead, P. Rigor, S. Bajracharya, C. Lopes,and P. Baldi, “Mining eclipse developer contributionsvia author-topic models,” in Fourth InternationalWorkshop on Mining Software Repositories.IEEE, 2007, pp. 30–30.
37. G. Yang, T. Zhang, and B. Lee, “Towardssemi-automatic bug triage and severity predictionbased on topic model and multi-feature ofbug reports,” in IEEE 38th annual Computersoftware and applications conference (COMPSAC).IEEE, 2014, pp. 97–106.
38. X. Xia, D. Lo, Y. Ding, J.M. Al-Kofahi, T.N.Nguyen, and X. Wang, “Improving automatedbug triaging with specialized topic model,” IEEETransactions on Software Engineering, Vol. 43,No. 3, 2017, pp. 272–297.
39. K. Somasundaram and G.C. Murphy, “Automaticcategorization of bug reports using latentdirichlet allocation,” in Proceedings of the 5thIndia Software Engineering Conference. ACM,2012, pp. 125–130.
40. R. Shokripour, Z.M. Kasirun, S. Zamani, andJ. Anvik, “Automatic bug assignment using informationextraction methods,” in InternationalConference on Advanced Computer Science Applicationsand Technologies (ACSAT). IEEE,2012, pp. 144–149.
41. R. Shokripour, J. Anvik, Z.M. Kasirun, andS. Zamani, “A time-based approach to automaticbug report assignment,” Journal of Systems andSoftware, Vol. 102, 2015, pp. 109–122.
42. D. Mohan, N. Sardana et al., “Visheshagya:Time based expertise model for bug report assignment,”in Ninth International Conference onContemporary Computing (IC3). IEEE, 2016, pp.1–6.
43. S. Zamani, S.P. Lee, R. Shokripour, and J. Anvik,“A noun-based approach to feature locationusing time-aware term-weighting,” Informationand Software Technology, Vol. 56, No. 8, 2014,pp. 991–1011.
44. T.T. Nguyen, A.T. Nguyen, and T.N. Nguyen,“Topic-based, time-aware bug assignment,” ACMSIGSOFT Software Engineering Notes, Vol. 39,No. 1, 2014, pp. 1–4.
45. H. Hosseini, R. Nguyen, and M.W. Godfrey, “Amarket-based bug allocation mechanism usingpredictive bug lifetimes,” in 16th European Conferenceon Software Maintenance and Reengineering (CSMR). IEEE, 2012, pp. 149–158.
46. W. Wu, W. Zhang, Y. Yang, and Q. Wang,“Drex: Developer recommendation withk-Nearest-Neighbor search and expertiseranking,” in 18th Asia Pacific SoftwareEngineering Conference (APSEC). IEEE, 2011,pp. 389–396.
47. T. Zhang and B. Lee, “An automated bugtriage approach: A concept profile and socialnetwork based developer recommendation,” inInternational Conference on Intelligent Computing.Springer, 2012, pp. 505–512.
48. J. Xuan, H. Jiang, Z. Ren, and W. Zou, “Developerprioritization in bug repositories,” in 34thInternational Conference on Software Engineering(ICSE). IEEE, 2012, pp. 25–35.
49. G. Yang, T. Zhang, and B. Lee, “Utilizing amulti-developer network-based developer recommendationalgorithm to fix bugs effectively,” inProceedings of the 29th Annual ACM Symposiumon Applied Computing. ACM, 2014, pp.1134–1139.
50. W. Zhang, S. Wang, Y. Yang, and Q. Wang,“Heterogeneous network analysis of developercontribution in bug repositories,” in InternationalConference on Cloud and Service Computing(CSC). IEEE, 2013, pp. 98–105.
51. P. Bhattacharya and I. Neamtiu, “Fine-grainedincremental learning and multi-feature tossinggraphs to improve bug triaging,” in IEEE InternationalConference on Software Maintenance(ICSM). IEEE, 2010, pp. 1–10.
52. G. Jeong, S. Kim, and T. Zimmermann, “Improvingbug triage with bug tossing graphs,” inProceedings of the the 7th joint meeting of theEuropean software engineering conference andthe ACM SIGSOFT symposium on The foundationsof software engineering. ACM, 2009, pp.111–120.
53. L. Chen, X. Wang, and C. Liu, “An approachto improving bug assignment with bug tossinggraphs and bug similarities.” JSW, Vol. 6, No. 3,2011, pp. 421–427.
54. A. Tamrawi, T.T. Nguyen, J. Al-Kofahi, and T.N.Nguyen, “Fuzzy set-based automatic bug triaging:NIER track,” in 33rd International Conferenceon Software Engineering (ICSE). IEEE,2011, pp. 884–887.
55. A. Tamrawi, T.T. Nguyen, J.M. Al-Kofahi, andT.N. Nguyen, “Fuzzy set and cache-based approachfor bug triaging,” in Proceedings of the19th ACM SIGSOFT symposium and the 13thEuropean conference on Foundations of softwareengineering. ACM, 2011, pp. 365–375.
56. A. Niknafs, J. Denzinger, and G. Ruhe, “A systematicliterature review of the personnel assignmentproblem,” in Proceedings of the InternationalMulticonference of Engineers and ComputerScientists, Hong Kong, 2013.
57. M.M. Rahman, S. Sohan, F. Maurer, andG. Ruhe, “Evaluation of optimized staffing forfeature development and bug fixing,” in Proceedingsof the 2010 ACM-IEEE InternationalSymposium on Empirical Software Engineeringand Measurement. ACM, 2010, p. 42.
58. D. Panagiotou, F. Paraskevopoulos, andL. Stojanovic, Specifications of developerprofile, (2010). [Online]. http://www.alertproject.eu/sites/portal2-alert.atosorigin.es/files/content-files/download/Specification%20of%20Developer%20Profile.pdf
59. N.K. Nagwani and S. Verma, “Rank-me: A Javatool for ranking team members in software bugrepositories,” Journal of Software Engineeringand Applications, Vol. 5, 2012, pp. 255–261.
60. D. Kim, Y. Tao, S. Kim, and A. Zeller, “Whereshould we fix this bug? a two-phase recommendationmodel,” IEEE Transactions on Software Engineering,Vol. 39, No. 11, 2013, pp. 1597–1610.
61. M. Linares-Vásquez, K. Hossen, H. Dang,H. Kagdi, M. Gethers, and D. Poshyvanyk,“Triaging incoming change requests: Bug or commithistory, or code authorship?” in 28th IEEEInternational Conference on Software Maintenance(ICSM). IEEE, 2012, pp. 451–460.
62. B. Ashok, J. Joy, H. Liang, S.K. Rajamani,G. Srinivasa, and V. Vangala, “DebugAdvisor: arecommender system for debugging,” in Proceedingsof the the 7th joint meeting of the Europeansoftware engineering conference and the ACMSIGSOFT symposium on The foundations ofsoftware engineering. ACM, 2009, pp. 373–382.
63. H. Kagdi and D. Poshyvanyk, “Who can help mewith this change request?” in 17th InternationalConference on Program Comprehension. IEEE,2009, pp. 273–277.
64. J. Anvik and G.C. Murphy, “Determining implementationexpertise from bug reports,” in Proceedingsof the Fourth International Workshop onMining Software Repositories. IEEE ComputerSociety, 2007, p. 2.
65. S. Minto and G.C. Murphy, “Recommendingemergent teams,” in Fourth International Workshopon Mining Software Repositories. IEEE,2007, pp. 5–5.
66. G. Gousios, E. Kalliamvakou, and D. Spinellis,“Measuring developer contribution from softwarerepository data,” in Proceedings of the 2008 in softwarerepositories. ACM, 2008, pp. 129–132.
67. T. Zhang, G. Yang, B. Lee, and A.T. Chan,“Guiding bug triage through developer analysisin bug reports,” International Journal of SoftwareEngineering and Knowledge Engineering,Vol. 26, No. 03, 2016, pp. 405–431.
68. T. Zhang, G. Yang, B. Lee, and I. Shin, “Roleanalysis-based automatic bug triage algorithm,”IPSJ SIG Technical Report, Tech. Rep., 2012.
69. X. Xia, D. Lo, X. Wang, and B. Zhou, “Dualanalysis for recommending developers to resolvebugs,” Journal of Software: Evolution and Process,Vol. 27, No. 3, 2015, pp. 195–220.
70. J. Helming, H. Arndt, Z. Hodaie, M. Koegel,and N. Narayan, “Automatic assignment of workitems,” in International Conference on Evaluationof Novel Approaches to Software Engineering.Springer, 2010, pp. 236–250.
71. W. Zou, Y. Hu, J. Xuan, and H. Jiang, “Towardstraining set reduction for bug triage,” in IEEE35th Annual Computer Software and ApplicationsConference (COMPSAC). IEEE, 2011, pp.576–581.
72. Z. Lin, F. Shu, Y. Yang, C. Hu, and Q. Wang,“An empirical study on bug assignment automationusing Chinese bug data,” in 3rd internationalsymposium on Empirical software engineeringand measurement. IEEE, 2009, pp.451–455.
73. S. Banitaan and M. Alenezi, “Tram: An approachfor assigning bug reports using their metadata,”in Third International Conference on Communicationsand Information Technology. IEEE, 2013,pp. 215–219.
74. J. Anvik, L. Hiew, and G.C. Murphy, “Whoshould fix this bug?” in Proceedings of the 28thinternational conference on Software engineering.ACM, 2006, pp. 361–370.
75. M. Sharma, M. Kumari, and V. Singh, “Bug assigneeprediction using association rule mining,”in International Conference on ComputationalScience and Its Applications. Springer, 2015, pp.444–457.
76. J. Park, M. Lee, J. Kim, S. Hwang, and S. Kim,“Costriage: A cost-aware triage algorithm forbug reporting systems,” in Proceedings of theNational Conference on Artificial Intelligence,2011, p. 139.
77. M.M. Rahman, G. Ruhe, and T. Zimmermann,“Optimized assignment of developers for fixingbugs an initial evaluation for eclipse projects,” inProceedings of the 2009 3rd International Symposiumon Empirical Software Engineering andMeasurement. IEEE Computer Society, 2009, pp.439–442.
78. R.K. Saha, S. Khurshid, and D.E. Perry, “An empiricalstudy of long lived bugs,” in Software EvolutionWeek-IEEE Conference on Software Maintenance,Reengineering and Reverse Engineering(CSMR-WCRE). IEEE, 2014, pp. 144–153.