Confusion detection in code reviews

F. Ebert, F. Castor, N. Novielli, A. Serebrenik

    Research output: Chapter in Book/Report/Conference proceedingConference contributionAcademicpeer-review

    32 Citations (Scopus)


    Code reviews are an important mechanism for assuring quality of source code changes. Reviewers can either add general comments pertaining to the entire change or pinpoint concerns or shortcomings about a specific part of the change using inline comments. Recent studies show that reviewers often do not understand the change being reviewed and its context.Our ultimate goal is to identify the factors that confuse code reviewers and understand how confusion impacts the efficiency and effectiveness of code review(er)s. As the first step towards this goal we focus on the identification of confusion in developers' comments. Based on an existing theoretical framework categorizing expressions of confusion, we manually classify 800 comments from code reviews of the Android project. We observe that confusion can be reasonably well-identified by humans: raters achieve moderate agreement (Fleiss' kappa 0.59 for the general comments and 0.49 for the inline ones). Then, for each kind of comment we build a series of automatic classifiers that, depending on the goals of the further analysis, can be trained to achieve high precision (0.875 for the general comments and 0.615 for the inline ones), high recall (0.944 for the general comments and 0.988 for the inline ones), or substantial precision and recall (0.696 and 0.542 for the general comments and 0.434 and 0.583 for the inline ones, respectively). These results motivate further research on the impact of confusion on the code review process. Moreover, other researchers can employ the proposed classifiers to analyze confusion in other contexts where software development-related discussions occur, such as mailing lists.
    Original languageEnglish
    Title of host publicationICSME 2017 : 33rd IEEE International Conference on Software Maintenance and Evolution, 17-24 September 2017, Shanghai, China
    Subtitle of host publicationNew Ideas and Emerging Results
    Place of PublicationPiscataway
    PublisherInstitute of Electrical and Electronics Engineers
    Number of pages5
    ISBN (Electronic)9781538609927
    ISBN (Print)978-1-5386-0992-7
    Publication statusPublished - 8 Nov 2017
    Event33rd IEEE International Conference on Software Maintenance and Evolution (ICSME 2017) - Shanghai, China
    Duration: 19 Sept 201722 Sept 2017
    Conference number: 33


    Conference33rd IEEE International Conference on Software Maintenance and Evolution (ICSME 2017)
    Abbreviated titleICSME2017
    Internet address


    • Code review
    • Confusion
    • Machine learning


    Dive into the research topics of 'Confusion detection in code reviews'. Together they form a unique fingerprint.

    Cite this