On the Adoption of a TODO Bot on GitHub: A Preliminary Study

Bots support different software maintenance and evolution activities, such as code review or executing tests. Recently, several bots have been proposed to help developers to keep track of postponed activities, expressed by means of TODO comments: e.g., TODO bot automatically creates a GitHub issue when a TODO comment is added to a repository, increasing visibility of TODO comments. In this work, we perform a preliminary evaluation of the impact of the TODO bot on software development practice. We conjecture that the introduction of the TODO bot would facilitate keeping track of the TODO comments, and hence encourage developers to use more TODO comments in their code changes.

To evaluate this conjecture, we analyze all the 2,208 repositories which have at least one GitHub issue created by the TODO bot. Firstly, we investigate to what extent the bot is being used and describe the repositories using the bot. We observe that the majority (54%) of the repositories which adopted the TODO bot are new, i.e., were created within less than one month of the first issue created by the bot, and from those, more than 60% have the issue created within three days. For the projects that have adopted the TODO bot at least one month after the repository has been created we compare the number of TODO comments before and after the adoption of the bot. We observe a statistically significant increase in the number of the TODO comments after the adoption of the bot, however with a small effect size. Our results suggest that the adoption of the TODO bot encourages developers to introduce TODO comments rendering the postponed decisions more visible. Nevertheless, it does not speed up the process of addressing TODO comments or corresponding GitHub issues.