Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot

Kei Koyanagi; Dong Wang; Kotaro Noguchi; Masanari Kondo; Alexander Serebrenik; Yasutaka Kamei; Naoyasu Ubayashi

doi:10.1145/3643991.3644917

Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot

Kei Koyanagi, Dong Wang, Kotaro Noguchi, Masanari Kondo, Alexander Serebrenik, Yasutaka Kamei, Naoyasu Ubayashi

Social Software Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

4 Citations (Scopus)

30 Downloads (Pure)

Abstract

GitHub Copilot is an AI-enabled tool that automates program synthesis. It has gained significant attention since its launch in 2021. Recent studies have extensively examined Copilot's capabilities in various programming tasks, as well as its security issues. However, little is known about the effect of different natural languages on code suggestion. Natural language is considered a social bias in the field of NLP, and this bias could impact the diversity of software engineering. To address this gap, we conducted an empirical study to investigate the effect of three popular natural languages (English, Japanese, and Chinese) on Copilot. We used 756 questions of varying difficulty levels from AtCoder contests for evaluation purposes. The results highlight that the capability varies across natural languages, with Chinese achieving the worst performance. Furthermore, regardless of the type of natural language, the performance decreases significantly as the difficulty of questions increases. Our work represents the initial step in comprehending the significance of natural languages in Copilot's capability and introduces promising opportunities for future endeavors.

Original language	English
Title of host publication	MSR '24
Subtitle of host publication	Proceedings of the 21st International Conference on Mining Software Repositories
Place of Publication	New York
Publisher	Association for Computing Machinery, Inc.
Pages	481-486
Number of pages	6
ISBN (Electronic)	979-8-4007-0587-8
DOIs	https://doi.org/10.1145/3643991.3644917
Publication status	Published - 2 Jul 2024
Event	21st International Conference on Mining Software Repositories, MSR 2024 - Lisbon, Portugal Duration: 15 Apr 2024 → 16 Apr 2024

Conference

Conference	21st International Conference on Mining Software Repositories, MSR 2024
Abbreviated title	MSR 2024
Country/Territory	Portugal
City	Lisbon
Period	15/04/24 → 16/04/24

Funding

We gratefully acknowledge the financial support of: (1) JSPS for the KAKENHI grants (JP21H04877, JP22K17874, JP22K18630, JP23K16864), and Bilateral Program grant JPJSBP120239929; and (2) the Inamori Research Institute for Science for supporting Yasutaka Kamei via the InaRIS Fellowship.

Funders	Funder number
Inamori Research Institute for Science
Japan Society for the Promotion of Science	JP23K16864, JP22K18630, JPJSBP120239929, JP21H04877, JP22K17874

Keywords

Code Suggestion
Empirical Study
GitHub Copilot

Access to Document

10.1145/3643991.3644917

pdfFinal published version, 513 KBLicence: TAVERNE

https://www.win.tue.nl/~aserebre/MSR2024.pdf

Cite this

Koyanagi, K., Wang, D., Noguchi, K., Kondo, M., Serebrenik, A., Kamei, Y., & Ubayashi, N. (2024). Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot. In MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories (pp. 481-486). Association for Computing Machinery, Inc.. https://doi.org/10.1145/3643991.3644917

@inproceedings{e102d41537964c6db73d7a7d44a7f359,

title = "Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot",

abstract = "GitHub Copilot is an AI-enabled tool that automates program synthesis. It has gained significant attention since its launch in 2021. Recent studies have extensively examined Copilot's capabilities in various programming tasks, as well as its security issues. However, little is known about the effect of different natural languages on code suggestion. Natural language is considered a social bias in the field of NLP, and this bias could impact the diversity of software engineering. To address this gap, we conducted an empirical study to investigate the effect of three popular natural languages (English, Japanese, and Chinese) on Copilot. We used 756 questions of varying difficulty levels from AtCoder contests for evaluation purposes. The results highlight that the capability varies across natural languages, with Chinese achieving the worst performance. Furthermore, regardless of the type of natural language, the performance decreases significantly as the difficulty of questions increases. Our work represents the initial step in comprehending the significance of natural languages in Copilot's capability and introduces promising opportunities for future endeavors.",

keywords = "Code Suggestion, Empirical Study, GitHub Copilot",

author = "Kei Koyanagi and Dong Wang and Kotaro Noguchi and Masanari Kondo and Alexander Serebrenik and Yasutaka Kamei and Naoyasu Ubayashi",

year = "2024",

month = jul,

day = "2",

doi = "10.1145/3643991.3644917",

language = "English",

pages = "481--486",

booktitle = "MSR '24",

publisher = "Association for Computing Machinery, Inc.",

address = "United States",

note = "21st International Conference on Mining Software Repositories, MSR 2024, MSR 2024 ; Conference date: 15-04-2024 Through 16-04-2024",

}

Koyanagi, K, Wang, D, Noguchi, K, Kondo, M, Serebrenik, A, Kamei, Y & Ubayashi, N 2024, Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot. in MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories. Association for Computing Machinery, Inc., New York, pp. 481-486, 21st International Conference on Mining Software Repositories, MSR 2024, Lisbon, Portugal, 15/04/24. https://doi.org/10.1145/3643991.3644917

Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot. / Koyanagi, Kei; Wang, Dong; Noguchi, Kotaro et al.
MSR '24: Proceedings of the 21st International Conference on Mining Software Repositories. New York: Association for Computing Machinery, Inc., 2024. p. 481-486.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot

AU - Koyanagi, Kei

AU - Wang, Dong

AU - Noguchi, Kotaro

AU - Kondo, Masanari

AU - Serebrenik, Alexander

AU - Kamei, Yasutaka

AU - Ubayashi, Naoyasu

PY - 2024/7/2

Y1 - 2024/7/2

N2 - GitHub Copilot is an AI-enabled tool that automates program synthesis. It has gained significant attention since its launch in 2021. Recent studies have extensively examined Copilot's capabilities in various programming tasks, as well as its security issues. However, little is known about the effect of different natural languages on code suggestion. Natural language is considered a social bias in the field of NLP, and this bias could impact the diversity of software engineering. To address this gap, we conducted an empirical study to investigate the effect of three popular natural languages (English, Japanese, and Chinese) on Copilot. We used 756 questions of varying difficulty levels from AtCoder contests for evaluation purposes. The results highlight that the capability varies across natural languages, with Chinese achieving the worst performance. Furthermore, regardless of the type of natural language, the performance decreases significantly as the difficulty of questions increases. Our work represents the initial step in comprehending the significance of natural languages in Copilot's capability and introduces promising opportunities for future endeavors.

AB - GitHub Copilot is an AI-enabled tool that automates program synthesis. It has gained significant attention since its launch in 2021. Recent studies have extensively examined Copilot's capabilities in various programming tasks, as well as its security issues. However, little is known about the effect of different natural languages on code suggestion. Natural language is considered a social bias in the field of NLP, and this bias could impact the diversity of software engineering. To address this gap, we conducted an empirical study to investigate the effect of three popular natural languages (English, Japanese, and Chinese) on Copilot. We used 756 questions of varying difficulty levels from AtCoder contests for evaluation purposes. The results highlight that the capability varies across natural languages, with Chinese achieving the worst performance. Furthermore, regardless of the type of natural language, the performance decreases significantly as the difficulty of questions increases. Our work represents the initial step in comprehending the significance of natural languages in Copilot's capability and introduces promising opportunities for future endeavors.

KW - Code Suggestion

KW - Empirical Study

KW - GitHub Copilot

UR - https://www.scopus.com/pages/publications/85185475301

U2 - 10.1145/3643991.3644917

DO - 10.1145/3643991.3644917

M3 - Conference contribution

SP - 481

EP - 486

BT - MSR '24

PB - Association for Computing Machinery, Inc.

CY - New York

T2 - 21st International Conference on Mining Software Repositories, MSR 2024

Y2 - 15 April 2024 through 16 April 2024

ER -

Exploring the Effect of Multiple Natural Languages on Code Suggestion Using GitHub Copilot

Abstract

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this