On Recruiting Experienced GitHub Contributors for Interviews and Surveys on Prolific

F. Ebert; Alexander Serebrenik; Christoph Treude; Nicole Novielli; Fernando Castor

On Recruiting Experienced GitHub Contributors for Interviews and Surveys on Prolific

F. Ebert, Alexander Serebrenik, Christoph Treude, Nicole Novielli, Fernando Castor

Software Engineering and Technology

Research output: Contribution to conference › Paper › Academic

Abstract

Software engineering researchers have been using general purpose online tools for crowd-sourcing for quite some time. Those tools can be useful to recruit participants for research studies as they are paid for their time. However, those tools should be used carefully. In this paper, we have described the issues we faced when recruiting participants on Prolific.

We used Prolific to recruit open-source developers with experience in submitting and reviewing pull requests (PRs). However, we did not succeed in obtaining valid participants for either the interview or the survey, which led us to change the approach of our study and not use Prolific anymore. For example, a major issue we faced on Prolific was that even using the specific filter related to participants with software development knowledge, the screening study revealed the majority of them did not have any development knowledge. Based on this experience, we formulate recommendations for future research when using such crowd-sourcing tools.

Original language	English
Publication status	Accepted/In press - 19 Feb 2022
Event	International Workshop on Recruiting Participants for Empirical Software Engineering - Virtual Duration: 17 May 2022 → 17 May 2022 https://ropes-workshops.github.io/ropes22/

Conference

Conference	International Workshop on Recruiting Participants for Empirical Software Engineering
Abbreviated title	RoPES
Period	17/05/22 → 17/05/22
Internet address	https://ropes-workshops.github.io/ropes22/

Access to Document

https://www.win.tue.nl/~aserebre/ROPES2022.pdf

Cite this

@conference{b6767956a50345788ec49afb139b2f57,

title = "On Recruiting Experienced GitHub Contributors for Interviews and Surveys on Prolific",

abstract = "Software engineering researchers have been using general purpose online tools for crowd-sourcing for quite some time. Those tools can be useful to recruit participants for research studies as they are paid for their time. However, those tools should be used carefully. In this paper, we have described the issues we faced when recruiting participants on Prolific.We used Prolific to recruit open-source developers with experience in submitting and reviewing pull requests (PRs). However, we did not succeed in obtaining valid participants for either the interview or the survey, which led us to change the approach of our study and not use Prolific anymore. For example, a major issue we faced on Prolific was that even using the specific filter related to participants with software development knowledge, the screening study revealed the majority of them did not have any development knowledge. Based on this experience, we formulate recommendations for future research when using such crowd-sourcing tools.",

author = "F. Ebert and Alexander Serebrenik and Christoph Treude and Nicole Novielli and Fernando Castor",

year = "2022",

month = feb,

day = "19",

language = "English",

note = "International Workshop on Recruiting Participants for Empirical Software Engineering, RoPES ; Conference date: 17-05-2022 Through 17-05-2022",

url = "https://ropes-workshops.github.io/ropes22/",

}

TY - CONF

T1 - On Recruiting Experienced GitHub Contributors for Interviews and Surveys on Prolific

AU - Ebert, F.

AU - Serebrenik, Alexander

AU - Treude, Christoph

AU - Novielli, Nicole

AU - Castor, Fernando

PY - 2022/2/19

Y1 - 2022/2/19

N2 - Software engineering researchers have been using general purpose online tools for crowd-sourcing for quite some time. Those tools can be useful to recruit participants for research studies as they are paid for their time. However, those tools should be used carefully. In this paper, we have described the issues we faced when recruiting participants on Prolific.We used Prolific to recruit open-source developers with experience in submitting and reviewing pull requests (PRs). However, we did not succeed in obtaining valid participants for either the interview or the survey, which led us to change the approach of our study and not use Prolific anymore. For example, a major issue we faced on Prolific was that even using the specific filter related to participants with software development knowledge, the screening study revealed the majority of them did not have any development knowledge. Based on this experience, we formulate recommendations for future research when using such crowd-sourcing tools.

AB - Software engineering researchers have been using general purpose online tools for crowd-sourcing for quite some time. Those tools can be useful to recruit participants for research studies as they are paid for their time. However, those tools should be used carefully. In this paper, we have described the issues we faced when recruiting participants on Prolific.We used Prolific to recruit open-source developers with experience in submitting and reviewing pull requests (PRs). However, we did not succeed in obtaining valid participants for either the interview or the survey, which led us to change the approach of our study and not use Prolific anymore. For example, a major issue we faced on Prolific was that even using the specific filter related to participants with software development knowledge, the screening study revealed the majority of them did not have any development knowledge. Based on this experience, we formulate recommendations for future research when using such crowd-sourcing tools.

M3 - Paper

T2 - International Workshop on Recruiting Participants for Empirical Software Engineering

Y2 - 17 May 2022 through 17 May 2022

ER -

On Recruiting Experienced GitHub Contributors for Interviews and Surveys on Prolific

Abstract

Conference

Access to Document

Fingerprint

Cite this