The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: A case study

Erik Driessen, C. Van Der Vleuten, L. Schuwirth, J. Van Tartwijk, J. Vermunt

Research output: Contribution to journalArticleAcademicpeer-review

118 Citations (Scopus)

Abstract

AIM: Because it deals with qualitative information, portfolio assessment inevitably involves some degree of subjectivity. The use of stricter assessment criteria or more structured and prescribed content would improve interrater reliability, but would obliterate the essence of portfolio assessment in terms of flexibility, personal orientation and authenticity. We resolved this dilemma by using qualitative research criteria as opposed to reliability in the evaluation of portfolio assessment. METHODOLOGY/RESEARCH DESIGN: Five qualitative research strategies were used to achieve credibility and dependability of assessment: triangulation, prolonged engagement, member checking, audit trail and dependability audit. Mentors read portfolios at least twice during the year, providing feedback and guidance (prolonged engagement). Their recommendation for the end-of-year grade was discussed with the student (member checking) and submitted to a member of the portfolio committee. Information from different sources was combined (triangulation). Portfolios causing persistent disagreement were submitted to the full portfolio assessment committee. Quality assurance procedures with external auditors were used (dependability audit) and the assessment process was thoroughly documented (audit trail). RESULTS: A total of 233 portfolios were assessed. Students and mentors disagreed on 7 (3%) portfolios and 9 portfolios were submitted to the full committee. The final decision on 29 (12%) portfolios differed from the mentor's recommendation. CONCLUSION: We think we have devised an assessment procedure that safeguards the characteristics of portfolio assessment, with credibility and dependability of assessment built into the judgement procedure. Further support for credibility and dependability might be sought by means of a study involving different assessment committees.

Original languageEnglish
Pages (from-to)214-220
Number of pages7
JournalMedical Education
Volume39
Issue number2
DOIs
Publication statusPublished - 1 Feb 2005
Externally publishedYes

Keywords

  • Clinical competence/*standards
  • Curriculum/ standards
  • Education, medical, undergraduate/*methods
  • Educational measurement/*methods
  • Mentors
  • Reproducibility of results
  • Students, medical/*psychology

Fingerprint Dive into the research topics of 'The use of qualitative research criteria for portfolio assessment as an alternative to reliability evaluation: A case study'. Together they form a unique fingerprint.

Cite this