The Trust Recovery Journey: The Effect of Timing of Errors on the Willingness to Follow AI Advice

Patricia Kahr; Gerrit Rooks; Chris C.P. Snijders; Martijn C. Willemsen

doi:10.1145/3640543.3645167

The Trust Recovery Journey: The Effect of Timing of Errors on the Willingness to Follow AI Advice

Patricia Kahr, Gerrit Rooks, Chris C.P. Snijders, Martijn C. Willemsen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

40 Downloads (Pure)

Abstract

Complementing human decision-making with AI advice offers substantial advantages. However, humans do not always trust AI advice appropriately and are overly sensitive to incidental AI errors, even in cases with overall good performance. Today’s research still needs to uncover the underlying aspects of trust decline and recovery over time in repeated human-AI interactions. Our work investigates the consequences of incidental AI error on (self-reported) trust and participants’ reliance on AI advice. Results from our experiment, where 208 participants evaluated 14 legal cases before and after receiving algorithmic advice, showed that trust significantly decreased after early and late errors but was rapidly restored in both scenarios. Reliance significantly dropped only for early errors but not for late errors. In both scenarios, reliance was able to be restored. Results suggest that late (compared to early) errors are less drastic in trust loss and allow quicker recovery. These findings align with an interpretation in which humans can build up trust over time if a system is performing well, making them more tolerant of incidental AI errors.

Original language	English
Title of host publication	IUI '24
Subtitle of host publication	Proceedings of the 29th International Conference on Intelligent User Interfaces
Publisher	Association for Computing Machinery, Inc
Pages	609-622
Number of pages	14
ISBN (Electronic)	979-8-4007-0508-3
DOIs	https://doi.org/10.1145/3640543.3645167
Publication status	Published - 5 Apr 2024
Event	IUI 2024 – International Conference on Intelligent User Interface (IUI '24) - Greenville, Greenville, United States Duration: 18 Mar 2024 → 21 Mar 2024

Conference

Conference	IUI 2024 – International Conference on Intelligent User Interface (IUI '24)
Abbreviated title	IUI
Country/Territory	United States
City	Greenville
Period	18/03/24 → 21/03/24

Funding

We thank Claudia Sprenger (Eindhoven University of Technology) for supporting the study setup, as well as Ralf Schmidt (Eindhoven University of Technology) for his guidance on the technical implementation of our study. Our gratitude extends to the European Supply Chain Forum (ESCF), the Department of Industrial Engineering and Innovation Sciences (IE&IS), the Eindhoven Artificial Intelligence Systems Institute (EAISI), and the Logistics Community Brabant for their sponsorship of the research project \"AI Planner of the Future\", which supports the PhD project \"Trust in AI over time\".

Funders	Funder number
Eindhoven University of Technology

Keywords

Collaborative Decision-Making
Trust Recovery
Trust in AI Over Time

Access to Document

10.1145/3640543.3645167Licence: CC BY

pdfFinal published version, 1.35 MBLicence: CC BY

Cite this

@inproceedings{0093339825054cd4b4894054ac25be32,

title = "The Trust Recovery Journey: The Effect of Timing of Errors on the Willingness to Follow AI Advice",

abstract = "Complementing human decision-making with AI advice offers substantial advantages. However, humans do not always trust AI advice appropriately and are overly sensitive to incidental AI errors, even in cases with overall good performance. Today{\textquoteright}s research still needs to uncover the underlying aspects of trust decline and recovery over time in repeated human-AI interactions. Our work investigates the consequences of incidental AI error on (self-reported) trust and participants{\textquoteright} reliance on AI advice. Results from our experiment, where 208 participants evaluated 14 legal cases before and after receiving algorithmic advice, showed that trust significantly decreased after early and late errors but was rapidly restored in both scenarios. Reliance significantly dropped only for early errors but not for late errors. In both scenarios, reliance was able to be restored. Results suggest that late (compared to early) errors are less drastic in trust loss and allow quicker recovery. These findings align with an interpretation in which humans can build up trust over time if a system is performing well, making them more tolerant of incidental AI errors.",

keywords = "Collaborative Decision-Making, Trust Recovery, Trust in AI Over Time",

author = "Patricia Kahr and Gerrit Rooks and Snijders, {Chris C.P.} and Willemsen, {Martijn C.}",

year = "2024",

month = apr,

day = "5",

doi = "10.1145/3640543.3645167",

language = "English",

pages = "609--622",

booktitle = "IUI '24",

publisher = "Association for Computing Machinery, Inc",

address = "United States",

note = "IUI 2024 – International Conference on Intelligent User Interface (IUI '24), IUI ; Conference date: 18-03-2024 Through 21-03-2024",

}

Kahr, P , Rooks, G , Snijders, CCP & Willemsen, MC 2024, The Trust Recovery Journey: The Effect of Timing of Errors on the Willingness to Follow AI Advice. in IUI '24: Proceedings of the 29th International Conference on Intelligent User Interfaces. Association for Computing Machinery, Inc, pp. 609-622, IUI 2024 – International Conference on Intelligent User Interface (IUI '24), Greenville, South Carolina, United States, 18/03/24. https://doi.org/10.1145/3640543.3645167

The Trust Recovery Journey: The Effect of Timing of Errors on the Willingness to Follow AI Advice. / Kahr, Patricia ; Rooks, Gerrit ; Snijders, Chris C.P. et al.
IUI '24: Proceedings of the 29th International Conference on Intelligent User Interfaces. Association for Computing Machinery, Inc, 2024. p. 609-622.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - The Trust Recovery Journey

T2 - IUI 2024 – International Conference on Intelligent User Interface (IUI '24)

AU - Kahr, Patricia

AU - Rooks, Gerrit

AU - Snijders, Chris C.P.

AU - Willemsen, Martijn C.

PY - 2024/4/5

Y1 - 2024/4/5

N2 - Complementing human decision-making with AI advice offers substantial advantages. However, humans do not always trust AI advice appropriately and are overly sensitive to incidental AI errors, even in cases with overall good performance. Today’s research still needs to uncover the underlying aspects of trust decline and recovery over time in repeated human-AI interactions. Our work investigates the consequences of incidental AI error on (self-reported) trust and participants’ reliance on AI advice. Results from our experiment, where 208 participants evaluated 14 legal cases before and after receiving algorithmic advice, showed that trust significantly decreased after early and late errors but was rapidly restored in both scenarios. Reliance significantly dropped only for early errors but not for late errors. In both scenarios, reliance was able to be restored. Results suggest that late (compared to early) errors are less drastic in trust loss and allow quicker recovery. These findings align with an interpretation in which humans can build up trust over time if a system is performing well, making them more tolerant of incidental AI errors.

AB - Complementing human decision-making with AI advice offers substantial advantages. However, humans do not always trust AI advice appropriately and are overly sensitive to incidental AI errors, even in cases with overall good performance. Today’s research still needs to uncover the underlying aspects of trust decline and recovery over time in repeated human-AI interactions. Our work investigates the consequences of incidental AI error on (self-reported) trust and participants’ reliance on AI advice. Results from our experiment, where 208 participants evaluated 14 legal cases before and after receiving algorithmic advice, showed that trust significantly decreased after early and late errors but was rapidly restored in both scenarios. Reliance significantly dropped only for early errors but not for late errors. In both scenarios, reliance was able to be restored. Results suggest that late (compared to early) errors are less drastic in trust loss and allow quicker recovery. These findings align with an interpretation in which humans can build up trust over time if a system is performing well, making them more tolerant of incidental AI errors.

KW - Collaborative Decision-Making

KW - Trust Recovery

KW - Trust in AI Over Time

UR - http://www.scopus.com/inward/record.url?scp=85190969348&partnerID=8YFLogxK

U2 - 10.1145/3640543.3645167

DO - 10.1145/3640543.3645167

M3 - Conference contribution

SP - 609

EP - 622

BT - IUI '24

PB - Association for Computing Machinery, Inc

Y2 - 18 March 2024 through 21 March 2024

ER -

The Trust Recovery Journey: The Effect of Timing of Errors on the Willingness to Follow AI Advice

Abstract

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this