It Seems Smart, but It Acts Stupid: Development of Trust in AI Advice in a Repeated Legal Decision-Making Task

Patricia K. Kahr; Gerrit Rooks; Martijn C. Willemsen; Chris C.P. Snijders

doi:10.1145/3581641.3584058

It Seems Smart, but It Acts Stupid: Development of Trust in AI Advice in a Repeated Legal Decision-Making Task

Patricia K. Kahr, Gerrit Rooks, Martijn C. Willemsen, Chris C.P. Snijders

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

6 Citations (Scopus)

105 Downloads (Pure)

Abstract

Humans increasingly interact with AI systems, and successful interactions rely on individuals trusting such systems (when appropriate). Considering that trust is fragile and often cannot be restored quickly, we focus on how trust develops over time in a human-AI-interaction scenario. In a 2x2 between-subject experiment, we test how model accuracy (high vs. low) and type of explanation (human-like vs. not) affect trust in AI over time. We study a complex decision-making task in which individuals estimate jail time for 20 criminal law cases with AI advice. Results show that trust is significantly higher for high-accuracy models. Also, behavioral trust does not decline, and subjective trust even increases significantly with high accuracy. Human-like explanations did not generally affect trust but boosted trust in high-accuracy models.

Original language	English
Title of host publication	IUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces
Place of Publication	New York
Publisher	Association for Computing Machinery, Inc
Pages	528–539
Number of pages	12
ISBN (Electronic)	9798400701061
ISBN (Print)	979-8-4007-0106-1
DOIs	https://doi.org/10.1145/3581641.3584058
Publication status	Published - 27 Mar 2023
Event	28th International Conference on Intelligent User Interfaces, IUI 2023 - Aerial Center, University of Technology Sydney (UTS), Sydney, Australia Duration: 27 Mar 2023 → 31 Mar 2023 Conference number: 28

Conference

Conference	28th International Conference on Intelligent User Interfaces, IUI 2023
Abbreviated title	IUI 2023
Country/Territory	Australia
City	Sydney
Period	27/03/23 → 31/03/23

Keywords

Collaborative Decision-Making
Human-AI Interaction
Trust Development
Trustworthy AI

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1145/3581641.3584058

pdfFinal published version, 690 KBLicence: CC BY

Cite this

Kahr, P. K., Rooks, G., Willemsen, M. C., & Snijders, C. C. P. (2023). It Seems Smart, but It Acts Stupid: Development of Trust in AI Advice in a Repeated Legal Decision-Making Task. In IUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces (pp. 528–539). Association for Computing Machinery, Inc. https://doi.org/10.1145/3581641.3584058

@inproceedings{1b9ddcd49b984fd3b58e7aec267bd538,

title = "It Seems Smart, but It Acts Stupid: Development of Trust in AI Advice in a Repeated Legal Decision-Making Task",

abstract = "Humans increasingly interact with AI systems, and successful interactions rely on individuals trusting such systems (when appropriate). Considering that trust is fragile and often cannot be restored quickly, we focus on how trust develops over time in a human-AI-interaction scenario. In a 2x2 between-subject experiment, we test how model accuracy (high vs. low) and type of explanation (human-like vs. not) affect trust in AI over time. We study a complex decision-making task in which individuals estimate jail time for 20 criminal law cases with AI advice. Results show that trust is significantly higher for high-accuracy models. Also, behavioral trust does not decline, and subjective trust even increases significantly with high accuracy. Human-like explanations did not generally affect trust but boosted trust in high-accuracy models.",

keywords = "Trust Development, Human-AI Interaction, Collaborative Decision-Making, Trustworthy AI, Collaborative Decision-Making, Human-AI Interaction, Trust Development, Trustworthy AI",

author = "Kahr, {Patricia K.} and Gerrit Rooks and Willemsen, {Martijn C.} and Snijders, {Chris C.P.}",

year = "2023",

month = mar,

day = "27",

doi = "10.1145/3581641.3584058",

language = "English",

isbn = "979-8-4007-0106-1",

pages = "528–539",

booktitle = "IUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces",

publisher = "Association for Computing Machinery, Inc",

address = "United States",

note = "28th International Conference on Intelligent User Interfaces, IUI 2023, IUI 2023 ; Conference date: 27-03-2023 Through 31-03-2023",

}

Kahr, PK , Rooks, G , Willemsen, MC & Snijders, CCP 2023, It Seems Smart, but It Acts Stupid: Development of Trust in AI Advice in a Repeated Legal Decision-Making Task. in IUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces. Association for Computing Machinery, Inc, New York, pp. 528–539, 28th International Conference on Intelligent User Interfaces, IUI 2023, Sydney, New South Wales, Australia, 27/03/23. https://doi.org/10.1145/3581641.3584058

It Seems Smart, but It Acts Stupid: Development of Trust in AI Advice in a Repeated Legal Decision-Making Task. / Kahr, Patricia K.; Rooks, Gerrit ; Willemsen, Martijn C. et al.
IUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces. New York: Association for Computing Machinery, Inc, 2023. p. 528–539.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - It Seems Smart, but It Acts Stupid

T2 - 28th International Conference on Intelligent User Interfaces, IUI 2023

AU - Kahr, Patricia K.

AU - Rooks, Gerrit

AU - Willemsen, Martijn C.

AU - Snijders, Chris C.P.

N1 - Conference code: 28

PY - 2023/3/27

Y1 - 2023/3/27

N2 - Humans increasingly interact with AI systems, and successful interactions rely on individuals trusting such systems (when appropriate). Considering that trust is fragile and often cannot be restored quickly, we focus on how trust develops over time in a human-AI-interaction scenario. In a 2x2 between-subject experiment, we test how model accuracy (high vs. low) and type of explanation (human-like vs. not) affect trust in AI over time. We study a complex decision-making task in which individuals estimate jail time for 20 criminal law cases with AI advice. Results show that trust is significantly higher for high-accuracy models. Also, behavioral trust does not decline, and subjective trust even increases significantly with high accuracy. Human-like explanations did not generally affect trust but boosted trust in high-accuracy models.

AB - Humans increasingly interact with AI systems, and successful interactions rely on individuals trusting such systems (when appropriate). Considering that trust is fragile and often cannot be restored quickly, we focus on how trust develops over time in a human-AI-interaction scenario. In a 2x2 between-subject experiment, we test how model accuracy (high vs. low) and type of explanation (human-like vs. not) affect trust in AI over time. We study a complex decision-making task in which individuals estimate jail time for 20 criminal law cases with AI advice. Results show that trust is significantly higher for high-accuracy models. Also, behavioral trust does not decline, and subjective trust even increases significantly with high accuracy. Human-like explanations did not generally affect trust but boosted trust in high-accuracy models.

KW - Trust Development

KW - Human-AI Interaction

KW - Collaborative Decision-Making

KW - Trustworthy AI

KW - Collaborative Decision-Making

KW - Human-AI Interaction

KW - Trust Development

KW - Trustworthy AI

UR - http://www.scopus.com/inward/record.url?scp=85152144514&partnerID=8YFLogxK

U2 - 10.1145/3581641.3584058

DO - 10.1145/3581641.3584058

M3 - Conference contribution

SN - 979-8-4007-0106-1

SP - 528

EP - 539

BT - IUI 2023 - Proceedings of the 28th International Conference on Intelligent User Interfaces

PB - Association for Computing Machinery, Inc

CY - New York

Y2 - 27 March 2023 through 31 March 2023

ER -

It Seems Smart, but It Acts Stupid: Development of Trust in AI Advice in a Repeated Legal Decision-Making Task

Abstract

Conference

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

28th International Conference on Intelligent User Interfaces, IUI 2023

Cite this

It Seems Smart, but It Acts Stupid: Development of Trust in AI Advice in a Repeated Legal Decision-Making Task

Abstract

Conference

Keywords

UN SDGs

Access to Document

Other files and links

Fingerprint

Activities

28th International Conference on Intelligent User Interfaces, IUI 2023

Cite this