Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on Their Long-Term Collaboration with an AI Planning System

Patricia Kahr; Gerrit Rooks; Chris C.P. Snijders; Martijn C. Willemsen

doi:10.1145/3706598.3713099

Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on Their Long-Term Collaboration with an AI Planning System

Patricia Kahr, Gerrit Rooks, Chris C.P. Snijders, Martijn C. Willemsen

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

23 Downloads (Pure)

Abstract

While research on trust in human-AI interactions is gaining recognition, much work is conducted in lab settings that, therefore, lack ecological validity and often omit the trust development perspective. We investigated a real-world case in which logistics experts had worked with an AI system for several years (in some cases since its introduction). Through thematic analysis, three key themes emerged: First, although experts clearly point out AI system imperfections, they still showed to develop trust over time. Second, however, inconsistencies and frequent efforts to improve the AI system disrupted trust development, hindering control, transparency, and understanding of the system. Finally, despite the overall trustworthiness, experts overrode correct AI decisions to protect their colleagues’ well-being. By comparing our results with the latest trust research, we can confirm empirical work and contribute new perspectives, such as understanding the importance of human elements for trust development in human-AI scenarios.

Original language	English
Title of host publication	CHI '25
Subtitle of host publication	Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems
Editors	Naomi Yamashita, Vanessa Evers, Koji Yatani, Xianghua (Sharon) Ding, Bongshin Lee, Marshini Chetty, Phoebe Toups-Dugas
Place of Publication	New York
Publisher	Association for Computing Machinery, Inc.
Number of pages	16
ISBN (Electronic)	979-8-4007-1394-1
DOIs	https://doi.org/10.1145/3706598.3713099
Publication status	Published - 25 Apr 2025
Event	ACM CHI 2025 Conference on Human Factors in Computing Systems - Yokohama, Japan Duration: 26 Apr 2025 → 1 May 2025 https://chi2025.acm.org/

Conference

Conference	ACM CHI 2025 Conference on Human Factors in Computing Systems
Abbreviated title	CHI2025
Country/Territory	Japan
City	Yokohama
Period	26/04/25 → 1/05/25
Internet address	https://chi2025.acm.org/

Funding

We thank DB Schenker for their support and for providing access to real-world insights.We particularly want to thank the interviewees, particularly the truck drivers, for their time and openness. This research was carried out as part of the AI Planner of the Future research program, with support from the Eindhoven Artificial Intelligence Systems Institute (EAISI), the European Supply Chain Forum (ESCF), and the Brabant Logistics Community (LCB).

Keywords

AI-supported decision-making
Trust (development) in human-AI-interactions

Access to Document

10.1145/3706598.3713099

pdfFinal published version, 1.02 MBLicence: CC BY

Cite this

Kahr, P., Rooks, G., Snijders, C. C. P., & Willemsen, M. C. (2025). Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on Their Long-Term Collaboration with an AI Planning System. In N. Yamashita, V. Evers, K. Yatani, X. Ding, B. Lee, M. Chetty, & P. Toups-Dugas (Eds.), CHI '25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems Article 1100 Association for Computing Machinery, Inc.. https://doi.org/10.1145/3706598.3713099

Kahr, Patricia ; Rooks, Gerrit ; Snijders, Chris C.P. et al. / Good Performance Isn't Enough to Trust AI : Lessons from Logistics Experts on Their Long-Term Collaboration with an AI Planning System. CHI '25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. editor / Naomi Yamashita ; Vanessa Evers ; Koji Yatani ; Xianghua (Sharon) Ding ; Bongshin Lee ; Marshini Chetty ; Phoebe Toups-Dugas. New York : Association for Computing Machinery, Inc., 2025.

@inproceedings{d77f6b3e7f654cf1b1f122c2ce430eae,

title = "Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on Their Long-Term Collaboration with an AI Planning System",

abstract = "While research on trust in human-AI interactions is gaining recognition, much work is conducted in lab settings that, therefore, lack ecological validity and often omit the trust development perspective. We investigated a real-world case in which logistics experts had worked with an AI system for several years (in some cases since its introduction). Through thematic analysis, three key themes emerged: First, although experts clearly point out AI system imperfections, they still showed to develop trust over time. Second, however, inconsistencies and frequent efforts to improve the AI system disrupted trust development, hindering control, transparency, and understanding of the system. Finally, despite the overall trustworthiness, experts overrode correct AI decisions to protect their colleagues{\textquoteright} well-being. By comparing our results with the latest trust research, we can confirm empirical work and contribute new perspectives, such as understanding the importance of human elements for trust development in human-AI scenarios.",

keywords = "AI-supported decision-making, Trust (development) in human-AI-interactions",

author = "Patricia Kahr and Gerrit Rooks and Snijders, \{Chris C.P.\} and Willemsen, \{Martijn C.\}",

year = "2025",

month = apr,

day = "25",

doi = "10.1145/3706598.3713099",

language = "English",

editor = "Naomi Yamashita and Vanessa Evers and Koji Yatani and Ding, \{Xianghua (Sharon)\} and Bongshin Lee and Marshini Chetty and Phoebe Toups-Dugas",

booktitle = "CHI '25",

publisher = "Association for Computing Machinery, Inc.",

address = "United States",

note = "ACM CHI 2025 Conference on Human Factors in Computing Systems, CHI2025 ; Conference date: 26-04-2025 Through 01-05-2025",

url = "https://chi2025.acm.org/",

}

Kahr, P , Rooks, G , Snijders, CCP & Willemsen, MC 2025, Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on Their Long-Term Collaboration with an AI Planning System. in N Yamashita, V Evers, K Yatani, X Ding, B Lee, M Chetty & P Toups-Dugas (eds), CHI '25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems., 1100, Association for Computing Machinery, Inc., New York, ACM CHI 2025 Conference on Human Factors in Computing Systems, Yokohama, Japan, 26/04/25. https://doi.org/10.1145/3706598.3713099

Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on Their Long-Term Collaboration with an AI Planning System. / Kahr, Patricia ; Rooks, Gerrit ; Snijders, Chris C.P. et al.
CHI '25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. ed. / Naomi Yamashita; Vanessa Evers; Koji Yatani; Xianghua (Sharon) Ding; Bongshin Lee; Marshini Chetty; Phoebe Toups-Dugas. New York: Association for Computing Machinery, Inc., 2025. 1100.

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › Academic › peer-review

TY - GEN

T1 - Good Performance Isn't Enough to Trust AI

T2 - ACM CHI 2025 Conference on Human Factors in Computing Systems

AU - Kahr, Patricia

AU - Rooks, Gerrit

AU - Snijders, Chris C.P.

AU - Willemsen, Martijn C.

PY - 2025/4/25

Y1 - 2025/4/25

N2 - While research on trust in human-AI interactions is gaining recognition, much work is conducted in lab settings that, therefore, lack ecological validity and often omit the trust development perspective. We investigated a real-world case in which logistics experts had worked with an AI system for several years (in some cases since its introduction). Through thematic analysis, three key themes emerged: First, although experts clearly point out AI system imperfections, they still showed to develop trust over time. Second, however, inconsistencies and frequent efforts to improve the AI system disrupted trust development, hindering control, transparency, and understanding of the system. Finally, despite the overall trustworthiness, experts overrode correct AI decisions to protect their colleagues’ well-being. By comparing our results with the latest trust research, we can confirm empirical work and contribute new perspectives, such as understanding the importance of human elements for trust development in human-AI scenarios.

AB - While research on trust in human-AI interactions is gaining recognition, much work is conducted in lab settings that, therefore, lack ecological validity and often omit the trust development perspective. We investigated a real-world case in which logistics experts had worked with an AI system for several years (in some cases since its introduction). Through thematic analysis, three key themes emerged: First, although experts clearly point out AI system imperfections, they still showed to develop trust over time. Second, however, inconsistencies and frequent efforts to improve the AI system disrupted trust development, hindering control, transparency, and understanding of the system. Finally, despite the overall trustworthiness, experts overrode correct AI decisions to protect their colleagues’ well-being. By comparing our results with the latest trust research, we can confirm empirical work and contribute new perspectives, such as understanding the importance of human elements for trust development in human-AI scenarios.

KW - AI-supported decision-making

KW - Trust (development) in human-AI-interactions

UR - http://www.scopus.com/inward/record.url?scp=105005736099&partnerID=8YFLogxK

U2 - 10.1145/3706598.3713099

DO - 10.1145/3706598.3713099

M3 - Conference contribution

BT - CHI '25

A2 - Yamashita, Naomi

A2 - Evers, Vanessa

A2 - Yatani, Koji

A2 - Ding, Xianghua (Sharon)

A2 - Lee, Bongshin

A2 - Chetty, Marshini

A2 - Toups-Dugas, Phoebe

PB - Association for Computing Machinery, Inc.

CY - New York

Y2 - 26 April 2025 through 1 May 2025

ER -

Kahr P , Rooks G , Snijders CCP , Willemsen MC. Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on Their Long-Term Collaboration with an AI Planning System. In Yamashita N, Evers V, Yatani K, Ding X, Lee B, Chetty M, Toups-Dugas P, editors, CHI '25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. New York: Association for Computing Machinery, Inc. 2025. 1100 doi: 10.1145/3706598.3713099

Good Performance Isn't Enough to Trust AI: Lessons from Logistics Experts on Their Long-Term Collaboration with an AI Planning System

Abstract

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this