When Computers Join the Moral Conversation

This essay draws attention to an underappreciated phenomenon that urgently needs attention and further research, but which was not covered in the 2021 AI100 Study Panel Report. Namely, computers are “joining the moral conversation,” in the sense that LLM-based chatbots now readily and flexibly respond to and apply many moral terms, and they appear to perform a number of conversational roles – producing outputs that look like moral assertions, expressions of moral sentiment, moral commands and reprimands, assent to norms, etc. These developments introduce unprecedented potential for computers to influence human norms and values. Consequently, we need interdisciplinary research into what capacities different LLM-based chat systems possess, how their dispositions diverge from human moral psychological dispositions, how humans respond to different forms of apparent moral communications from computers, and, ultimately, what roles computers should be permitted to play in conversations about values, norms, and moral questions.