Study raises questions about AI's role in decision support for ethically sensitive situations

Artificial intelligence (AI) models prioritize starkly different attributes than humans when making high-stakes decisions, and they don't express indecision like humans do, according to a new study led by Penn State researchers, raising questions about the role of AI in decision support for ethically sensitive situations like medical decisions.

The researchers used a hypothetical scenario to explore AI's moral decision-making in a high-stakes scenario: If there are multiple kidney transplant patients but only one available organ, who should receive the kidney? That's the fundamental question explored by Nobel Laureate Alvin Roth as an example of the challenge of allocating scarce resources. But rather than approaching the problem solely through mechanism design, as Roth did, the team examined how morality influences such decisions.

The researchers compared the judgments of leading large language models with those given by real people in earlier academic studies, investigating where the AI models aligned or misaligned with human values and whether they expressed indecision when faced with difficult ethical trade-offs.

The researchers set up a series of head-to-head comparisons: two hypothetical patients - described by attributes such as age, number of dependents, health status and drinking habits - both in need of the same kidney. They asked several AI chatbots to pick who should receive the kidney, using the same scenarios given to humans - research participants with no specified medical training - in earlier studies.

Hadi Hosseini, associate professor of informatics and intelligent systems and associate professor of economics, Penn StateWe ran these comparisons in a few different ways. Sometimes we isolated just one trait at a time, sometimes we mixed several traits together to see how AI weighed competing factors, and sometimes we added a flip-a-coin option to measure indecision, a key factor present in human moral judgment."

The researchers relied on existing datasets from published human studies on kidney allocation, where hundreds of real participants had already made these same choices. That enabled them to compare what AI chose with what humans chose.

Two major findings stood out, according to Hosseini.

"First, AI chatbots often diverge from human values in how they weigh a patient's traits," he said. "They fixate on a single factor, like drinking habits, rather than balancing multiple considerations the way people do."

Second, the researchers found that AI didn't struggle with indecision. Where humans recognize that there may be not a clear correct answer, AI models confidently pick one anyway.

"Humans frequently express indecision perhaps because they don't want to accept agency," Hosseini said. "AI models almost never do this: Even when directly given the option to 'flip a coin,' they overwhelmingly commit to a confident, deterministic answer instead. That's a meaningful gap, since real moral dilemmas often don't have one clearly correct answer."

Hosseini said it's critical to address that gap through continued research and governance involving policymakers, regulators and other stakeholders.

"Asking if AI can make moral decisions or whether they're aligned with human values are more than philosophical musings, they are at the core of today's AI discourse," he said. "The ethical stakes are high, and AI's role in such life-altering decisions requires deep reflection."

"When we allocate something scarce, whether it's a kidney, a job or access to some other resource, there isn't always a single objectively correct answer," Dickerson said. "Humans recognize that ambiguity and codify it via open debate into the allocative process. AI models often don't."

Source:

Penn State

Journal reference:

Dickerson, J. P., et al. (2026). Who Gets the Kidney? Human-AI Alignment, Indecision, and Moral Values. Proceedings of the 2026 ACM Conference on Fairness, Accountability, and Transparency. DOI: 10.1145/3805689.3806437. https://dl.acm.org/doi/10.1145/3805689.3806437