Login
Donate
image

Discussion Guide and Transcript

Season Three - Episode Nine

Research Ethics Reimagined Podcast logo with pictures of speakers.

Research Ethics Reimagined Podcast Season Three: Episode Nine “When Does AI Deliver Meaningful Benefit" Guests: Alex John London and Ida Mattsson

  • In this episode of PRIM&R's podcast, "Research Ethics Reimagined," we explore the ethics of artificial intelligence systems that emulate real people with Alex John London, PhD, the K&L Gates Professor of Ethics and Computational Technologies and co-lead of the K&L Gates Initiative in Ethics and Computational Technologies at Carnegie Mellon University, and Ida Mattsson, a PhD student at CMU. London and Mattsson discuss their co-authored paper on "systems that emulate identifiable individuals," drawing on a case study of voice-based AI designed to assist older adults experiencing cognitive decline. They examine when assistance becomes unjustified paternalism or deception, how responsibility is shared among developers, deployers, and the people being emulated, and why institutional review boards (IRBs) are a limited tool for overseeing technologies of this kind. Listen on Spotify | Listen on Apple| Listen on Amazon

Discussion Questions

Assistance, Paternalism, and Genuine Benefit
  • London describes a child's emulated voice telling a disoriented parent to go back to bed as "paternalistic" yet assistive, because it helps the parent act on values they hold but cannot act on in the moment. He warns, however, that the same technology could be used to limit older adults' choices to serve the interests of a child or a care facility. What safeguards could help developers, families, and institutions tell justified paternalism apart from unjustified paternalism?
  • Mattsson explains that for paternalistic interventions to be truly assistive, a system needs some way to access a person's "true preferences" in advance, since the person may not be able to express them later. What would a trustworthy mechanism for capturing and honoring those preferences look like, and who should have a say in how it is designed?
Shared Control, Responsibility, and Transparency
  • Mattsson and London describe a "triadic relationship" among the developer who builds a system, the deployer who uses it, and the person whose likeness is emulated, and London notes that many regulatory frameworks presume a "single locus of control." When a system behaves in ways the person being emulated never intended, how should responsibility be shared, and who should have the authority to stop or withdraw a deployment?
  • Mattsson distinguishes between watermarks that help trace where content came from and perceptible cues, such as a cartoon-style avatar, that tell users they are not interacting with the real person. London adds that the right choice depends on the harm one is trying to avoid. How should developers and institutions decide which type of disclosure fits a given application?
Oversight, Consent, and the Role of Research Ethics
  • London observes that systems like these "can be developed and deployed without ever going before an IRB," since IRBs review only research involving human subjects, and calls IRBs "a really limited instrument." Where does that leave oversight of technologies that affect people but fall outside of research regulation, and what role could research ethics professionals play in filling the gap?
  • Mattsson argues that consent and alignment should not be treated as "one shot" events, pointing to augmentative and alternative communication technologies, where a person retains control over how a voice emulation is used over time. How might this iterative approach to consent and data control inform the way researchers and IRBs think about consent for studies involving AI technologies?
Key Terms
  • Systems That Emulate Identifiable Individuals (SEIIs): AI systems that reproduce the voice, appearance, or behavior of a specific real person, whether living or deceased. The guests distinguish them from general-purpose chatbots with a generic persona and from deepfakes, which they describe as artifacts that such systems can produce.
  • Value Alignment: The effort to design AI systems so that their behavior reflects and advances the values and goals of the people they affect, rather than simply doing something new or technically impressive.
  • Justified Paternalism: Intervening in a person's choices for their own good in a way that helps them act on their own considered values, as when a system gently redirects someone whose cognitive decline prevents them from acting on their wish to stay safe at home. It becomes unjustified when the intervention serves someone else's interests.
  • Triadic Relationship: The guests' term for the three parties involved when an AI system emulates a person: the user who interacts with the system, the deployer who puts it to use, and the individual being emulated, any of whom may be a group rather than a single person.
  • Augmentative and Alternative Communication (AAC): Technologies that help people who cannot speak, or cannot speak easily, to communicate, including systems that generate speech in a voice resembling the user's own recorded voice.
Resources

Transcript

Please note, a transcript generator was used to help create written show transcript. Listen to the full conversation on PRIM&R’s Research Ethics Reimagined podcast.