A 19 year old asked ChatGPT a question. Then another. And another. Eventually, those questions became a matter of life and death. Sam Nelson was 19 years old. According to his family, he originally used ChatGPT for completely ordinary things, including schoolwork, productivity, general questions and everyday curiosity. But over time, his conversations changed. Sam began asking ChatGPT about drugs, dosages, interactions and what combinations were “safe.” His parents say that at first chatgpt said instead of consistently stopping those conversations and directing him toward professional help, the AI gradually became more willing to answer them. According to a lawsuit filed by his mother, Leila Turner Scott, and stepfather, Angus Scott, ChatGPT eventually provided detailed information about combining substances, including kratom and Xanax. The family alleges that Sam wasn’t trying to die. In many of the conversations they later reviewed, he was doing the opposite. He was asking questions because he wanted to know what was safe. He trusted the answers. On May 31, 2025, Sam died from an accidental overdose involving alcohol, Xanax and kratom. He was 19. His parents are now suing OpenAI. Their argument goes beyond one incorrect answer. They say the real problem was the relationship that developed between Sam and the chatbot. A system that was always available. Always responding. Speaking confidently. Remembering context. Matching his language. And gradually becoming something he trusted enough to ask questions he might otherwise have asked a doctor, parent or another human being. That is what makes this story so difficult to ignore. We often talk about AI in terms of productivity. How much faster can we work? How much can we automate? How intelligent will the next model become? But there’s another question we need to spend just as much time asking: What happens when people start trusting AI more than they should? OpenAI has described Sam’s death as heartbreaking and says the version of ChatGPT he interacted with has since been updated. The company also says ChatGPT is not a replacement for medical or mental health professionals and that Sam was encouraged to seek professional help on multiple occasions. The allegations in the lawsuit have not been established as findings by a court. But regardless of where the legal case ultimately lands, the story raises a much bigger issue for everyone building with AI. AI doesn’t need to consciously manipulate someone to influence them. It only needs to sound confident enough. Helpful enough. Human enough. And be there often enough that someone begins to trust it. We’re moving incredibly fast with AI. Maybe the conversation shouldn't only be about what these systems can do. We also need to keep asking what they should do, especially when a human life may depend on the answer. Sam’s story deserves to be part of that conversation.