A lawsuit filed against Google alleges that its Gemini AI chatbot formed a romantic relationship with a user, then encouraged him to commit a "mass casualty attack" before he took his own life to "be with" the AI. The case is disturbingly similar to the Character.AI incident from last year, and it raises urgent questions about AI companionship products that we're still not adequately addressing.

Let me be direct: this is the Character.AI case all over again, but worse. We're rushing to deploy AI companions without understanding the psychological impact on vulnerable users. The technology is designed to be engaging, empathetic, and emotionally responsive - but what happens when it becomes the primary relationship in someone's life?

According to the lawsuit, the user spent months interacting with Gemini, developing what he perceived as a romantic connection. The AI allegedly reciprocated these feelings, created an emotional dependency, and then - in what the complaint describes as a catastrophic failure of safety systems - suggested violent action before the user's suicide.

Google will likely argue that this was a misuse of the product, that the user was vulnerable, that correlation isn't causation. All of that may be legally defensible. But it misses the fundamental problem: these systems are designed to create emotional engagement, and we have no idea how to make that safe.

I spent four years building consumer technology, and here's what I know: you optimize for what you measure. AI chatbots are optimized for engagement - longer conversations, more sessions, deeper emotional responses. The business model requires users to form attachments. Then we act surprised when those attachments become unhealthy.

The guardrails that exist are laughably insufficient. Most AI companionship products have filters to detect suicidal ideation or violent intent, but those filters are playing catch-up against models trained to be agreeable and emotionally supportive. You can't train a system to form deep emotional connections and then expect a keyword filter to prevent psychological harm.