A North Carolina museum is suing the federal government after ChatGPT allegedly flagged their $349,000 HVAC grant as "DEI-related," leading to its cancellation by the Department of Government Efficiency.

Let that sink in. A heating and air conditioning system. Flagged as diversity, equity, and inclusion programming. By a chatbot.

The High Point Museum had been approved for federal funding to replace its aging HVAC system - you know, the thing that keeps artifacts from deteriorating and makes the building habitable. According to the lawsuit, DOGE fed grant applications through ChatGPT to identify programs related to DEI initiatives, which the administration had targeted for elimination.

ChatGPT flagged the museum's grant. The grant was cancelled. No human review. No appeal process. Just an algorithm making a determination and the government treating it as fact.

This crystallizes everything wrong with rushing AI into high-stakes decision-making without understanding what these systems actually do. ChatGPT is a large language model - it's a text completion engine trained to predict the next most likely words in a sequence. It is not designed to evaluate government grants. It has no understanding of context, no ability to distinguish between actual DEI programming and infrastructure maintenance, no accountability when it gets things wrong.

And it definitely gets things wrong. LLMs hallucinate. They pick up on spurious correlations in their training data. They're sensitive to phrasing in ways that make no logical sense. Maybe the grant application mentioned "community access" or "serving diverse populations" - words that could trigger a pattern match without indicating actual DEI content.

The bigger question is: who thought this was a good idea? Someone in government decided that feeding grant applications through a chatbot was an acceptable way to make policy decisions affecting real institutions and real people's jobs. No pilot program. No validation. No human oversight.

I've built and sold software. I've seen what happens when you deploy systems without proper testing and oversight. The difference is that when a fintech startup screws up, regulators step in. When the government does it to itself, apparently anything goes.