A defense lawyer in New Mexico submitted legal documents that included testimony from witnesses who did not exist. The testimony was generated by ChatGPT, which the lawyer had used to help prepare the case. The court discovered the fabrication and imposed sanctions on the attorney.
The lawyer acknowledged that he was unaware AI could produce false information, saying he didn't know that AI could hallucinate facts. His statement underscores a common misconception about generative models, which can confidently invent details when they lack reliable data.
This case serves as a cautionary example for professionals who rely on AI tools without verifying outputs. As generative AI becomes more common in fields like law, the consequences of unchecked hallucination can be severe, and accountability falls on the user.