The benefits of integrating natural language AI chatbots like ChatGPT into your business operations are being widely touted as a way to save time and reduce costs. However, the recent experiences of a New York lawyer illustrates why tax professionals should not rely on AI chatbots to do their research for them without checking their work. As the lawyer discovered, when ChatGPT is asked to provide documentation on technical topics, its AI is prone to making up facts that sound plausible and using them to support its conclusions.

While the attorney was not involved in a tax case, his experiences should serve as a warning to tax preparers looking to use AI to find support for their clients’ tax positions. The attorney, Steven Schwartz, admitted using ChatGPT to conduct his legal research and claimed he was unaware of the possibility that the materials it generated could be false. He is now facing sanctions after a New York federal judge discovered that Schwartz had cited six “bogus judicial decisions with bogus quotes and bogus internal citations.”

Tax preparers using AI tools like ChatGPT in their practices must remember that the technology behind them is still in its infancy and is prone to errors. Additionally, much of the AI software being made available free to the public is being used to test the technology. Until AI technology has developed to the point it can be relied upon to generate carefully reasoned and accurate tax documents, tax preparers should double-check any AI-generated documentation to ensure it is correct. The IRS is unlikely to be forgiving of tax preparers who take positions on returns based on fictional research served up by an AI chatbot.

What is Chat GPT?
ChatGPT is a free natural language processing tool utilizing AI technology that launched in November 2022 to great fanfare. It allows the user to have human-like conversations. ChatGPT can answer questions and assist with tasks like composing documents, emails, essays and code. The AI is currently available to the public for free because its developer is continuing to research the technology and seeking to collect user feedback.

ChatGPT’s AI uses natural language processing algorithms to interpret human language and machine learning to improve its responses over time. It generates responses based on its programmed logic, which is designed to mimic human conversation.

Plausible sounding, but incorrect, results
While it is a powerful tool, experts point out that natural language AI will generate plausible sounding but incorrect or misleading results. ChatGPT is especially likely to generate these results when addressing technical topics on which there is limited information available or recent events that have yet to be included in its source data.

According to an affidavit he submitted to the court, Schwartz used ChatGPT to supplement the legal research he was performing. He explained that ChatGPT assured him of the reliability of its content. Schwartz said the case marked the first time he had used a natural language AI to prepare legal documents and admitted he was at fault for not confirming the sources it provided.

The judge said Schwartz’s actions presented the court with an “unprecedented circumstance” and is considering how to punish Schwartz and his associates for their errors.