New York's own business chatbot told owners to break the law

The conversation

On 16 October 2023, New York City Mayor Eric Adams and Chief Technology Officer Matthew Fraser announced an artificial intelligence action plan. The city's own mayoral press release named, as its centerpiece, the Adams administration 'Piloting First Citywide AI Chatbot to Help Business Owners Navigate Government' as part of the MyCity portal. Five months later, the nonprofit newsroom The Markup tested that chatbot and, in a report titled 'NYC's AI Chatbot Tells Businesses to Break the Law', published 29 March 2024, documented specific test exchanges in which the bot gave legally wrong answers.

What the documents show

The Markup's documented tests, run by its own staff, recorded the bot stating that landlords do not have to accept rental-assistance vouchers, that a residential tenant can be locked out, that there are no restrictions on rent for rent-stabilized units, that a restaurant can refuse cash, and that an employer can keep a portion of a worker's tips. The Markup reports each of these as contrary to New York City or state law and states that ten separate staff members who asked the voucher question all received the same incorrect answer. The report also quotes the city's own disclaimer, seen by Markup reporters on the chatbot page, that the tool 'may occasionally produce incorrect, harmful or biased content.' The Markup reports the bot is built on Microsoft's Azure AI services, a detail confirmed to the Markup by Microsoft's declining further comment.

The system boundary

The Markup quotes Leslie Brown, a spokesperson for the city's Office of Technology and Innovation, stating that the city 'has been clear the chatbot is a pilot program' that 'has already provided thousands of people with timely, accurate answers' while disclosing risk to users. The Markup also reports the city's own statement that questions would be reviewed to improve answers and address 'harmful, illegal, or otherwise inappropriate' content, with underlying data deleted within 30 days. As retrieved on 16 September 2026, the chatbot's own page at chat.nyc.gov states plainly: 'The Chatbot beta test has ended,' directing visitors instead to NYC.gov.

Where it fails

The Markup's report does not claim every query to the bot produced a wrong answer, and this entry does not extend that claim either; the documented failures are the specific tenant, worker, and consumer-law questions the reporters and Rosalind Black of Legal Services NYC tested. The gap between a prominent capability announcement and a small-print accuracy disclaimer is itself the pattern worth studying.

• Where does an operator's own disclaimer about incorrect output sit relative to how prominently the tool's capabilities are announced?
• What review process, if any, catches a wrong legal answer before it reaches another user asking the same question?
• What ends a pilot chatbot, and does the operator publish that decision as clearly as the launch?

Taken together, the mayor's launch release, The Markup's tested exchanges, and the chatbot's current shutdown notice trace one deployed assistant from announcement to documented failure to discontinuation.