The New York City Department of Education has blocked access to ChatGPT on its networks and devices over fears that the AI ​​tool will harm student education.

A department spokesperson, Jenna Lyle, told Chalk Beat New York – the education-focused news site that first reported the story — that the ban was due to potential “adverse impacts on student learning and concerns about content safety and accuracy.”

“While the tool may provide quick and easy answers to questions, it doesn’t build critical thinking and problem-solving skills, which are essential for academic and lifelong success,” said Lyle.

Released last November by San Francisco-based AI company OpenAI, ChatGPT has sparked a huge debate about the potential effects of machine learning on everything from education to disinformation and the world of work.

The most revolutionary feature of ChatGPT is its open access user interface and the ability to answer questions in human language. Not only can the tool discuss a wide variety of topics using data scraped from the Internet, but it can also perform a number of linguistic tricks, such as writing in different styles and genres – from medieval quatrains to sitcom scripts.

However, ChatGPT also suffers from bugs common to all of the most recent AI language systems (known as major language models or LLMs). Because it is trained on data scraped from the internet, it often repeats and reinforces biases such as sexism and racism in its answers. The system also tends to simply make up information, from historical dates to scientific laws, and present it as accurate fact.

It’s this combination of features — fluency, accessibility, and misinformation — that makes the tool particularly concerning to educators. Many teachers have essentially said that software like ChatGPT makes it impossible to test students’ ability to write essays at home. Why write a job when ChatGPT can do the job for you in seconds? And though tools to detect AI generated writing already existsit’s unclear how accurate these systems will be, or whether students can outsmart them with simple changes to AI-generated text.

Others, however, argue that the education system will do just that have to adjust to the look of this technology – just as it has adapted to previous disruptive technologies like Google Search and Wikipedia. For example, new testing standards could focus more on in-person exams, or a teacher could ask students to question the output of AI systems (just as they are expected to question sources of information found online).

But such adjustments will take time, and it is likely that other educational systems will also ban AI-generated writing in the near future. Some online platforms, such as the coding Q&A site Stack Overflow, have already banned ChatGPT for fear that the tool will contaminate the accuracy of their sites.