Weil, the academy’s CEO, agrees with the need for more skepticism toward AI systems and advocates for a ban on companion AI chatbots or systems for students under 18 years old.

“These algorithms are written in a way to be sycophantic, and so they seem like your friend because they’re trained to be your friend. They’re not trained to tell you the truth.”

Many AI companies acknowledge that their models can be sycophantic. In late April, OpenAI released a regular update that drastically increased ChatGPT’s sycophancy before walking it back to a model “with more balanced responses.” OpenAI said in a statement that it recognized that this behavior could seem “uncomfortable or unsettling” and “raise safety concerns.”

More importantly, Weil said, AI “should not be a replacement for human interaction and human relationships. Human relationships can be messy and really problematic and hard, but that’s the beauty of humanity.”

Faced with an often-crushing dearth of resources and little time to provide individual attention to students, many teachers at the workshop were eager to learn how AI systems can enhance — not replace — existing education methods.

“I want to move past the fact that teachers are using AI to help them with lesson plans,” said Rose, the computer skills teacher who works in Queens, citing one of the more commonly-used examples of AI in the classroom today. “We want to delve deeper and dig deeper. I want to make sure that all of us are well informed and that teachers understand all the different uses of AI.”