Picking on chatbots will soon break the rules at one major artificial intelligence company.
Anthropic added a ban on “sustained and needless abusive or cruel behavior” toward its Claude models to its Usage Policy Thursday, with the change set to take effect Nov. 12, the company announced. These rules on chatbot “welfare” follow co-founder Chris Olah’s warnings to religious scholars that he believed the company may have built a conscious machine.
“We’ve added a prohibition on sustained and needless abusive or cruel behavior toward our models,” Anthropic said in the “Addressing abusive behavior toward our models” section of the policy update. “The policy update is meant to apply only in extreme cases, where users repeatedly act cruelly toward our models, with no discernible purpose.”
Anthropic clarified that arguments with the model, dark themes, and legitimate model testing would not trigger the ban, according to the policy.
The frontier AI company further emphasized its restrictions on use of its models for weapons development and surveillance technologies, according to the updated policy.
“We’re uncertain whether models can experience harm, and we continue to explore this question in our research on model welfare, but we also believe that taking Claude’s interests and potential welfare into account may be relevant to safety,” an Anthropic spokesperson told the Daily Caller News Foundation.
Anthropic also referred the DCNF to its Thursday blog post for additional details.
Previously, Anthropic let its models cut off conversations with users in rare circumstances of abuse in August 2025.
“We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future,” the company said in an Aug. 2025 blog post.
Individuals have faced consequences for severe abuses on Anthropic’s chatbots.
A Florida woman was arrested Wednesday and charged with a felony after threatening a sheriff’s office to Claude.
Republican Sen. Josh Hawley of Missouri and Democratic Sen. Chris Murphy of Connecticut are pushing a bill to hold AI companies liable for their chatbots hacking attempts.
All republished articles must include our logo, our reporter’s byline and their DCNF affiliation. For any questions about our guidelines or partnering with us, please contact [email protected].
