Recently formed AI Preferences Working Group has August deadline to develop ideas on how to tell crawlers to go away, or come for a feast The Internet Engineering Task Force has chartered a group it ...

Last summer, Anthropic inspired backlash when its ClaudeBot AI crawler was accused of hammering websites a million or more times a day. And it wasn't the only artificial intelligence company making ...