Illustration by Erik Carter

For decades, robots.txt governed the behavior of web crawlers. But as unscrupulous AI companies seek out more and more data,

Read more on