New York moves to curb undisclosed news scraping by AI bots

New York lawmakers have passed legislation aimed at restricting ‘stealth crawlers’, automated bots that access and scrape content from news websites without identifying themselves. If signed by Governor Kathy Hochul, New York would become the first US state to impose such transparency requirements.

The bill would require companies operating such bots to identify themselves when accessing the websites of news organisations. It would also prohibit activity that damages, impairs or places undue burdens on news websites, or otherwise causes economic harm to publishers.

Supporters, including the New York State Broadcasters Association and the New York News Publishers Association, argue that undisclosed scraping allows technology companies to use journalistic content for AI and other automated services while reducing traffic and revenue opportunities for publishers.

The legislation would authorise the New York Attorney General’s office to take enforcement action against non-compliant companies, with civil penalties of up to $15,000 per day for violations. The measure was passed by lawmakers in New York and now awaits the governor’s decision.

Why does it matter?

The legislation reflects growing tensions between news publishers and technology companies over the use of online content for AI training, search services and other automated applications. Publishers increasingly argue that large-scale content scraping can generate commercial value for technology firms while undermining the business models that support journalism.

If enacted, the measure could establish one of the first state-level transparency frameworks governing automated content collection in the United States. It may also influence broader debates about AI training data, web scraping practices, publisher rights and the relationship between technology platforms and news organisations.

Would you like to learn more about AI, tech and digital diplomacy? If so, ask our Diplo chatbot!