Amazon launches inquiry into allegations of unauthorized data scraping

Tech & AI | June 27, 2024, 7:43 p.m.

Amazon's cloud division is investigating AI search startup Perplexity for potentially violating Amazon Web Services rules by scraping websites that prohibited access. The investigation comes after reports that Perplexity relies on content from scraped websites that utilize the Robots Exclusion Protocol, a web standard indicating which pages should not be accessed by automated bots and crawlers. While the protocol is not legally binding, terms of service generally are. Perplexity, backed by the Jeff Bezos family fund and Nvidia and valued at $3 billion, has been accused of stealing articles and engaging in scraping abuse and plagiarism. Despite claims by Perplexity CEO Aravind Srinivas that the scraping was conducted by a third-party company, investigations have revealed extensive crawling of news websites that forbid bots, prompting AWS to launch an investigation into potential breaches of its terms of service.