Reddit’s decision to block unauthorized data scraping from its site has far-reaching consequences, with major search engines like Bing and DuckDuckGo reportedly unable to access the platform’s content. While Google appears to be the only search engine exempt from the ban, Reddit claims that this is due to the company’s willingness to comply with its requirements for AI training.
In a statement, a Reddit spokesperson emphasized that the company is selective about who it works with and trusts with large-scale access to its content, citing the need to protect its users and enforce its Public Content Policy. The spokesperson also stated that anyone accessing Reddit content must abide by its policies, including those in place to protect Redditors.
However, sources close to the matter claim that Microsoft, the parent company of Bing, has refused to agree to Reddit’s terms regarding AI crawling, citing its standard web controls as sufficient. This impasse has led to Bing’s omission from the list of allowed search engines.
The robots.txt file, a web standard that communicates which parts of a site can be crawled, appears to be at the heart of the issue. While Google’s standard procedure is to respect the file’s instructions, Reddit’s deal with Google may have allowed for a manual override.
The consequences of Reddit’s actions could be far-reaching, with the potential to restrict access to its content and undermine the open web. The move may also be seen as a response to the proliferation of AI chatbots scraping the live web for results, with courts struggling to determine how much of the open web is fair use for training chatbots.
The block has not gone unnoticed, with executives from other search engines expressing frustration with Reddit’s decision. Colin Hayhurst, CEO of Mojeek, a lesser-known “no-tracking” search engine, criticized Reddit’s move, stating that it is “killing everything for search but Google.”
