• Reddit Blocks All Web Crawlers

  • Reddit Blocks All Web Crawlers (4th July 2024)

    In an unexpected turn of events, Reddit has made a significant change to its robots.txt file, effectively blocking all web crawlers from indexing its site. This change, which took place on 4th July 2024, prevents search engines and other automated bots from accessing Reddit content. This decision comes just days after a reported deal with Google to use Reddit's content for artificial intelligence (AI) training.

    Card image cap

    Search Posts

    Related Posts

    The Current Robots.txt File

    The updated robots.txt file now simply states:

    # Welcome to Reddit's robots.txt # Reddit believes in an open internet, but not the misuse of public content.
    # See https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy Reddit's Public Content Policy for access and use restrictions to Reddit content.
    # See https://www.reddit.com/r/reddit4researchers/ for details on how Reddit continues to support research and non-commercial use.
    # policy: https://support.reddithelp.com/hc/en-us/articles/26410290525844-Public-Content-Policy

    User-agent: * Disallow: /

    Previous Robots.txt Configurations

    Until very recently, Reddit's robots.txt included a variety of specific rules allowing certain file types and excluding others. On 29th June 2024, the file was more selective but still permitted significant site indexing. Even as of 3rd July 2024, the robots.txt allowed many kinds of search engine access. The rapid shift to a blanket "Disallow: /" has raised questions among industry experts.

    Implications for Google's AI Training

    Earlier this year, Reddit confirmed a $60 million annual deal with Google, granting the tech giant access to Reddit's vast trove of data for AI training. This development was seen as a lucrative revenue stream for Reddit as it prepared for a highly anticipated stock market launch. However, the recent changes to the robots.txt file put this agreement under scrutiny.

    Was The Deal Too Good To Be True?

    By blocking all web crawlers, Reddit has also blocked Google's bots, which could significantly affect data access for AI models. This dramatic shift could be seen as a protective measure to guard proprietary content or a response to the misuse of data. While this move aims to prevent "misuse of public content," it could also hinder Google's ability to scrape data, potentially undermining the $60 million-per-year agreement.

    Reddit Robots Text on July 4th

    Impact on Search Engine Visibility

    Reddit's decision to block all web crawlers has broader implications for the site's visibility on search engines. Traditionally, a substantial portion of Reddit's traffic comes from search engine referrals. By excluding search engine crawlers, Reddit could see a decrease in discoverability for new users, affecting community growth and engagement.

    Final Thoughts

    Reddit's decision to block all robots with a sweeping change to its robots.txt file has raised eyebrows in the tech community. While aimed at preventing misuse of content, this move has potential repercussions for their AI licensing agreement with Google and their overall search engine visibility.

    jburns-cut-2

    Author

    Jason Burns is the Head of SEO at SEO Moves, a leading digital marketing agency servicing clients in the UK. With over 14 years of experience in the SEO industry, Jason has helped numerous small to medium sized businesses improve their online visibility and achieve significant growth under the digital practice of SEO. He holds a 1st Class Honours Bachelor's degree in Business & Marketing and is a certified Google Partner, demonstrating his expertise in search engine optimisation and the very best digital marketing practices.

    Jason is a frequent speaker at conferences and has been featured as an SEO expert in several online publications. He is committed to staying up-to-date with the latest trends and algorithm updates, ensuring that SEO Moves provides cutting-edge strategies and techniques to help clients achieve their online marketing goals. Jason's dedication to delivering exceptional results and providing top-notch customer service has earned him a reputation as a trusted and authoritative figure in the SEO community.

    About Us Case Studies

    Recent Blogs

    Request A Call

    Speak to an SEO Consultant for a free 30 minute discussion on how we can help you rank highly on Google.

    Speak to a member of our team at a time suited to you.

    Free SEO Analysis

    Speak to an SEO Consultant for a free 30 minute discussion on how we can help you rank highly on Google.

    Speak to a member of our team at a time suited to you.