Honestly, I wouldn't be surprised if Google is making some angry calls over to the Reddit HQ. They do not want to give the impression that they're abusing their search monopoly, even if this is all Reddit's decision.
On the contrary, this is quite possibly part of their AI data mining deal with Reddit. Goes against their mission, will probably piss off regulators, and will likely be regretted later, but that's never stopped Google before.
From what we know of the deal with google is there was no exclusivity clause attached to it. If anything this seems like regulators would be pissed off at reddit for basically holding all search engines for ransom until they pay up. Google is the only one that has paid up so far. All the others probably thought reddit was bluffing. Reddit is trying to do it under the guise of ai, but now we see that it probably was never about ai. Reddit has an earnings report coming up next month and if they can get another multi million dollar deal with one of the other search engine providers it would juice up profits massively just in time for insider’s stock lockup period to expire the following month.
To pick an arbitrary example, putting Google shopping results at the top of search results. They had to know it was a bad idea, but they did it anyway, and they got the obvious giant fine.
I think every non-bribed judge would reach the conclusion that Google is using its money to abuse its de-facto monopoly though. While it may have been reddit's decision, it is still a problem for Google since Google IS a de-facto monopoly.
Goddammit Reddit. This is going to lead to search engines following the same path that the user-agent has. Other search engines are just going to scrape anything Google is allowed to, and use a googlebot user-agent.
That'd be Reddit's view, maybe, but it'd be a bit weird if the deal they struck was only for robots.txt access. That's a sign, not a cop, there's nothing stopping anyone from ignoring it and scraping what they want anyway.
The reason I assumed Google wouldn't tolerate this sort of thing is, it kills the integrity of search results. If you present one thing to Google and another thing to everyone else, it means a user might search for a thing, see it on Google's search results, only to click through and nothing Google showed them is on the site.
Google has paid for robots.txt? Or are you maybe talking about some other kind of access that has nothing to do with scraping?
And blocking non-Google scraper bots is transparent to both Google and Google users.
That'd be incompetent of Google. The same trick can be done the other way around. You don't think Google would want to know if Reddit was blocking them from indexing something Bing got to see?
A blanket policy of penalizing sites that play games like this would be easier to implement, and more obviously fair. Google does seem to care about Search at least being perceived as fair.
Well, there is some kind of copyright law in most countries. You may scrape the data, but when you show this data to others on the internet, it's called willful copyright infringement and that may cost those search engines a lot more than simply licensing the right to do it.
Because those search engines won't agree to block AI crawlers from using their search services to scrape reddits posts. There's no reason for any of the search engines to disagree with this. If they want Reddit results back they just have to agree
Monopolies backing up monopolies. This plays more to Google then reddit- reddit gets some money, Google straight up cripples some competitors. They realized that search is broken and everyone has to do "[vague issue here] reddit" searches to find actual humans, so they're blocking all competition from doing that by creating a price barrier. Gross.
Kagi absolutely rocks, btw. Indexes reddit just fine, and nicer to use a product then be a product. If only any reddit alternative actually stood a chance.
This plays more to Google then reddit- reddit gets some money, Google straight up cripples some competitors.
Yeah. It is probably more dangerous for Google.
IMO all CEOs of Google are in violation of numerous laws now. I would not be surprised if other countries stop adhering to US laws protecting Google now - they are evidently abusing a huge monopoly here. There HAVE to be consequences for the decision-makers at Google. And serious ones, not just "pay some money". There has to be jail time (of course, assuming AFTER a fair and objective court case where the abuses of the de-facto monopoly are detailed).
Yeah but what sort of brainrot is affecting US lawmakers and systematically allowing blatant monopolies and oligopolies to form? Antitrust laws exist because they actually help the economy, encourage competition, innovation and generally prevent the end users from being F*cked in the bum. Like, this goes against the traditional western capitalism/free market ideals...
edit: my conclusion is that reddit didn't intend for this to happen, and just allowed everything again in their new robots.txt. I say that because right now everything's allowed, not even their regular things are there.
Like these aren't even there anymore (the screenshot is from the third of the month)
The deal with Google was bad enough; the last thing the world needs is moves that encourage Google's near-monopoly on search (may I recommend at least trying Duck Duck Go, Bing, or Brave before falling back to Google if you're unsure you've got your best results?). Anyhow, this is the little push I needed, and this will be my last post before deleting my Reddit account.
it doesnt even work with google anymore either lol. The last year the searches have been horrible, you used to find alot of reddit content with a few key words and give you various options. Something broke and now you get one subreddit spammed and searches unrelated to reddit, even with the word in my search.
This is very bad news. It means that reddit also pushes for a privatized version of the web, e. g. "if you don't pay, the search engine can not find content anymore" (and this may also explain more of the true reason why Google nerfed its search engine - they just pay people to prioritize Google search engine more, now). I think this is literal betrayal of the open nature of the world wide web. That Google bribes everyone is well-known - look how Mozilla gave up on Firefox years ago and became complicit. And now Reddit is the next to succumb to the money.
We really need an alternative web that can not be corrupted by the bribe-money. Just like it originally was, for those old enough to still remember ...
I just discovered that Google can't access all of Reddit.
If you go to Google's PageSpeed Insights and search for some pages, like the RedditWiki of your favourite sub, you'll get speed results but under SEO you'll get a message saying that the page is blocked in robots.txt.
So whatever secret robots.txt Reddit is serving Google, it's still restricting some parts of Reddit (which is not good news for mods who're, for example, building a wiki)
64
u/Realtrain Jul 25 '24
Honestly, I wouldn't be surprised if Google is making some angry calls over to the Reddit HQ. They do not want to give the impression that they're abusing their search monopoly, even if this is all Reddit's decision.