Google won’t comment on a potentially massive leak of its search algorithm documentation ( www.theverge.com )

A purported leak of 2,500 pages of internal documentation from Google sheds light on how Search, the most powerful arbiter of the internet, operates.

The leaked documents touch on topics like what kind of data Google collects and uses, which sites Google elevates for sensitive topics like elections, how Google handles small websites, and more. Some information in the documents appears to be in conflict with public statements by Google representatives, according to Fishkin and King.

NutWrench ,
@NutWrench@lemmy.world avatar

Here's the sooper-secret search result algorithm for whatever you type into Google:

YouTube results, followed by Reddit results, followed by "Sponsored" results, followed by AI-written Bot results, then a couple pages of Amazon results and finally, on page 10 or so, a ten-year-old result that's probably no longer relevant.

pyre ,

awesome, now we can make our own search engine that is filled with complete trash and isn't concerned with helping the user at all.

muntedcrocodile , (edited )

Someone got a torrent for us?

douglasg14b , (edited )
@douglasg14b@lemmy.world avatar

Where can one get a hold of these documents?

This appears to be the original blog post, but I'm not finding a way to download this. https://sparktoro.com/blog/an-anonymous-source-shared-thousands-of-leaked-google-search-api-documents-with-me-everyone-in-seo-should-see-them/

Is this not leaked past this one person?

Edit 2: No, these appear to be normal public docs.

Edit: seems these are the docs? https://hexdocs.pm/google_api_content_warehouse/0.4.0/GoogleApi.ContentWarehouse.V1.Model.QualityNavboostCrapsCrapsData.html

redcalcium ,

Grab it while it's still up: https://github.com/yoshi-code-bot/elixir-google-api/commit/d7a637f4391b2174a2cf43ee11e6577a204a161e

iopq ,

I want a federated social bookmarking site. Not for news or discussion of recent stuff, but to keep some good sites in your account and to share with others.

Searching those and getting results with attached upvotes/downvotes would be ideal

nucleative ,

Interesting concept. Like if you could upvote/downvoted the SERP and it actually mattered and wasn't easy to manipulate.

rottingleaf ,

Was going to say that I was dreaming of such platform, but then it can be used for more than just links, and work as a decentralized Usenet, and what's more important, as a rating system potentially more resilient to abuse (by bots or by people whose votes you don't care about). Then noticed that you wrote "federated".

RizzRustbolt ,

Oh thank the tiny gods...

I thought this was going to be a leak of actual information.

deweydecibel , (edited )

Rand Fishkin, who worked in SEO for more than a decade, says a source shared 2,500 pages of documents with him with the hopes that reporting on the leak would counter the “lies” that Google employees had shared about how the search algorithm works.

Am I supposed to care that the poor SEO assholes that need to get their ads more visibility weren't being given all the instructions on how to do that by the search engine?

Most of this article is SEO "experts" complaining that some of the guidelines they were given didn't match what's in the internal documents.

Google is shit, but SEO is a cancer too. I can't be too bothered by Google jacking them around a bit.

lvxferre , (edited )
@lvxferre@mander.xyz avatar

And I supposed to care that the poor SEO assholes that need to get their ads more visibility weren’t being given all the instructions on how to do that by the search engine?

No. You're supposed to care that a company is pointlessly* lying, thus it's extremely likely to deceive, mislead and lie when it gets some benefit out of it.

In other words: SEO arseholes can ligma, Google is lying to you and me too.

*I say "pointlessly" because not disclosing info would achieve practically the same result as lying.

brbposting ,

need to get their ads more visibility

I occasionally encounter the desire for a search engine to surface non-advertisement content :)

Now if they lied to advertisers and told small bloggers, reputable news agencies, fediverse admins, etc. the insider secrets… now we’re talkin’!

WindyRebel , (edited )

Edit: If you’re going to downvote me, please take the time to explain why you think I’m wrong. Stop being the hive mind.

Tell me you don’t know shit about SEO without telling me you don’t know shit about SEO.

Just because there are people who do bad things doesn’t mean the industry is bad or have bad intentions. SEO isn’t ads. Advertorials can be a tactic of SEO, but it’s not SEO as a whole. Same with clickbait because it works, and I guarantee you also fall for it constantly.

SEO is about understanding what someone needs and creating an experience to ensure that someone finds the answer to what they need through content and/or a product to solve their needs.

This can be achieved through copywriting, researching search trends and queries, technical analysis of websites and how they render, providing guidance on helpful assets (photos, pdfs, videos, form, copy, etc), PR outreach because links are how people move around online or discover things, social planning because social media are a form of search engines, and more.

And finally, SEOs are not responsible for how Google treats shit. That’s Google who is responsible. Google is the one that tweaks the algorithm and doesn’t catch spammy shit. In fact many SEOs catch it and report it to Google’s reps, but they are the ones who can ensure the right team(s) fix the issue.

nick ,

Now, where to download these, for science.

Tronn4 ,

Use Bing 😅😅

NoIWontPickAName ,

Google it.

mlg ,
@mlg@lemmy.world avatar

Google has been pretty crap for a decade now.

I still remember demoing how easily they can manipulate people by searching "Pakistan News" and the results being exclusively all Indian media outlet propaganda way back in 2016.

I really feel like they never got properly exposed for this just because it's a search engine and not a social media, so people didn't care enough about it. Also because Google was still top of the game in most results compared to other sites back then.

SirEDCaLot ,

My thought exactly. If this was back in like 2010, it would be a real oh shit moment, The key to the kingdom has been leaked. Now I don't think anybody really cares other than SEO spammers who will game the system even more than they already are.

Google search is crap and has been crap for some time. Not sure any others are better. But it started going downhill with the Google Plus social network, when they removed "+" as a search operator so you could better search for 'Google+' that was the first time they messed with Search to further some other business goal. It wasn't the last time.
Back when Google was good, they publicly said their goal was to get you off their site as fast as possible. Now the results reek of engagement algorithm bullshit.

JoMiran , (edited )
@JoMiran@lemmy.ml avatar

https://lemmy.ml/pictrs/image/23f9edc7-6ddc-4cf2-8224-92ad5483d37d.jpeg

AllNewTypeFace ,
@AllNewTypeFace@leminal.space avatar

Does Google still have a search algorithm? I thought they now just feed everything into a huge LLM and let it regurgitate statistically plausible answers.

Mereo ,
@Mereo@lemmy.ca avatar

Google Search uses a regular search algorithm. Google AI overview will be a product that feeds from Google search.

FaceDeer ,
@FaceDeer@fedia.io avatar

No, don't you see, everything we hate must be interrelated somehow.

Benardsmart ,

I saw an advert about a lady on YouTube news so I followed up to her IG page indeed she’s worth to be praised with just $1000 I can now boost of $17,099 Expert Eloise Wilbert on IG✅✅✅✅✅❎❎

glimse ,

Whoever paid you to spam this shit got scammed so hard lol

Did you really think that would work on the fediverse?

iamanurd ,

Now I’m super curious. All of their comments have been removed.

lvxferre ,
@lvxferre@mander.xyz avatar

Modlog to the rescue. It stinks scam from a distance. And spam. Scam spam.

thisbenzingring ,

Check the modlog

https://lemmy.sdf.org/modlog?page=1&userId=11610631 this is what I found on my instance for that username

Atropos ,

I dunno man, if it had one more checkmark I'd have been sold.

flappy ,

Can't wait for selfhosted web search to become better.

paraphrand ,

What are the current contenders?

nossaquesapao ,

the only one I know that isn't a proxy search is yacy

muntedcrocodile ,

I was looking at it the other day unfortunatly its got quite poor results

brbposting ,

That tracks

https://sh.itjust.works/pictrs/image/f976a748-ab26-466a-967f-886caaf4dfa1.jpeg

LittleBobbyTables ,

YaCy, Mwmbl, Alexandria, Stract, Marginalia to name a few.

brbposting ,

What it looks like beyond Google and Bing

It would be much harder to know what exists beyond "GBY" (Google, Bing, Yandex) and how it all works without the work of Rohan “Seirdy” Kumar. For three years, Kumar has been updating a heavily annotated list of search engines with their own indexes. It is 7,000 words, but only a portion of it deals with engines offering general indexing, in the English language. You can read Kumar's evaluation methodology for a better understanding of how he compared and assessed sites.

What stands out? Mojeek ("it's not bad… I'd live") and Stract ("a useful supplement to more major engines") are two of Kumar's favorites. Right Dao has "very fast, good results," in part because its crawler starts off from Wikipedia. Yep reaches farther out, showing results that link to and back from sites related to your query and also promises to share ad revenue with creators. All of them show promise, but you get the sense that they're a second car, or a third bicycle, rather than a primary transport.

There are far smaller-scoped engines in other sections of Kumar's post. If you're wondering where that one other search engine you've heard about is, it's probably in the "Semi-independent indexes" section, because it uses a GBY index when its own results are not strong enough. Here, you'll find cryptocurrency-friendly, controversy-courting-founder-having Brave, a few engines that either "resell" GBY results or stuff affiliate links into them, and "the most interesting entry," according to Kumar, Kagi.

Kagi requires an account and uses its own index, Teclis, in combination with Google, Bing, Yandex, Mojeek, and others, including, notably, Brave. Kagi's founder has strong opinions on the AI-based future of search and responding to harmful searches in ways that are not "scalable." How much of that does or does not bother you will vary, but it's worth noting that Kagi also suffers when the GBY triumvirate is restricted.

Ars Technica this week: Bing outage shows just how little competition Google search really has

The referenced search engine comparison by Rohan “Seirdy” Kumar

Mojeek ,
@Mojeek@lemmy.ml avatar

can't emphasise too much that this piece is a very necessary read for anyone who wants to know about search; not just because it says good things about us, but because of the depth of research which has been put in here. Most times you encounter an article about indexes they are just taking whatever a (meta)search engine says about themselves, not even looking at privacy policies for "relationships with microsoft" etc. or doing any comparative work.

Fish ,

I've been using Kagi and really like it so far. It's not good for local stuff, but afaik only Google and Bing have the resources and userbase for things like maps and reviews. It's designed to be an ad-free 'premium' search engine and only earns revenue from users paying for membership.

neblem ,

OpenStreetMap's platform is the only real way to compete against Google and Apple and it's why Microsoft even though it has Bing Maps, has licenced to them resources like satellite imagery for mapping. It's awesome in bigger population areas but there's still a lot to map in rural places outside the EU.

Review is harder. Right now the leading open platform afaik is Open Reviews (aka Mangrove Reviews) which has tie-ins to OSM projects like MapComplete. OsmAnd and OrganicMaps have open tickets to hook into that ecosystem. You're right about the userbase problem though, I think it (or a successor) needs AP federation to really take off. That being said there's several active non-Google nonfree alternatives like Yelp and TripAdvisor as well as niche sites for things like camping, parks, and schools.

Peppycito ,
@Peppycito@sh.itjust.works avatar

If they're taking tips from Google, why would they get better?

FaceDeer , (edited )
@FaceDeer@fedia.io avatar

Google actually was good, so there's probably some good information in this documentation. If nothing else we can perhaps figure out what "went wrong."

Edit: I've been reading the blog post that appears to be the main person the leak was shared with and there's a lot of in-depth analysis being done there, but I'm not seeing a link to the actual documents. This is a huge article, though, I might be overlooking it.

sbv ,

That was an interesting read. Thanks for linking to it.

jonne ,

You mean hosting your own crawler/indexer? That doesn't really sound like a thing you could do cost-effectively.

warmaster ,

Federated bookmarks?

Paradox ,
@Paradox@lemdro.id avatar

Federated directories. We're going back to Yahoo like it's 1995

NoIWontPickAName ,

Webrings!!!

AbidanYre ,

<under_construction.gif>

harrys_balzac ,

I loved Geocities!

neblem ,

Neocities is trying to be a modern reincarnation https://neocities.org/

oce ,
@oce@jlai.lu avatar

Yahoo patiently plotting its return from Japan.

Audacious ,

I'm so ready for something like this. I've cleaned up my bookmarks and been waiting for alternatives to search engines.

4am ,

SearxNG

Im_old ,

Look up the yacy repo in github

interdimensionalmeme ,

No problem we crowdsource the crawling torrent style.

We outsourced that to google for reasonnable performance reason. But they shit the bed so now there's no choice but to do it ourselves.

bamfic ,

ooh that might be an interesting app to run on veilid

brbposting ,

Right!

Before his company was able to block more of Microsoft's own tracking scripts, DuckDuckGo CEO and founder Gabriel Weinberg explained in a Reddit reply why firms like his weren't going the full DIY route:

“… [W]e source most of our traditional links and images privately from Bing … Really only two companies (Google and Microsoft) have a high-quality global web link index (because I believe it costs upwards of a billion dollars a year to do), and so literally every other global search engine needs to bootstrap with one or both of them to provide a mainstream search product. The same is true for maps btw -- only the biggest companies can similarly afford to put satellites up and send ground cars to take streetview pictures of every neighborhood.”

Ars

zutto ,
@zutto@lemmy.fedi.zutto.fi avatar

Surprisingly, it's very doable, requires basic technical knowledge and relatively minimal computing resources (runs in the background on your computer).

https://yacy.net/ Github

I have tampermonkey script that sends yacy to crawl any websites that I visit, and it's keeping up relatively good index for personal use of the visited websites. Combine yacy with ~300gb of Kiwix databases, add searxng as a frontend and you have pretty strong self hosted search engine.

Of course you need to supplement your searches from other search engines, as yacy does not crawl the whole web, just what you tell it to.

I encourage anyone who's even slightly interested on this stuff to try Yacy, it's ancient piece of software, but it still works very well and is not an abandoned project yet!

--

I personally use Yacy mostly on private mode, but it does have the distributed network there as well. Yacy current freeworld status

jonne ,

Yeah, I guess the P2P component sort of solves part of the issue I was imagining by distributing indexes and crawling. I was thinking that people were trying to run all of Google on a raspberry pi at home.

Cheradenine ,

You could use Common Crawl, it's run by a non profit

https://en.wikipedia.org/wiki/Common_Crawl

Jako301 ,

How is that even supposed to work? These search engines need per definition massive databanks to search through. Either you need your own crawler and indexer which is more than just inefficient, or you are limited to a relatively short list of curated static results.

Fedizen ,

rubs hands together

woelkchen ,
@woelkchen@lemmy.world avatar

Some information in the documents appears to be in conflict with public statements by Google representatives

I would have never guessed that.

CosmoNova ,

Crazy how self regulation always winds up like this. By crazy I mean predictable of course.

SuckMyWang ,

Libertarians assemble!

barsquid ,

Listen, the problem is too many regulations prevented the Invisible Hand from manifesting. If we remove even more regulations the free market will work this time, I swear.

deweydecibel ,

This doesn't have anything to with regulation. This is mainly a bunch of SEO and marketing people whining that Google hasn't been honest with them in telling them exactly how to game their search engine.

iopq ,

You're supposed to move to a different search engine for the market to work. I already have, have you?

Mnemnosyne ,

I did years ago when Google started censoring my search results even with safe search off.

Unfortunately Bing is doing it too now and I can't find a search engine that isn't, though I would love to learn about one that isn't.

NeatNit ,

This approach is doomed to fail, so long as the general public isn't aware of the problem or its scale. Government regulation is the only way.

cm0002 ,

Well I am just shocked, SHOCKED. Well, not that shocked.

Dsklnsadog , (edited )
@Dsklnsadog@lemmy.dbzer0.com avatar

Surprise MF

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • kbinchat
  • All magazines
    •