If you ever wanted to know too much about where the majority of our search results come from and the many niche alternatives trying something different.....
I had a similar idea: Could search engines be broken up and distributed instead of being just a couple of monoliths?
Reading the HN thread, the short answer is: NO.
Still, its fun to imagine what it might look like if only......
I think the OP is looking for an answer to the problem of Google having a monopoly that gives them the power to make it impossible to be challenged. The cost to replicate their search service is just so astronomical that its basically impossible to replace them.
Would the OP be satisfied if we could make cheaper components that all fit together to make a competing but decentralized search service?
Breaking down the technical problems is just the first step, the basic concepts for me are:
Crawling -> Indexing -> Storing/host index -> Ranking
All of them are expensive because the internet is massive!
If each of these were isolated but still interoperable then we get some interesting possibilities: Basically you could have many smaller specialized companies that can focus on better ranking algorithms for example.
What if crawling was done by the owners of each website and then submitted to an index database of their choice? This flips the model around so things like robots.txt might become less relevant. Bad actors and spam however now don't need any SEO tricks to flood a database or mislead as to their actual content, they can just submit whatever they like!. These concerns feed into the next step:
What if there were standard indexing functions similar to how you have many standard hash functions. How a site is indexed plays an important role in how ranking will work (or not) later. You could have a handful of popular general purpose index algorithms that most sites would produce and then submit (e.g. keywords, images, podcasts, etc.) combined with many more domain specific indexing algorithms (e.g. product listings, travel data, mapping, research). Also if the functions were open standards then it would be possible for a browser to run the index function on the current page and compare the result to the submitted index listing. It could warn users that the page they are viewing is probably either spam or misconfigured in some way to make the index not match what was submitted.
What if the stored indexes were hosted in a distributed way similar to DNS? Sharing the database would lower individual costs. Companies with bigger budgets could replicate the database to provide their users with a faster service. Companies with fewer resources would be able to use the publicly available indexes yet still be competitive.
Enabling more competition between different ranking methods will hopefully reduce the effectiveness of SEO gaming (or maybe make it worse as the same content is repackaged for each and every index/rank combination). Ranking could happen locally (although this would probably not be efficient at all but that fact that it might even be possible at all is quite a novel thought)
Inspired by a post since deleted, I feel bad for probably coming off judgemental about the poster's taste in the movie that drove him to consider sailing....
Was just trying to watch the original Star Wars from when I was young and found out that it is simply not available for sale. My money is no good! Then I found this Project 4K77.
California socialite Rebecca Grossman sentenced to 15 to life for killing 2 kids in crosswalk ( www.nbcnews.com )
cross-posted from: https://lemmy.world/post/16402340...
A look at search engines with their own indexes ( seirdy.one )
If you ever wanted to know too much about where the majority of our search results come from and the many niche alternatives trying something different.....
The Mac vs. PC war is back on? ( www.theverge.com )
I find it hard to believe that, outside of work computers, many people would be choosing Windows over Mac or Linux, especially is AI is their goal....
Ask HN: Can we create a new internet where search engines are irrelevant? ( news.ycombinator.com )
What drew you to the high seas?
Inspired by a post since deleted, I feel bad for probably coming off judgemental about the poster's taste in the movie that drove him to consider sailing....