It's been discussed at length the extent to which Big Tech search engines/indexers are basically garbage at this point. Even using a wrapper like Duckduckgo, which re-arranges the results from a Google + Bing search, the usefulness of these tools is really in decline compared to how well they *used to work* (that is to say, we know for a fact that a better quality is possible on a technical level!)
It's been of course noted that the feasibility of running your own web indexer is simply too high of a bar for typical folks to clear. The web is literally just too big. There's too many links, too many pages, and all too much text to store in a cache. Not even to mention the work required to keep it up to date by doing regular crawling.
So, what about a search engine that only aims at a portion of the web? When we remove all of the big players like social media (Reddit, Twitter, etc), large forums (StackOverflow, etc), major wikis (Wikipedia, fandom wiki, etc), and large news outlets (The New York Times, AP News, etc) -- what's left?
Say we let W be the set of all pages on the web. We know that len(W) is too big to store on something that isn't a giant datacenter. What about storing (W - M), the set of all pages *minus* all the pages from major web domains (as discussed above)? How big is len(W - M)? Is it reasonable, or would you have to truncate even further, taking only a subset of (W - M)?
🧵️ (1/2)
#webSearch #search #web #indieWeb #smallWeb #marginalia #google #bing #yandex #openSource