Marginalia Search seems to be doing this to some extent¹, but it's of course not quite perfect. It's incredible what it's able to do, but at the same time, I still often have to dig around quite a bit longer before finding anything particularly useful.
I wonder if a Duckduckgo-style approach to this kind of "indie search index" might prove useful? Somebody could prop up a service that compiles together the results of Marginalia alongside the bigger results from the Google, Bing, or Yandex indexers.
I'm really just spitballing here. I wonder if anyone has been doing any thinking about this topic, particularly the way forward for indie search engines, and whether or not it would be possible for them to be an actually competitive alternative to our existing options.
Feel free to link any blogs on the topic, or share your own ideas in the comments. I think this is an idea I'd like to explore some more...
---
¹ note: I don't actually know what the set of websites/pages Marginalia is operating on. My (W - M) example was of course overly reductive, so I would doubt it's doing all of that. My own experience is that it has some list of whitelisted domains that it crawls, and that's it. I don't know how in particular it adds new domains, but I know you can request to add your own site through a form -- which is a thing that I did in the last year or so with my own site.