Remote profile

There's been a lot of criticism thrown out lately about the gradual ceding of ground to AI programming agents within open source projects -- notably within crucial digital infrastructure such as the Linux kernel, and various system utilities or tools.

I'm curious. Has anybody with the expertise to do so gone through various commits to open source projects, in order to analyze the good/bad within these codebases since introducing AI agents to the process?

I feel like a resource that actually dissects *real code* -- not just theoretical concerns about tech debt -- would be useful in the overall discussion around this stuff.

I know it's somewhat dubious to even properly try to figure out with 100% certainty which commits are produced by AI agents vs which were handwritten unless the agent specifically is included in the git commit. But, I still do wonder if anybody has done anything like this.

Marginalia Search seems to be doing this to some extent¹, but it's of course not quite perfect. It's incredible what it's able to do, but at the same time, I still often have to dig around quite a bit longer before finding anything particularly useful.

I wonder if a Duckduckgo-style approach to this kind of "indie search index" might prove useful? Somebody could prop up a service that compiles together the results of Marginalia alongside the bigger results from the Google, Bing, or Yandex indexers.

I'm really just spitballing here. I wonder if anyone has been doing any thinking about this topic, particularly the way forward for indie search engines, and whether or not it would be possible for them to be an actually competitive alternative to our existing options.

Feel free to link any blogs on the topic, or share your own ideas in the comments. I think this is an idea I'd like to explore some more...

---

¹ note: I don't actually know what the set of websites/pages Marginalia is operating on. My (W - M) example was of course overly reductive, so I would doubt it's doing all of that. My own experience is that it has some list of whitelisted domains that it crawls, and that's it. I don't know how in particular it adds new domains, but I know you can request to add your own site through a form -- which is a thing that I did in the last year or so with my own site.

It's been discussed at length the extent to which Big Tech search engines/indexers are basically garbage at this point. Even using a wrapper like Duckduckgo, which re-arranges the results from a Google + Bing search, the usefulness of these tools is really in decline compared to how well they *used to work* (that is to say, we know for a fact that a better quality is possible on a technical level!)

It's been of course noted that the feasibility of running your own web indexer is simply too high of a bar for typical folks to clear. The web is literally just too big. There's too many links, too many pages, and all too much text to store in a cache. Not even to mention the work required to keep it up to date by doing regular crawling.

So, what about a search engine that only aims at a portion of the web? When we remove all of the big players like social media (Reddit, Twitter, etc), large forums (StackOverflow, etc), major wikis (Wikipedia, fandom wiki, etc), and large news outlets (The New York Times, AP News, etc) -- what's left?

Say we let W be the set of all pages on the web. We know that len(W) is too big to store on something that isn't a giant datacenter. What about storing (W - M), the set of all pages *minus* all the pages from major web domains (as discussed above)? How big is len(W - M)? Is it reasonable, or would you have to truncate even further, taking only a subset of (W - M)?

🧵️ (1/2)

#webSearch #search #web #indieWeb #smallWeb #marginalia #google #bing #yandex #openSource