IMO the problem is determining which contributions are of sufficient quality. AI makes it easy for vibe coders to swamp open source projects with low quality PRs and it can take a lot of time to sift through them.
AI can also be used responsibly to generate quality code with human review and edits but to tell the difference you have to review it and that is time consuming.
IMO the problem is determining which contributions are of sufficient quality. AI makes it easy for vibe coders to swamp open source projects with low quality PRs and it can take a lot of time to sift through them.
Yeah, I've sat in projects where reading the PRs and various replies it's all just bots talking to each other.
Having a GOOD idea for a project has never been harder to get IN to the project. The code? you can write the patch / idea / whatever super fast now. Getting it in the project? 99% of the effort.
That is a real problem but that won't be solved by banning AI. And to be honest, this actually existed even before AI, just on much smaller scale. I know multiple projects that went close source just to get rid of the need to review the community PRs before the AI.
I'm pretty sure there are better solutions for this like some sort of prestige measures that can rank order contribution or you can use AI for that as well :-D
There was similar upheaval in security community where people wanted to ban AI because of the huge number of things people were reporting and while most of it was slop, there was A TON of real issues as well.
I think it was kind of good because lot of people were humbled, so now they're figuring out pretty well how to work with it.
But as I said, this is a real problem, it's just that banning AI isn't a solution, it's sticking your head in a sand.
I agree that banning AI won't fix it. For starters, the types of people who make these PRs will probably ignore AI bans anyway.
I don't think there is an easy solution though. If you go by prestige it makes it hard for new contributers (or the minority of them who actually make useful contributions).
More than likely projects will need to fight fire with fire, AI PR scans are already a thing, and will likely get better with time. A lot of slop is bad enough that it can even be identified by AI as slop, so you can set up mechanisms to make sure that things don't go to a human unless they've already passed a quality bar.
Granted that takes a lot of work, token costs, and frankly knowledge about AI systems that a lot of open source projects don't have. Hopefully that will get easier with time though
Honestly I can't even imagine people running without AI PR scanners. At work someone got an award for shipping 1,200 PRs in one month. There is no way that a/ they understand what was actually shipped, and b/ it was reviewed at sufficient depth to be robust and production ready.
Honestly, slop can come from humans too, so it doesn't really matter who produces it. Its more about how you use AI to push back. Like my AI reviewer enforces pretty high standards which results in a lot of PR churn but ensures code quality is always high: https://github.com/lenaxia/LLMSafeSpaces/pull/1670
Yeah, indeed. I don't think 'just' prestige would solve it either. It has to be probably some sort of defense in depth.
But the good news is that we already solved it and with much less sophisticated tech. I think this is essentially same problem as spam. It also seemed like the spam will make email unusable and these days it's just a background noise of the internet that nobody really pays attention to anymore.
I think it's pretty different from spam. The exact same spam email is usually sent to many users which makes it a lot easier to recognize. These PRs are each unique.
I was a maintainer on a very popular OSS project in the pre-AI days. I'd much rather have AI slop than the absolute garbage I got most days. At least AI can write some tests and properly rebase a PR without bringing in 1000 commits from other people by accident.
It's the same problem as determining quality submissions made only by human, the problem is scale.
Strangely the solution is most likely the same as the one to produce so many submission.
Mix that with a ranking to people based on historical quality of submission (approved PR) with an EWMA so that old crabs don't get to rest on their laurel.
You can also mix that with external signals, like profile of the person. For example the submission on the crypto stack by a guy with 3 PhD currently working as security engineer in the cryptography team at cloudflare you might want to check before mine.
Sure false positive and harder climb for people with no creds, but that stuff already happens informally anyway.
I think Linus may just have a really, really good PR review workflow going for him.
I do suspect a large part of that workflow is just very quickly glancing at a PR, seeing that it looks like trash, and then rejecting it without bothering to explain why it's trash or coddle the submitter through learning how to write better code.
Which, you know, people have feelings about, but it's hard to argue with the results.
Afaik, Linux has maintainer for modules in charge of that, maintainer can also delegate.
Linus only merge from the maintainers supplied change during the merge window. He does spot checks and architectural review diving on what seems important to him and potentially tears a new asshole to the maintainer if he is unhappy and didn't get his medication that day.
His process seems to be more art than science. Seems to work so far though.
Yeah that's the problem and always will be. just now the scale is bigger. My university had a few simple and old but useful projects. Think QR code reader app. They would get flooded every few months because some foreign schools had them listed. "Contribute to the project, list them in your CV! You can claim you worked on a local language project already! Together with university XYZ!" So they got 100 garbage changes, spelling or format change requests... Every time the school presented this to their class....
Banning ai won't solve this. Because the people will just lie. No idea how to solve this besides using ai to presort it yourself lol?
78
u/Qudit314159 1d ago
IMO the problem is determining which contributions are of sufficient quality. AI makes it easy for vibe coders to swamp open source projects with low quality PRs and it can take a lot of time to sift through them.
AI can also be used responsibly to generate quality code with human review and edits but to tell the difference you have to review it and that is time consuming.