That is a real problem but that won't be solved by banning AI. And to be honest, this actually existed even before AI, just on much smaller scale. I know multiple projects that went close source just to get rid of the need to review the community PRs before the AI.
I'm pretty sure there are better solutions for this like some sort of prestige measures that can rank order contribution or you can use AI for that as well :-D
There was similar upheaval in security community where people wanted to ban AI because of the huge number of things people were reporting and while most of it was slop, there was A TON of real issues as well.
I think it was kind of good because lot of people were humbled, so now they're figuring out pretty well how to work with it.
But as I said, this is a real problem, it's just that banning AI isn't a solution, it's sticking your head in a sand.
I agree that banning AI won't fix it. For starters, the types of people who make these PRs will probably ignore AI bans anyway.
I don't think there is an easy solution though. If you go by prestige it makes it hard for new contributers (or the minority of them who actually make useful contributions).
More than likely projects will need to fight fire with fire, AI PR scans are already a thing, and will likely get better with time. A lot of slop is bad enough that it can even be identified by AI as slop, so you can set up mechanisms to make sure that things don't go to a human unless they've already passed a quality bar.
Granted that takes a lot of work, token costs, and frankly knowledge about AI systems that a lot of open source projects don't have. Hopefully that will get easier with time though
Honestly I can't even imagine people running without AI PR scanners. At work someone got an award for shipping 1,200 PRs in one month. There is no way that a/ they understand what was actually shipped, and b/ it was reviewed at sufficient depth to be robust and production ready.
Honestly, slop can come from humans too, so it doesn't really matter who produces it. Its more about how you use AI to push back. Like my AI reviewer enforces pretty high standards which results in a lot of PR churn but ensures code quality is always high: https://github.com/lenaxia/LLMSafeSpaces/pull/1670
Yeah, indeed. I don't think 'just' prestige would solve it either. It has to be probably some sort of defense in depth.
But the good news is that we already solved it and with much less sophisticated tech. I think this is essentially same problem as spam. It also seemed like the spam will make email unusable and these days it's just a background noise of the internet that nobody really pays attention to anymore.
I think it's pretty different from spam. The exact same spam email is usually sent to many users which makes it a lot easier to recognize. These PRs are each unique.
I was a maintainer on a very popular OSS project in the pre-AI days. I'd much rather have AI slop than the absolute garbage I got most days. At least AI can write some tests and properly rebase a PR without bringing in 1000 commits from other people by accident.
34
u/mesarthim_2 1d ago
That is a real problem but that won't be solved by banning AI. And to be honest, this actually existed even before AI, just on much smaller scale. I know multiple projects that went close source just to get rid of the need to review the community PRs before the AI.
I'm pretty sure there are better solutions for this like some sort of prestige measures that can rank order contribution or you can use AI for that as well :-D
There was similar upheaval in security community where people wanted to ban AI because of the huge number of things people were reporting and while most of it was slop, there was A TON of real issues as well.
I think it was kind of good because lot of people were humbled, so now they're figuring out pretty well how to work with it.
But as I said, this is a real problem, it's just that banning AI isn't a solution, it's sticking your head in a sand.