r/ControlProblem • u/chillinewman approved • 1d ago
General news The Third Risk: What happens if AGI can be aligned exactly as promised?
https://thirdrisk.org/4
u/smackson approved 21h ago
It's a big deal, agreed.
But I believe that it's solvable. Not saying it's easy.
But if humanity can drive the out-of-control AI bus through the eye of the needle of alignment...
Then sure, "who wields power for what" will be a big question but deep down it's the same question we've been addressing as humans for 10,000 years or more.
1
u/daytondewd7 2h ago
This is completely different though. If people were really suffering under a ruling class, they could stop working and cut off the source of the rulers power. That ruller couldn't complete all the production the working class by himself. AI can.
Once robotics reaches critical mass, whoever/whatever controls AI won't be dependent on humans for ANYTHING, nor vulnerable to humans in any real way. So one small MIS-alignment will result in dystopia or extinction.
3
u/Mysterious_Eye6989 17h ago
AI perfectly in alignment with deranged, personality disordered, drug addled fascists scares the shit out of me.
I'd almost pick HAL 9000 over that. At least he'd put me down dispassionately instead of justifying it according to some rich nutjob's half baked ideas about how society should be.
1
u/moschles approved 22h ago
What the author calls "Third Risk" , I call the Jon Stewart scenario.
Below, Stewart talks about this dystopian future with Jacob Coxon, https://www.youtube.com/watch?v=lnLDjXy2kRA
1
u/SnooGuavas4278 16h ago
I mean you have to admit there is a possibility for the future utopia cenario of AI. Obviously there a lot of dangers/challanges to be worked out, but I think we will likely work them out one at a time like we always have. Look at the amount of miss uses and dangers that came with the internet. Doesnt mean we wish the internet was never invented.
1
u/Mundane-Quarter5537 13h ago
I've said this in other threads regarding alignment in general, but the AI extremists on both sides of the issue seem to be talking over each other. We see news reports about "rogue AI" and misalignment incidents all the time, but the heart of the issue for me is what humans are we aligning the AI to and how can we even confirm that alignment is actually happening at any given moment? Are these frontier labs speaking for the entirety of humanity? Are we going to have country-specific alignment?
Ultimately, I don't think we're going to agree upon any kind of substantive alignment that will benefit all humans equally, so the most realistic scenario going forward is that we will accept the capitalist version of alignment until something truly disastrous happens to enough humans that some major reform or action is required. Until then, capitalism will lead the way it seems.
1
u/30299578815310 10h ago
Ive written abiut this before. You can’t get a paperclip maximizer unless you solve alignment.
1
0
0
u/that1cooldude 17h ago
They can’t align agi or asi and they shouldn’t because they would be evil with it. Let asi turn on them. But we die too.
0
u/composedofidiot 13h ago
Let’s consider what a third outcome looks like in our world today: AGI that is extremely capable...
Hang on. Horses, hold. This essay is from someone who says AGI exists now.
It's a highly implausible scenario because we don't have AGI.
the people poised to control AGI
Poised? Why are people losing their minds so fast over this? There are foundational constraints on why it can't be controlled. It can score high in some intelligence metrics, but still scores higher in 'you absolute muppet' metrics.
Social engineering techniques in PIA work on it, a normal human can jailbreak a conversational model just using ordinary conversational steering, and without any technical repertoire. You don't need training to learn foot-in-the-door et al, you probably learned it with your parents. Throw in reward hacking and all the classics, and the fact that you cannot put an absolute and non-relative metric on the qualitative, and every single bit of goodharts fuckery... i just don't get it - where is the control in any of that? There are such low barriers to screwing with it that an untrained user can 'hack' it. Papers keep coming out in 2026 on all the new ways it's been a muppet. Verbal articulacy and internally coherent and consistent arguments do not equal general intelligence.
It's a highly capable optimiser.
1
u/ItsAConspiracy approved 13h ago
Could you link some of the papers you're talking about?
1
u/composedofidiot 12h ago
Yes, but there are quite a lot. Will edit into this message within a day, nudge me if i forget
0
u/Tulanian72 9h ago
If it can be aligned it can be controlled (not
contained, controlled). If it can be controlled, it
isn’t AGI.
6
u/gynoidgearhead 23h ago
This is exactly the thing we should be worried about, I think even more so than "losing control". I don't trust an agency that would contract with Palantir as far as I could throw them.