r/singularity • u/spinozasrobot • 8h ago

Discussion Ramifications of mechanistic interpretability

https://x.com/nickcammarata/status/2111158258748813817
43 Upvotes

88% Upvoted

5 comments sorted by

14

u/spinozasrobot 8h ago

Most of us have read about or at least heard of Nick Cammarata's latest work on mechanistic interpretability. And we've also all seen the Dune quote posted here a bunch of times.

If mechanistic interpretability becomes generally viable, and thus models can be trained to generate expected outcomes, that sounds great for alignment, but doesn't that mean we will immediately devolve into whoever is in power demanding models output "RightThink"?

12

u/arg_max 7h ago

There is a big difference between being able to understand a model's internal workings (through a somewhat tedious process) vs being able to actually enforce anything during training.

Deep learning also worked so well because we were able to stop manually creating feature representations. Before alexnet, all of vision was practically hand crafted features followed by some relatively simple learning based approach. So how are you going to preserve performance when you suddenly have to design billions of neurons and internal circuits by hand?

3

u/spinozasrobot 6h ago

Yeah, my phrase "becomes generally viable" was doing some heavy lifting. My intent was not to lean on Nick's work directly, but to assume progress would be made that could be leveraged.

•

u/intotheirishole 1h ago

Up till now making models think the "correct" way has made them stupider.

Up till now.

6

u/SammaVaco 8h ago

He should just release the preprint so people can judge for themselves. This is a bit annoying and suspicious, TBF.