Superheroification ₄

I guess I should make clear my stance on superintelligence: We’re less likely to create independent volitional entities than we are to create superpowered human-AI centaurs. This does not solve the alignment problem, as we have misaligned human coordination mechanisms already.

On the whole, though, this is probably good. We have surpassed the limits of our previous coordination mechanisms and the limits of human cognition. If we’re going to make it through the bottleneck of the century, we’re going to need new powers.

This is why comic book heroes are our modern mythology, by the way. We’re undergoing the process of superheroification and we are grappling with the implications of that, moral and strategic both.

Novacene doesn’t directly take on the question of superintelligent agents versus centaurs, but it does make the case for the bottleneck: If we don’t avert the climate collapse, if life ends on this planet, the sun is too hot now for it to begin again.

Novacene - it’s a good book, short and to the point. The point is kind of weird though.

?t=tHVJa2y6Pj7dpkx7YeOucw

This essay seems to be about centaurs maintaining a competitive edge against pure AI in finite games with static rule sets. But that’s not what human coordination problems actually look like. We will play new games.

Yeah, there’s not going to be a hard line as much as an accelerating curve. I’m sending these messages to thousands of people worldwide instantly, by mumbling into my pocket computer.

To clarify: I don’t think that AI safety and alignment work is unimportant. But I think it will accelerate along with AI progress, and tend to lead to centaurs, as we develop better interpretability measures and AIs reinforce on our desires.

View original