Even worse: if simulations are used, you now have two problems - formulating correct incentives and protecting against abusing flaws in the simulation.

Isn’t this true about all systems, not just “AI”? The definition of a software bug is an unintended behavior. In a large system, myriad intents overlap and combine in unexpected ways. You might imagine a complex enough system where the confidence that a modification doesn’t introduce an unintended behavior is near zero.

I think it’s true for many systems, not just AI that’s true.

AI is worth calling out in this regard because, if the field is successful enough, it can create dangerous systems that don’t behave how we want.

Building a safe general AI is much harder than building a general AI, which is why it’s worth considering AI as it’s own problem domain.