The claim that fear of punishment or repercussions affects people’s actions shouldn’t be a controversial thing to say.
I didn't say it was controversial - I said it's pretty useless as a tool to predict a given society's behavior with. Plenty of tyrants have discovered that the hard way.
demonstrated its orders-of-magnitude greater intelligence to you
The ability to ace IQ tests will never impress me... and it's unlikely to make up for the fact that it needs a box.
simulate your mind, and create a hell for you where you’ll be tortured for literal eternity
That argument is no different than the ones co-opted religion has been making for thousands of years - and it still hasn't managed to tame us much.
Of course, you’re always free to call its bluff,
Calling power's bluff is something we do as a matter of course - the history books are filled with it. This doesn't make power less dangerous - but there is no such thing as "unknowable" power.