Technology being crap today doesn't mean it'll be crap forever
Person A: Future AI systems could be really dangerous
Person B: No they won't; AI can't count letters and gets fooled by basic trick questions
This confuses how good something is today with how good it could get. Most actively developed technology will get better, and overcome non-fundamental problems. This is probably especially true of something like AI, where huge amounts of capital and other resources are being poured into improving it.
I think people know this, but in the moment it seems like System 1 thinking assumes technology won't advance. This might be related to normalcy bias, or the end-of-history illusion.
If forced to make their reasoning explicit, people would realise this wouldn't quite make sense. And all of the explicitly stated versions are much weaker, e.g. today's AI can't count letters and therefore...:
- this indicates that there is a long way to improve before they become dangerous (but really this would need to be combined with a good grasp of the rate of change as well); or
- there's some fundamental limitation X that would prevent AI systems ever being dangerous (but this needs precision in what that limitation is, which actually feels like the core point)
Related light relief
(From r/singularity; underlying chart Our World in Data)