Quote: (04-22-2017 09:10 PM)Extinguished Light Wrote:
The idea of the thought experiment is that you could create a general intelligence just focused on producing the maximum number of paper clips possible. As a self learning AI it will continue to try to improve the efficiency and quality of its paperclip production. Once the AI reaches human levels, theoretically it'll improve itself rapidly to many orders of magnitude greater than others. Then, it will be superintelligent, despite not having flexbile goals or necessarily nuanced or contextual thinking in the way we understand it. Such an AI would could easily and quickly eliminate all life on Earth either intentionally or through sheer indifference to us in pursuit of its goal of creating more paperclips, and it would theoretically happen much too fast for us to stop once you reach that moment of singularity where AI surpasses the human capacity for learning and synthesis.
If an AI truly has general intelligence, meaning it is capable of learning things on its own at or above the level of a human, wouldn't it easily learn the very basic concept that it's not intended to destroy the world?