I bought and read "The Emperor's New Mind" in the early 90s, mostly for "knowing your enemy." I have slowly but surely come to mostly agree with Penrose: I think there is something magical and "quantum mechanical" about brain consciousness (including animals).
I believe in eventual real AI, but I would guess that it will not be on current computer hardware.
I only have a BS in Physics (UCSB) so this may be rough: about 10 years ago I went to a Quantum Mechanics and Consciousness conference and the gist is that there is a quantum effect that is required consciousness. My dad taught physics at Berkeley and his friend from Berkeley Henry Stapp presented an interesting paper on this theory. The philosopher David Chalmers was also there and he talked about the difficult question of consciousness: why did evolution favor the development of consciousness and qualia (an inner mental life)?
There are several obvious explanations like that we evolved conciousness in order to be able to explain ourselves to other humans. But I thought Chalmers believe that the physical world was "causally closed" and that therefore a purely material feedback process like evolution couldn't select for qualia?
I believe in eventual real AI, but I would guess that it will not be on current computer hardware.