Neural networks are based on an oversimplified model of neuron cells.
As a programmer who has studied neuroanatomy and the structure/function of neurons themselves, I remain astonished at how not like real biological nervous systems computer neural networks still are. It's like the whole field is based on one person's poor understanding of the state of biological knowledge in the late 1970s. That doesn't mean it's not effective in some ways as it is, but you'd think there'd be more experimentation in neural networks based on current biological knowledge.