When we scale a neural network to billions of parameters, new capabilities emerge. But we still lack a complete explanation of why certain capabilities emerge or how they arise. Well, this is not unique to neural networks!
I recently came across a beautiful essay, “More Is Different,” by Philip W. Anderson. Anderson argued that as systems become larger and more complex, new principles emerge. The fundamental laws remain valid, but they are often insufficient to explain behaviour at higher levels of organisation.
For example,
- We can often explain the behaviour of an individual, but understanding families, societies, nations, and humanity requires new concepts and models.
- We understand the laws governing individual atoms, yet explaining molecules, cells, living organisms, and even the universe requires additional levels of description.
- We understand how individual neurons function. However, when billions of artificial neurons are connected into a large neural network, new capabilities emerge. These capabilities are difficult to predict from the behaviour of a single neuron or a much smaller network.
We often use reductionism (breaking systems down into their smallest components). It has transformed physics, chemistry, biology,… but constructionism asks a different question. How do simple components, when combined, give rise to entirely new behaviour?
That question remains open across many disciplines, from physics and biology to neuroscience and artificial intelligence. Perhaps AI is reminding us of a lesson nature has been teaching all along.
P. W. Anderson, “More Is Different,” Science, Vol. 177, No. 4047 (1972), pp. 393–396.