What are the differences between a shallow network and a deep network?
A shallow neural network has only one hidden layer between the input and output layers, while a deep neural network has multiple hidden layers.
One way to think about it is like a hierarchy of decision-making. Just like how a human brain makes decisions by processing information in layers, a deep neural network learns to make decisions by processing information through multiple hidden layers. On the other hand, a shallow network is like having just one layer of decision-making, which might not be enough to capture the complexity of the problem at hand. It's like trying to understand a complex book by just reading the summary on the back cover.
A shallow network might be used for simple tasks like image classification, while a deep network might be used for more complex tasks like image segmentation or natural language processing. For instance, a shallow neural network with a single hidden layer can be used to recognize handwritten digits from the MNIST dataset. But for a more complex task like identifying different objects in an image, a deep neural network like the popular ResNet architecture might be used.
The main advantage of a shallow network is that it is computationally less expensive to train, and can be sufficient for simple tasks. However, it may not be powerful enough to capture complex patterns in the data. A deep network, on the other hand, can capture more complex patterns in the data and potentially achieve higher accuracy, but it is more computationally expensive to train and may require more data to avoid overfitting. Additionally, deep networks can be more challenging to design and optimize than shallow networks.