Deep Learning Fundamentals with Keras
Chapter 1: Introduction to Deep Learning Concepts
What is Deep Learning?
Artificial Neurons and Network Structure
Supervised Learning with Neural Networks
Introduction to Keras
Setting Up Your Keras Environment
Hands-on Practical: Environment Verification
Chapter 2: Building Neural Networks with Keras
The Sequential API
Common Layer Types: Dense
Activation Functions
Specifying Input Shape
The Functional API
Model Summary and Visualization
Practice: Building Your First Network with Keras
Quiz for Chapter 2
Chapter 3: Training Deep Neural Networks
The Compilation Step
Understanding Loss Functions
Optimization Algorithms
Backpropagation
The Training Loop: fit() Method
Batches and Epochs
Validation Data and Monitoring Performance
Model Evaluation: evaluate() Method
Hands-on Practical: Training a Simple Classifier
Quiz for Chapter 3
Chapter 4: Convolutional Neural Networks (CNNs)
Introduction to Convolutional Networks
Convolutional Layers (Conv2D)
Pooling Layers (MaxPooling2D)
Building a Simple CNN Architecture
Flattening and Dense Layers in CNNs
Working with Image Data in Keras
Understanding Feature Maps
Practice: Implementing a CNN for Image Classification
Quiz for Chapter 4
Chapter 5: Recurrent Neural Networks (RNNs)
Introduction to Sequence Data
Recurrent Neural Network Concepts
SimpleRNN Layer in Keras
The Vanishing Gradient Problem
Long Short-Term Memory (LSTM) Networks
LSTM Layer in Keras
Gated Recurrent Units (GRUs)
Preparing Sequence Data for RNNs
Practice: Building an RNN/LSTM for Text Classification
Quiz for Chapter 5
Chapter 6: Improving Model Performance and Workflow
Understanding Overfitting and Underfitting
Regularization Techniques: L1, L2, Dropout
Data Augmentation
Using Callbacks
Saving and Loading Models
Introduction to TensorBoard
Hyperparameter Tuning Concepts
Hands-on Practical: Applying Improvement Techniques
Quiz for Chapter 6
The Vanishing Gradient Problem
Was this section helpful?
- Learning long-term dependencies with gradient descent is difficult, Yoshua Bengio, Patrice Simard, Paolo Frasconi, 1994 IEEE Transactions on Neural Networks, Vol. 5 (IEEE) DOI: 10.1109/72.279181 - A foundational paper that first explicitly identified and analyzed the vanishing and exploding gradient problems in recurrent neural networks.
- Long Short-Term Memory, Sepp Hochreiter, Jürgen Schmidhuber, 1997 Neural Computation, Vol. 9 (MIT Press) DOI: 10.1162/neco.1997.9.8.1735 - The original paper introducing Long Short-Term Memory (LSTM) networks, which effectively mitigate the vanishing gradient problem through their gating mechanisms.
- Deep Learning, Ian Goodfellow, Yoshua Bengio, Aaron Courville, 2016 (MIT Press) - A comprehensive textbook covering the theoretical foundations and practical aspects of deep learning, including detailed explanations of recurrent neural networks and the vanishing/exploding gradient problems.
- On the difficulty of training recurrent neural networks, Razvan Pascanu, Tomas Mikolov, Yoshua Bengio, 2013 Proceedings of the 30th International Conference on Machine Learning, Vol. 28 (PMLR) - This paper thoroughly investigates the vanishing and exploding gradient problems in RNNs and proposes practical solutions, such as gradient clipping, for stable training.