While the Keras Sequential and Functional APIs provide convenient ways to build a wide array of common model architectures, you might encounter situations where you need more flexibility. Perhaps you are implementing a novel layer described in a research paper, or you need a layer with complex internal state, or your model's forward pass logic doesn't fit neatly into the standard directed acyclic graph structure of the Functional API.

For these scenarios, TensorFlow allows you to create custom layers and custom models by subclassing the base classes provided by Keras. This object-oriented approach gives you complete control over the component's behavior.

Creating Custom Layers

A custom layer is essentially a Python class that inherits from tf.keras.layers.Layer. By subclassing Layer, your custom component integrates smoothly with the rest of the Keras ecosystem. You can use it within Sequential or Functional models just like any built-in layer (e.g., Dense, Conv2D).

To create a custom layer, you typically need to implement three methods:

__init__(self, **kwargs): This is the constructor. Use it to define layer-specific attributes that do not depend on the input shape. This includes hyperparameters like the number of units, activation functions, or regularization strengths. Call super().__init__(**kwargs) first. Any arguments needed for configuration should be accepted here.
build(self, input_shape): This method is where you define the layer's weights (trainable variables). It's called automatically the first time the layer is used, precisely because weight shapes often depend on the shape of the input data, which isn't known when the layer is instantiated. Use the self.add_weight() method inside build to create weights. The input_shape argument (a tf.TensorShape object) provides the necessary information. It's good practice to set self.built = True at the end of this method.
call(self, inputs, **kwargs): This method defines the layer's forward pass logic. It takes the input tensor(s) as arguments (along with optional keyword arguments like training=None for layers behaving differently during training and inference, such as Dropout) and returns the output tensor(s). All the computations involving inputs and weights happen here.

Here's a conceptual sketch of a custom layer structure:

import tensorflow as tf

class CustomLayer(tf.keras.layers.Layer):
    def __init__(self, config_param1, config_param2=None, **kwargs):
        super().__init__(**kwargs)
        # Store configuration parameters that don't depend on input shape
        self.config_param1 = config_param1
        self.config_param2 = config_param2
        # ... other initializations

    def build(self, input_shape):
        # Called once, the first time the layer is used.
        # Use input_shape to determine weight dimensions.
        # Example: Creating a trainable weight matrix 'w'
        self.w = self.add_weight(
            name='kernel', # A meaningful name
            shape=(input_shape[-1], self.config_param1), # Shape depends on input features and a config param
            initializer='random_normal', # How to initialize weights
            trainable=True # This weight should be updated during training
        )
        # Example: Creating a non-trainable bias 'b' (less common, just for illustration)
        self.b = self.add_weight(
            name='bias',
            shape=(self.config_param1,),
            initializer='zeros',
            trainable=False # This weight won't be updated
        )
        # Mark the layer as built
        self.built = True

    def call(self, inputs, training=None):
        # Define the forward pass logic using self.w, self.b, and inputs
        # Example: A simple linear transformation
        output = tf.matmul(inputs, self.w) + self.b
        # Apply activation if specified during __init__
        # if self.activation: output = self.activation(output)

        # Handle behavior differences during training vs inference if needed
        # if training:
        #     # Apply dropout, etc.
        # else:
        #     # Use inference behavior

        return output

    # Optional: Implement get_config for serialization
    def get_config(self):
        config = super().get_config()
        config.update({
            "config_param1": self.config_param1,
            "config_param2": self.config_param2,
        })
        return config

Creating Custom Models

Just as you can subclass tf.keras.layers.Layer to create custom layers, you can subclass tf.keras.Model to create custom models. This is useful when you need more control over the model's architecture or its training loop than Sequential or the Functional API provide.

The process is similar to creating a custom layer:

__init__(self, **kwargs): Define the layers your model will use as attributes of the class instance. These can be standard Keras layers or your own custom layers.
call(self, inputs, **kwargs): Define the forward pass of the entire model. You'll call the layers defined in __init__ here, specifying how data flows through them.

Subclassing tf.keras.Model provides maximum flexibility. You essentially define the model's forward pass as a Python function. While you can still use the standard model.compile() and model.fit() methods with custom models, subclassing Model also opens the door to writing fully custom training loops if needed (though that's a more advanced topic).

Here's a conceptual sketch:

class CustomModel(tf.keras.Model):
    def __init__(self, num_units_l1, num_units_l2, num_classes, **kwargs):
        super().__init__(**kwargs)
        # Define the layers the model will use
        self.layer1 = tf.keras.layers.Dense(num_units_l1, activation='relu')
        self.layer2 = CustomLayer(num_units_l2) # Using our custom layer
        self.classifier = tf.keras.layers.Dense(num_classes, activation='softmax')

    def call(self, inputs, training=None):
        # Define the forward pass using the layers
        x = self.layer1(inputs)
        x = self.layer2(x, training=training) # Pass training flag if layer needs it
        outputs = self.classifier(x)
        return outputs

# Instantiate the custom model
# model = CustomModel(num_units_l1=128, num_units_l2=64, num_classes=10)
# You can then compile and fit this model as usual

When to Use Custom Components

While you should generally prefer the straightforwardness of the Sequential and Functional APIs when possible, consider creating custom layers or models when:

You need to implement a layer with operations not available in standard Keras layers.
You are recreating an architecture from a research paper that uses non-standard components.
Your layer requires complex internal state management.
Your model architecture involves intricate connectivity, conditional logic within the forward pass, or recursive structures that are difficult to express with the Functional API.
You require fine-grained control over weight creation or the exact sequence of operations in the forward pass.

This introduction provides a glimpse into the flexibility offered by Keras subclassing. While we won't implement complex custom components in this introductory course, understanding that this capability exists is valuable as you encounter more advanced machine learning architectures. For most common tasks, the Sequential and Functional APIs will be sufficient and often easier to work with.