EfficientNet: Compound Scaling for Models

Previous sections discussed architectures like ResNet and DenseNet, which achieved significant performance gains primarily by increasing network depth and improving gradient flow. However, simply making networks deeper or wider doesn't always lead to the best performance or efficiency. The question arose: Is there a more principled way to scale up Convolutional Neural Networks (CNNs) to achieve better accuracy and efficiency? EfficientNet provides a compelling answer through a strategy called compound scaling.

The Challenge of Scaling CNNs

Traditionally, CNNs have been scaled up in one of three dimensions:

1. Depth: Increasing the number of layers (e.g., ResNet-18 to ResNet-200). Deeper networks can capture more complex features but suffer from vanishing gradients and require techniques like residual connections.
2. Width: Increasing the number of channels or filters in each layer (e.g., Wide ResNets). Wider networks can capture finer-grained features and are often easier to train, but the quadratic increase in computation cost with width can be prohibitive.
3. Resolution: Using higher-resolution input images. Higher resolution inputs provide more detail but increase computational cost and require networks with larger receptive fields (often achieved through more layers) to process effectively.

Scaling only one of these dimensions often leads to diminishing returns. For instance, making a network excessively deep without increasing width or input resolution might struggle to capture sufficient spatial detail. Similarly, widening a shallow network might not leverage the potential of complex feature hierarchies. The authors of EfficientNet observed that these dimensions are interdependent and should be balanced for optimal results.

Compound Scaling: Balancing Depth, Width, and Resolution

The central idea behind EfficientNet is compound scaling. Instead of arbitrarily scaling one dimension, EfficientNet proposes scaling network depth ($d$), width ($w$), and image resolution ($r$) simultaneously using a fixed set of scaling coefficients.

The scaling is controlled by a single compound coefficient, $\phi$. Given a baseline network (EfficientNet-B0), scaling up involves increasing $\phi$. The depth, width, and resolution are scaled according to these rules:

• Depth: $d = \alpha^\phi$
• Width: $w = \beta^\phi$
• Resolution: $r = \gamma^\phi$

Subject to the constraint:

$$\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$$

Here, $\alpha$, $\beta$, and $\gamma$ are constants determined through a grid search on the baseline model. They represent how much to scale each dimension. The constraint $\alpha \cdot \beta^2 \cdot \gamma^2 \approx 2$ ensures that for every increase in $\phi$, the total floating-point operations per second (FLOPS) required increases by approximately $2^\phi$.

The intuition is that if the input image resolution ($r$) is increased, the network needs more layers (depth $d$) to increase the receptive field and more channels (width $w$) to capture finer patterns in the larger input image. Compound scaling provides a systematic way to balance these factors.

The EfficientNet-B0 Baseline and MBConv

The effectiveness of compound scaling relies on having a strong, efficient baseline architecture. EfficientNet uses a baseline network (EfficientNet-B0) found through Neural Architecture Search (NAS). This search optimized for both accuracy and FLOPS.

The core building block of EfficientNet-B0 is the Mobile Inverted Bottleneck Convolution (MBConv), similar to the one used in MobileNetV2, but potentially enhanced with Squeeze-and-Excitation (SE) optimization. An MBConv block typically includes:

1. A $1 \times 1$ convolution to expand the number of channels (the "inverted" part, opposite to ResNet bottlenecks).
2. A depthwise separable convolution to perform spatial filtering efficiently.
3. A Squeeze-and-Excitation block (optional but used in EfficientNet) to perform channel-wise attention, recalibrating channel feature responses.
4. A $1 \times 1$ convolution to project the channels back down.
5. A skip connection (similar to ResNet) if the input and output dimensions match.

The EfficientNet Family

By starting with EfficientNet-B0 and applying the compound scaling rules with increasing values of the compound coefficient $\phi$ (typically integer values starting from 1), a family of models (EfficientNet-B1, B2, ..., B7, etc.) is generated. Each subsequent model uses approximately twice the FLOPS of the previous one but aims for higher accuracy while maintaining high parameter efficiency.

Comparison showing how EfficientNet models (blue) typically achieve higher accuracy for a given amount of computation compared to models scaled conventionally (gray).

Advantages and Considerations

EfficientNets demonstrated state-of-the-art performance on ImageNet and several other transfer learning benchmarks upon their release, often achieving comparable accuracy to much larger models with significantly fewer parameters and lower computational cost (FLOPS).

• Efficiency: Their primary advantage is superior accuracy per parameter and per FLOP.
• Scalability: The compound scaling method provides a clear recipe for generating larger, more capable models from a well-designed baseline.
• Transfer Learning: Due to their efficiency and strong performance, pre-trained EfficientNets are excellent choices for transfer learning tasks.

Some considerations include:

• Training Complexity: While efficient at inference, training very large EfficientNets can still be resource-intensive, potentially benefiting from techniques like mixed-precision training discussed in Chapter 2.
• NAS Dependency: The baseline architecture and scaling factors were discovered using NAS, which is computationally very expensive. However, practitioners typically use the pre-defined B0-B7 models and scaling factors.

EfficientNet represents a significant step in designing effective and scalable CNN architectures. By carefully balancing depth, width, and resolution through compound scaling, it provides a powerful framework for developing models that push the boundaries of accuracy and efficiency in computer vision. Pre-trained versions are widely available in frameworks like TensorFlow (via tf.keras.applications.EfficientNetB0 to B7) and PyTorch (often via third-party libraries like timm), making them readily usable for practical applications.