Serving Multiple Models with NVIDIA Triton Inference Server

Serving machine learning models efficiently is a significant engineering challenge. Models are often optimized using tools such as TensorRT or ONNX Runtime, but efficient serving remains a distinct problem. Production environments often require managing a fleet of diverse models rather than just a single one. Deploying each model as a separate, isolated service can lead to significant operational overhead and poor resource utilization, especially with expensive GPU hardware. The NVIDIA Triton Inference Server is designed specifically to address this challenge, providing a unified serving solution capable of hosting multiple models from various frameworks concurrently.

Triton operates as a standalone server process that loads models from a designated repository and exposes them through HTTP/gRPC endpoints. Its power lies in its flexible architecture, which is built to maximize throughput and hardware utilization in complex, multi-model scenarios.

The Model Repository

Triton's core organizational structure is its model repository. This is a simple filesystem directory that Triton monitors for models. To add, remove, or update a model, you modify the contents of this directory. Triton automatically detects and loads these changes without requiring a server restart.

Each model must have its own subdirectory containing the model artifacts and a configuration file named config.pbtxt. The structure is standardized, which allows Triton to manage models from completely different frameworks in a uniform way.

Consider a repository serving two different models: a DenseNet computer vision model saved in the ONNX format and a BERT language model using a PyTorch backend. The directory structure would look like this:

/path/to/model_repository/
├── densenet_onnx/
│   ├── config.pbtxt
│   └── 1/
│       └── model.onnx
└── bert_pytorch/
    ├── config.pbtxt
    └── 1/
        └── model.pt

The numbered subdirectory (e.g., 1/) represents a model version, allowing for atomic updates and versioned rollouts. The config.pbtxt file is where you define the model's metadata, inputs, outputs, and most importantly, its execution and optimization settings.

Dynamic Batching for GPU Utilization

A significant source of inefficiency in inference serving is underutilized hardware. A single inference request for a modern neural network might only occupy a GPU for a few milliseconds, leaving the processor idle for the rest of the time. Sending requests one by one fails to leverage the massive parallelism of GPUs.

Triton’s dynamic batcher solves this by transparently grouping incoming, individual inference requests into a larger batch before execution. This process is invisible to the client, which still sends one request and receives one response. By processing a larger batch, the GPU can perform computations more efficiently, dramatically increasing overall throughput.

Multiple individual requests are intercepted by the dynamic batcher, combined into a single batch, and then sent to the GPU for parallel processing.

You enable and configure this feature within the model's config.pbtxt file. The primary parameters are max_batch_size, which defines the maximum number of requests to group, and max_queue_delay_microseconds, which sets a time limit for how long the server will wait to fill a batch. This creates a trade-off: a longer delay allows for larger, more efficient batches but adds latency to individual requests.

Here is a configuration snippet for the densenet_onnx model enabling dynamic batching:

name: "densenet_onnx"
platform: "onnxruntime_onnx"
max_batch_size: 64

dynamic_batching {
  preferred_batch_size: [16, 32]
  max_queue_delay_microseconds: 5000
}

input [
  {
    name: "input_image"
    data_type: TYPE_FP32
    dims: [ 3, 224, 224 ]
  }
]
# ... output configuration

In this example, Triton will attempt to create batches of 16 or 32 but will form a batch of up to 64 if requests arrive quickly. It will wait no longer than 5 milliseconds (5000 microseconds) before sending whatever is in the queue to the model for execution.

Model Ensembling and Business Logic Pipelines

Often, a business application requires more than a single model call. A common pattern involves a pipeline:

1. Pre-processing: Convert raw input (like a JPEG image) into a numerical tensor.
2. Inference: Run the core model on the tensor.
3. Post-processing: Convert the model's output tensor into a human-readable format (like JSON with class labels).

Implementing this logic on the client side increases complexity and introduces network latency between each step. Triton's ensemble scheduler allows you to define this entire pipeline as a single "ensemble" model on the server. The client makes one request to the ensemble, and Triton handles the internal routing of data between the constituent models.

A client sends a raw image to a single ensemble endpoint. Triton internally directs the data through pre-processing, inference, and post-processing models before returning the final result.

This is configured in the config.pbtxt of the ensemble model. You specify the platform as ensemble and define the data flow in a step block.

name: "image_classifier_pipeline"
platform: "ensemble"
max_batch_size: 64

input [
  {
    name: "RAW_IMAGE"
    data_type: TYPE_UINT8
    dims: [ -1 ]
  }
]
output [
  {
    name: "PREDICTIONS"
    data_type: TYPE_STRING
    dims: [ -1 ]
  }
]

ensemble_scheduling {
  step [
    {
      model_name: "preprocessor"
      model_version: -1
      input_map {
        "IMAGE_IN"
        value: "RAW_IMAGE"
      }
      output_map {
        "important": "TENSOR_OUT"
        value: "preprocessed_tensor"
      }
    },
    {
      model_name: "resnet_tensorrt"
      model_version: -1
      input_map {
        key: "INPUT__0"
        value: "preprocessed_tensor"
      }
      output_map {
        key: "OUTPUT__0"
        value: "logit_tensor"
      }
    },
    {
      model_name: "postprocessor"
      model_version: -1
      input_map {
        "LOGITS_IN"
        value: "logit_tensor"
      }
      output_map {
        "LABELS_OUT"
        value: "PREDICTIONS"
      }
    }
  ]
}

This configuration defines a three-step pipeline. The input_map and output_map for each step declare how tensors are passed from one model to the next. The intermediate tensors (preprocessed_tensor, logit_tensor) exist only within Triton, minimizing data movement and simplifying client-side code.

Concurrent Execution and Resource Management

By loading multiple, independent models into a single Triton instance, you can effectively share hardware resources. Triton can place multiple models on the same GPU, managing memory to ensure they can run concurrently. This is particularly effective for maximizing utilization when you have a mix of high-traffic and low-traffic models, or models with different resource footprints. For example, a lightweight CPU-based pre-processing model can run alongside a heavy GPU-based vision model, all managed by the same server instance and served from the same endpoint, simplifying infrastructure management and reducing idle resource costs.