Theory provides the foundation, but translating project requirements into a tangible list of components is where infrastructure engineering becomes a practical discipline. Creating a hardware specification sheet is the primary document that bridges the gap between an AI workload's needs and a functional on-premise server. This process involves applying established principles to make informed decisions and justify choices based on performance goals and constraints.
Scenario: Fine-Tuning an Internal Q&A Model
Imagine you are an engineer at a company that wants to build an internal knowledge-base assistant. The goal is to fine-tune a powerful open-source large language model on your company's private documentation.
- Model: Llama 3 70B. This is a large model that requires significant VRAM for both training and inference.
- Task: Supervised fine-tuning on a proprietary dataset.
- Dataset: 500 GB of curated text and code documents.
- Performance Target: Complete a full fine-tuning run in under 48 hours to allow for frequent experimentation.
- Team: A team of three machine learning engineers will share this server.
- Budget: The budget is substantial but requires clear justification for high-cost components. The focus is on long-term value and performance, not finding the absolute cheapest parts.
The Task: Build the Specification Sheet
Your task is to fill out a hardware specification sheet for a server designed to handle the scenario above. The sheet should not only list the components but also provide a clear justification for each choice, connecting it back to the project's requirements.
Below is a template for your specification sheet. We will walk through filling it out for the LLM scenario, explaining the reasoning behind each selection.
| Category |
Component |
Selected Specification |
Quantity |
Justification |
| Compute |
Graphics Processing Unit (GPU) |
Your Choice & Rationale |
# |
Explain why this GPU model, VRAM, and quantity are appropriate. |
|
Central Processing Unit (CPU) |
Your Choice & Rationale |
1 |
Explain why this CPU is a good match for the GPUs and workload. |
| Platform |
Motherboard |
Your Choice & Rationale |
1 |
Justify based on GPU support, PCIe lanes, and CPU compatibility. |
|
Server Chassis |
Your Choice & Rationale |
1 |
Justify based on form factor, cooling, and component compatibility. |
| Memory |
System RAM |
Your Choice & Rationale |
# |
Calculate the required RAM for data preprocessing and system overhead. |
| Storage |
Primary Storage (Hot Data) |
Your Choice & Rationale |
# |
Explain the choice of technology (e.g., NVMe) and capacity for the active dataset and OS. |
|
Secondary Storage (Cold Data) |
Your Choice & Rationale |
# |
Explain the choice for storing the full dataset archive and model checkpoints. |
| Network |
Network Interface Card (NIC) |
Your Choice & Rationale |
1 |
Justify the required network speed. |
| Power |
Power Supply Unit (PSU) |
Your Choice & Rationale |
1 |
Calculate the total power draw and select a PSU with an appropriate wattage and efficiency rating, allowing for overhead. |
Worked Example: Completing the Specification Sheet
Let's populate the template based on our LLM fine-tuning scenario.
Compute: GPU and CPU
-
Graphics Processing Unit (GPU): NVIDIA RTX 4090, Quantity 4.
- Justification: The Llama 3 70B model requires substantial VRAM. A single 70B parameter model at full precision (FP32) needs 280 GB of VRAM (70B params×4 bytes/param). Even with mixed-precision (FP16/BF16), this is still 140 GB. Using four RTX 4090 GPUs, each with 24 GB of VRAM, provides a total of 96 GB. This is sufficient for fine-tuning the 70B model using quantization (e.g., 4-bit) and techniques like LoRA. The four GPUs will work in parallel, drastically reducing the training time to meet the sub-48-hour target. While professional cards like the A100 offer more VRAM and NVLink, the RTX 4090 provides an excellent performance-to-cost ratio for this budget. Communication will occur over PCIe 4.0, which is fast enough for this scale.
-
Central Processing Unit (CPU): AMD Ryzen Threadripper 7960X (24-Core)
- Justification: The primary role of the CPU here is to feed the GPUs with data and manage the system. It does not need the highest core count, but it absolutely must provide enough PCIe lanes. The Threadripper platform offers a high number of PCIe 5.0 lanes, which is essential to provide maximum bandwidth to all four GPUs simultaneously without creating a bottleneck. The 24 cores are more than sufficient for background OS tasks and the data loading pipeline that prepares batches for the GPUs.
Diagram of the proposed server architecture. The CPU provides dedicated high-bandwidth PCIe lanes to each GPU and the primary NVMe storage, ensuring minimal communication overhead.
Platform and Memory
Storage, Network, and Power
-
Primary Storage (Hot Data): 4 TB NVMe Gen4 SSD
- Justification: Speed is critical for the active dataset. An NVMe SSD provides extremely fast read speeds, ensuring the data pipeline can keep the GPUs saturated. 4 TB offers enough space for the OS, software, and the 500 GB dataset with room to spare for intermediate files.
-
Secondary Storage (Cold Data): 8 TB SATA SSD
- Justification: Model checkpoints for a 70B model can be very large (over 100 GB each). An 8 TB SATA SSD provides a cost-effective solution for storing multiple checkpoints, experiment logs, and a backup of the dataset. It's slower than NVMe but much faster and more reliable than a hard disk drive (HDD).
-
Network Interface Card (NIC): 10 GbE (10 Gigabit Ethernet)
- Justification: The 500 GB dataset needs to be transferred to the server, and results need to be shared. A standard 1 GbE connection would be a significant bottleneck (taking over an hour for the dataset alone). 10 GbE provides a 10x speedup, making data management efficient for the team.
-
Power Supply Unit (PSU): 2000W 80+ Titanium
- Justification: Power requirements are high. Each RTX 4090 can draw up to 450W, and the CPU can draw over 300W under load.
- Total GPU Power: 4×450W=1800W.
- This calculation is a simplification; peak power can be higher. A 2000W PSU provides enough headroom. An 80+ Titanium rating ensures high power efficiency, reducing waste heat and electricity costs, which is a critical consideration for a server running intensive jobs for 48-hour stretches.
Your Turn: Practice Scenario
Now, use the empty template to create your own specification sheet for the following, more constrained scenario.
- Project Goal: Develop and serve a real-time object detection model for a manufacturing quality control system.
- Model: A custom model based on the YOLOv8 architecture.
- Task: Training on a new dataset and deploying for low-latency inference.
- Dataset: 2 TB of high-resolution images.
- Performance Target: Inference must be faster than 30ms per frame. Training speed is less of a priority than inference performance and server cost.
- Budget: The budget is moderate. The priority is cost-effectiveness for a 24/7 inference workload.
Consider how these different requirements will change your component selection. For example, will you still need four GPUs? Is system RAM or VRAM more important for this task? How does the 24/7 inference requirement affect your choice of components and PSU? This exercise will solidify your ability to align hardware choices with specific workload characteristics.
