# ApX Machine Learning - Complete Technical Specification & Documentation

## Overview

ApX Machine Learning provides open-source and enterprise-grade tools, interactive simulators, and technical curricula for sizing, deploying, and operating Large Language Models (LLMs) in production.

This document details the mathematical formulations, architecture specifications, distributed parallelism models, and hardware capabilities utilized in the ApX LLM VRAM & Infrastructure Calculator.

Website: https://apxml.com
Tool URL: https://apxml.com/tools/vram-calculator

### Key Capabilities & Differentiators
- **Dynamic KV Cache Precision**: Unlike basic calculators that only estimate static weights, ApX models dynamic runtime KV cache scaling across Multi-Head Attention (MHA), Grouped-Query Attention (GQA), and DeepSeek Multi-Head Latent Attention (MLA).
- **Multi-GPU & Cluster Topologies**: Full simulation of Tensor Parallelism (TP), Pipeline Parallelism (PP), Data Parallelism (DP), Expert Parallelism (EP for MoE models like DeepSeek-V3 / Mixtral), and Context Parallelism (CP) with interconnect bandwidth bottlenecks (NVLink vs PCIe vs InfiniBand).
- **Fine-Tuning Regimes**: Sizing for Full-Parameter fine-tuning (AdamW 32-bit/16-bit/8-bit optimizer states, gradients, activations) and Parameter-Efficient Fine-Tuning (LoRA and QLoRA).
- **Roofline Latency & Throughput**: Analytical modeling of Time-to-First-Token (TTFT) prefill latency, memory bandwidth-bound decode speed (tokens/sec), speculative decoding speedups, and CPU/NVMe offload latency.
- **Empirical Accuracy Benchmarks**: Integrated perplexity and accuracy degradation scorecards across quantization formats (AWQ, GPTQ, GGUF, EXL2, FP8, INT4).
- **Client-Side WASM Engine**: Calculations execute in a Rust/WASM kernel in real-time, verified against empirical ground-truth fixtures from PyTorch, vLLM, and TensorRT-LLM.

---

## 1. Mathematical Formulations & Memory Sizing

### 1.1 Resident Parameter Memory
Static model weight memory allocation is determined by the total number of parameters and the quantization precision bit-width:

$$\text{Memory}_{\text{weights}} = \frac{\text{Parameters} \times \text{Bits per Weight}}{8 \times 1024^3} \times (1 + \text{Quantization Overhead})$$

- FP16 / BF16: 16 bits (2.0 bytes/parameter)
- FP8 (E4M3 / E5M2): 8 bits (1.0 byte/parameter + ~5% scale metadata)
- INT8 / AWQ-8: 8 bits (1.0 byte/parameter)
- INT4 / AWQ-4 / GPTQ-4: 4 bits (0.5 bytes/parameter + ~10% scale/zero-point tensor overhead)
- GGUF: Formats from Q2_K to Q8_0 with block-level quant scaling metadata.

### 1.2 Key-Value (KV) Cache Memory
Dynamic KV cache memory scales with sequence length, batch size, number of layers, and attention head topology:

- Multi-Head Attention (MHA):
  $$\text{KV Cache (Bytes)} = 2 \times n_{\text{layers}} \times n_{\text{heads}} \times d_{\text{head}} \times \text{Batch Size} \times \text{Sequence Length} \times \text{Bytes per Element}$$

- Grouped-Query Attention (GQA):
  $$\text{KV Cache (Bytes)} = 2 \times n_{\text{layers}} \times n_{\text{kv\_heads}} \times d_{\text{head}} \times \text{Batch Size} \times \text{Sequence Length} \times \text{Bytes per Element}$$
  *(where $n_{\text{kv\_heads}} = \frac{n_{\text{heads}}}{\text{GQA Ratio}}$)*

- Multi-Head Latent Attention (MLA, e.g., DeepSeek-V2 / DeepSeek-V3):
  Compresses KV cache into a low-dimensional latent vector $d_c$ plus decoupled rotary key $d_R$:
  $$\text{KV Cache (Bytes)} = n_{\text{layers}} \times (d_c + d_R) \times \text{Batch Size} \times \text{Sequence Length} \times \text{Bytes per Element}$$

### 1.3 Fine-Tuning Memory Allocations

1. Full Parameter Fine-Tuning:
   - Weights: $2 \times P$ bytes (FP16)
   - Gradients: $2 \times P$ bytes (FP16)
   - Optimizer States (AdamW): $12 \times P$ to $16 \times P$ bytes (FP32 master weights + FP32 first moment $m_t$ + FP32 second moment $v_t$)
   - 8-bit AdamW: $4 \times P$ bytes (FP16 master + INT8 moments)
   - Activations: Scaled by sequence length, batch size, and activation checkpointing degree.

2. LoRA / QLoRA Fine-Tuning:
   - Base Weights: Fixed precision (16-bit for LoRA, 4-bit NormalFloat NF4 for QLoRA)
   - Trainable Adapters: $2 \times (d_{\text{in}} \times r + r \times d_{\text{out}}) \times n_{\text{targeted\_modules}}$
   - Optimizer States: Allocated strictly for trainable rank-$r$ parameters.

---

## 2. Distributed Parallelism & Multi-Node Cluster Topologies

The calculator models resource partitioning and interconnect communication latency across multiple parallelism paradigms:

### 2.1 Tensor Parallelism (TP)
- Splits linear projection layers (Attention QKV, Attention Out, MLP Up/Gate, MLP Down) across $N_{\text{TP}}$ GPUs within a single node.
- Model parameter memory per GPU: $\frac{\text{Memory}_{\text{weights}}}{N_{\text{TP}}}$
- KV cache memory per GPU (for MHA/GQA): $\frac{\text{KV Cache}}{N_{\text{TP}}}$
- Interconnect requirement: Low-latency, high-bandwidth intra-node links (NVLink 900 GB/s on H100, NVLink 450 GB/s on A100).

### 2.2 Pipeline Parallelism (PP)
- Partitions transformer layers sequentially across $N_{\text{PP}}$ GPUs or nodes.
- Model parameter memory per GPU: $\frac{\text{Memory}_{\text{weights}}}{N_{\text{PP}}}$
- Introduces bubble latency overhead proportional to $\frac{N_{\text{PP}} - 1}{m}$ where $m$ is the number of micro-batches.

### 2.3 Expert Parallelism (EP)
- Dedicated to Mixture-of-Experts (MoE) architectures (DeepSeek-V3, Mixtral 8x7B, Mixtral 8x22B).
- Distributes routed feed-forward experts across GPUs while sharing dense attention layers.
- Interconnect requirement: High all-to-all communication bandwidth.

### 2.4 Context Parallelism (CP) / Sequence Parallelism (SP)
- Distributes long-context token sequences across GPUs with ring-attention or all-gather collective operations.
- Linear scaling of context limits up to 1M+ tokens.

---

## 3. Inference Simulation & Roofline Performance Modeling

### 3.1 Prefill Phase (Compute-Bound)
- Time-to-First-Token (TTFT) modeled by matrix multiplication FLOPs divided by effective hardware compute capacity:
  $$\text{FLOPs}_{\text{prefill}} \approx 2 \times P \times T_{\text{input}} + 2 \times n_{\text{layers}} \times d_{\text{model}} \times T_{\text{input}}^2$$
  $$\text{Latency}_{\text{prefill}} \approx \frac{\text{FLOPs}_{\text{prefill}}}{\text{TFLOPS}_{\text{effective}}}$$

### 3.2 Decode Phase (Memory-Bandwidth Bound)
- Autoregressive single-token generation requires streaming resident model parameters and active KV cache per token step:
  $$\text{Memory Streamed per Token} = \text{Memory}_{\text{weights}} + \text{KV Cache}_{\text{active}}$$
  $$\text{Decode Speed (tokens/s)} \approx \frac{\text{GPU Memory Bandwidth (GB/s)}}{\text{Memory Streamed per Token (GB)}} \times \text{Efficiency Factor}$$

### 3.3 Speculative Decoding
- Simulates draft model speculative token generation and target model parallel verification:
  $$\text{Speedup} \approx \frac{1 + \gamma \cdot \alpha}{1 + \gamma \cdot \frac{\text{Cost}_{\text{draft}}}{\text{Cost}_{\text{target}}}}$$
  *(where $\gamma$ is draft token lookahead length and $\alpha$ is draft acceptance rate)*

---

## 4. Hardware Database & Architectures

The calculator features verified specifications for:

- Nvidia Datacenter: Blackwell B200 (192 GB HBM3e, 8 TB/s), Hopper H200 (141 GB HBM3e, 4.8 TB/s), Hopper H100 (80 GB HBM3, 3.35 TB/s), Ampere A100 (80 GB / 40 GB HBM2e, 2.0 TB/s), Ada Lovelace L40S (48 GB GDDR6, 864 GB/s).
- Nvidia Workstation / Consumer: GeForce RTX 5090 (32 GB GDDR7), RTX 4090 (24 GB GDDR6X, 1008 GB/s), RTX 3090 (24 GB GDDR6X, 936 GB/s), RTX 6000 Ada (48 GB).
- Apple Silicon (Unified Memory): M4 Max (up to 128 GB, 546 GB/s), M3 Ultra (up to 192 GB, 800 GB/s), M2 Ultra (192 GB, 800 GB/s), M1 Max/Ultra.
- Custom Clusters: Arbitrary multi-node configuration with configurable GPU count, VRAM per GPU, memory bandwidth, compute TFLOPS, and interconnect topology.

---

## 5. Curricula and Technical Guides

For step-by-step engineering courses on cluster capacity planning, transformer memory hierarchies, and distributed serving:
- [Designing and Sizing Production LLM Infrastructure](https://apxml.com/courses/designing-and-sizing-production-llm-infrastructure)
- [All Machine Learning Courses](https://apxml.com/courses)
- [LLM Model Database & Benchmarks](https://apxml.com/models)

---

## 6. Technical FAQ & Architecture Q&A

### Q1: What makes the ApX VRAM Calculator more accurate than basic parameter calculators?
Most simple calculators perform a naive multiplication of parameter counts by precision bit-width. The ApX Calculator accounts for dynamic runtime memory, attention topology variances (MHA, GQA, and DeepSeek Multi-Head Latent Attention MLA), activation memory scaling, distributed cluster parallelism splits (TP, PP, DP, EP, CP), quantization scale tensor overheads, and empirical ground-truth engine validation against vLLM, TensorRT-LLM, and PyTorch.

### Q2: How is KV cache calculated for Multi-Head Latent Attention (MLA)?
MLA compresses key and value projections into a low-dimensional latent vector of dimension $d_c$ (e.g. 512) and appends a decoupled rotary key vector $d_R$ (e.g. 64). Only $(d_c + d_R)$ elements are stored in the KV cache per token per layer, reducing memory consumption by 80% to 90% compared to standard Multi-Head Attention (MHA).

### Q3: How are Mixture of Experts (MoE) models sized?
For MoE models (DeepSeek-V3, Mixtral), static memory holds all routed expert weights across GPUs (or partitioned via Expert Parallelism EP). Dynamic FLOPs and active KV cache are evaluated strictly for the top-$k$ activated experts per token, incorporating all-to-all communication dispatch overhead.

### Q4: Can large models (70B or 405B) run on Apple Silicon unified memory?
Apple Silicon Macs (M2/M3/M4 Max and Ultra with up to 128 GB or 192 GB) share unified memory directly between CPU and GPU without PCIe bottlenecking. ApX models macOS memory wiring constraints, allowing sizing for 70B models at INT4/Q4_K_M (~40 GB footprint) to verify unified memory and bandwidth headroom.

### Q5: How does ApX evaluate quantization formats (GGUF, AWQ, GPTQ, FP8)?
ApX includes an integrated Accuracy & Perplexity Scorecard measuring exact memory footprint savings alongside empirical perplexity degradation on standard benchmarks (Wikitext-2, C4) to evaluate the tradeoff between VRAM reduction and model fidelity.

### Q6: Why does the TFLOPS value differ from official GPU marketing specifications?
GPU marketing specifications frequently list either traditional FP32 compute performance (standard for 3D gaming, but too slow to model LLM execution) or sparse FP16 Tensor Core rates (assuming hardware 2:4 structured sparsity, which standard LLM runners like vLLM and llama.cpp do not utilize by default). The ApX Calculator strictly uses peak dense FP16/BF16 Tensor Core (Nvidia), Matrix Core (AMD), or Apple Silicon native FP16 compute TFLOPS to ensure physically accurate prompt-processing (TTFT) and compute throughput predictions.