ApX logoApX logo

Command R

Parameters

35B

Context Length

128K

Modality

Text

Architecture

Dense

License

CC-BY-NC

Release Date

11 Mar 2024

Knowledge Cutoff

Jun 2024

System Requirements

VRAM requirements for different quantization methods and context sizes

1,024 tokens

75.18 GB VRAM

Consumer

4x RTX 4090

24GB VRAM

Datacenter

1x NVIDIA A100

80GB VRAM

Apple Silicon

1x Apple M3 Max

128GB VRAM

128,000 tokens

97.02 GB VRAM

Consumer

5x RTX 4090

24GB VRAM

Datacenter

2x NVIDIA A100

80GB VRAM

Apple Silicon

1x Apple M3 Max

128GB VRAM

Architecture Diagram

Input TokensToken EmbeddingPosition: AbsoluteHidden: 8.2k · Context: 128Kx 40 layersLayerNormPre-AttentionMulti-Head Attention64Q / 8KV headsHead dim: 128+LayerNormPre-FFNFeed-Forward NetworkSwiGLU+Final LayerNormOutput Logits

Evaluation Benchmarks

Rank

#141

BenchmarkScoreRank

General Text

Text Arena

1226

155

Rankings

Overall Rank

#141

Coding Rank

-

About Command R

Cohere Command R is a generative language model architected specifically for high-performance enterprise workloads, with an emphasis on long-context processing and tool-augmented workflows. Built on an optimized decoder-only Transformer framework, the model utilizes Grouped Query Attention (GQA) to maintain a significant 128,000-token context window while reducing the memory overhead typically associated with large-scale attention mechanisms. It is designed to facilitate the transition from experimental prototypes to production-grade deployments by offering a balance between inference efficiency and high-fidelity output.

The model undergoes a multi-stage training process including extensive pre-training on a diverse multilingual corpus and subsequent alignment via supervised fine-tuning and preference optimization. A defining architectural feature is its native training for grounded generation, which allows the model to produce responses with precise inline citations from external document sources. This makes it particularly effective for retrieval-augmented generation (RAG) pipelines, where maintaining factual consistency and source traceability is a primary requirement. Furthermore, Command R supports sophisticated multi-step tool use, enabling it to act as an agent that can reason through complex tasks by interacting with external APIs, databases, and software tools.

Optimized for global business applications, Command R provides native support for 10 languages and is trained on 23 in total, ensuring versatility across international markets. The architecture incorporates advanced components such as Rotary Positional Embeddings (RoPE) and Layer Normalization to ensure stability and coherence when handling massive input sequences. By focusing on practical utility in tasks like document summarization, complex reasoning, and structured data analysis, Command R serves as a scalable backbone for automated enterprise systems and intelligent agentic workflows.

Technical Specifications

Attention

Attention Structure

Multi-Head Attention

Attention Heads

64

Key-Value Heads

8

Attention Head Dimension

-

Position Embedding

Absolute Position Embedding

RoPE Theta

-

Sliding Window Attention

-

Sliding Window Size

-

Sliding Window Ratio

-

Linear Attention

-

Linear Attention Ratio

-

Normalization

Layer Normalization

Activation Function

SwigLU

Dimensions

Hidden Dimension Size

8,192

Number of Layers

40

FFN Intermediate Size (Dense)

-

Multi-Token Prediction Heads

-

Tokenizer

Vocabulary Size

-

About Command


Other Command Models