SmolVLM2 256M Video Instruct

Parameters

260M

Context Length

8K

Modality

Text

Architecture

Dense

KV Cache Rate

23 KB/tok

License

Apache 2.0

Release Date

11 Feb 2025

Knowledge Cutoff

-

API Pricing (per 1M)

Self-hosted only

Hardware & System Requirements

Precision

Native

Context

Workload Memory: 2.80 GB

INT4 · 8K context

Base Weights: 0.12 GB

KV Cache: 0.18 GB

Runtime & Graphs: 2.50 GB

Minimum Required Hardware

1x NVIDIA L4 24GB

2.8 / 21.6 GB usable

1x RTX 3060 12GB

2.8 / 10.8 GB usable

Apple M4 24GB

2.8 / 18.0 GB usable

Determine requirements for your specific workload:

Open in VRAM Calculator

Runtime requirements

Checkpoint format

BF16

CUDA

>= 11.8

PyTorch

>= 2.1

Transformers

>= 4.47.1

ArchitecturePrecisionAttention kernel

Blackwell, Hopper

Native BF16

FlashAttention-3

Ada Lovelace, Ampere

Native BF16

FlashAttention-2

Architecture Diagram

Input TokensToken EmbeddingPosition: RoPEHidden: 576 · Context: 8K · Vocab: 49.3kx 30 layersRMSNormPre-AttentionGrouped-Query Attention9Q / 3KV headsHead dim: 64+RMSNormPre-FFNFeed-Forward NetworkSwiGLUIntermediate: 1.5k+Final RMSNormOutput Logits

Evaluation Benchmarks

No evaluation benchmarks for SmolVLM2 256M Video Instruct available.

Rankings

Overall Rank

-

Coding Rank

-

About SmolVLM2 256M Video Instruct

SmolVLM2 256M Video Instruct is a lightweight compact multimodal model tailored for video comprehension and instruction following on edge devices. It supports efficient multimodal reasoning with low memory and computational overhead.

Specifications

Attention

Attention Structure

Grouped-Query Attention

Attention Heads

9

Key-Value Heads

3

Attention Head Dimension

64

Position Embedding

ROPE

RoPE Theta

100,000

Normalization

RMS Normalization

Activation Function

SwigLU

Dimensions

Base Precision

BF16

Hidden Dimension Size

576

Number of Layers

30

FFN Intermediate Size (Dense)

1,536

Tokenizer

Vocabulary Size

49,280

About SmolLM Family

SmolLM open-weight language models (e.g. SmolLM3)


Other SmolLM Family Models