ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing
ApX logoApX logo
ModelsCalculatorHardwareCoursesPricing

All Courses

Getting Started with Local LLMs
Chapter 1: Introduction to Large Language Models
What Is a Large Language Model (LLM)?
A Simple View of How LLMs Work
Understanding Tokens and Text Generation
Why Run LLMs Locally?
Local vs. Cloud-Based LLMs
Quiz for Chapter 1
Chapter 2: Preparing Your Local Environment
Hardware Considerations: CPU
Hardware Considerations: RAM
Hardware Considerations: GPU and VRAM
Checking Your System Specifications
Operating System Compatibility
Installing Python (Optional but Recommended)
Introduction to the Command Line / Terminal
Quiz for Chapter 2
Chapter 3: Finding and Selecting Local LLMs
Where to Find LLM Models: Hugging Face Hub
Understanding Model Sizes and Parameters
Model Formats: GGUF and Others
Quantization: Making Models Smaller
Reading Model Cards for Information
Model Licenses and Usage Restrictions
Choosing Your First Model
Quiz for Chapter 3
Chapter 4: Running Your First Local LLM
Introduction to Local LLM Runners
Setting up Ollama
Downloading a Model with Ollama
Running a Model with Ollama (Command Line)
Setting up LM Studio
Finding and Downloading Models in LM Studio
Loading and Chatting with a Model in LM Studio
Introduction to llama.cpp (Concept)
Hands-on Practical: Running a Model
Quiz for Chapter 4
Chapter 5: Basic Interaction and Prompting
What is a Prompt?
Your First Prompt: Simple Questions
Giving Instructions
Understanding Context Window
Basic Prompt Formatting Tips
Temperature and Creativity
Common Interaction Patterns
Practice: Simple Prompting Techniques
Quiz for Chapter 5
Building a Customized Local AI Assistant

Model Formats: GGUF and Others

A primary aspect of working with Large Language Models is understanding their file format. Model formats can be thought of like different types of files on your computer, such as .txt, .docx, or .pdf. Each format has its own structure and purpose, and not all software can read every format. For Large Language Models, especially when running them locally, the format significantly affects how easily you can use them and how well they perform on your machine.

Models are typically trained using large, complex software frameworks like PyTorch or TensorFlow. While these frameworks are powerful for developing and training models, the resulting model files (often with extensions like .pth, .pt, or stored in specific directory structures like SavedModel) aren't always optimized for running efficiently on typical desktop or laptop hardware, particularly CPUs. They might require installing large software libraries just to load and run the model.

Enter GGUF: Optimized for Local Use

For the specific goal of running LLMs efficiently on your own computer, a format called GGUF (Georgi Gerganov Universal Format) has become very popular. You'll encounter files ending in .gguf frequently when looking for models to run locally.

GGUF is the successor to an earlier format called GGML. It was developed specifically to make running large models feasible on consumer-grade hardware. Here's why it's suitable for beginners and local setups:

  1. Designed for CPU Efficiency: While it can leverage GPUs, GGUF is highly optimized to run well even if you only have a standard CPU. This makes it accessible to more users.
  2. Quantization Integration: GGUF works hand-in-hand with quantization (which we'll discuss in the next section). Quantization reduces the model's size and computational needs. GGUF files often come pre-quantized, meaning the size reduction is already done for you.
  3. Self-Contained (Mostly): GGUF files aim to bundle the model's parameters (its "knowledge") and necessary metadata (information about the model, like its architecture and how text should be processed) into a single file. This simplifies loading and running the model.
  4. Wide Tool Support: Many popular tools designed for easy local LLM execution, such as Ollama, LM Studio, and the underlying llama.cpp library, are built to work directly with the GGUF format. This means you can often download a .gguf file and run it with minimal fuss.

A simplified view showing how models trained in frameworks like PyTorch or TensorFlow are often converted into the GGUF format for easier use with local LLM running tools.

Other Formats You Might See

While GGUF is prevalent for local, easy-to-run models, you might occasionally see references to other formats:

  • PyTorch (.pt, .pth): These are native formats for models trained with the PyTorch framework. Running them usually requires installing PyTorch and potentially writing Python code to load and interact with the model.
  • TensorFlow (SavedModel, .pb): Similar to PyTorch formats, these are native to the TensorFlow framework and typically require installing TensorFlow libraries.
  • Safetensors (.safetensors): This is a newer format gaining traction, designed for safely and efficiently saving and loading model weights. It's often used alongside frameworks like PyTorch but is more secure than some older formats like Python's pickle (which .pth files sometimes use). While safer, it might still require the underlying framework (like PyTorch) to run the model logic.

For getting started without needing to install large development frameworks or write code, focusing on models available in the GGUF format is generally the most direct path. Tools like Ollama and LM Studio abstract away much of the complexity, often relying on GGUF models behind the scenes. As you progress, you might interact with other formats, but GGUF provides an excellent starting point for running LLMs on your personal computer.

  • GGUF File Format, Georgi Gerganov and llama.cpp contributors, 2024 - Official specification and explanation of the GGUF file format, developed for efficient local LLM inference within the llama.cpp project.
  • safetensors Documentation, Hugging Face and safetensors contributors, 2024 - Official documentation for the safetensors library, which provides a secure and efficient way to save and load model weights, offering an alternative to traditional serialization methods.
  • Saving and Loading Models, Matthew Inkawhich, 2024 (PyTorch Foundation) - The official PyTorch tutorial on saving and loading models, demonstrating how models trained with PyTorch are typically stored and retrieved using native framework formats.
  • Save and load Keras models, TensorFlow Team and contributors, 2024 - The official TensorFlow guide for saving and loading models using the Keras API, covering formats such as SavedModel, which are native to the TensorFlow ecosystem.

© 2026 ApX Machine Learning