Data Models and Schemas

While traditional databases excel at storing and retrieving structured data based on exact matches or range queries on predefined fields, vector databases are built around a different core principle: managing and searching high-dimensional vectors based on similarity. This necessitates distinct data models and schema concepts optimized for this task.

At the foundation of a vector database's data model lies the concept of a vector record (sometimes called an item, point, or document). This is the fundamental unit of data storage. Each vector record typically bundles together several important pieces of information:

1. A Unique Identifier (ID): Just like in most databases, each vector record needs a unique ID for direct retrieval, updates, or deletion. This is often a string or an integer provided by the user or generated by the database.

2. The Vector Embedding: This is the core numerical representation of the data point, usually stored as an array or list of floating-point numbers. For example, a 768-dimensional text embedding would be stored as an array containing 768 numbers. A critical aspect here is dimensionality consistency. Within a specific collection or index in the database, all vectors typically must have the same dimension ($d$). Trying to insert vectors of varying dimensions into the same index is usually not supported or requires special handling, as the underlying ANN algorithms rely on operating within a consistent vector space.

"3. Metadata Payload: This is arguably what makes vector databases truly powerful for applications. Vectors rarely exist in isolation; they represent something concrete, like a document, an image, a product, or a user profile. The metadata payload stores attributes associated with the vector, providing context and enabling more refined searches. This payload is often structured like a JSON object or a dictionary, containing key-value pairs."

Representing Metadata

The metadata associated with a vector can include a wide range of data types, similar to those found in NoSQL or relational databases:

• Strings: For storing text information like product names, categories, filenames, or document titles.
• Numbers: Integers or floating-point numbers for storing quantities, prices, timestamps, ratings, or numerical properties.
• Booleans: For representing true/false flags, such as is_active, on_sale, or is_processed.
• Lists/Arrays: Often used for tags, keywords, or lists of associated IDs.
• Geospatial Data: Some vector databases support specialized types for storing geographical coordinates, enabling location-based filtering.
• Nested Objects: Some systems allow for more complex, nested structures within the metadata.

Consider an example of a vector record representing a chunk of text from a document:

{
  "id": "doc1_chunk3",
  "vector": [0.013, -0.245, ..., 0.912], // A 512-dimensional vector
  "metadata": {
    "document_source": "/path/to/research_paper.pdf",
    "page_number": 15,
    "chunk_length": 350, // Number of characters or tokens
    "topic": "Machine Learning",
    "keywords": ["vector database", "ANN", "similarity search"],
    "published_year": 2023
  }
}

In this example, id uniquely identifies the record, vector holds the embedding, and metadata contains contextual information about the text chunk.

Schemas: Defining the Structure

How strictly this structure (vector dimension, metadata fields, data types) must be defined beforehand varies between different vector database systems.

• Explicit Schemas: Some databases require you to define a schema upfront when creating a collection or index. This involves specifying the expected vector dimension and defining the names and data types for all allowed metadata fields. This approach provides data consistency and can sometimes optimize storage and filtering performance, as the database knows exactly what structure to expect.
• Schema-on-Write/Flexible Schemas: Other systems offer more flexibility, similar to many NoSQL document stores. You might only need to specify the vector dimension, and the metadata fields can be added more dynamically during data insertion. The database infers the schema based on the data it receives. While offering flexibility, this might require careful management to ensure consistency if needed for filtering logic.

Regardless of the approach, understanding the structure of your vector records, the fixed vector dimensionality and the available metadata fields and their types, is essential. This structure directly influences how you build indexes (covered in the next chapter) and, critically, how you query the database. The power of vector databases often comes from combining vector similarity search with precise filtering based on these metadata attributes (e.g., "find documents semantically similar to this query, but only those published after 2022 and tagged with 'ANN'"). This dual capability is enabled by a data model that tightly couples the high-dimensional vector with its descriptive metadata.