Data Versioning Strategies

Managing data effectively is fundamental to creating reproducible machine learning workflows. Standard version control systems like Git excel at tracking changes in text-based code files, but they struggle with the scale and nature of typical ML datasets. So, how do teams manage evolving datasets? Let's examine some common strategies, ranging from simple manual methods to more sophisticated approaches, highlighting their strengths and weaknesses.

Manual Naming Conventions and Copying

Perhaps the most basic approach is simply renaming files or directories to indicate different versions. You might have encountered folders like data_processed_v1, data_processed_v2, or files such as features_final.csv, features_final_really_final.csv.

• Pros: Requires no special tools; easy to understand initially for very small projects.
• Cons: This method quickly becomes unmanageable.
  • Error-Prone: It's easy to make mistakes, forget the exact changes between versions, or accidentally overwrite files.
  • Scalability Issues: Disk space usage balloons as you duplicate large datasets for minor changes.
  • Lack of Provenance: It's difficult to reliably link a specific version of the data to the exact code commit or experiment run that used or produced it. Collaboration becomes a nightmare.
  • No Audit Trail: There's no systematic record of why changes were made or who made them.

While simple, manual versioning lacks the rigor needed for serious ML development and reproducibility.

Cloud Storage Snapshots or Versioning

Cloud storage providers (AWS S3, Google Cloud Storage, Azure Blob Storage) often offer their own versioning features. You can enable versioning on a storage bucket, and the provider will keep previous versions of objects when they are overwritten or deleted.

• Pros: Uses existing cloud infrastructure; provides backups and rollback capabilities.
• Cons:
  • Decoupled from Code: Cloud storage versioning is independent of your Git repository. Linking a specific data version to a code commit requires manual tracking or complex scripting.
  • Granularity: Often operates at the object (file) level. Managing versions of a dataset composed of many files as a single coherent unit can be awkward.
  • ML Context Missing: These systems don't understand ML workflows, dependencies between data processing steps, or experiment tracking.
  • Checkout Complexity: Retrieving a specific, consistent set of data files corresponding to a past project state can be cumbersome.

Cloud storage versioning is useful for backup and disaster recovery but doesn't directly address the tight coupling needed between code, data, and experiments in ML.

Git Large File Storage (LFS)

Git LFS is a Git extension designed to handle large files more efficiently. Instead of storing large binary files directly in the Git repository history (which bloats the repository quickly), Git LFS stores pointers (small text files) in Git. The actual large files are stored on a separate LFS server (which could be self-hosted or provided by services like GitHub, GitLab, Bitbucket). When you check out a commit, Git LFS downloads the required large files based on the pointers.

• Pros:
  • Git Integration: Works relatively smoothly within the standard Git workflow (git add, git commit, git push, git pull).
  • Improved Repository Size: Keeps the core Git repository small and fast.
• Cons:
  • Not Data-Specific: Git LFS treats data like any other large asset (e.g., graphics, binaries). It lacks awareness of dataset structure, dependencies, or ML-specific operations.
  • Storage Management: Requires managing a separate LFS server or paying for LFS bandwidth/storage on Git hosting platforms.
  • Potential for Large Checkouts: Checking out a branch might still trigger large downloads if the data files changed significantly.
  • Limited Pipeline Awareness: Git LFS doesn't help define or manage data processing pipelines or track data provenance through multiple steps.

Git LFS is a definite improvement over storing large files directly in Git, but it's a general-purpose solution for large files, not a tailored solution for the specific needs of data versioning in machine learning.

Specialized Data Versioning Tools (e.g., DVC)

Recognizing the limitations of the above methods, specialized tools have emerged specifically for versioning data and models within ML projects. Data Version Control (DVC), the focus of this chapter, is a prime example. These tools typically work alongside Git, leveraging Git for code versioning while providing dedicated mechanisms for data.

• Pros:
  • Designed for ML: Built with the ML workflow in mind, understanding concepts like datasets, models, pipelines, and metrics.
  • Git-like Experience: Often provide a command-line interface similar to Git (e.g., dvc add, dvc push, dvc pull).
  • Storage Agnostic: Integrate with various storage backends (cloud, local, network drives) without storing data directly in Git.
  • Reproducibility Focus: Enable tracking data dependencies and creating reproducible data pipelines.
  • Data Provenance: Maintain clear links between code versions, data versions, and experiment results.
• Cons:
  • Learning Curve: Requires learning a new tool and its concepts.
  • Added Complexity: Introduces another layer of tooling to the project setup.

Here's a simplified comparison:

Feature	Manual Copying	Cloud Versioning	Git LFS	Specialized Tools (DVC)
Git Integration	None	Poor	Good	Excellent (Alongside)
Storage	Local copies	Cloud Provider	Separate LFS Server	Flexible (Cloud/Local)
Reproducibility	Very Low	Low	Moderate	High
ML Pipeline Aware	No	No	No	Yes
Scalability	Poor	Good	Moderate	Good
Granularity	Manual	File-level	File-level	File/Directory/Dataset

Given the challenges outlined, specialized tools offer the most comprehensive solution for managing data in a reproducible ML setting. They bridge the gap left by Git and generic file storage systems. In the following sections, we will explore how DVC implements these principles, providing a practical and powerful way to version your data alongside your code.