Switching Between Data Versions

DVC tracks data files using .dvc metadata files, which are committed to Git. Data content synchronizes between local machines and remote storage using dvc push and dvc pull. This setup enables switching between different versions of your data alongside your code, a primary aspect of DVC's utility.

Machine learning projects often involve experimenting with different datasets or different versions of the same dataset (e.g., before and after cleaning, or with different feature sets). Being able to reliably revert your project's data to a specific historical state is essential for reproducibility, debugging, and comparing results. DVC, working hand-in-hand with Git, makes this process straightforward.

The Git and DVC Partnership for Versioning

The core idea is that Git tracks the evolution of your code and the pointers to your data, while DVC manages the actual data files associated with those pointers.

When you run dvc add data.csv and then git add data.csv.dvc, you are telling Git to track the metadata file (data.csv.dvc). This file contains information like the hash of the actual data.csv content, but not the content itself. When you make a Git commit, you capture the state of your code and the specific version of the .dvc file(s) at that moment.

Think of it like this:

• Git Commit: A snapshot of your codebase and a list of instructions (the .dvc files) specifying which version of the data belongs with that code snapshot.
• DVC Cache/Remote Storage: The actual storage location for all the different versions of your data files, indexed by their content hashes.

Workflow for Switching Data Versions

Switching between data versions tied to specific points in your project's history involves a two-step process using standard Git commands followed by a DVC command:

1. Use git checkout: Check out the Git commit, branch, or tag that corresponds to the desired project state. This action updates the files tracked by Git in your working directory, including the .dvc files. Your code will revert to the state of that commit, and crucially, the .dvc files will now point to the data versions associated with that commit.
2. Use dvc checkout: After Git has updated the .dvc files, run dvc checkout. This command instructs DVC to read the current .dvc files in your working directory and synchronize the actual data files. DVC will find the corresponding data content (identified by the hash in the .dvc file) in its local cache and place it in your workspace, overwriting the previous version if necessary.

Let's illustrate with an example. Imagine you have two significant commits in your Git history:

• commit-A: Used data/raw_images_v1.zip. The data/raw_images_v1.zip.dvc file in this commit points to the hash of this specific dataset version.
• commit-B: Updated the dataset to data/raw_images_v2.zip (perhaps after some cleaning or additions). The data/raw_images_v1.zip.dvc file in this commit points to the hash of this new dataset version.

Suppose you are currently working on the state corresponding to commit-B, but you want to go back and examine the results or rerun an analysis using the data from commit-A.

Step 1: Switch Git history

# Ensure your current work is saved or committed
git status

# Check out the previous commit
git checkout commit-A

At this point, Git updates your codebase and restores the data/raw_images_v1.zip.dvc file as it existed in commit-A. However, the actual data/raw_images_v1.zip file in your workspace might still be the v2 version (or might even be missing if you just cloned the repository). Your workspace data is momentarily out of sync with the .dvc metadata file.

Step 2: Synchronize data with DVC

# Tell DVC to update the workspace data based on the current .dvc file
dvc checkout data/raw_images_v1.zip.dvc
# Or, to update all DVC-tracked files in the repository:
# dvc checkout

DVC reads data/raw_images_v1.zip.dvc (which now points to the v1 hash), finds the corresponding v1 data in its cache, and places it into your data/ directory, replacing the v2 content. Your workspace now accurately reflects the project state, both code and data, as it was in commit-A.

To return to the state of commit-B, you would simply reverse the process:

git checkout commit-B
dvc checkout

What if the Data Isn't in the Local Cache?

The dvc checkout command relies on the required data version being present in the DVC local cache (typically located in .dvc/cache). If you check out a Git commit whose associated data has not yet been downloaded from remote storage (e.g., after cloning a repository or switching to a very old branch), dvc checkout might report that the data is missing.

In this scenario, you first need to fetch the required data from your configured DVC remote storage using dvc pull:

# After git checkout commit-A
# Attempt to sync workspace data
dvc checkout

# If it fails due to missing cache, pull from remote:
dvc pull data/raw_images_v1.zip.dvc
# Or pull all data for the current commit:
# dvc pull

# Now the data is cached locally, so checkout will succeed
# (Note: dvc pull often performs the checkout implicitly,
# but running dvc checkout ensures the workspace is synced)
dvc checkout

dvc pull fetches the data corresponding to the current .dvc files from the remote into the local cache, and often updates the workspace file directly as well. Running dvc checkout afterward is a good practice to ensure the workspace is correctly synchronized with the cache according to the .dvc files checked out by Git.

By combining git checkout for managing code and metadata history with dvc checkout (and dvc pull when needed) for synchronizing the associated large data files, you gain a powerful and reliable method for navigating the complete history of your machine learning project. This ability to precisely restore past states is fundamental for reproducibility and collaborative development.