Data Versioning and Experiment Tracking for Machine Learning
Chapter 1: The Need for Reproducibility in Machine Learning
Challenges in Managing ML Projects
Why Git Alone Is Not Sufficient
Defining Reproducibility in ML
Components of a Reproducible ML Workflow
Introduction to Data Versioning Concepts
Introduction to Experiment Tracking Concepts
Quiz for Chapter 1
Chapter 2: Versioning Data with DVC
Data Versioning Strategies
Introducing Data Version Control (DVC)
Setting Up DVC in a Project
Tracking Data Files and Directories
Storing and Retrieving Data Versions
Connecting DVC to Remote Storage (S3, GCS, Azure Blob)
Switching Between Data Versions
Hands-on Practical: Versioning a Dataset
Quiz for Chapter 2
Chapter 3: Tracking Experiments with MLflow
The Importance of Experiment Tracking
Introducing MLflow Tracking
Setting up MLflow
Logging Parameters and Metrics
Logging Artifacts (Models, Plots, Files)
Organizing Runs with Experiments
Using the MLflow UI
Comparing Experiment Runs
Practice: Tracking a Training Run
Chapter 4: Integrating DVC and MLflow for Reproducible Workflows
Connecting Data Versions to Experiments
Structuring Projects for Integration
Logging DVC Metadata in MLflow
Creating DVC Pipelines
Reproducing DVC Pipelines
Tracking DVC Pipeline Metrics
Combining DVC Pipelines and MLflow Tracking
Best Practices for Integrated Workflows
Hands-on Practical: Building an Integrated Pipeline
Introduction to Experiment Tracking Concepts
Was this section helpful?
- MLflow Documentation, The MLflow Team, 2024 - Provides comprehensive guidance on using MLflow for experiment tracking, including logging parameters, metrics, and artifacts, and managing the ML lifecycle.
- Designing Machine Learning Systems: An Iterative Process for Production-Ready Applications, Chip Huyen, 2022 (O'Reilly Media) - Discusses the principles and practices for building robust ML systems, covering the importance of experiment tracking and reproducibility in iterative development.
- The ICLR 2021 Reproducibility Challenge Report, Sinha, Koustuv, Dodge, Jesse, Luccioni, Sasha, Forde, Jessica Zosa, Raparthy, Sharath Chandra, Pineau, Joelle, Stojnic, Robert, 2022 ReScience C, Vol. 8 (ReScience C) DOI: 10.5281/zenodo.6574723 - Examines the challenges and practices related to reproducibility in machine learning research, providing context for the need for systematic tracking.
- Practical MLOps: How to Take a Machine Learning Model from Experimentation to Production, Noah Gift, Alfredo Kelly, 2021 (O'Reilly Media) - Offers practical guidance on implementing MLOps, with a focus on integrating experiment tracking into a complete ML lifecycle workflow.