Introduction to Data Cleaning and Preprocessing
Chapter 1: The Importance of Clean Data
What is Data Cleaning?
What is Data Preprocessing?
Common Sources of Dirty Data
Impact of Poor Data Quality
The Data Cleaning Workflow Overview
Quiz for Chapter 1
Chapter 2: Identifying and Handling Missing Data
What Are Missing Values?
Methods for Detecting Missing Data
Visualizing Missing Data Patterns
Strategy 1: Deleting Rows (Listwise Deletion)
Strategy 2: Deleting Columns
Strategy 3: Basic Imputation (Mean/Median/Mode)
Considerations for Choosing a Strategy
Handling Missing Data: Hands-on Practical
Quiz for Chapter 2
Chapter 3: Dealing with Duplicate Data
What Constitutes Duplicate Data?
Why Remove Duplicates?
Identifying Complete Duplicate Rows
Identifying Duplicates Based on Specific Columns
Removing Duplicate Rows
Handling Duplicates: Practice
Quiz for Chapter 3
Chapter 4: Correcting Data Types
Common Data Types in Datasets
Why Correct Data Types Matter
Identifying Incorrect Data Types
Converting to Numeric Types (Integer, Float)
Handling Errors During Numeric Conversion
Converting to Datetime Types
Converting to Categorical or String Types
Data Type Correction: Hands-on Practical
Quiz for Chapter 4
Chapter 5: Basic Data Formatting and Standardization
Importance of Consistent Formatting
Standardizing Text Case (Upper/Lower)
Removing Leading/Trailing Whitespace
Simple String Replacements
Basic Unit Conversion Example
Formatting Practice
Quiz for Chapter 5
Impact of Poor Data Quality
Was this section helpful?
- Data Mining: Concepts and Techniques, Jiawei Han, Micheline Kamber, Jian Pei, 2011 (Morgan Kaufmann) - Provides a foundational overview of data preprocessing, including data quality problems and their impact on analysis.
- Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow: Concepts, Tools, and Techniques to Build Intelligent Systems, Aurélien Géron, 2019 (O'Reilly Media) - Explains the practical importance of data preparation for machine learning models and the consequences of poor data quality on model performance.
- The impact of data quality on business processes and decision-making: A case study approach, Marvin Haffke, Maximilian Kaltenbach, Clemens Stöhr, 2022 Procedia Computer Science, Vol. 200 (Elsevier) DOI: 10.1016/j.procs.2022.01.403 - Presents a recent academic perspective on how poor data quality specifically affects business processes and organizational decision-making.
- Data quality problems: What happens when they don't get fixed?, Thomas C. Redman Jr., 2013 MIT Sloan Management Review, Vol. 54 (Massachusetts Institute of Technology) DOI: 10.1177/154165181305400405 - A highly respected article that thoroughly discusses the consequences and costs incurred when data quality issues are ignored in an organization.