Reliability engineering emphasizes the "shift left" philosophy. This approach argues that the cost of fixing a defect increases exponentially the further it travels through the delivery pipeline. By the time a schema error or a malformed SQL query reaches the production data warehouse, it may have already corrupted downstream dashboards or machine learning models. Pre-commit hooks serve as the first line of defense, validating code and configuration on the local workstation before it ever enters the version control system.
Git hooks are scripts that Git executes before or after events such as commit, push, or receive. In the context of data engineering, the pre-commit hook is the most significant. When an engineer attempts to commit changes, this hook triggers a series of automated checks against the staged files. If any check fails, the commit is rejected. The engineer must resolve the issue and attempt the commit again.
This mechanism enforces a standard of hygiene across the team without requiring human intervention during code review. Instead of a senior engineer pointing out trailing whitespace or invalid YAML syntax in a pull request, the system rejects the code immediately on the author's machine.
Mathematically, we can view the pre-commit process as a filter function applied to a set of staged files . The commit is accepted if and only if:
If the result is false for any file, the state of the repository remains unchanged.
The workflow of a pre-commit hook. The process intercepts the commit command, acting as a gatekeeper that ensures only compliant code enters the local history.
While you can write Git hooks manually using Bash scripts, maintaining them across a team is difficult. The industry standard solution is the pre-commit framework, a multi-language package manager for pre-commit hooks. It allows you to define your checks in a single YAML configuration file (.pre-commit-config.yaml) at the root of your repository.
When a developer clones the repository and runs pre-commit install, the framework configures the Git hooks to point to this configuration. This ensures that every team member runs the exact same checks with the exact same versions of the validation tools.
SQL is the dominant language in data engineering, yet it often lacks the tooling rigor associated with Python or Java. A common issue in data teams is "SQL drift," where inconsistent formatting and syntax styles make code reviews difficult and debugging tedious.
To address this, we integrate SQL linters like sqlfluff into the pre-commit config. These tools parse SQL files into an Abstract Syntax Tree (AST) to validate structure, keywords, and even dialect-specific rules (e.g., ensuring BigQuery or Snowflake compliance).
A configuration ensures that no SQL file is committed unless it meets specific criteria:
SELECT * in production views which can break downstream dependencies if the schema changes.For example, a hook might enforce that all keywords are uppercase and that commas are placed at the end of lines. If a developer commits a query with lowercase keywords, the hook can automatically modify the file to fix the capitalization and then fail the commit, prompting the user to stage the corrected file.
Data pipelines written in Python (such as Airflow DAGs or PySpark jobs) require standard software engineering validations. Code that follows PEP 8 standards is easier to read and maintain.
Common hooks for Python data reliability include:
DataFrame do not receive a Dict or None.In a data context, we also validate configuration files. Pipelines are often driven by YAML or JSON configurations. A syntax error in a YAML file can cause an orchestration server to crash. Pre-commit hooks can validate YAML syntax and ensuring that specific schemas are followed before the file leaves the local environment.
One of the most critical roles of pre-commit hooks is security. It is common for engineers to accidentally commit access keys, passwords, or internal API tokens. Once these secrets are pushed to a remote repository, they must be considered compromised.
Tools like detect-secrets or gitleaks scan the staged changes for high-entropy strings and known patterns of API keys (such as those starting with AKIA for AWS). If a potential secret is detected, the commit is blocked.
Additionally, hooks should prevent the accidental commitment of large data files. Data engineers frequently work with CSV or Parquet extracts locally. Committing a 500MB CSV file to Git bloats the repository size and slows down cloning for everyone. A check-added-large-files hook enforces a hard limit (e.g., 10MB) on file sizes.
To operationalize these checks, you create a configuration file that lists the repositories and specific hooks to run. The execution order matters; generally, formatters run first to fix simple issues, followed by linters and security checks.
Below is an example of how a configuration is structured to secure a data pipeline repository. Note that we define the specific version (rev) for each tool to ensure determinism; all engineers check against the same version of the rules.
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.4.0
hooks:
- id: trailing-whitespace
- id: check-yaml
- id: check-added-large-files
args: ['--maxkb=10000']
- repo: https://github.com/psf/black
rev: 23.3.0
hooks:
- id: black
- repo: https://github.com/sqlfluff/sqlfluff
rev: 2.1.0
hooks:
- id: sqlfluff-lint
args: ['--dialect', 'snowflake']
This configuration achieves three objectives: it maintains basic file hygiene, forces Python code into a standard format, and validates SQL against Snowflake syntax rules. By automating these assertions locally, we reduce the noise in the Continuous Integration (CI) environment. The CI server serves as a final gatekeeper, but the pre-commit hook acts as the immediate feedback loop for the engineer.
Was this section helpful?
pre-commit framework documentation, Anthony Sottile and `pre-commit` contributors, 2023 - The official guide for setting up and managing multi-language pre-commit hooks.© 2026 ApX Machine LearningEngineered with