Data rarely remains static or originates exactly where it is needed. A core task in data engineering involves constructing pathways to move, process, and refine data for analysis or application use. These pathways are known as data pipelines.

This chapter introduces the fundamental structure and operation of data pipelines. We will examine two common architectural patterns: Extract Transform Load (ETL) and Extract Load Transform (ELT). You will learn about the distinct phases within these patterns including:

• Methods for data extraction from various sources.
• Basic operations used for data transformation, such as cleaning and formatting.
• Techniques for loading data into target storage systems.
• Simple approaches to scheduling and managing pipeline execution (orchestration).

Finally, you will apply these ideas by sketching a basic data pipeline based on a given scenario.