Data Sources and Collection Methods

Data exists in various formats, such as structured, semi-structured, and unstructured types. A primary aspect of data engineering involves identifying where this data originates and the methods engineers use to gather it. Pinpointing the origin and collection method is often the main step when designing systems to store and process data. Think of it like knowing where your ingredients come from before you start cooking.

Data can originate from a surprisingly diverse set of places. As a data engineer, you'll encounter many of these regularly. Let's look at some common categories:

Common Data Sources

1. Operational Databases: These are the workhorses behind many applications. They store the current state of business operations. Examples include:

   • A retail company's database holding customer information, product inventory, and sales records.
   • A web application's database storing user accounts and preferences. These databases are often relational (like PostgreSQL, MySQL, SQL Server) which store data in structured tables, or NoSQL (like MongoDB, Cassandra) which offer more flexible data models. We'll explore databases more in the next section. The main point here is that they are a primary source of structured or semi-structured business data.

2. Logs: Almost every computer system generates logs. These are records of events that occurred, often used for troubleshooting, monitoring performance, or understanding usage patterns. Examples include:

   • Web server logs recording every request (IP address, page visited, time).
   • Application logs noting errors, warnings, or specific actions taken within the software.
   • System logs tracking operating system events. Log data is often semi-structured (e.g., lines in a specific format) or unstructured (e.g., free-form error messages) and can accumulate very rapidly.

3. User Activity and Interaction Data: Modern applications, especially web and mobile apps, generate enormous amounts of data based on user behavior. This includes:

   • Clicks on buttons or links.
   • Pages viewed and time spent on them.
   • Search queries entered.
   • Items added to a shopping cart. This data, sometimes called clickstream data, is invaluable for understanding user engagement, personalizing experiences, and performing A/B testing. It's often captured as events and can be semi-structured (like JSON objects).

4. APIs (Application Programming Interfaces): APIs are like contracts that allow different software systems to request and exchange data in a predefined way. Data engineers use APIs to fetch data from:

   • Third-party services: Getting weather forecasts, stock prices, social media mentions, or map data from external providers.
   • Internal microservices: In modern architectures, different parts of a larger application might communicate via internal APIs. For instance, an ordering service might use an API to get product details from an inventory service. We'll touch upon APIs again later in this chapter.

5. Files: Sometimes, data simply arrives as files. This is common for:

   • Data exported from legacy systems.
   • Reports generated by partners or departments (e.g., spreadsheets).
   • Data collected from sensors or devices that periodically dump readings to a file. These files can be in various formats like CSV (Comma Separated Values), JSON (JavaScript Object Notation), XML (Extensible Markup Language), Parquet, or plain text.

6. Streaming Sources: Unlike data sitting in databases or files, some data arrives as a continuous flow, or stream. Think of:

   • Real-time sensor readings from IoT devices.
   • Financial market data (stock tickers).
   • Live social media feeds.
   • Events from online gaming platforms. Handling streaming data requires different approaches compared to processing data in batches, which we'll discuss in Chapter 5.

Data Collection Methods

Knowing the source is half the battle; the other half is knowing how to collect the data. The method often depends on the source:

1. Database Queries: For operational databases, the most direct method is often querying them using their native language, typically SQL for relational databases or specific query languages for NoSQL databases. Data might be extracted in bulk periodically (e.g., every night) or more frequently.

2. Log Shipping / Aggregation: Collecting logs from potentially hundreds or thousands of servers requires specialized tools. Log shippers (like Fluentd, Filebeat, or cloud-native agents) run on servers, read log files as they are written, and forward the log events to a central storage or processing system.

3. API Calls: To get data from an API, your system needs to make a network request (often an HTTP GET request) to a specific URL endpoint provided by the API. The API then sends back the requested data, usually in a format like JSON. Collection might involve calling the API on a schedule (e.g., every hour) to get updates. Practical aspects include handling authentication (proving you have permission to access the API) and respecting rate limits (how many requests you can make in a given time).

4. File Transfer / Ingestion: When data comes as files, collection might involve:

   • Scheduled jobs that transfer files using protocols like SFTP (Secure File Transfer Protocol).
   • Users uploading files through a web interface.
   • Systems automatically detecting new files dropped into a specific storage location (like an Amazon S3 bucket).

5. Streaming Ingestion: Collecting data from streaming sources typically involves using message queues or dedicated streaming platforms (like Apache Kafka, Google Pub/Sub, or AWS Kinesis). Producers (the sources) send data continuously to these platforms, and consumers (your data processing applications) read from them in real-time or near real-time.

6. Change Data Capture (CDC): Instead of repeatedly querying an entire database table to find changes, CDC techniques focus on capturing only the changes (inserts, updates, deletes) as they happen in the source database, often by reading the database's transaction log. This can be much more efficient for keeping downstream systems synchronized. While the implementation details can be complex, it's useful to know this approach exists.

7. Web Scraping: This involves writing programs (scrapers) to automatically browse websites and extract information directly from HTML pages. While sometimes necessary if no API is available, it should be approached with caution. Websites change frequently, breaking scrapers, and scraping can raise ethical and legal questions or violate a site's terms of service. It's generally less reliable and less preferred than using a formal API.

Here's a diagram illustrating how different sources are typically accessed:

Different data sources often require specific collection methods. Databases are typically queried, logs are shipped, APIs are called, files are transferred, and streams are ingested.

Understanding these sources and collection methods is fundamental. When you need to build a data pipeline (which we'll cover in Chapter 3), your first questions will often be: "Where does the data live?" and "How can we get it?". Having this map of possibilities allows you to choose the right tools and techniques for the job. Next, we'll look a bit deeper into one of the most common data storage structures: databases.