Feature Creation from Date/Time Data

Raw timestamp or date string columns, while readable by humans, often contain rich temporal information that machine learning models cannot directly interpret in their raw form. Treating dates merely as identifiers or continuous numbers usually misses underlying patterns like seasonality, trends, or specific time-based behaviors. Extracting specific components from date/time features allows models to capture these temporal dynamics effectively.

Extracting Temporal Components with Pandas

The most common approach involves breaking down a datetime object into its constituent parts. If you're working with data in a Pandas DataFrame, the first step is to ensure your date column is actually stored as a datetime type. You can usually achieve this using pd.to_datetime(). Once in the correct format, Pandas provides the convenient .dt accessor to extract various components.

Consider a DataFrame df with a column event_timestamp:

import pandas as pd

# Sample data
data = {'event_timestamp': ['2023-01-15 08:30:00', '2023-07-22 14:05:00', '2024-12-01 21:00:00'],
        'value': [10, 20, 15]}
df = pd.DataFrame(data)

# Ensure correct dtype
df['event_timestamp'] = pd.to_datetime(df['event_timestamp'])

# Extract components
df['year'] = df['event_timestamp'].dt.year
df['month'] = df['event_timestamp'].dt.month
df['day'] = df['event_timestamp'].dt.day
df['hour'] = df['event_timestamp'].dt.hour
df['day_of_week'] = df['event_timestamp'].dt.dayofweek # Monday=0, Sunday=6
df['day_of_year'] = df['event_timestamp'].dt.dayofyear
df['week_of_year'] = df['event_timestamp'].dt.isocalendar().week.astype(int) # Use isocalendar for ISO week
df['quarter'] = df['event_timestamp'].dt.quarter
df['is_weekend'] = df['day_of_week'].isin([5, 6]).astype(int) # Saturday=5, Sunday=6

print(df)

This simple process generates several new numerical features that a model can use. For instance:

• year can capture long-term trends.
• month and quarter can capture seasonality (e.g., higher sales in Q4).
• day_of_week and is_weekend can model weekly patterns (e.g., different user activity on weekdays vs. weekends).
• hour can capture intra-day variations (e.g., peak website traffic times).

These extracted features are often much more informative for typical machine learning algorithms than the original timestamp.

Handling Cyclical Features

Some extracted temporal features are cyclical. For example, month goes from 1 to 12 and then wraps back to 1. Similarly, day_of_week cycles from 0 to 6. Representing these directly as simple integers (1, 2, ..., 12 or 0, 1, ..., 6) can be problematic for some models, especially distance-based ones or linear models. The model might interpret month 12 as being very far from month 1, when in reality they are adjacent in the cycle.

A common technique to represent cyclical features is to map them onto a circle using sine and cosine transformations. This creates two features that together preserve the cyclical distance information.

For a feature x with a maximum value max_val (e.g., 12 for month, 6 for day of week starting at 0), the transformations are:

$x_{sin} = sin(2 \pi \frac{x}{max\_val + 1})$ $x_{cos} = cos(2 \pi \frac{x}{max\_val + 1})$

If the feature is 1-indexed (like month 1-12), the denominator might be just max_val. If it's 0-indexed (like dayofweek 0-6), the denominator is often max_val + 1. Ensure the range of x aligns with the denominator used.

Let's apply this to the month and day_of_week features:

import numpy as np

# Month (1-12)
df['month_sin'] = np.sin(2 * np.pi * df['month'] / 12)
df['month_cos'] = np.cos(2 * np.pi * df['month'] / 12)

# Day of week (0-6)
df['dow_sin'] = np.sin(2 * np.pi * df['day_of_week'] / 7)
df['dow_cos'] = np.cos(2 * np.pi * df['day_of_week'] / 7)

# Drop original cyclical features if desired
# df = df.drop(['month', 'day_of_week'], axis=1)

print(df[['event_timestamp', 'month_sin', 'month_cos', 'dow_sin', 'dow_cos']].head())

Now, months 12 and 1 will have similar values in the (month_sin, month_cos) space, reflecting their cyclical proximity. The same applies to days of the week.

Sine and Cosine transformation maps months onto a circle, preserving the adjacency of December (12) and January (1).

Calculating Time Differences (Durations)

Another powerful type of feature derived from date/time data is the duration or time elapsed between events or relative to a specific reference point.

• Time Between Events: If you have multiple timestamp columns (e.g., order_date, ship_date), you can calculate the difference:

  # Assuming df['order_date'] and df['ship_date'] are datetime objects
  df['processing_time'] = df['ship_date'] - df['order_date']
  # Convert Timedelta to a numerical unit, e.g., days
  df['processing_days'] = df['processing_time'].dt.total_seconds() / (60*60*24)
  

• Time Since a Reference Point: Calculate time relative to a fixed date or the date of the first/last event in the dataset. This is useful for creating features like "customer account age" or "time since last interaction".

  # Example: Time since the earliest event in the dataset
  reference_date = df['event_timestamp'].min()
  df['time_since_start'] = (df['event_timestamp'] - reference_date).dt.total_seconds()
  
  # Example: Time relative to a specific date (e.g., today, analysis date)
  analysis_date = pd.to_datetime('2024-01-01')
  df['days_until_analysis'] = (analysis_date - df['event_timestamp']).dt.days
  

These duration features can capture information about process efficiency, customer tenure, recency of events, and other time-dependent factors.

Additional Notes

• Time Zones: If your data spans multiple time zones or involves daylight saving time transitions, ensure consistent handling. Pandas Timestamps can be timezone-aware. Standardizing to UTC is often a good practice.
• Holidays and Special Events: While not directly extracted from the timestamp itself, identifying holidays or other significant events relevant to your domain (e.g., promotion days) can be very beneficial. This often requires an external calendar or domain knowledge and falls partly under domain-specific feature engineering. You might create binary flags for holidays falling on or near the event date.
• Feature Interaction: Combine extracted temporal features (e.g., hour, is_weekend) with other existing features (e.g., product_category) to create interaction terms that might reveal more specific patterns (e.g., sales of certain products peak during specific hours on weekends).

Extracting meaningful information from date/time columns is a common and impactful feature engineering task. By converting raw timestamps into components representing trends, seasonality, cycles, and durations, you provide your machine learning models with valuable signals to improve their predictive performance. Remember to choose the components and transformations that are most relevant to the specific problem you are trying to solve.