You've learned about Autoregressive (AR) models, which predict future values based on past values, and Moving Average (MA) models, which predict future values based on past forecast errors. While useful individually, many real-world stationary time series exhibit correlation structures that aren't perfectly captured by a pure AR or pure MA process alone. The dynamics might depend both on recent values and recent shocks or errors.

This is where the Autoregressive Moving Average (ARMA) model comes in. It provides a more flexible framework by combining both AR and MA components into a single model. An ARMA model assumes the current value of the series depends linearly on its own previous values and on previous error terms.

The ARMA(p, q) Model Structure

An ARMA model is denoted as ARMA(p, q), where:

p is the order of the Autoregressive (AR) part.
q is the order of the Moving Average (MA) part.

The equation for an ARMA(p, q) model for a time series $Y_t$ is:

Y_t = c + \phi_1 Y_{t-1} + \phi_2 Y_{t-2} + ... + \phi_p Y_{t-p} + \theta_1 \epsilon_{t-1} + \theta_2 \epsilon_{t-2} + ... + \theta_q \epsilon_{t-q} + \epsilon_t

Let's break this down:

$Y_t$ : The value of the time series at time $t$ .
$c$ : A constant term (baseline or intercept).
$\phi_1, ..., \phi_p$ : The coefficients for the AR terms ( $p$ past values $Y_{t-1}$ to $Y_{t-p}$ ).
$\theta_1, ..., \theta_q$ : The coefficients for the MA terms ( $q$ past error terms $\epsilon_{t-1}$ to $\epsilon_{t-q}$ ).
$\epsilon_t$ : The white noise error term at time $t$ . This represents the unpredictable part of the series at time $t$ .
$\epsilon_{t-1}, ..., \epsilon_{t-q}$ : Past error terms. These are the differences between the actual values and the model's forecasts at previous time points.

Essentially, the model states that the current observation $Y_t$ is a weighted sum of $p$ past observations (the AR part), a weighted sum of $q$ past forecast errors (the MA part), a constant, and the current error term.

Flow of an ARMA(p, q) model, showing how past values and past errors contribute to predicting the current value.

Stationarity Assumption

A significant prerequisite for applying an ARMA model directly is that the time series $Y_t$ must be stationary. Recall from Chapter 2 that stationarity implies the series has a constant mean, constant variance, and constant autocorrelation structure over time. If your series exhibits trends or seasonality, it's non-stationary, and an ARMA model is not appropriate without first transforming the data to achieve stationarity (which leads us towards ARIMA models, discussed next).

Connecting ARMA to ACF and PACF

In Chapter 3, you learned how Autocorrelation Function (ACF) and Partial Autocorrelation Function (PACF) plots help identify the structure of a stationary time series. Let's revisit their characteristic patterns in the context of ARMA models:

Pure AR(p) Model:
- ACF: Tails off gradually (exponentially or sinusoidally).
- PACF: Cuts off sharply after lag p.
Pure MA(q) Model:
- ACF: Cuts off sharply after lag q.
- PACF: Tails off gradually.
ARMA(p, q) Model:
- ACF: Tails off gradually after lag q.
- PACF: Tails off gradually after lag p.

For a mixed ARMA process, both the ACF and PACF typically decay gradually towards zero. The specific rates of decay can offer hints about p and q, but interpreting these patterns for ARMA models is generally more challenging than for pure AR or MA processes. Often, you'll use these plots to get an initial idea for plausible p and q values, and then rely on model diagnostics and information criteria (like AIC or BIC, covered in Chapter 6) to refine the model order during the fitting process.

By combining AR and MA terms, ARMA models can capture a wider range of temporal dependencies in stationary data than either AR or MA models alone. This makes them a valuable tool in your time series analysis toolkit. However, the requirement for stationarity is a limitation we'll address next by introducing the 'Integrated' component, leading us to the full ARIMA model.