Improving Model Accuracy

1. Data Preprocessing and Feature Engineering

Before delving into model tuning, it's crucial to ensure that your data is well-prepared. Data preprocessing involves handling missing values, scaling features, and encoding categorical variables. Feature engineering, on the other hand, involves creating new features or modifying existing ones to better capture the underlying patterns in the data.

from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.pipeline import Pipeline

# Creating a pipeline for data preprocessing
preprocessing_pipeline = Pipeline([
    ('imputer', SimpleImputer(strategy='mean')),  # Fill missing values
    ('scaler', StandardScaler())  # Standardize features
])

# Fit and transform the data
X_preprocessed = preprocessing_pipeline.fit_transform(X)

Data preprocessing pipeline

Effective feature engineering can significantly enhance model accuracy. Consider polynomial features for linear models or interaction terms that capture the relationship between different variables.

2. Model Selection

Choosing the right model is crucial. Different algorithms have varying strengths and weaknesses depending on the data characteristics. For instance, decision trees might capture non-linear patterns well but can overfit, while linear models might generalize better with simpler patterns.

Use Scikit-Learn's model selection utilities, such as GridSearchCV or RandomizedSearchCV, to find the optimal algorithm and hyperparameters:

from sklearn.model_selection import GridSearchCV
from sklearn.ensemble import RandomForestClassifier

# Define parameter grid
param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [None, 10, 20, 30]
}

# Initialize a Random Forest Classifier
rf_model = RandomForestClassifier()

# Grid search for best parameters
grid_search = GridSearchCV(estimator=rf_model, param_grid=param_grid, cv=5)
grid_search.fit(X_preprocessed, y)

# Best parameters and accuracy
print("Best Parameters:", grid_search.best_params_)
print("Best Accuracy:", grid_search.best_score_)

3. Hyperparameter Tuning

Hyperparameters are external configurations to a model, influencing how the model is trained. Tuning these values can significantly impact model accuracy. Techniques like grid search or random search allow you to explore different configurations systematically.

from sklearn.model_selection import RandomizedSearchCV

# Randomized search for hyperparameter tuning
random_search = RandomizedSearchCV(estimator=rf_model, param_distributions=param_grid, n_iter=10, cv=5)
random_search.fit(X_preprocessed, y)

print("Best Parameters from Randomized Search:", random_search.best_params_)

4. Ensemble Methods

Ensemble methods, such as bagging and boosting, combine predictions from multiple models to improve accuracy. Bagging methods like Random Forests reduce variance, while boosting methods like AdaBoost or Gradient Boosting focus on errors from previous iterations.

from sklearn.ensemble import GradientBoostingClassifier

# Training a Gradient Boosting Classifier
gb_model = GradientBoostingClassifier(n_estimators=100, learning_rate=0.1)
gb_model.fit(X_preprocessed, y)

# Evaluating model
accuracy = gb_model.score(X_test_preprocessed, y_test)
print("Gradient Boosting Classifier Accuracy:", accuracy)

Gradient Boosting ensemble method

5. Cross-Validation

Cross-validation is indispensable for evaluating model performance robustly. It helps prevent overfitting by ensuring that the model generalizes well to unseen data. Use k-fold cross-validation to assess the stability of your model's accuracy.

from sklearn.model_selection import cross_val_score

# 5-fold cross-validation
cv_scores = cross_val_score(gb_model, X_preprocessed, y, cv=5)

print("Cross-Validation Scores:", cv_scores)
print("Mean Accuracy:", cv_scores.mean())

Cross-validation scores across folds

6. Handling Class Imbalance

Class imbalance can skew model accuracy. Techniques such as resampling, synthetic data generation (SMOTE), or adjusting class weights are effective strategies to address this issue.

from imblearn.over_sampling import SMOTE

# Applying SMOTE for handling class imbalance
sm = SMOTE(random_state=42)
X_resampled, y_resampled = sm.fit_resample(X_preprocessed, y)

# Verify resampling
print("Original dataset shape:", y.shape)
print("Resampled dataset shape:", y_resampled.shape)

Class imbalance before resampling

Class balance after SMOTE resampling

By leveraging these techniques, you can systematically improve your model's accuracy, leading to more reliable and robust predictive insights. Remember that improving accuracy often requires an iterative approach, where you continuously refine your data, model selection, and tuning strategies.