Nabid In Motion Nabid In Motion

Study interactive :: Progress tools open in the Study Hub reader.

Common Pitfalls and Best Practices in Machine Learning

Essential guide to avoiding common mistakes and following best practices when building ML models.

Table of Contents

Pitfall 1: Data Leakage

What it is: Using information from test set or future data during training.

Examples:

# WRONG: Scaling before splitting
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # Uses all data including test!
X_train, X_test = train_test_split(X_scaled, y)

# CORRECT: Split first, then scale
X_train, X_test, y_train, y_test = train_test_split(X, y)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Fit only on train
X_test_scaled = scaler.transform(X_test)  # Transform test

Common Leakage Sources:

How to Avoid:

Pitfall 2: Insufficient Data

Problem, not enough data for model to learn patterns.

Signs:

Solutions:

Pitfall 3: Imbalanced Data

Problem: Classes are not equally represented.

Example:

# 99% class A, 1% class B
# Model predicts A all the time → 99% accuracy but useless!

Solutions:

Pitfall 4: Poor Data Quality

Problems:

Solutions:

Model Training Pitfalls

Pitfall 5: Overfitting

What it is: Model memorizes training data, fails on new data.

Signs:

train_accuracy = 0.99  # Perfect on training
test_accuracy = 0.65  # Poor on test
# Large gap = overfitting!

Solutions:

Pitfall 6: Underfitting

What it is: Model too simple to capture patterns.

Signs:

train_accuracy = 0.55  # Poor on training
test_accuracy = 0.53   # Poor on test
# Both low = underfitting

Solutions:

Pitfall 7: Wrong Algorithm Choice

Problem: Using algorithm that doesn't fit the problem.

Examples:

Solution: Follow algorithm selection guide, start simple.

Pitfall 8: Ignoring Hyperparameters

Problem: Using default hyperparameters without tuning.

Impact: Suboptimal performance

Solution: Use validation set or cross-validation to tune hyperparameters.

# WRONG: Use defaults without checking
model = RandomForestClassifier()
model.fit(X_train, y_train)

# BETTER: Tune hyperparameters
from sklearn.model_selection import GridSearchCV

param_grid = {
    'n_estimators': [50, 100, 200],
    'max_depth': [3, 5, 10]
}
grid_search = GridSearchCV(model, param_grid, cv=5)
grid_search.fit(X_train, y_train)
best_model = grid_search.best_estimator_

Evaluation Pitfalls

Pitfall 9: Wrong Evaluation Metric

Problem: Using metric that doesn't match the problem.

Examples:

Solutions:

Pitfall 10: No Validation Set

Problem: Only using train/test split, no validation for tuning.

Impact: Overfitting to test set, unrealistic performance estimates

Solution: Use train/validation/test split or cross-validation.

# CORRECT: Three-way split
X_train, X_temp, y_train, y_temp = train_test_split(X, y, test_size=0.4)
X_val, X_test, y_val, y_test = train_test_split(X_temp, y_temp, test_size=0.5)

# Use validation for tuning
# Use test only for final evaluation

Pitfall 11: Testing on Training Data

Problem: Evaluating model on data it was trained on.

Why it's wrong: Gives overly optimistic results, doesn't reflect real performance.

Solution: Always use separate test set that model has never seen.

Pitfall 12, not Checking for Data Drift

Problem: Model performance degrades over time as data changes.

Solution: Monitor model performance, retrain periodically.

Deployment Pitfalls

Pitfall 13, not Versioning Models

Problem: Can't reproduce results or rollback if needed.

Solution: Version control for models, data, and code.

# Use MLflow or similar
import mlflow
mlflow.log_model(model, "iris_classifier")
mlflow.log_params(model.get_params())

Pitfall 14: Ignoring Model Interpretability

Problem: Can't explain predictions when needed.

Solution: Use interpretable models or explainability tools (SHAP, LIME).

Pitfall 15: No Monitoring

Problem: Don't know when model performance degrades.

Solution: Set up monitoring for:

Best Practices

1. Start Simple

Principle: Begin with the simplest solution that works.

Why:

Example:

1. Try Linear Regression first
2. If not good enough, try Random Forest
3. If still not enough, try XGBoost
4. Last resort: Neural Networks

2. Follow the Workflow

Structured Approach:

  1. Problem definition
  2. Data collection and exploration
  3. Data preparation
  4. Model selection and training
  5. Evaluation
  6. Deployment
  7. Monitoring

Don't skip steps!

3. Validate Everything

Check:

4. Use Cross-Validation

Why:

from sklearn.model_selection import cross_val_score

scores = cross_val_score(model, X, y, cv=5)
print(f"Mean CV Score: {scores.mean():.4f} (+/- {scores.std()*2:.4f})")

5. Document Everything

Document:

6. Version Control

Track:

7. Test on Unseen Data

Always:

8. Consider Business Context

Ask:

9. Iterate and Improve

Process:

  1. Build baseline model
  2. Evaluate and identify issues
  3. Improve (more data, better features, different model)
  4. Repeat

10. Understand Your Model

Know:

Checklist

Before Training

During Training

After Training

Before Deployment

Resources

Key Takeaways

  1. Avoid Data Leakage: Always split before preprocessing
  2. Start Simple: Don't overcomplicate
  3. Validate Properly: Use appropriate metrics and validation
  4. Document: Keep track of everything
  5. Iterate: ML is iterative, not one-shot
  6. Understand: Know your model and data

Try next: Pick one pitfall from this page and show it failing on a tiny toy dataset.

Recall ::

When must you split train/test relative to preprocessing?

Split first. Fit preprocessors only on train to avoid leakage.