Classification Quick Reference Guide

Quick reference for classification algorithms, metrics, and best practices.

Table of Contents

• Algorithm Selection
• Code Snippets
• Evaluation Metrics
• Common Issues & Solutions
• Best Practices Checklist

Algorithm Selection

Quick Decision Tree

Algorithm Comparison

Code Snippets

Basic Classification Pipeline

from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score, classification_report

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Scale features (for Logistic Regression, SVM, KNN)
scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

# Train
model = LogisticRegression(random_state=42, max_iter=1000)
model.fit(X_train_scaled, y_train)

# Predict
y_pred = model.predict(X_test_scaled)

# Evaluate
accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.3f}")
print(classification_report(y_test, y_pred))

Logistic Regression

from sklearn.linear_model import LogisticRegression

# Basic
model = LogisticRegression(random_state=42, max_iter=1000)
model.fit(X_train_scaled, y_train)

# With class weights (for imbalanced data)
model = LogisticRegression(
    class_weight='balanced',
    random_state=42,
    max_iter=1000
)

# Multiclass
model = LogisticRegression(
    multi_class='multinomial',
    solver='lbfgs',
    random_state=42
)

Decision Trees

from sklearn.tree import DecisionTreeClassifier

# Basic
tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_train, y_train)

# With hyperparameters
tree = DecisionTreeClassifier(
    max_depth=5,
    min_samples_split=10,
    min_samples_leaf=5,
    criterion='gini',  # or 'entropy'
    random_state=42
)

# Visualize tree
from sklearn.tree import plot_tree
import matplotlib.pyplot as plt
plt.figure(figsize=(20, 10))
plot_tree(tree, filled=True, feature_names=feature_names)
plt.show()

Random Forests

from sklearn.ensemble import RandomForestClassifier

# Basic
rf = RandomForestClassifier(n_estimators=100, random_state=42)
rf.fit(X_train, y_train)

# With hyperparameters
rf = RandomForestClassifier(
    n_estimators=200,
    max_depth=10,
    min_samples_split=5,
    min_samples_leaf=2,
    max_features='sqrt',
    class_weight='balanced',  # For imbalanced data
    random_state=42
)

# Feature importance
feature_importance = pd.DataFrame({
    'feature': feature_names,
    'importance': rf.feature_importances_
}).sort_values('importance', ascending=False)

SVM

from sklearn.svm import SVC

# Linear SVM
svm = SVC(kernel='linear', random_state=42)
svm.fit(X_train_scaled, y_train)

# RBF kernel (most common)
svm = SVC(
    kernel='rbf',
    C=1.0,
    gamma='scale',
    probability=True,  # Enable predict_proba
    random_state=42
)

# Polynomial kernel
svm = SVC(kernel='poly', degree=3, random_state=42)

KNN

from sklearn.neighbors import KNeighborsClassifier

# Basic
knn = KNeighborsClassifier(n_neighbors=5)
knn.fit(X_train_scaled, y_train)

# With hyperparameters
knn = KNeighborsClassifier(
    n_neighbors=5,
    weights='distance',  # or 'uniform'
    metric='euclidean',  # or 'manhattan', 'minkowski'
    algorithm='auto'
)

XGBoost

import xgboost as xgb

# Basic
xgb_clf = xgb.XGBClassifier(random_state=42)
xgb_clf.fit(X_train, y_train)

# With hyperparameters
xgb_clf = xgb.XGBClassifier(
    n_estimators=100,
    max_depth=5,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    random_state=42
)

Naive Bayes

from sklearn.naive_bayes import GaussianNB, MultinomialNB, BernoulliNB

# Gaussian (continuous data)
gnb = GaussianNB()
gnb.fit(X_train, y_train)

# Multinomial (discrete counts, text)
mnb = MultinomialNB()
mnb.fit(X_train_counts, y_train)  # Use CountVectorizer/TF-IDF

# Bernoulli (binary features)
bnb = BernoulliNB()
bnb.fit(X_binary, y_train)

Multi-Class Classification Strategies

from sklearn.multiclass import OneVsRestClassifier, OneVsOneClassifier

# One-vs-Rest (OvR): Train N binary classifiers
ovr = OneVsRestClassifier(LogisticRegression())
ovr.fit(X_train, y_train)

# One-vs-One (OvO): Train N(N-1)/2 binary classifiers
ovo = OneVsOneClassifier(SVC())
ovo.fit(X_train, y_train)

# Most algorithms handle multi-class automatically
# LogisticRegression: multi_class='multinomial' or 'ovr'
# SVC: decisi or 'ovo'

Evaluation Metrics

Confusion Matrix

from sklearn.metrics import confusion_matrix, ConfusionMatrixDisplay

cm = confusion_matrix(y_test, y_pred)
print(cm)

# Visualize
disp = ConfusionMatrixDisplay(confusion_matrix=cm)
disp.plot()
plt.show()

Accuracy

from sklearn.metrics import accuracy_score

accuracy = accuracy_score(y_test, y_pred)
print(f"Accuracy: {accuracy:.3f}")

# Warning: Misleading for imbalanced data!

Precision, Recall, F1-Score

from sklearn.metrics import precision_score, recall_score, f1_score

# Binary classification
precision = precision_score(y_test, y_pred)
recall = recall_score(y_test, y_pred)
f1 = f1_score(y_test, y_pred)

# Multiclass (weighted average)
precision = precision_score(y_test, y_pred, average='weighted')
recall = recall_score(y_test, y_pred, average='weighted')
f1 = f1_score(y_test, y_pred, average='weighted')

ROC-AUC (Binary Only)

from sklearn.metrics import roc_auc_score, roc_curve

# Get probabilities
y_pred_proba = model.predict_proba(X_test)[:, 1]

# Calculate AUC
auc = roc_auc_score(y_test, y_pred_proba)
print(f"AUC: {auc:.3f}")

# Plot ROC curve
fpr, tpr, _ = roc_curve(y_test, y_pred_proba)
plt.plot(fpr, tpr, label=f'AUC = {auc:.3f}')
plt.plot([0, 1], [0, 1], 'k--')
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.legend()
plt.show()

Classification Report

from sklearn.metrics import classification_report

print(classification_report(y_test, y_pred))
print(classification_report(y_test, y_pred, target_names=class_names))

Complete Evaluation Function

def evaluate_classification(y_true, y_pred, y_pred_proba=None):
    """Comprehensive classification evaluation"""
    from sklearn.metrics import (accuracy_score, precision_score, 
                                recall_score, f1_score, roc_auc_score)
    
    metrics = {
        'accuracy': accuracy_score(y_true, y_pred),
        'precision': precision_score(y_true, y_pred, average='weighted'),
        'recall': recall_score(y_true, y_pred, average='weighted'),
        'f1': f1_score(y_true, y_pred, average='weighted')
    }
    
    if y_pred_proba is not None and len(np.unique(y_true)) == 2:
        metrics['roc_auc'] = roc_auc_score(y_true, y_pred_proba)
    
    return metrics

Common Issues & Solutions

Issue 1: Imbalanced Data

Problem: Model always predicts majority class

Solutions:

# Solution 1: Class weights
model = LogisticRegression(class_weight='balanced')

# Solution 2: SMOTE oversampling
from imblearn.over_sampling import SMOTE
smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

# Solution 3: Use appropriate metrics
# Don't use accuracy! Use F1-score or AUC
f1 = f1_score(y_test, y_pred)
auc = roc_auc_score(y_test, y_pred_proba)

Issue 2: Low Accuracy

Problem: Model performs poorly

Solutions:

# Solution 1: Try different algorithms
models = [LogisticRegression(), RandomForestClassifier(), SVC()]

# Solution 2: Feature engineering
from sklearn.preprocessing import PolynomialFeatures
poly = PolynomialFeatures(degree=2)
X_poly = poly.fit_transform(X)

# Solution 3: Hyperparameter tuning
from sklearn.model_selection import GridSearchCV
param_grid = {'C': [0.1, 1, 10]}
grid_search = GridSearchCV(model, param_grid, cv=5)
grid_search.fit(X_train, y_train)

Issue 3: Overfitting

Problem: High training accuracy, low test accuracy

Solutions:

# Solution 1: Regularization
model = LogisticRegression(C=0.1)  # Lower C = more regularization

# Solution 2: Reduce model complexity
tree = DecisionTreeClassifier(max_depth=5, min_samples_split=10)

# Solution 3: Use ensemble methods
rf = RandomForestClassifier(n_estimators=100, max_depth=10)

# Solution 4: Cross-validation
from sklearn.model_selection import cross_val_score
scores = cross_val_score(model, X_train, y_train, cv=5)

Issue 4: Slow Training

Problem: Model takes too long to train

Solutions:

# Solution 1: Reduce dataset size
X_train_small = X_train[:10000]
y_train_small = y_train[:10000]

# Solution 2: Use faster algorithms
# Logistic Regression > Random Forest > SVM

# Solution 3: Reduce features
from sklearn.feature_selection import SelectKBest
selector = SelectKBest(k=50)
X_selected = selector.fit_transform(X_train, y_train)

# Solution 4: Parallel processing
model = RandomForestClassifier(n_jobs=-1)  # Use all CPUs

Issue 5: Memory Error

Problem: Out of memory when training

Solutions:

# Solution 1: Use batch processing
# Process data in chunks

# Solution 2: Use memory-efficient algorithms
# Logistic Regression uses less memory than Random Forest

# Solution 3: Reduce data size
# Sample data or use feature selection

Best Practices Checklist

Data Preparation

• Check for missing values
• Handle missing values appropriately
• Encode categorical variables
• Scale features (for Logistic Regression, SVM, KNN)
• Check class distribution (imbalanced data?)
• Use stratified train-test split

Model Selection

• Start with simple baseline (Logistic Regression)
• Try multiple algorithms
• Compare models using cross-validation
• Choose model based on problem requirements

Evaluation

• Use appropriate metrics (F1, AUC for imbalanced data)
• Don't rely only on accuracy
• Check confusion matrix
• Use cross-validation for robust evaluation
• Plot ROC curve for binary classification

Improvement

• Handle class imbalance if present
• Perform feature engineering
• Tune hyperparameters
• Try ensemble methods
• Regularize to prevent overfitting

Deployment

• Save model and preprocessing steps
• Document model performance
• Create prediction function
• Test on new data
• Monitor model performance

Quick Tips

1. Always use stratified split for imbalanced data
2. Scale features for distance-based algorithms (SVM, KNN, Logistic Regression)
3. Use F1-score or AUC instead of accuracy for imbalanced data
4. Start simple - Logistic Regression is a great baseline
5. Cross-validate for robust evaluation
6. Visualize - Confusion matrix, ROC curve, feature importance
7. Handle imbalance - Class weights, SMOTE, or threshold tuning
8. Tune hyperparameters - Grid search or random search
9. Try ensembles - Random Forest, XGBoost often perform better
10. Save everything - Model, scaler, feature selector

Common Mistakes to Avoid

1. Using accuracy for imbalanced data
2. Not scaling features for SVM/KNN
3. Data leakage (scaling before split)
4. Overfitting (too complex model)
5. Not using cross-validation
6. Ignoring class imbalance
7. Not trying multiple algorithms
8. Using test set for hyperparameter tuning
9. Not saving preprocessing steps
10. Not checking confusion matrix

Resources

• Scikit-learn Classification Guide
• Classification Metrics
• Handling Imbalanced Data

Try next: Report confusion matrix plus one business metric, not accuracy alone.