Model Stacking in PyTorch Tabular¶
This page demonstrates how to use model stacking functionality in PyTorch Tabular to combine multiple models for better predictions.
Setup and Imports¶
import warnings
warnings.filterwarnings("ignore")
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from pytorch_tabular import TabularModel
from pytorch_tabular.models import (
CategoryEmbeddingModelConfig,
FTTransformerConfig,
TabNetModelConfig
)
from pytorch_tabular.config import DataConfig, OptimizerConfig, TrainerConfig
from pytorch_tabular.models.stacking import StackingModelConfig
from pytorch_tabular.utils import make_mixed_dataset
Create synthetic classification dataset & split into train, validation and test sets¶
Common configurations¶
data_config = DataConfig(
target=["target"],
continuous_cols=num_col_names,
categorical_cols=cat_col_names,
)
trainer_config = TrainerConfig(
batch_size=1024,
max_epochs=20,
early_stopping="valid_accuracy",
early_stopping_mode="max",
early_stopping_patience=3,
checkpoints="valid_accuracy",
load_best=True,
)
optimizer_config = OptimizerConfig()
Configure individual models¶
model_config_1 = CategoryEmbeddingModelConfig(
task="classification",
layers="128-64-32",
activation="ReLU",
learning_rate=1e-3
)
model_config_2 = FTTransformerConfig(
task="classification",
input_embed_dim=32,
num_attn_blocks=2,
num_heads=4,
learning_rate=1e-3
)
model_config_3 = TabNetModelConfig(
task="classification",
n_d=8,
n_a=8,
n_steps=3,
learning_rate=1e-3
)
Configure Stacking Model¶
Now let's set up the stacking configuration that will combine these models:
Train Stacking Model¶
Evaluate Results¶
Compare with individual models¶
def train_and_evaluate_model(model_config, name):
model = TabularModel(
data_config=data_config,
model_config=model_config,
optimizer_config=optimizer_config,
trainer_config=trainer_config,
)
model.fit(train=train, validation=valid)
metrics = model.evaluate(test)
print(f"\n{name} Metrics:")
print(metrics)
return metrics
ce_metrics = train_and_evaluate_model(model_config_1, "Category Embedding")[0]
ft_metrics = train_and_evaluate_model(model_config_2, "FT Transformer")[0]
tab_metrics = train_and_evaluate_model(model_config_3, "TabNet")[0]
ce_acc = ce_metrics["test_accuracy"]
ft_acc = ft_metrics["test_accuracy"]
tab_acc = tab_metrics["test_accuracy"]
Save the stacking model & load it¶
Key Points About Stacking¶
- The stacking model combines predictions from multiple base models into a final prediction
- Each base model can have its own architecture and hyperparameters
- The head layer combines the outputs from all base models
- Base models are trained simultaneously
- The stacking model can often achieve better performance than individual models
Tips for Better Stacking Results¶
- Use diverse base models that capture different aspects of the data
- Experiment with different head architectures
- Consider using cross-validation for more robust stacking
- Balance model complexity with training time
- Monitor individual model performances to ensure they contribute meaningfully
This example demonstrates basic stacking functionality. For production use cases, you may want to: - Use cross-validation - Implement more sophisticated ensemble techniques - Add custom metrics - Tune hyperparameters for both base models and stacking head