Importing the Library¶
Hyperparameter Tuning¶
data_config = DataConfig(
target=[
"target"
], # target should always be a list. Multi-targets are only supported for regression. Multi-Task Classification is not implemented
continuous_cols=num_col_names,
categorical_cols=cat_col_names,
)
trainer_config = TrainerConfig(
batch_size=1024,
max_epochs=100,
early_stopping="valid_loss", # Monitor valid_loss for early stopping
early_stopping_mode="min", # Set the mode as min because for val_loss, lower is better
early_stopping_patience=5, # No. of epochs of degradation training will wait before terminating
checkpoints="valid_loss", # Save best checkpoint monitoring val_loss
load_best=True, # After training, load the best checkpoint
progress_bar="none", # Turning off Progress bar
trainer_kwargs=dict(enable_model_summary=False), # Turning off model summary
)
optimizer_config = OptimizerConfig()
head_config = LinearHeadConfig(
layers="", dropout=0.1, initialization="kaiming" # No additional layer in head, just a mapping layer to output_dim
).__dict__ # Convert to dict to pass to the model config (OmegaConf doesn't accept objects)
model_config = CategoryEmbeddingModelConfig(
task="classification",
layers="1024-512-512", # Number of nodes in each layer
activation="LeakyReLU", # Activation between each layers
learning_rate=1e-3,
head="LinearHead", # Linear Head
head_config=head_config, # Linear Head Config
)
Grid Search¶
Note: For demonstration we are using the test split for tuning, but in real problems, please use a separate validation set for tuning purposes. Otherwise, you will be overfitting to the test set and have falsely high performance estimates.
Define the Hyperparameter Space¶
The hyperparameter space is defined as a dictionary. The keys are the hyperparameter names and the values are the list of values to be tried. The hyparameter names follow the below convention:
- model_config__<hyperparameter_name> for model hyperparameters
- model_config.head_config__<hyperparameter_name> for head hyperparameters
- trainer_config__<hyperparameter_name> for trainer hyperparameters
- optimizer_config__<hyperparameter_name> for optimizer hyperparameters
- We can't use data module hyperparameters for tuning as the datamodule is already fitted and we can't change it's hyperparameters.
search_space = {
"model_config__layers": ["1024-512-512", "1024-512-256", "1024-512-128"],
"model_config.head_config__dropout": [0.1, 0.2, 0.3],
"trainer_config__batch_size": [1024, 2048, 4096],
"optimizer_config__optimizer": ["RAdam", "AdamW"],
}
# Any other parameter which is not part of the search_space, will be kept constant during the search
tuner = TabularModelTuner(
data_config=data_config,
model_config=model_config,
optimizer_config=optimizer_config,
trainer_config=trainer_config
)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
result = tuner.tune(
train=train,
validation=test,
search_space=search_space,
strategy="grid_search",
# cv=5, # Uncomment this to do a 5 fold cross validation
metric="accuracy",
mode="max",
progress_bar=True,
verbose=False # Make True if you want to log metrics and params each iteration
)
Result is a namedtuple with trials_df, best_params, best_score and best_model\
- trials_df: A dataframe with all the hyperparameter combinations and their corresponding scores
- best_params: The best hyperparameter combination
- best_score: The best score
- best_model: If return_best_model is True, return best_model otherwise return None
Random Search¶
Note: For demonstration we are using the test split for tuning, but in real problems, please use a separate validation set for tuning purposes. Otherwise, you will be overfitting to the test set and have falsely high performance estimates.
Define the Hyperparameter Space¶
The hyperparameter space is defined as a dictionary. The keys are the hyperparameter names and the values are the list of values for categorical and distributions for continuous. The hyparameter names follow the below convention:
- model_config__<hyperparameter_name> for model hyperparameters
- model_config.head_config__<hyperparameter_name> for head hyperparameters
- trainer_config__<hyperparameter_name> for trainer hyperparameters
- optimizer_config__<hyperparameter_name> for optimizer hyperparameters
- We can't use data module hyperparameters for tuning as the datamodule is already fitted and we can't change it's hyperparameters.
from scipy.stats import uniform, randint, loguniform
search_space = {
"model_config__layers": ["1024-512-512", "1024-512-256", "1024-512-128"],
"model_config.head_config__dropout": uniform(0, 0.5),
"trainer_config__batch_size": randint(128, 2048),
"optimizer_config__optimizer": ["RAdam", "AdamW"],
}
# Any other parameter which is not part of the search_space, will be kept constant during the search
tuner = TabularModelTuner(
data_config=data_config,
model_config=model_config,
optimizer_config=optimizer_config,
trainer_config=trainer_config
)
with warnings.catch_warnings():
warnings.simplefilter("ignore")
result = tuner.tune(
train=train,
validation=test, # Need not give validation is we use CV
search_space=search_space,
n_trials=10,
strategy="random_search",
# cv=5, # Uncomment this to do a 5 fold cross validation
metric="accuracy",
mode="max",
progress_bar=True,
verbose=False # Make True if you want to log metrics and params each iteration
)
Result is a namedtuple with trials_df, best_params, best_score and best_model\
- trials_df: A dataframe with all the hyperparameter combinations and their corresponding scores
- best_params: The best hyperparameter combination
- best_score: The best score
- best_model: If return_best_model is True, return best_model otherwise return None