Training a Deep Learning model can get arbritarily complex. PyTorch Tabular, by inheriting PyTorch Lightning, offloads the whole workload onto the underlying PyTorch Lightning Framework. It has been made to make training your models easy as a breeze and at the same time give you the flexibility to make the training process your own.
The trainer in PyTorch Tabular, although haven't inherited all the features of the Pytorch Lightning trainer, have adopted quite a bit.
The parameters that you would set most frequently are:
batch_size: int: Number of samples in each batch of training. Defaults to
max_epochs: int: Maximum number of epochs to be run. The maximum is in case of Early Stopping where this becomes the maximum and without Early Stopping, this is the number of epochs that will be run Defaults to
gpus: int: The index of the GPU to be used. If zero, will use CPU. Defaults to
load_best: int: Flag to load the best model saved during training. This will be ignored if checkpoint saving is turned off. Defaults to True
trainer_config = TrainerConfig(batch_size=64, max_epochs=10, gpus=1)
PyTorch Tabular uses Early Stopping by default and monitors
valid_loss to stop training. Checkpoint saving is also turned on by default, which monitors
valid_loss and saved the best model in a folder
saved_models. All of these are configurable as we will see in the next section.
Early Stopping and Checkpoint Saving
Early Stopping is turned on by default. But you can turn it off by setting
None. On the other hand, if you want to monitor some other metric, you just need to give that metric name in the
early_stopping parameter. Few other paramters that controls early stopping are:
early_stopping_min_delta: float: The minimum delta in the loss/metric which qualifies as an improvement in early stopping. Defaults to
early_stopping_mode: str: The direction in which the loss/metric should be optimized. Choices are
min. Defaults to
early_stopping_patience: int: The number of epochs to wait until there is no further improvements in loss/metric. Defaults to
min_epochs: int: Minimum number of epochs to be run. This many epochs are run regardless of the stopping criteria. Defaults to
Checkpoint Saving is also turned on by default and to turn it off you can set the
checkpoints parameter to
None. If you want to monitor some other metric, you just need to give that metric name in the
early_stopping parameter. Few other paramters that controls checkpoint saving are:
checkpoints_path: str: The path where the saved models will be. Defaults to
checkpoints_mode: str: The direction in which the loss/metric should be optimized. Choices are
min. Defaults to
checkpoints_save_top_k: int: The number of best models to save. If you want to save more than one best models, you can set this parameter to >1. Defaults to
Make sure the name of the metric/loss you want to track exactly matches the ones in the logs. Recommended way is to run a model and cehck the results by evaluating the model. From the resulting dictionary, you can pick up a key to track during training.
Learning Rate Finder
First proposed in this paper Cyclical Learning Rates for Training Neural Networks and the subsequently popularized by fast.ai, is a technique to reach the neighbourhood of optimum learning rate without costly search. PyTorch Tabular let's you find the optimal learning rate(using the method proposed in the paper) and automatically use that for training the network. All this can be turned on with a sinple flag
Controlling the Gradients/Optimization
While training, there can be situations where you need to have a heavier control on the gradient optimization process. For eg. if the gradients are exploding, you might want to clip gradient values before each update.
gradient_clip_val let's you do that.
Sometimes, you might want to accumulate gradients across multiple batches before you do a backward propoagation(may be because a larger batch size does not fit in your GPU). PyTorch Tabular let's you do this with
Many times, you will need to debug a model and see why it is not performing as it is supposed to. Or even, while developing new models, you will need to debug the model a lot. PyTorch Lightning has a few features for this usecase, which Pytorch Tabular has adopted.
To find out performance bottle necks, we can use:
profiler: Optional[str]: To profile individual steps during training and assist in identifying bottlenecks. Choices are:
advanced. Defaults to
To check if the whole setup runs without errors, we can use:
fast_dev_run: Optional[str]: Quick Debug Run of Val. Defaults to
If the model is not learning properly:
overfit_batches: float: Uses this much data of the training set. If nonzero, will use the same training set for validation and testing. If the training dataloaders have shuffle=True, Lightning will automatically disable it. Useful for quickly debugging or trying to overfit on purpose. Defaults to
track_grad_norm: bool: This is only used if experiment tracking is setup. Track and Log Gradient Norms in the logger. -1 by default means no tracking. 1 for the L1 norm, 2 for L2 norm, etc. Defaults to
False. If the gradient norm falls to zero quickly, then we have a problem.
For a complete list of parameters refer to the API Docs