What are the common strategies for hyperparameter tuning in deep learning?
Answer
Hyperparameter tuning optimizes the configuration settings that control the learning process, such as learning rate, batch size, and architecture choices. Because each evaluation requires training a model, the goal is to find strong configurations with as few trials as possible, which makes sample-efficient search strategies essential.
(1) Manual/Heuristic Search: Start with values from prior work or common practice and iteratively adjust based on validation performance.
(2) Grid Search: Exhaustively evaluate all combinations over a predefined discrete grid; simple but scales poorly with dimensionality.
(3) Random Search: Randomly sample values from predefined ranges; covers more distinct values per hyperparameter than grid search for the same budget.
(4) Bayesian Optimization: Use a probabilistic surrogate model to intelligently suggest the next configuration, balancing exploration vs exploitation.
Figure 1: With 9 trials, grid search tests only 3 distinct learning rates, while random search covers 9 distinct values, which matters when one hyperparameter is much more important than the others.
Mathematical Formulation:
![EI(x) = \mathbb{E}\left[\max\left(f(x) - f(x^*),\ 0\right)\right]](https://s0.wp.com/latex.php?latex=EI%28x%29+%3D+%5Cmathbb%7BE%7D%5Cleft%5B%5Cmax%5Cleft%28f%28x%29+-+f%28x%5E%2A%29%2C%5C+0%5Cright%29%5Cright%5D&bg=ffffff&fg=000&s=3&c=20201002)
Where:
is the Expected Improvement acquisition function: the expected amount by which 
exceeds the best observed score 
(for maximization).
is the acquisition function (e.g., EI or UCB) computed from the surrogate model fitted on past trials 
; maximizing it picks the next configuration to evaluate.
is the Expected Improvement acquisition function: the expected amount by which 
exceeds the best observed score 
(for maximization).
is the acquisition function (e.g., EI or UCB) computed from the surrogate model fitted on past trials 
; maximizing it picks the next configuration to evaluate.
Figure 2: Bayesian optimization iterates: fit a surrogate model, pick the next point via an acquisition function, evaluate it, and update the model, repeating until the trial budget is spent.
Hyperparameter Interactions: Validation accuracy is non-monotonic in learning rate, and the optimum shifts with batch size. That is one reason joint search beats tuning one hyperparameter at a time.
Figure 3: Accuracy peaks in the same best lr region for both batch sizes, but the curves differ: hyperparameters interact and should be tuned jointly.
Leave a Reply