Automated Hyperparameter Tuning

Was this section helpful?

References

Random Search for Hyper-Parameter Optimization, James Bergstra and Yoshua Bengio, 2012 Journal of Machine Learning Research, Vol. 13 DOI: 10.5555/2330613.2330674 - This paper introduces random search as an effective and computationally efficient alternative to grid search for hyperparameter optimization, especially when only a few hyperparameters are truly influential.
Practical Bayesian Optimization of Machine Learning Algorithms, Jasper Snoek, Hugo Larochelle, and Ryan P. Adams, 2012 Advances in Neural Information Processing Systems 25 (NIPS 2012), Vol. 25 (Curran Associates, Inc.) DOI: 10.5555/2999486.2999602 - A seminal paper presenting a practical framework for Bayesian optimization using Gaussian processes, demonstrating its superior sample efficiency for tuning machine learning algorithms.
Hyperband: A Novel Bandit-Based Approach to Hyperparameter Optimization, Lisha Li, Kevin Jamieson, Giulia DeSalvo, Afshin Rostamizadeh, and Ameet Talwalkar, 2017 Journal of Machine Learning Research, Vol. 18 DOI: 10.5555/3305381.3305404 - Introduces Hyperband, an efficient hyperparameter optimization algorithm that intelligently allocates resources using a bandit-based strategy and successive halving to prune unpromising configurations early.
Optuna: A Next-generation Hyperparameter Optimization Framework, Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama, 2019 Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD '19) DOI: 10.1145/3292500.3330705 - The original research paper introducing Optuna, an open-source hyperparameter optimization framework with a define-by-run API and efficient sampling and pruning algorithms, specifically mentioned in the section.
Hyperparameter Optimization: A Review of Methods and Applications, Bob Shahriari, Kevin Swersky, Ziyu Wang, Ryan P. Adams, and Nando de Freitas, 2016 Proceedings of the IEEE, Vol. 104 (IEEE) DOI: 10.1109/JPROC.2015.2494218 - A comprehensive review paper covering a broad range of hyperparameter optimization techniques, including grid search, random search, and Bayesian optimization, providing a strong theoretical and practical overview.