Bayesian Optimization for Branching and Nested Hyperparameters in Deep Learning

Jiazhao Zhang

doi:10.1080/01621459.2026.2625450

Bayesian Optimization for Branching and Nested Hyperparameters in Deep Learning

2026-04-22

0

PRE

AI

DOI：10.1080/01621459.2026.2625450

Original

Original request for help

Save

Abstract

En 中文

Hyperparameter optimization plays a crucial role in the success of neural networks as hyperparameters directly control the behavior and performance of the training algorithms. To obtain efficient tuning, Bayesian optimization based on Gaussian process is widely used. Despite numerous applications in deep learning, the existing methods rely on a convenient but restrictive assumption that the tuning parameters are independent of each other. However, tuning parameters with conditional dependence are common in practice. In this article, we focus on two types of them: branching and nested parameters. Nested parameters refer to those tuning parameters that exist only within a particular setting of another tuning parameter, and a parameter that contains other nested parameters is referred to as a branching parameter. To capture the conditional dependence between branching and nested parameters, a unified Bayesian optimization framework is proposed. The sufficient conditions are rigorously derived to guarantee the validity of the kernel function, and the asymptotic convergence of the proposed optimization framework is proven under the continuum-armed-bandit setting. The new model, which accounts for the dependent structure among input variables through a new kernel function, provides a higher prediction accuracy and a better optimization efficiency in simulations and real data applications in neural networks. Supplementary materials for this article are available online, including a standardized description of the materials available for reproducing the work.

Keywords:

Bayesian optimization

Deep learning

Experimental design

Gaussian process

Bayesian Optimization for Branching and Nested Hyperparameters in Deep Learning

Abstract

Journal

Organization

Cited Papers