Contractivity of neural ODEs: An eigenvalue optimization problem

We propose a novel methodology to solve a key eigenvalue optimization problem which arises in the contractivity analysis of neural ordinary differential equations (ODEs). When looking at contractivity properties of a one-layer weight-tied neural ODE u ˙ ( t ) = σ ( A u ( t ) + b ) \dot {u}(t)=\sigma (Au(t)+b) (with u , b ∈ R n u,b \in \mathbb {R}^n , A A is a given n × n n \times n matrix, σ : R → R \sigma : \mathbb {R}\to \mathbb {R} denotes an activation function and for a vector z ∈ R n z \in \mathbb {R}^n , σ ( z ) ∈ R n \sigma (z) \in \mathbb {R}^n has to be interpreted entry-wise), we are led to study the logarithmic norm of a set of products of type D A D A , where D D is a diagonal matrix such that d i a g ( D ) ∈ σ ′ ( R n ) diag(D) \in \sigma ’(\mathbb {R}^n) . Specifically, given a real number c c (usually c = 0 c=0 ), the problem consists in finding the largest positive interval I ⊆ [ 0 , ∞ ) \mathrm {I}\subseteq \mathbb [0,\infty ) such that the logarithmic norm μ ( D A ) ≤ c \mu (DA) \le c for all diagonal matrices D D with D i i ∈ I D_{ii}\in \mathrm {I} . We propose a two-level nested methodology: an inner level where, for a given I \mathrm {I} , we compute an optimizer D ⋆ ( I ) D^\star (\mathrm {I}) by a gradient system approach, and an outer level where we tune I \mathrm {I} so that the value c c is reached by μ ( D ⋆ ( I ) A ) \mu (D^\star (\mathrm {I})A) . We extend the proposed two-level approach to the general multilayer, and possibly time-dependent, case u ˙ ( t ) = σ ( A k ( t ) … σ ( A 1 ( t ) u ( t ) + b 1 ( t ) ) … + b k ( t ) ) \dot {u}(t) = \sigma ( A_k(t) \ldots \sigma ( A_{1}(t) u(t) + b_{1}(t) ) \ldots + b_{k}(t) ) and we propose several numerical examples to illustrate its behaviour, including its stabilizing performance on a one-layer neural ODE applied to the classification of the MNIST handwritten digits dataset.