Convex Formulations of Finite-Width Neural Networks

Essay is available at link to pdf.

The essay is structured around two complementary viewpoints on neural network architecture:

convex reformulations of finite-width two-layer ReLU networks and geometric interpretations based on Clifford algebra. A central part of the work involved understanding the relationship between these formulations. I provided simple examples and geometric illustrations of the selected features in a simple two-dimensional setting, for both the convex formulation and the formulation derived using Clifford Algebra.

Below is the illustration. ((x) denotes data points and (k) is related to optimal weights)

I also conducted numerical experiments to compare convex optimisation with stochastic gradient descent in both fixed- and random-design settings. These experiments illustrate the gap between the global optimum of the convex formulation and the solutions obtained by non-convex gradient descent and stochastic gradient descent across various settings.

Through this project, I developed a deeper understanding of how optimisation theory interacts with the neural network minimization problem and how geometric ideas can provide insight into the behaviour of hidden neurons. In particular, I found it interesting that finite-width neural networks admit convex reformulations under suitable conditions, which helps explain their strong empirical performance.

I still have not figured out theoretical explanations for my numerical experiment results. (For example, the minimised training loss reached its global minimum when networks are trained on noisy data, but does not reach it when networks are trained on noiseless data.) The theoretical explanations behind these outcomes can be explored in future research.

Codes are available at GitHub repository.