Learning goal: universal approximation concerns existence, not guaranteed learning. At fixed architecture and training budget, changing from one to 32 cycles produces very different errors: this setup learns lower frequencies more readily. A dense MLP also lacks a built-in spatial-neighborhood rule, motivating architectures with different inductive biases.

Dashed gray: target sin(2πkx); points: training data; solid: current network prediction.

Log training MSE versus steps for k=1,4,16,32; horizontal 0.5 line: no-learning reference.

k = 1 Width = 32 Training MSE at k=1 = — k=4 = — k=16 = — k=32 = —

Observe: the four runs start together; k=1 quickly falls below 10⁻³ while k=32 stays near 0.5 in this experiment. Increasing width from 32 to 128 can help the high-frequency target, but a substantial gap remains here. This illustrates representation, capacity, and optimization interacting under a fixed budget. An existence theorem does not guarantee that this training procedure finds suitable weights.