Task Complexity Shapes Internal Representations and Robustness in Neural Networks

A new research paper published on arXiv investigates how task complexity influences the internal representations and robustness of multilayer perceptrons (MLPs). The study introduces five data-agnostic probes, including pruning, binarization, and noise injection, to analyze MLPs as signed, weighted bipartite graphs. By comparing easy and hard classification tasks on MNIST and Fashion-MNIST datasets, the authors demonstrate that binarizing weights in hard-task models causes accuracy to collapse to chance levels, while easy-task models remain robust. The research reveals a sharp phase-transition in performance when pruning low-magnitude edges in binarized hard-task models. Additionally, moderate noise injection was found to enhance accuracy through a stochastic-resonance effect. The findings suggest that preserving only the sign structure of weights is sufficient for maintaining high accuracy in certain contexts. This work defines a model-agnostic measure of task complexity based on the performance gap between full-precision and simplified networks, offering new strategies for model compression and interpretability aligned with task difficulty.