Mechanistic Analysis of Transferable Representations in Cohort-Trained Implicit Neural Representations
A new research paper published on arXiv investigates the internal mechanisms of cohort-trained Implicit Neural Representations (INRs), specifically focusing on which layers encode transferable information and how to optimize them for new signal fitting. The study examines two standard backbones, SIREN and Fourier-feature MLPs (FFMLP), revealing that the optimal layer for freezing during initialization coincides with the layer exhibiting the highest weight stable rank. This approach matches or exceeds standard fine-tuning performance. Furthermore, the authors employ sparse autoencoders (SAEs) to decompose INR activations into interpretable dictionary atoms. The analysis shows distinct differences between architectures: SIREN learns localized, coordinate-based atoms independent of content, while FFMLP learns image-spanning atoms that trace memorized signal contours. Ablation studies confirm these findings, demonstrating that single FFMLP atoms significantly impact global image quality, whereas SIREN atoms affect only local regions. This work provides the first mechanistic account of transfer in cohort-trained INRs, offering tools to distinguish between generalization and memorization in neural network architectures.
Wire timeline
Mechanistic Analysis of Transferable Representations in Cohort-Trained Implicit Neural Representations
A new research paper published on arXiv investigates the internal mechanisms of cohort-trained Implicit Neural Representations (INRs), specifically focusing on which layers encode transferable information and how to optimize them for new signal fitting. The study examines two standard backbones, SIREN and Fourier-feature MLPs (FFMLP), revealing that the optimal layer for freezing during initialization coincides with the layer exhibiting the highest weight stable rank. This approach matches or exceeds standard fine-tuning performance. Furthermore, the authors employ sparse autoencoders (SAEs) to decompose INR activations into interpretable dictionary atoms. The analysis shows distinct differences between architectures: SIREN learns localized, coordinate-based atoms independent of content, while FFMLP learns image-spanning atoms that trace memorized signal contours. Ablation studies confirm these findings, demonstrating that single FFMLP atoms significantly impact global image quality, whereas SIREN atoms affect only local regions. This work provides the first mechanistic account of transfer in cohort-trained INRs, offering tools to distinguish between generalization and memorization in neural network architectures.
cs.AI updates on arXiv.org