Game Theoretic Free Energy Analysis of Higher Order Synergy in LLM Attention Heads
A new research paper introduces the Game Theoretic Free Energy Principle (GTFEP) to analyze interactions among attention heads in large language models (LLMs). By treating attention heads as bounded rational agents minimizing variational free energy, the study decomposes collective behavior into Harsanyi dividends. The analysis reveals that triple dividends are consistently negative across BERT, GPT2, and Llama models, indicating higher-order redundancy rather than synergy. Leveraging the Nash FEP correspondence, the authors demonstrate that heads with negligible marginal contributions can be pruned with minimal performance impact. Experimental results on GPT2 show that pruning 20% of attention heads reduces computational costs by 18% and increases throughput by 22%, while only modestly increasing perplexity. This work provides a principled theoretical foundation for optimizing transformer architectures, offering a method to enhance efficiency without significant degradation in model accuracy. The findings suggest that understanding higher-order interactions through game theory can lead to more efficient AI systems.
Wire timeline
Game Theoretic Free Energy Analysis of Higher Order Synergy in LLM Attention Heads
A new research paper introduces the Game Theoretic Free Energy Principle (GTFEP) to analyze interactions among attention heads in large language models (LLMs). By treating attention heads as bounded rational agents minimizing variational free energy, the study decomposes collective behavior into Harsanyi dividends. The analysis reveals that triple dividends are consistently negative across BERT, GPT2, and Llama models, indicating higher-order redundancy rather than synergy. Leveraging the Nash FEP correspondence, the authors demonstrate that heads with negligible marginal contributions can be pruned with minimal performance impact. Experimental results on GPT2 show that pruning 20% of attention heads reduces computational costs by 18% and increases throughput by 22%, while only modestly increasing perplexity. This work provides a principled theoretical foundation for optimizing transformer architectures, offering a method to enhance efficiency without significant degradation in model accuracy. The findings suggest that understanding higher-order interactions through game theory can lead to more efficient AI systems.
cs.AI updates on arXiv.org