Study Identifies Topological Bias as Key for Transformer Reasoning on Knowledge Graphs
A new research paper titled 'What Structural Inductive Bias Helps Transformers Reason Over Knowledge Graphs? A Study with Tabula RASA' investigates the specific structural mechanisms that enable transformers to perform multi-hop reasoning over knowledge graphs. Through controlled ablations of a minimal transformer modification, the authors isolated four components: sparse adjacency masking, edge-type biases, query scaling, and value gating. The study reveals that sparse adjacency masking is the primary driver of performance improvements, accounting for significant gains across benchmarks like MetaQA, WebQSP, and CWQ. In contrast, learned relation parameters provided only modest refinements and sometimes hindered performance without structural guidance. Zero-shot experiments further corroborated that masking-based attention is significantly more robust than relation-specific weights when edge types are withheld. The findings suggest that the useful inductive bias for multi-hop knowledge graph question answering is predominantly topological rather than relational. This research provides critical insights for optimizing transformer architectures in complex reasoning tasks, emphasizing the importance of structural topology over semantic relation embeddings.
Wire timeline
Study Identifies Topological Bias as Key for Transformer Reasoning on Knowledge Graphs
A new research paper titled 'What Structural Inductive Bias Helps Transformers Reason Over Knowledge Graphs? A Study with Tabula RASA' investigates the specific structural mechanisms that enable transformers to perform multi-hop reasoning over knowledge graphs. Through controlled ablations of a minimal transformer modification, the authors isolated four components: sparse adjacency masking, edge-type biases, query scaling, and value gating. The study reveals that sparse adjacency masking is the primary driver of performance improvements, accounting for significant gains across benchmarks like MetaQA, WebQSP, and CWQ. In contrast, learned relation parameters provided only modest refinements and sometimes hindered performance without structural guidance. Zero-shot experiments further corroborated that masking-based attention is significantly more robust than relation-specific weights when edge types are withheld. The findings suggest that the useful inductive bias for multi-hop knowledge graph question answering is predominantly topological rather than relational. This research provides critical insights for optimizing transformer architectures in complex reasoning tasks, emphasizing the importance of structural topology over semantic relation embeddings.
cs.AI updates on arXiv.org