LinkBERT: Enhancing Language Model Training with Document Links
The Stanford Machine Learning Group introduces LinkBERT, a novel pretraining method for language models published at ACL 2022. Traditional language models like BERT and GPT typically process documents in isolation, missing rich dependencies and multi-hop knowledge contained in hyperlinks and citations. LinkBERT addresses this limitation by treating text corpora as graphs of interconnected documents rather than independent lists. By incorporating document link information during self-supervised learning, the model captures knowledge that spans across multiple sources. For example, linking a Wikipedia article about the Tidal Basin to one about the Cherry Blossom Festival allows the model to infer that Japanese cherry trees are present at the basin. This approach enables language models to acquire deeper world knowledge, significantly improving performance on knowledge-intensive downstream tasks such as question answering and knowledge discovery. The research highlights the importance of leveraging structural relationships within data to enhance artificial intelligence systems' understanding and reasoning capabilities.
Wire timeline
LinkBERT: Enhancing Language Model Training with Document Links
The Stanford Machine Learning Group introduces LinkBERT, a novel pretraining method for language models published at ACL 2022. Traditional language models like BERT and GPT typically process documents in isolation, missing rich dependencies and multi-hop knowledge contained in hyperlinks and citations. LinkBERT addresses this limitation by treating text corpora as graphs of interconnected documents rather than independent lists. By incorporating document link information during self-supervised learning, the model captures knowledge that spans across multiple sources. For example, linking a Wikipedia article about the Tidal Basin to one about the Cherry Blossom Festival allows the model to infer that Japanese cherry trees are present at the basin. This approach enables language models to acquire deeper world knowledge, significantly improving performance on knowledge-intensive downstream tasks such as question answering and knowledge discovery. The research highlights the importance of leveraging structural relationships within data to enhance artificial intelligence systems' understanding and reasoning capabilities.
The Stanford AI Lab Blog