Image credit: OpenAIAbstract
Recent advances in the development of large language models have led to substantial enhancements in performance across an array of downstream tasks. Remarkably, these models, trained with straightforward end-to-end objectives, have demonstrated an inherent ability to manage language tasks. Not long ago, tackling language tasks heavily depended on our in-depth understanding of language. The convergence of these trends provides an excellent opportunity to delve into their relationship. Specifically, we pose the question, can contemporary deep neural network (DNN) based end-to-end language modeling paradigms provide us with insights into language? In this paper, we focus on a long-standing linguistic debate, can syntax and semantics be separated? We argue that by incorporating an inductive bias for labor division, the separation between syntax and semantics naturally emerges in the English language. To demonstrate this, we employ a two-tower language model setup. Here, two language models with identical configurations are trained collaboratively in parallel. Intriguingly, this configuration results in a spontaneously emerging preference where specific tokens are consistently better predicted by one tower, while others by the second tower. This pattern remains qualitatively consistent across different model structures and reflects separation of syntax and semantics. Our findings show the potential of DNN-based end-to-end trained language models in deepening our comprehension of the properties of natural language.
