The impact of syntactic complexity on sentence parsing for humans and LLMs

Date worked on:

November 2024

Research context:

For the culminating experience of COGS 50.05 (Psycholinguistics), I proposed an experiment investigating how sentence complexity impacts human cognitive load and computational parsing difficulty.

My involvement:

Lead researcher

Collaborators:

Dr. Samantha Wray (course professor)

I proposed a study that investigates how sentence complexity impacts human cognitive load and computational parsing difficulty, as well as the syntactic processing parallels that may exist between the two systems.

The study will be made up of two experiments, and the independent variable for both will be sentence complexity (simple sentences, sentences with nested clauses, garden-path sentences, and ambiguous sentences). In Experiment 1, human participants will read sentences of varying syntactic complexity while their reading times, eye-tracking data, and comprehension accuracy are recorded. In Experiment 2, a sequence-to-sequence (seq2seq) computational parser will process the same sentences, while their parsing time, perplexity, and parsing accuracy are measured. I will run ANOVAs to investigate the effects of complexity on each dependent variable, with p-values used to assess statistical significance.

I hypothesize that both human and computational parsing performance will be negatively correlated with sentence complexity, with overall performance, comprehension, and parsing accuracy decreasing as complexity increases. I also expect that reading and parsing times will be positively associated with sentence complexity, and that similar patterns of parsing difficulty will be observed across conditions between humans and computational parsers.

These results would indicate that the two systems face similar syntactic processing challenges, potentially due to similar underlying mechanisms. Additionally, this study would deepen our understanding of the relationship between syntax and semantics in human cognition and inform the design of artificial intelligence (AI) systems regarding human language comprehension. Ultimately, this study would contribute to research and development in psycholinguistics and natural language processing (NLP). 

Read the full version here!