Assessment of L2 student writing: Does teacher disciplinary background matter?

Abstract

This preliminary study examines the rating behavior of five composition and five ESL writing teachers while evaluating a text from a university-level non-native (L2) English speaking student. Using an eye tracker, we measured raters’ dwell times and reading behaviors across four areas of interest—rhetoric, organization, vocabulary, and grammar. Results indicate that raters with differing disciplinary backgrounds read the text differently. L2 writing teachers tended to spend more time on and re-read the rhetorical, lexical, and grammatical features of the text while skipping over more of the grammar errors, while composition teachers read the text more deliberately. The findings suggest L2 writing teachers were more prone to skim and scan for information on which to base a grade while composition teachers delayed rating decisions until after reviewing the entire text, which is corroborated in previous research. These findings can expand our understanding of how disciplinary background can influence rating processes, which can inform rater training procedures, especially in disciplinary writing contexts where L2 writing is judged by individuals with and without expertise in composition or second language writing. Moreover, it demonstrates the utility of eye-tracking methods to examine the cognitive processes associated with reading and scoring student writing.