Making sense of L2 written argumentation with keystroke logging
DOI:
https://doi.org/10.17239/jowr-2024.15.03.01Keywords:
L2 written argumentation, Keystroke logging, Cognitive activitiesAbstract
This study examines associations between writing behaviors manifested by keystroke analytics and the formulation of argument elements in L2 undergraduate writers' writing processes. Ninety-nine persuasive essays written by L2 undergraduate writers were human annotated for Toulmin argument elements. The corresponding keystroke logs were segmented and analyzed to characterize the dynamics of writing processes for different categories of the elements. A multinomial mixed-effects logistic regression model was built to predict argument categories using the keystroke analytics. The study reported that L2 undergraduate writers' text production for final claims and primary claims featured P-bursts (execution processes delimited by pauses exceeding 2 seconds) of longer spans but lower production fluency compared to that for data. In addition, fewer revisions were observed when L2 writers were constructing final claims than when they were formulating data. These findings shed light on the varying cognitive loads and activities L2 undergraduate writers may experience when building different argument elements in written argumentation.
References
Andriessen, J., Baker, M., & Suthers, D. (2003). Argumentation, computer support, and the educational context of confronting cognitions. In J. Andriessen, M. Baker, & D. Suthers (Eds.), Arguing to learn: Confronting cognitions in computer-supported collaborative learning environments (pp. 1–25). Boston: Kluwer. https://doi.org/10.1007/978-94-017-0781-7
Alvès, R. A., Castro, S. L., & Olive, T. (2008). Execution and pauses in writing narratives: Processing time, cognitive effort and typing skill. International Journal of Psychology, 43, 969-979. https://doi.org/10.1080/00207590701398951
Barkaoui, K. (2019). What can L2 writers’ pausing behavior tell us about their L2 writing processes?. Studies in Second Language Acquisition, 41(3), 529-554. https://doi.org/10.1017/s027226311900010x
Bayerl, P. S., Lüngen, H., Gut, U., & Paul, K. I. (2003, October). Methodology for reliable schema development and evaluation of manual annotations. In Proceedings of the Workshop on Knowledge Markup and Semantic Annotation at the Second International Conference on Knowledge Capture (K-CAP 2003).
Bell, P., & Linn, M. C. (2000). Scientific arguments as learning artifacts: Designing for learning from the web with KIE. International Journal of Science Education, 22(8), 797–818. https://doi.org/10.1080/095006900412284
Breuer, E. O. (2019). Fluency in L1 and FL writing: An analysis of planning, essay writing and final revision. In E. Lindgren & K. P. H. Sullivan (Eds.), Observing Writing (pp. 190-211). Brill. https://doi.org/10.1163/9789004392526_010
Brooks, S. (1998). Markov chain Monte Carlo method and its application. Journal of the royal statistical society: series D (the Statistician), 47(1), 69-100. https://doi.org/10.1111/1467-9884.00117
Chan, S. (2017). Using keystroke logging to understand writers’ processes on a reading-into-writing test. Language Testing in Asia, 7(1), 1-27. https://doi.org/10.1186/s40468-017-0040-5
Chandrasegaran, A. (2008). NNS students’ arguments in English: Observations in formal and informal contexts. Journal of Second Language Writing, 17(4), 237–254. https://doi.org/10.1016/j.jslw.2008.04.003
Chenoweth, N. A., & Hayes, J. R. (2001). Fluency in writing: Generating text in L1 and L2. Written Communication, 18, 80-98. https://doi.org/10.1177/0741088301018001004
Chukharev-Khudilaynen, E. (2014). Pauses in spontaneous written communication: A keystroke logging study. Journal of Writing Research, 6(1), 61. https://doi.org/10.17239/jowr-2014.06.01.3
Chukharev-Hudilainen, E. (2019). Empowering automated writing evaluation with keystroke logging. In E. Lindgren & K. P. H. Sullivan (Eds.), Observing writing (pp.125-142). Brill. https://doi.org/10.1163/9789004392526_007
Chukharev-Hudilainen, E., Saricaoglu, A., Torrance, M., & Feng, H. H. (2019). Combined deployable keystroke logging and eyetracking for investigating L2 writing fluency. Studies in Second Language Acquisition, 41(3), 583-604. https://doi.org/10.1017/s027226311900007x
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46.
https://doi.org/10.1177/001316446002000104
Coirier, P., Andriessen, J., & Chanquoy, L. (1999). From planning to translating: The specificity of argumentative writing. Foundations of argumentative text processing, 1-28. https://doi.org/10.5117/9789053563403
Crammond, J. G. (1998). The uses and complexity of argument structures in expert and student persuasive writing. Written Communication, 15(2), 230-268. https://doi.org/10.1177/0741088398015002004
Crossley, S., Tian, Y., & Wan, Q. (2022). Argumentation Features and Essay Quality: Exploring Relationships and Incidence Counts. Journal of Writing Research. https://doi.org/10.17239/jowr-2022.14.01.01
Dich, N., & Pedersen, B. (2013). Native Language Effects on Spelling in English as a Foreign Language: A Time-Course Analysis. Canadian Journal of Applied Linguistics/Revue canadienne de linguistique appliquee, 16(1), 51-68.
Ellis, R. (2003). Task-based language learning and teaching. Oxford: Oxford University Press.
Favart, M., & Coirier, P. (2006). Acquisition of the linearization process in text composition in third to ninth graders: effects of textual superstructure and macrostructural organization. Journal of psycholinguistic research, 35(4), 305. https://doi.org/10.1007/s10936-006-9017-8
Ferretti, R. P., & Fan, Y. (2016). Argumentative writing. In C. A. MacArthur, S. Graham, & J. Fitzgerald (Eds.), Handbook of writing research (2nd ed., pp. 301–315). New York: Guilford Press. https://doi.org/10.17239/jowr-2014.06.02.5
Ferretti, R. P., MacArthur, C. A., & Dowdy, N. S. (2000). The effects of an elaborated goal on the persuasive writing of students with learning disabilities and their normally achieving peers. Journal of Educational Psychology, 92(4), 694. https://doi.org/10.1037/0022-0663.92.4.694
Ferretti, R. P., Lewis, W. E., & Andrews-Weckerly, S. (2009). Do goals affect the structure of students’ argumentative writing strategies?. Journal of Educational Psychology, 101(3), 577. https://doi.org/10.1037/a0014702
Fisher, A. (2011). Critical thinking: An introduction (2nd ed.). Cambridge: Cambridge University Press.
Flower, L., & Hayes, J. R. (1981). A cognitive process theory of writing. College composition and communication, 32(4), 365-387. https://doi.org/10.2307/356600
Galbraith, D., & Baaijen, V. M. (2019). Aligning keystrokes with cognitive processes in writing. In E. Lindgren & K. P. H. Sullivan (Eds.), Observing Writing (pp. 306-325). Brill. https://doi.org/10.1163/9789004392526_015
Godo, A. (2008). Cross-cultural aspects of academic writing: A study of Hungarian and North American college students LI argumentative essays. International Journal of English Studies, 8, 2, pp. 65-111.
Hadfield, J. D. (2010). MCMC methods for multi-response generalized linear mixed models: the MCMCglmm R package. Journal of statistical software, 33, 1-22. https://doi.org/10.18637/jss.v033.i02
Hayes, J. R. (1996). A new framework for understanding cognition and affect in writing. In M. Levy & S. Ransdell (Eds.), The science of writing (pp. 1-27). Mahwah, NJ: Erlbaum.
Hayes, J. R. (2012). Modeling and remodeling writing. Written Communication, 29(3), 369–388. https://doi.org/10.1177/0741088312451260
Hillocks, G., Jr. (2011). Teaching argument writing, grades 6-12. Portsmouth, NH: Heinemann.
Hirvela, A. (2017). Argumentation & second language writing: Are we missing the boat? Journal of Second Language Writing, 36, 69–74. https://doi.org/10.1016/j.jslw.2017.05.002
Hoang, H. (2019). Metaphorical language in second language learners' texts: Additional baggage of the writing journey? In E. Lindgren & K. P. H. Sullivan (Eds.), Observing Writing (pp. 236-257). Brill. https://doi.org/10.1163/9789004392526_012
Jimenez-Aleixandre, M., Rodriguez, M., & Duschl, R. A. (2000). “Doing the lesson” or “doing science”: Argument in high school genetics. Science Education, 84(6), 757–792. https://doi.org/10.1002/1098-237x(200011)84:6<757::aid-sce5>3.0.co;2-f
Jonassen, D. H., & Kim, B. (2010). Arguing to learn and learning to argue: Design justifications and guidelines. Educational Technology Research and Development, 58(4), 439-457. https://doi.org/10.1007/s11423-009-9143-8
Kaufer, D. S., Hayes, J. R., & Flower, L. (1986). Composing written sentences. Research in the Teaching of English, 20, 121-140.
Kellogg, R. T. (1994). The psychology of writing. Oxford: Oxford University Press.
Kim, M., Tian, Y., & Crossley, S. A. (2021). Exploring the relationships among cognitive and linguistic resources, writing processes, and written products in second language writing. Journal of Second Language Writing, 53, 100824. https://doi.org/10.1016/j.jslw.2021.100824
Knudson, R. E. (1992). Analysis of argumentative writing at two grade levels. The Journal of Educational Research, 85(3), 169-179. https://doi.org/10.1080/00220671.1992.9944434
Kubota, R. (2010). Cross-cultural perspectives on writing. In N. H. Hornberger & S. Lee Mckay (Eds.), Sociolinguistics and language education (pp. 265-289) UK: Multilingual Matters. https://doi.org/10.21832/9781847692849-012
Kuhn, M., & Johnson, K. (2013). Applied predictive modeling, New York, NY: Springer.
Kunnan, A. J. (2010). Test fairness and Toulmin’s argument structure. Language Testing, 27, 183–189. https://doi.org/10.1177/0265532209349468
Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. biometrics, 159-174. https://doi.org/10.2307/2529310
Lee, S. H. (2014). Argument structure as an interactive resource by undergraduate students. Linguistics & the Human Sciences, 9(3).
Leijten, M., & Van Waes, L. (2013). Keystroke logging in writing research: Using Inputlog to analyze and visualize writing processes. Written Communication, 30(3), 358-392. https://doi.org/10.1177/0741088313491692
Leijten, M., Van Waes, L., & Ransdell, S. (2010). Correcting text production errors: Isolating the effects of writing mode from error span, input mode, and lexicality. Written Communication, 27(2), 189-227. https://doi.org/10.1177/0741088309359139
Levishina, N. (2015). Bayesian logistic models with MCMCglmm: A brief tutorial. Retrieved from http://www.natalialevshina.com/Documents/MCMCglmm_Tutorial.pdf
Limpo, T., & Alvès, R. A. (2017). Tailoring Multicomponent Writing Interventions: Effects of Coupling Self-Regulation and Transcription Training. Journal of Learning Disabilities, 51(4), 381-398. https://doi.org/10.1177/0022219417708170
Lindgren, E., Sullivan, K. P. H., & Stevenson, M. (2008). Supporting the reflective language learner with computer keystroke logging. In B. Barber & F. Zhang (Eds.), Handbook of research on computer enhanced language acquisition and learning (pp. 189–204). Hershey, NY: Information Science Reference, IGI Global. https://doi.org/10.4018/978-1-59904-895-6.ch011
Lindgren, E. & Sullivan K. P. H. (2006). Analyzing on-line revision. In G. Rijlaarsdam (Series Ed.) and K. P. H. Sullivan, & E. Lindgren. (Vol. Eds.), Studies in Writing, Vol.18, Computer Keystroke Logging: Methods and Applications, (157–188). Oxford: Elsevier. https://doi.org/10.1163/9789004392526
Lindgren, E., & Sullivan, K. P. H. (Eds.). (2019). Observing writing: Insights from keystroke logging and handwriting. Brill. https://doi.org/10.1163/9789004392526
Linnemann, M. (2019). Application of audience during writing. In E. Lindgren & K. P. H. Sullivan (Eds.), Observing Writing (pp. 326-345). Brill.
Liu, F., & Stapleton, P. (2014). Counterargumentation and the cultivation of critical thinking in argumentative writing: Investigating washback from a high-stakes test. System, 45, 117–128. https://doi.org/10.1016/j.system.2014.05.005
MacArthur C. A., & Graham, S. (2016). Writing research from a cognitive perspective. In C. A. MacArthur, S. Graham, S., & J. Fitzgerald, J. (Eds.), Handbook of writing research (pp. 24–40). NY: Guilford Press. https://doi.org/10.17239/jowr-2014.06.02.5
MacArthur, C. A., Graham, S., & Fitzgerald, J. (Eds.). (2008). Handbook of writing research. Guilford Press.
McCutchen, D. (1996). A capacity theory of writing: Working memory in composition. Educational psychology review, 8(3), 299-325. https://doi.org/10.1007/bf01464076
McElreath, R. (2020). Statistical rethinking: A Bayesian course with examples in R and Stan. Chapman and Hall/CRC. https://doi.org/10.1201/9780429029608
Medimorec, S., & Risko, E. F. (2016). Effects of disfluency in writing. British Journal of Psychology, 107(4), 625-650. https://doi.org/10.1111/bjop.12177
Newell, G. E., Beach, R., Smith, J., VanDerHeide, J., Kuhn, D., & Andriessen, J. (2011). Teaching and Learning Argumentative Reading and Writing: A Review of Research. Reading Research Quarterly, 46(3), 273–304.
Nippold, M. A., & Ward-Lonergan, J. M. (2010). Argumentative writing in pre-adolescents: The role of verbal reasoning. Child Language Teaching and Therapy, 26, 238–248. https://doi.org/10.1177/0265659009349979
Nussbaum, E.M. (2008).Using argumentation vee diagrams (AVDs) for promoting argument-counterargument integration in reflective writing. Journal of Educational Psychology, 100, 3, 549–565. https://doi.org/10.1037/0022-0663.100.3.549
Nussbaum, E. M., & Kardash, C. M. (2005). The Effects of Goal Instructions and Text on the Generation of Counterarguments During Writing. Journal of Educational Psychology, 97(2), 157–169. https://doi.org/10.1037/0022-0663.97.2.157
Nussbaum, E. M., & Schraw, G. (2007). Promoting argument-counterargument integration in students' writing. The Journal of Experimental Education, 76(1), 59-92. https://doi.org/10.3200/jexe.76.1.59-92
Olive, T., Alvès, R. A., & Castro, S. L. (2009). Cognitive processes in writing during pause and execution periods. European Journal of Cognitive Psychology, 21(5), 758-785. https://doi.org/10.1080/09541440802079850
Qin, J., & Karabacak, E. (2010). The analysis of Toulmin elements in Chinese EFL university argumentative writing. System, 38(3), 444-456. https://doi.org/10.1016/j.system.2010.06.012
Révész, A., Kourtali, N. E., & Mazgutova, D. (2017). Effects of task complexity on L2 writing behaviors and linguistic complexity. Language Learning, 67(1), 208-241. https://doi.org/10.1111/lang.12205
Sampson, V., & Clark, D. B. (2008). Assessment of the ways students generate arguments in science education: Current perspectives and recommendations for future directions. Science Education, 92(3), 447–472. https://doi.org/10.1002/sce.20276
Schilperoord, J. (1996). It’s about time. Temporal aspects of cognitive processes in text production. Amsterdam, the Netherlands: Rodopi. https://doi.org/10.1163/9789004458598
Schwarz, B. B., & Asterhan, C. S. (2010). Argumentation and reasoning. International handbook of psychology in education, 137-176.
Shehab, H. M., & Nussbaum, E. M. (2015). Cognitive load of critical thinking strategies. Learning and Instruction, 35, 51-61. https://doi.org/10.1016/j.learninstruc.2014.09.004
Simon, D., & Holyoak, K. J. (2002). Structural dynamics of cognition: From consistency theories to constraint satisfaction. Personality and social psychology review, 6(4), 283-294. https://doi.org/10.1207/s15327957pspr0604_03
Spelman Miller, K. S. (2000). Academic writers on-line: Investigating pausing in the production of text. Language Teaching Research, 4, 123-148. https://doi.org/10.1191/136216800675510135
Spelman Miller, K. S., Lindgren, E., & Sullivan, K. P. (2008). The psycholinguistic dimension in second language writing: Opportunities for research and pedagogy using computer keystroke logging. TESOL Quarterly, 42(3), 433-454.
https://doi.org/10.1002/j.1545-7249.2008.tb00140.x
Stapleton, P., & Wu, Y. A. (2015). Assessing the quality of arguments in students’ persuasive writing: A case study analyzing the relationship between surface structure and substance. Journal of English for Academic Purposes, 17, 12–23. https://doi.org/10.1016/j.jeap.2014.11.006
Stevenson, M., Schoonen, R., & De Glopper, K. (2006). Revising in two languages: A multi-dimensional comparison of online writing revisions in L1 and FL. Journal of Second Language Writing, 15(3), 201-233. https://doi.org/10.1016/j.jslw.2006.06.002
Sullivan, K., & Lindgren, E. (2006). Computer Key-Stroke Logging and Writing: Methods and Applications (Studies in Writing). Elsevier Science Inc.. https://doi.org/10.1163/9780080460932
Tanimoto, T. T. (1958). An elementary mathematical theory of classification and prediction. Internal Rport IBM Corp.
Thorson, H. (2000). Using the computer to compare foreign and native language writing processes: A statistical and case study approach. Modern Language Journal, 84(ii), 155–170. https://doi.org/10.1111/0026-7902.00059
Torrance, M. (2016) Understanding planning in text production. In C. A. MacArthur, S. Graham, S., & J. Fitzgerald, J. (Eds.), Handbook of writing research (pp. 72–87). NY: Guilford Press.
Toulmin, S. (1958). The uses of argument. Cambridge: Cambridge University Press.
Toulmin, S. E. (2003). The uses of argument. Cambridge: Cambridge university press
Van Waes, L. V., & Leijten, M. (2015). Fluency in Writing: A Multidimensional Perspective on Writing Fluency Applied to L1 and L2. Computers and Composition, 38, 79-95. https://doi.org/10.1016/j.compcom.2015.09.012
Van Waes, L., Leijten, M., Pauwaert, T., & Van Horenbeeck, E. (2019). A multilingual copy task: Measuring typing and motor skills in writing with Inputlog. Journal of open research software.-2013, currens, 7(30), 1-8. https://doi.org/10.5334/jors.234
Van Waes, L. & Schellens, P. J. (2003). Writing Profiles: The Effect of the Writing Mode on Pausing and Revision Patterns of Experienced Writers. Journal of Pragmatics, 35:6, 829- 853. https://doi.org/10.1016/s0378-2166(02)00121-2
Van Waes, L., Van Weijen, D., & Leijten, M. (2014). Learning to write in an online writing center: The effect of learning styles on the writing process. Computers & Education, 73, 60-71. https://doi.org/10.1016/j.compedu.2013.12.009
Varghese, S. A., & Abraham, S. A. (1998). Undergraduates arguing a case. Journal of Second Language Writing, 7, 287-306. https://doi.org/10.1016/s1060-3743(98)90018-2
Voss, J. F., & Means, M. L. (1991). Learning to reason via instruction in argumentation. Learning and instruction, 1(4), 337-350. https://doi.org/10.1016/0959-4752(91)90013-x
Wengelin, Å. (2006). Examining pauses in writing: Theories, methods and empirical data. In K.P.H. Sullivan & E. Lindgren (Eds.), Computer Key-Stroke Logging and Writing: Methods and Applications (pp. 107-130). Amsterdam, the Netherlands: Elsevier. https://doi.org/10.1163/9780080460932_008
Wingate, U. (2012). ‘Argument’ helping students understand what essay writing is about. Journal of English for Academic Purposes, 11, 145-154. https://doi.org/10.1016/j.jeap.2011.11.001
Zhang, S. (1987). Cognitive complexity and written production in English as a second language. Language Learning, 37(4), 469-481. https://doi.org/10.1111/j.1467-1770.1987.tb00580.x
Published
Issue
Section
License
Copyright (c) 2023 Yu Tian, Minkyung Kim, Scott Crossley
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 Unported License.
