Argumentation features and essay quality: Exploring relationships and incidence counts
DOI:
https://doi.org/10.17239/jowr-2022.14.01.01Keywords:
argumentation, rhetorical structure theory, writing assessmentAbstract
This study examines links between human ratings of writing quality and the incidence of argumentative features (e.g., claims, data) in persuasive essays along with relationships among these features and their distance from one another within an essay. The goal is to better understand how argumentation elements in persuasive essays combine to model human ratings of essay quality. The study finds that, in most cases, it is not the presence of argumentation features that is predictive of writing quality but rather the relationships between superordinate and subordinate features, parallel features, and the distances between features. This finding has not only theoretical value but also practical value in terms of pedagogical approaches and automated writing feedback.
References
Applebee, A. N.(1984). Writing and reasoning. Review of Educational Research Winter, 54(4), 577-596. https://doi.org/10.3102/00346543054004577
Azar, M. (1999). Argumentative text as rhetorical structure: An application of rhetorical structure theory. Argumentation, 13(1), 97-114.
Brassart, D. G. (1996). Didactique de l'argumentation écrite: Approches psycho-cognitives. Argumentation, 10(1), 69-87. https://doi.org/10.1007/bf00126160
Brassart, D. G. (1996). Does a prototypical argumentative schema exist? Text recall in 8 to 13 years olds. Argumentation, 10(2), 163-174. https://doi.org/10.1007/bf00180723
Berrill, D. P. (1996). Reframing argument from the metaphor of war. In P. Berrill (Ed.), Perspectives on written argument (pp. 171–187). Creskill, NJ: Hampton Press.
Breiman, L. (2001). Decision-tree forests. Machine Learning, 45(1), 5-32.
Brown, H. D., & Abeywickrama, P. (2010). Language assessment: Principles and classroom practices (Vol. 10). White Plains, NY: Pearson Education.
Cohen J. (1960). A coefficient of agreement for nominal scales. Educational Psychology Measurement, 20, 37–46. https://doi.org/10.1177/001316446002000104
Cooper, C.R., Cherry, R., Copley, B., Fleischer, S., Pollard, R., Sartisky, M. (1984). Studying the writing abilities of a university freshman class: strategies from a case study. In Beach, R.,Bridwell, L.S. (Eds.), New Directions in Composition Research. The Guilford Press, New York, pp. 19-52. https://doi.org/10.1017/s0142716400006263
Crammond, J. (1998). The uses and complexity of argument structures in expert and student persuasive writing. Written Communication, 15, 230-268.
https://doi.org/10.1177/0741088398015002004
Crasnich, S., & Lumbelli, L. (2005). The reflection-response in enhancing argumentation ability. L1-educational Studies in Language and Literature, 5(2), 147-174.
https://doi.org/10.1007/s10674-005-0918-5
Currie, P. (1996). Fullness and sound reasoning: Argument and evaluation in a university content course. In P. Berrill (Ed.), Perspectives on written argument (pp. 121-137). Creskill, NJ: Hampton Press.
Ferretti, R. P., Lewis, W. E., & Andrews-Weckerly, S. (2009). Do goals affect the structure of students’ argumentative writing strategies?. Journal of Educational Psychology, 101(3), 577.
https://doi.org/10.1037/a0014702
Ferretti, R. P., MacArthur, C. A., & Dowdy, N. S. (2000). The effects of an elaborated goal on the persuasive writing of students with learning disabilities and their normally achieving peers. Journal of Educational Psychology, 92(4), 694.
https://doi.org/10.1037/0022-0663.92.4.694
Fleiss, J. L., Levin, B., & Paik, M. C. (2003). Statistical methods for rates and proportions Wiley-Interscience. Hoboken, NJ. https://doi.org/10.1002/0471445428
Gárate, M., & Melero, A. (2005). Teaching How to Write Argumentative Texts at Primary School. Studies In Writing Effective Learning and Teaching of Writing, 323–337.
https://doi.org/10.1007/978-1-4020-2739-0_22
Green, N. L. (2010). Representation of argumentation in text with rhetorical structure theory. Argumentation, 24(2), 181-196. https://doi.org/10.1007/s10503-009-9169-4
Hillocks, G. (2010). " EJ" in Focus: Teaching Argument for Critical Thinking and Writing: An Introduction. The English Journal, 99(6), 24-32.
Hillocks, G. J. (2011). Teaching argumentative writing, Grades 6-12: Supporting claims with relevant evidence and clear reasoning. Portsmouth: Heinemann.
Hirvela, A. (2017). Argumentation & second language writing: Are we missing the boat? Journal of Second Language Writing, 36, 69–74. https://doi.org/10.1016/j.jslw.2017.05.002
Hyland, K. (1997). A genre description of the argumentative essays. RELC Journal 21 (1): 66–78. http://dx.doi.org/10.1177/003368829002100105
Jonassen, D. H., & Kim, B. (2010). Arguing to learn and learning to argue: Design justifications and guidelines. Educational Technology Research and Development, 58(4), 439-457.
https://doi.org/10.1007/s11423-009-9143-8
Kuhn, M., Wing, J., Weston, S., Williams, A., Keefer, C., Engelhardt, A., ... & Kenkel, B. the R Core Team, Benesty M, Lescarbeau R, Ziem A, Scrucca L, Tang Y, Candan C (2016) Caret: classification and regression training. R package version, 6-0.
Kunnan, A. J. (2010). Test fairness and Toulmin’s argument structure. Language Testing, 27, 183–189. https://doi.org/10.1177/0265532209349468
Lee, J. J., & Deakin, L. (2016). Interactions in L1 and L2 undergraduate student writing: Interactional metadiscourse in successful and less-successful argumentative essays. Journal of Second Language Writing, 33, 21–34. https://doi.org/10.1016/j.jslw.2016.06.004
Liu, F., & Stapleton, P. (2014). Counterargumentation and the cultivation of critical thinking in argumentative writing: Investigating washback from a high-stakes test. System, 45, 117-128.
https://doi.org/10.1016/j.system.2014.05.005
Mann, W. C., & Matthiessen, C. M. (1991). Functions of language in two frameworks. Word, 42(3), 231-249. https://doi.org/10.1080/00437956.1991.11435839
Mann, W. C., & Thompson, S. A. (1986). Relational propositions in discourse. Discourse processes, 9(1), 57-90. https://doi.org/10.1080/01638538609544632
Mann, W. C., & Thompson, S. A. (1987). Rhetorical structure theory: A theory of text organization (pp. 87-190). Los Angeles: University of Southern California, Information Sciences Institute.
Mann, W. C., & Thompson, S. A. (1988). Rhetorical structure theory: Toward a functional theory of text organization. Text, 8(3), 243-281. https://doi.org/10.1515/text.1.1988.8.3.243
McCann, T. M. (1989). Student argumentative writing knowledge and ability at three grade levels. Research in the Teaching of English, 62-76.
Nemeth, N., & Kormos, J. (2001). Pragmatic aspects of task-performance: the case of argumentation. Language Teaching Research, 5, 213-240. https://doi.org/10.1177/136216880100500303
Newell, G. E., Beach, R., Smith, J., & VanDerHeide, J. (2011). Teaching and learning argumentative reading and writing: A review of research. Reading Research Quarterly, 46(3), 273-304.
Nippold, M. A., & Ward-Lonergan, J. M. (2010). Argumentative writing in pre-adolescents: The role of verbal reasoning. Child Language Teaching and Therapy, 26, 238–248.
https://doi.org/10.1177/0265659009349979
Nussbaum, E. M., & Kardash, C. M. (2005). The Effects of Goal Instructions and Text on the Generation of Counterarguments During Writing. Journal of Educational Psychology, 97(2), 157–169.
https://doi.org/10.1037/0022-0663.97.2.157
Nussbaum, E. M., & Schraw, G. (2007). Promoting argument-counterargument integration in students' writing. The Journal of Experimental Education, 76(1), 59-92.
https://doi.org/10.3200/jexe.76.1.59-92
Pessoa, S., Mitchell, T. D., & Miller, R. T. (2017). Emergent arguments: A functional approach to analyzing student challenges with the argument genre. Journal of Second Language Writing, 38, 42–55.
https://doi.org/10.1016/j.jslw.2017.10.013
Qin, J., & Karabacak, E. (2010). The analysis of Toulmin elements in Chinese EFL university argumentative writing. System, 38(3), 444–456. https://doi.org/10.1016/j.system.2010.06.012
R Core Team (2017). R: A language and environment for statistical computing. R Found. Stat. Comput. Vienna, Austria.
Ramage, J. D., & Bean, J. C. (1989). Writing arguments: A rhetoric with readings. New York: Macmillan.
Sampson, V., & Clark, D. B. (2008). Assessment of the ways students generate arguments in science education: Current perspectives and recommendations for future directions. Science Education, 92(3), 447–472. https://doi.org/10.1002/sce.20276
Shehab, H.M. & Nussbaum, E.M. (2015). Cognitive load of critical thinking strategies. Learning and Instruction 35, 51-61. https://doi.org/10.1016/j.learninstruc.2014.09.004
Simon, S. (2008). Using Toulmin's Argument Pattern in the evaluation of argumentation in school science. International Journal of Research & Method in Education, 31, 277-289.
https://doi.org/10.1080/17437270802417176
Song, Y., Deane, P., & Beigman Klebanov, B. (2017). Toward the automated scoring of written arguments: Developing an innovative approach for annotation. ETS Research Report Series, 2017(1), 1-15.
https://doi.org/10.1002/ets2.12138
Stapleton, P. (2001). Assessing critical thinking in the writing of Japanese university students: Insights about assumptions and content familiarity. Written communication, 18(4), 506-548. https://doi.org/10.1177/0741088301018004004
Stapleton, P., & Wu, Y. A. (2015). Assessing the quality of arguments in students’ persuasive writing: A case study analyzing the relationship between surface structure and substance. Journal of English for Academic Purposes, 17, 12–23.
https://doi.org/10.1016/j.jeap.2014.11.006
Strobl, C., Ailhaud, E., Benetos, K., Devitt, A., Kruse, O., Proske, A., & Rapp, C. (2019). Digital support for academic writing: A review of technologies and pedagogies. Computers & Education, 131, 33-48.
https://doi.org/10.1016/j.compedu.2018.12.005
Toulmin, S. E. (1958). The uses of argument. Cambridge: Cambridge University Press.
Toulmin, S. E. (2003). The uses of argument. Cambridge: Cambridge university press.
Townsend, M. A., Hicks, L., Thompson, J. D., Wilton, K. M., Tuck, B. F., & Moore, D. W. (1993). Effects of introductions and conclusions in assessment of student essays. Journal of Educational Psychology, 85(4), 670. https://doi.org/10.1037/0022-0663.85.4.670
Van Daal, T., Lesterhuis, M., Coertjens, L., Donche, V., & De Maeyer, S. (2016). Validity of comparative judgement to assess academic writing: Examining implications of its holistic character and building on a shared consensus. Assessment in Education: Principles, Policy & Practice.
https://doi.org/10.1080/0969594x.2016.1253542
Varghese, S. A., & Abraham, S. A. (1998). Undergraduates arguing a case. Journal of Second Language Writing, 7, 287-306. https://doi.org/10.1016/s1060-3743(98)90018-2
Walton, D., Reed, C., and Macagno, F. (2008), Argumentation Schemes, Cambridge: Cambridge University Press.
Wolcott, W., & Legg, S. M. (1998). An Overview of Writing Assessment: Theory, Research, and Practice. Urbana, IL: National Council of Teachers of English.
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Scott Crossley, Yu Tian, Qian Wan
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 Unported License.
