Distinguishing effective writing styles in the PERSUADE corpus
Keywords:
Writing Styles, Natural Language Processing, Cluster Analysis, SALATAbstract
Many linguistic studies of writing assume a single linear relationship between linguistic features in the text and human judgments of writing quality. However, writing quality may be better understood as a complex latent construct that can be constructed in a number of different ways through different linguistic profiles of high-quality writing styles as shown in Crossley et al. (2014). This study builds on the exploratory study reported by Crossley et al. by analyzing a representational corpus of 4,170 highly rated persuasive essays written by secondary-school students. The study uses natural language processing tools to derive quantitative representations for the linguistic features found in the texts. These linguistic features inform a k-means cluster analysis which indicates that a four-cluster profile best fits the data. By examining the indices most and least distinctive of each cluster, the study identifies a structured writing style, a conversational writing style, a reportive writing style, and an academic writing style. The findings support the notion that writers can employ a variety of writing profiles to successfully write an argumentative essay.
References
Aryadoust, V., Ng, L. Y., & Sayama, H. (2021). A comprehensive review of Rasch measurement in language assessment: Recommendations and guidelines for research. Language Testing, 38(1), 6–40. https://doi.org/10.1177/0265532220927487
Attali, Y., & Burstein, J. (2006). Automated essay scoring with e-rater® V. 2. The Journal of Technology, Learning and Assessment, 4(3).
https://ejournals.bc.edu/index.php/jtla/article/view/1650
Au, W., Brown, A. L., Calderón, D., & Dumas, M. (2016). Reclaiming the multicultural roots of U.S. curriculum: Communities of color and official knowledge and education. Teachers College Press. https://doi.org/10.1086/705262
Biber, D. (1991). Variation across speech and writing. Cambridge University Press. https://doi.org/10.1017/cbo9780511621024
Biber, D., Gray, B., & Poonpon, K. (2011). Should We Use Characteristics of Conversation to Measure Grammatical Complexity in L2 Writing Development? TESOL Quarterly, 45(1), 5–35. https://doi.org/10.5054/tq.2011.244483
Cambria, E., & Hussain, A. (2015). SenticNet. In E. Cambria & A. Hussain, Sentic Computing (pp. 23–71). Springer International Publishing. https://doi.org/10.1007/978-3-319-23654-4_2
Chen, D., & Manning, C. D. (2014). A fast and accurate dependency parser using neural networks. Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 740–750. https://aclanthology.org/D14-1082.pdf
Cohen, J. (1988). Set Correlation and Contingency Tables. Applied Psychological Measurement, 12(4), 425–434. https://doi.org/10.1177/014662168801200410
Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155–159. https://doi.org/10.1037/0033-2909.112.1.155
Coxhead, A. (2000). A New Academic Word List. TESOL Quarterly, 34(2), 213.
https://doi.org/10.2307/3587951
Crossley, S. A., Baffour, P., Tian, Y., Picou, A., Benner, M., & Boser, U. (2022). The persuasive essays for rating, selecting, and understanding argumentative and discourse elements (PERSUADE) corpus 1.0. Assessing Writing, 54, 100667. https://doi.org/10.1016/j.asw.2022.100667
Crossley, S. A., & Kim, M. (2022). Linguistic Features of Writing Quality and Development: A Longitudinal Approach. The Journal of Writing Analytics, 6(1), 59–93. https://doi.org/10.37514/JWA-J.2022.6.1.04
Crossley, S. A., Kyle, K., & Dascalu, M. (2019). The Tool for the Automatic Analysis of Cohesion 2.0: Integrating semantic similarity and text overlap. Behavior Research Methods, 51(1), 14–27. https://doi.org/10.3758/s13428-018-1142-4
Crossley, S. A., Kyle, K., & McNamara, D. S. (2016). The tool for the automatic analysis of text cohesion (TAACO): Automatic assessment of local, global, and text cohesion. Behavior Research Methods, 48(4), 1227–1237. https://doi.org/10.3758/s13428-015-0651-7
Crossley, S. A., Kyle, K., & McNamara, D. S. (2017). Sentiment Analysis and Social Cognition Engine (SEANCE): An automatic tool for sentiment, social cognition, and social-order analysis. Behavior Research Methods, 49(3), 803–821. https://doi.org/10.3758/s13428-016-0743-z
Crossley, S. A., & McNamara, D. S. (2010). Cohesion, Coherence, and Expert Evaluations of Writing Proficiency. Proceedings of the Annual Meeting of the Cognitive Science Society, 32(32). https://escholarship.org/uc/item/6n5908qx
Crossley, S. A., & McNamara, D. S. (2014). Does writing development equal writing quality? A computational investigation of syntactic complexity in L2 learners. Journal of Second Language Writing, 26, 66–79. https://doi.org/10.1016/j.jslw.2014.09.006
Crossley, S. A., Roscoe, R., & McNamara, D. S. (2011). Predicting Human Scores of Essay Quality Using Computational Indices of Linguistic and Textual Features. In G. Biswas, S. Bull, J. Kay, & A. Mitrovic (Eds.), Artificial Intelligence in Education (Vol. 6738, pp. 438–440). Springer Berlin Heidelberg. https://doi.org/10.1007/978-3-642-21869-9_62
Crossley, S. A., Roscoe, R., & McNamara, D. S. (2014). What Is Successful Writing? An Investigation Into the Multiple Ways Writers Can Write Successful Essays. Written Communication, 31(2), 184–214. https://doi.org/10.1177/0741088314526354
Crossley, S. A., Tian, Y., Baffour, P., Franklin, A., Kim, Y., Morris, W., Benner, M., Picou, A., & Boser, U. (2023). The English Language Learner Insight, Proficiency and Skills Evaluation (ELLIPSE) Corpus. International Journal of Learner Corpus Research, 9(2), 250–271. https://doi.org/10.1075/ijlcr.22026.cro
Crossley, S., & McNamara, D. (2016). Say more and be more coherent: How text elaboration and cohesion can increase writing quality. Journal of Writing Research, 7(3 (February 2016)), 351–370. https://doi.org/10.17239/jowr-2016.07.3.02
Dascalu, M., McNamara, D., Crossley, S., & Trausan-Matu, S. (2016). Age of Exposure: A Model of Word Learning. Proceedings of the AAAI Conference on Artificial Intelligence, 30(1). https://doi.org/10.1609/aaai.v30i1.10372
Davies, M. (2008). The Corpus of Contemporary American English (COCA). https://www.english-corpora.org/coca/
De Los Ríos, C. V. (2020). Writing Oneself Into the Curriculum: Photovoice Journaling in a Secondary Ethnic Studies Course. Written Communication, 37(4), 487–511. https://doi.org/10.1177/0741088320938794
De Smedt, F., Landrieu, Y., De Wever, B., & Van Keer, H. (2022). Do cognitive processes and motives for argumentative writing converge in writer profiles? The Journal of Educational Research, 115(4), 258–270. https://doi.org/10.1080/00220671.2022.2122020
Devitt, A. J. (2015). Genre performances: John Swales’ Genre Analysis and rhetorical-linguistic genre studies. Journal of English for Academic Purposes, 19, 44–51.
https://doi.org/10.1016/j.jeap.2015.05.008
Gaies, S. J. (1980). T-Unit Analysis in Second Language Research: Applications, Problems and Limitations. TESOL Quarterly, 14(1), 53. https://doi.org/10.2307/3586808
Gan, G., & Ng, M. K.-P. (2017). K -means clustering with outlier removal. Pattern Recognition Letters, 90, 8–14. https://doi.org/10.1016/j.patrec.2017.03.008
Garner, J., Crossley, S., & Kyle, K. (2019). N-gram measures and L2 writing proficiency. System, 80, 176–187. https://doi.org/10.1016/j.system.2018.12.001
González, M. C. (2017). The Contribution of Lexical Diversity to College‐Level Writing. TESOL Journal, 8(4), 899–919. https://doi.org/10.1002/tesj.342
Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2), 193–202. https://doi.org/10.3758/BF03195564
Guo, L., Crossley, S. A., & McNamara, D. S. (2013). Predicting human judgments of essay quality in both integrated and independent second language writing samples: A comparison study. Assessing Writing, 18(3), 218–238. https://doi.org/10.1016/j.asw.2013.05.002
Halliday, M. A. K., & Hasan, R. (1976). Cohesion in English (0 ed.). Routledge.
https://doi.org/10.4324/9781315836010
Hammill, D. D., Mather, N., Allen, E. A., & Roberts, R. (2002). Using Semantics, Grammar, Phonology, and Rapid Naming Tasks to Predict Word Identification. Journal of Learning Disabilities, 35(2), 121–136. https://doi.org/10.1177/002221940203500204
Hartigan, J. A., & Wong, M. A. (1979). Algorithm AS 136: A K-Means Clustering Algorithm. Applied Statistics, 28(1), 100. https://doi.org/10.2307/2346830
Hartley, J., & Branthwaite, A. (1989). The psychologist as wordsmith: A questionnaire study of the writing strategies of productive British psychologists. Higher Education, 18(4), 423–452. https://doi.org/10.1007/BF00140748
Hunt, K. W. (1965). A Synopsis of Clause-to-Sentence Length Factors. The English Journal, 54(4), 300. https://doi.org/10.2307/811114
Jarvis, S. (2013). Capturing the Diversity in Lexical Diversity: Lexical Diversity. Language Learning, 63, 87–106. https://doi.org/10.1111/j.1467-9922.2012.00739.x
Jarvis, S., Grant, L., Bikowski, D., & Ferris, D. (2003). Exploring multiple profiles of highly rated learner compositions. Journal of Second Language Writing, 12(4), 377–403. https://doi.org/10.1016/j.jslw.2003.09.001
Jnoub, N., Al Machot, F., & Klas, W. (2020). A Domain-Independent Classification Model for Sentiment Analysis Using Neural Models. Applied Sciences, 10(18), 6221. https://doi.org/10.3390/app10186221
Jung, Y. J., Crossley, S., & McNamara, D. (2019). Predicting Second Language Writing Proficiency in Learner Texts Using Computational Tools. The Journal of AsiaTEFL, 16(1), 37–52. https://doi.org/10.18823/asiatefl.2019.16.1.3.37
Kim, M., & Crossley, S. A. (2018). Modeling second language writing quality: A structural equation investigation of lexical, syntactic, and cohesive features in source-based and independent writing. Assessing Writing, 37, 39–56. https://doi.org/10.1016/j.asw.2018.03.002
Klein, D., & Manning, C. D. (2003). Accurate unlexicalized parsing. Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, 423–430. https://doi.org/10.3115/1075096.1075150
Kodinariya, T. M., Makwana, P. R., & others. (2013). Review on determining number of Cluster in K-Means Clustering. International Journal, 1(6), 90–95.
Kyle, K. (2016). Measuring syntactic development in L2 writing: Fine grained indices of syntactic complexity and usage-based indices of syntactic sophistication [Doctoral Dissertation].
Kyle, K., & Crossley, S. (2016). The relationship between lexical sophistication and independent and source-based writing. Journal of Second Language Writing, 34, 12–24. https://doi.org/10.1016/j.jslw.2016.10.003
Kyle, K., & Crossley, S. A. (2015). Automatically Assessing Lexical Sophistication: Indices, Tools, Findings, and Application. TESOL Quarterly, 49(4), 757–786. https://doi.org/10.1002/tesq.194
Kyle, K., & Crossley, S. A. (2018). Measuring Syntactic Complexity in L2 Writing Using Fine-Grained Clausal and Phrasal Indices. The Modern Language Journal, 102(2), 333–349. https://doi.org/10.1111/modl.12468
Kyle, K., Crossley, S. A., & Jarvis, S. (2021). Assessing the Validity of Lexical Diversity Indices Using Direct Judgements. Language Assessment Quarterly, 18(2), 154–170.
https://doi.org/10.1080/15434303.2020.1844205
Kyle, K., Crossley, S., & Berger, C. (2018). The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behavior Research Methods, 50(3), 1030–1046. https://doi.org/10.3758/s13428-017-0924-4
Kyle, K., & Eguchi, M. (2021). 6 Automatically Assessing Lexical Sophistication Using Word, Bigram, and Dependency Indices. In S. Granger (Ed.), Perspectives on the L2 Phrasicon (pp. 126–151). Multilingual Matters. https://doi.org/10.21832/9781788924863-007
Lasswell, H. D., & Namenwirth, J. Z. (1969). The Lasswell value dictionary. New Haven.
Laufer, B., & Nation, P. (1995). Vocabulary Size and Use: Lexical Richness in L2 Written Production. Applied Linguistics, 16(3), 307–322. https://doi.org/10.1093/applin/16.3.307
Levy, R., & Andrew, G. (2006). Tregex and Tsurgeon: Tools for querying and manipulating tree data structures. LREC, 2231–2234.
Liu, B. (2022). Sentiment analysis and opinion mining. Springer Nature. https://www.cs.uic.edu/~liub/FBS/liub-SA-and-OM-book.pdf
Lu, X. (2010). Automatic analysis of syntactic complexity in second language writing. International Journal of Corpus Linguistics, 15(4), 474–496. https://doi.org/10.1075/ijcl.15.4.02lu
Lu, X., Casal, J. E., & Liu, Y. (2021). Towards the Synergy of Genre- and Corpus-Based Approaches to Academic Writing Research and Pedagogy: International Journal of Computer-Assisted Language Learning and Teaching, 11(1), 59–71. https://doi.org/10.4018/IJCALLT.2021010104
MacArthur, C. A., Jennings, A., & Philippakos, Z. A. (2019). Which linguistic features predict quality of argumentative writing for college basic writers, and how do those features change with instruction? Reading and Writing, 32(6), 1553–1574. https://doi.org/10.1007/s11145-018-9853-6
Macqueen, J. (1967). Some Methods for Classification and Analysis of Multivariate Observations. Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, 1, 281–297.
McCarthy, P. M., & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaches to lexical diversity assessment. Behavior Research Methods, 42(2), 381–392. https://doi.org/10.3758/BRM.42.2.381
McNamara, D. S., Allen, L. K., Crossley, S. A., Dascalu, M., & Perret, C. A. (2017). Natural Language Processing and Learning Analytics. In C. Lang, G. Siemens, A. Wise, New York University, USA, & D. Gasevic (Eds.), Handbook of Learning Analytics (First, pp. 93–104). Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17.008
McNamara, D. S., Crossley, S. A., & McCarthy, P. M. (2010). Linguistic Features of Writing Quality. Written Communication, 27(1), 57–86. https://doi.org/10.1177/0741088309351547
McNamara, D. S., Crossley, S. A., & Roscoe, R. (2013). Natural language processing in an intelligent writing strategy tutoring system. Behavior Research Methods, 45(2), 499–515. https://doi.org/10.3758/s13428-012-0258-1
McNamara, D. S., Crossley, S. A., Roscoe, R. D., Allen, L. K., & Dai, J. (2015). A hierarchical classification approach to automated essay scoring. Assessing Writing, 23, 35–59. https://doi.org/10.1016/j.asw.2014.09.002
Medhat, W., Hassan, A., & Korashy, H. (2014). Sentiment analysis algorithms and applications: A survey. Ain Shams Engineering Journal, 5(4), 1093–1113.
https://doi.org/10.1016/j.asej.2014.04.011
Mohammad, S. M. (2016). Sentiment Analysis. In Emotion Measurement (pp. 201–237). Elsevier. https://doi.org/10.1016/B978-0-08-100508-8.00009-6
Mohammad, S. M., & Turney, P. D. (2013). Crowdsourcing a Word-Emotion Association Lexicon. Computational Intelligence, 29(3), 436–465. https://doi.org/10.1111/j.1467-8640.2012.00460.x
Moskal, B. M. (2000). Scoring Rubrics: What, When and How? Practical Assessment, Research and Evaluation, 7(3). https://doi.org/10.7275/A5VQ-7Q66
Muangkammuen, P., & Fukumoto, F. (2020). Multi-task learning for automated essay scoring with sentiment analysis. Proceedings of the 1st Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 10th International Joint Conference on Natural Language Processing: Student Research Workshop, 116–123. https://aclanthology.org/2020.aacl-srw.17/
Myhill, D. (2008). Towards a Linguistic Model of Sentence Development in Writing. Language and Education, 22(5), 271–288. https://doi.org/10.1080/09500780802152655
Nakayama, M., Sears, C. R., & Lupker, S. J. (2008). Masked priming with orthographic neighbors: A test of the lexical competition assumption. Journal of Experimental Psychology: Human Perception and Performance, 34(5), 1236–1260. https://doi.org/10.1037/0096-1523.34.5.1236
National Center for Educational Statistics. (2012). The Nation׳s Report Card: Writing 2011. https://nces.ed.gov/nationsreportcard/pdf/main2011/2012470.pdf
Olinghouse, N. G., & Wilson, J. (2013). The relationship between vocabulary and writing quality in three genres. Reading and Writing, 26(1), 45–65. https://doi.org/10.1007/s11145-012-9392-5
Omuya, E. O., Okeyo, G., & Kimwele, M. (2023). Sentiment analysis on social media tweets using dimensionality reduction and natural language processing. Engineering Reports, 5(3), e12579. https://doi.org/10.1002/eng2.12579
Ortega, L. (2003). Syntactic Complexity Measures and their Relationship to L2 Proficiency: A Research Synthesis of College-level L2 Writing. Applied Linguistics, 24(4), 492–518. https://doi.org/10.1093/applin/24.4.492
Perin, D., & Lauterbach, M. (2018). Assessing Text-Based Writing of Low-Skilled College Students. International Journal of Artificial Intelligence in Education, 28(1), 56–78. https://doi.org/10.1007/s40593-016-0122-z
R Core Team. (2021). R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing. https://www.R-project.org/
Richards, B. (1987). Type/Token Ratios: What do they really tell us? Journal of Child Language, 14(2), 201–209. https://doi.org/10.1017/S0305000900012885
Rudner, L. M., Garcia, V., & Welch, C. (2006). An evaluation of IntelliMetricTM essay scoring system. The Journal of Technology, Learning and Assessment, 4(4).
https://ejournals.bc.edu/index.php/jtla/article/view/1651
Saito, K., Webb, S., Trofimovich, P., & Isaacs, T. (2016). Lexical Profiles of Comprehensible Second Language Speech: The Role of Appropriateness, Fluency, Variation, Sophistication, Abstractness, and Sense Relations. Studies in Second Language Acquisition, 38(4), 677–701. https://doi.org/10.1017/S0272263115000297
Salsbury, T., Crossley, S. A., & McNamara, D. S. (2011). Psycholinguistic word information in second language oral discourse. Second Language Research, 27(3), 343–360. https://doi.org/10.1177/0267658310395851
Schwartz, M. (1983). Revision Profiles: Patterns and Implications. College English, 45(6), 549. https://doi.org/10.2307/377139
Seyoum, W. M., Yigzaw, A., & Bewuketu, H. K. (2022). Students’ Attitudes and Problems on Question-Based Argumentative Essay Writing Instruction. Journal of English Language Teaching and Learning, 3(2), 58–63. https://doi.org/10.33365/jeltl.v3i2.2106
Shermis, M. D., Burstein, J., Higgins, D., & Zechner, K. (2010). Automated essay scoring: Writing assessment and instruction. International Encyclopedia of Education, 4(1), 20–26. https://doi.org/10.1016/b978-0-08-044894-7.00233-5
Sinclair, J. (1991). Corpus, concordance, collocation (4. impr). Oxford Univ. Pr. https://doi.org/10.2307/330144
Stone, P. J., Dunphy, D. C., & Smith, M. S. (1966). The general inquirer: A computer approach to content analysis. https://psycnet.apa.org/record/1967-04539-000
Struthers, L., Lapadat, J. C., & MacMillan, P. D. (2013). Assessing cohesion in children’s writing: Development of a checklist. Assessing Writing, 18(3), 187–201.
https://doi.org/10.1016/j.asw.2013.05.001
Swales, J. (1990). Genre analysis: English in academic and research settings (1. publ., 14. print). Cambridge Univ. Pr. https://doi.org/10.2307/416471
Torrance, M., Thomas, G., & Robinson, E. (1999). Individual differences in the writing behaviour of undergraduate students. British Journal of Educational Technology, 69, 189–199. https://doi.org/10.1348/000709999157662
Torrance, M., Thomas, G. V., & Robinson, E. J. (1994). The writing strategies of graduate research students in the social sciences. Higher Education, 27(3), 379–392.
https://doi.org/10.1007/BF03179901
Torrance, M., Thomas, G. V., & Robinson, E. J. (2000). Individual differences in undergraduate essay-writing strategies: A longitudinal study. Higher Education, 39(2), 181–200.
https://doi.org/10.1023/A:1003990432398
Torruella, J., & Capsada, R. (2013). Lexical Statistics and Tipological Structures: A Measure of Lexical Richness. Procedia - Social and Behavioral Sciences, 95, 447–454.
https://doi.org/10.1016/j.sbspro.2013.10.668
Tywoniw, R., & Crossley, S. (2019). The effect of cohesive features in integrated and independent L2 writing quality and text classification. Language Education and Assessment, 2(3), 110–134. https://doi.org/10.29140/lea.v2n3.151
Uccelli, P., Dobbs, C. L., & Scott, J. (2013). Mastering Academic Language: Organization and Stance in the Persuasive Writing of High School Students. Written Communication, 30(1), 36–62. https://doi.org/10.1177/0741088312469013
Van Waes, L., & Schellens, P. J. (2003). Writing profiles: The effect of the writing mode on pausing and revision patterns of experienced writers. Journal of Pragmatics, 35(6), 829–853. https://doi.org/10.1016/S0378-2166(02)00121-2
Vengadasalam, S. S. (2020). Transformative Pedagogy and Student Voice: Using S.E.A. Principles in Teaching Academic Writing. Journal of Effective Teaching in Higher Education, 3(2), 12–27. https://doi.org/10.36021/jethe.v3i2.95
Verhavert, S., Bouwer, R., Donche, V., & De Maeyer, S. (2019). A meta-analysis on the reliability of comparative judgement. Assessment in Education: Principles, Policy & Practice, 26(5), 541–562. https://doi.org/10.1080/0969594X.2019.1602027
Warschauer, M., & Ware, P. (2006). Automated writing evaluation: Defining the classroom research agenda. Language Teaching Research, 10(2), 157–180.
https://doi.org/10.1191/1362168806lr190oa
Weigle, S. C. (2007). Teaching writing teachers about assessment. Journal of Second Language Writing, 16(3), 194–209. https://doi.org/10.1016/j.jslw.2007.07.004
Wiseman, C. S. (2012). A comparison of the performance of analytic vs. Holistic scoring rubrics to assess L2 writing. International Journal of Language Testing, 2(1), 59–92.
Xanthopoulos, P., Pardalos, P. M., & Trafalis, T. B. (2013). Linear Discriminant Analysis. In P. Xanthopoulos, P. M. Pardalos, & T. B. Trafalis, Robust Data Mining (pp. 27–33). Springer New York. https://doi.org/10.1007/978-1-4419-9878-1_4
Zhang, M., Zhu, M., Deane, P., & Guo, H. (2019). Identifying and Comparing Writing Process Patterns Using Keystroke Logs. In M. Wiberg, S. Culpepper, R. Janssen, J. González, & D. Molenaar (Eds.), Quantitative Psychology (Vol. 265, pp. 367–381). Springer International Publishing. https://doi.org/10.1007/978-3-030-01310-3_32
Zhao, C. G., & Wu, J. (2022). Perceptions of authorial voice: Why discrepancies exist. Assessing Writing, 53, 100632. https://doi.org/10.1016/j.asw.2022.100632
Published
Issue
Section
License
Copyright (c) 2025 Wesley Morris, Scott Crossley, Langdon Holmes, Joon Choi
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 Unported License.
