Efficient measurement of writing knowledge with forced-choice tasks: Preliminary data using the student knowledge of writing tests
DOI:
https://doi.org/10.17239/jowr-2023.15.02.06Keywords:
writing knowledge, writing quality, assessment, late elementaryAbstract
Much of the research that has examined the writing knowledge of school-age students has relied on interviews to ascertain this information, which is problematic because interviews may underestimate breadth and depth of writing knowledge, require lengthy interactions with participants, and do not permit a direct evaluation of a prescribed array of constituent knowledge elements. For these reasons, our goal in this study is to report the development, piloting, and field testing, using a sample of 335 students from grades 4 and 5, of four alternate versions of a writing knowledge assessment—the Student Knowledge of Writing Test (SKOWT)—that uses forced-choice responses to evaluate students’ knowledge of writing processes, genre elements, and linguistic features of written language. All versions of the SKOWT demonstrated adequate internal consistency reliability and construct validity based on exploratory factor analyses following deletion of some items. In addition, there was acceptable predictive criterion validity based on associations of SKOWT scores with subtests from the Test of Written Language-4 and measures of narrative, opinion, and informative essay quality. We discuss how the SKOWT might be used in future research and educational practice.
References
Albin, M. L., Benton, S. L., & Khramtsova, I. (1996). Individual differences in interest and narrative writing. Contemporary Educational Psychology, 21(4), 305-324. https://doi.org/10.1006/ceps.1996.0024
Alexander, P. A. (1997). Stages and phases of domain learning: The dynamics of subject-matter knowledge, strategy knowledge, and motivation. In C. E. Weinstein & B. L. McCombs (Eds.), Skill, will, and self-regulation. Erlbaum.
Alexander, P. A. (2003). The development of expertise: The journey from acclimation to proficiency. Educational Researcher, 32(8), 10-14. https://doi.org/10.3102/0013189X032008010
Apel, K., Masterson, J., & Hart, P. (2004). Integration of language components in spelling: Instruction that maximizes students’ learning. In E. R. Silliman & L. C. Wilkinson (Eds.), Language and literacy learning in schools (pp. 292-315). Guilford.
Baker, F. B., & Kim, S. (2017). The basics of item response theory using R. Springer.
Barbeiro, L. F. (2011). What happens when I write? Pupils’ writing about writing. Reading and Writing: An Interdisciplinary Journal, 24, 813-834. https://doi.org/10.1007/s11145-010-9226-2
Behizadeh, N., & Pang, M. E. (2016). Awaiting a new wave: The status of state writing assessment in the United States. Assessing Writing, 29, 25-41. https://doi.org/10.1016/j.asw.2016.05.003
Bereiter, C., & Scardamalia, M. (1984). Learning about writing from reading. Written Communication, 1, 163-88. https://doi.org/10.1177/0741088384001002001
Berman, R. A., & Slobin, D. I. (1994). Relating events in narrative: A cross-linguistic developmental study. Erlbaum.
Benton, S. L., Corkill, S. J., Sharp, J. M., Downy, R., & Khramstova, I. (1995). Knowledge, interest, and narrative writing. Journal of Educational Psychology, 87, 66¬¬¬¬¬¬¬¬-79. https://doi.org/10.1037/0022-0663.87.1.66
Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149
Bourassa, D., Treiman, R., & Kessler, B. (2006). Use of morphology in spelling by children with dyslexia and typically developing children. Memory and Cognition, 34, 703-714. https://doi.org/10.3758/BF03193589
Bouwer, R., Béguin, A., Sanders, T., & van den Bergh, H. (2015). Effect of genre on the generalizability of writing scores. Language Testing, 32 (1), 83-100. https://doi.org/10.1177/0265532214542994
Brown, T. (2014). Confirmatory factor analysis for applied research. Guildford Press.
Chi, M. T. H. (1978). Knowledge structures and memory development. In R. S. Siegler (Ed.), Children's thinking: What develops? (pp. 73-96). Erlbaum.
Costello, A. B., & Osborne, J. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, and Evaluation, 10(7), 1-9. https://doi.org/10.7275/jyj1-4868
Couzijn, M. (1999). Learning to write by observation of writing and reading processes: Effects on learning and transfer. Learning and Instruction, 9, 109-142. https://doi.org/10.1016/S0959-4752(98)00040-1
DeGroff, L. C. (1987). The influence of prior knowledge on writing, conferencing, and revising. The Elementary School Journal, 88, 105-118. https://doi.org/10.1086/461527
Donovan, C. A. (2001). Children's development and control of written story and informational genres: Insights from one elementary school. Research in the Teaching of English, 35, 394-447. https://www.jstor.org/stable/40171493
Donovan, C. A., & Smolkin, L. B. (2005). Elementary school children's written scientific explanations: An examination of developing abilities. Paper presented at the annual meeting of the National Reading Conference, Miami, FL.
Duke, N. K. (1999). The scarcity of informational texts in first grade. (Report No.1-007). Center for the Improvement of Early Reading Achievement. University of Michigan.
Englert, S., Raphael, T., Fear, K., & Anderson, L. (1988). Students' metacognitive knowledge about how to write informational texts. Learning Disability Quarterly, 11, 18-46. https://doi.org/10.2307/1511035
Fidalgo, R., Torrance, M., & Garcia, J. (2008). The long-term effects of writing strategy instruction for young writers. Contemporary Educational Psychology, 33, 672-693. https://doi.org/10.1016/j.cedpsych.2007.09.001
Fitzgerald, J., & Markham, L. R. (1987). Teaching children about revision in writing. Cognition and Instruction, 4, 3-24. https://doi.org/10.1207/s1532690xci0401_1
Fitzgerald, J., & Teasley, A. (1986). Effects of instruction in narrative structure on children’s writing. Journal of Educational Psychology, 78, 424-432. https://doi.org/10.1037/0022-0663.78.6.424
Franklin, S. B., Gibson, D. J., Robertson, P. A., Pohlmann, J. T., & Fralish, J. S. (1995). Parallel analysis: A method for determining significant principal components. Journal of Vegetation Science, 6(1), 99-106. https://doi.org/10.2307/3236261
Gambrell, L. B., & Chasen, S. P. (1991). Explicit story structure instruction and the narrative writing of fourth- and fifth-grade below-average readers. Reading Research and Instruction, 31, 54-62. https://doi.org/10.1080/19388079109558071
Gillespie, A., Olinghouse, N., & Graham, S. (2013). Fifth-grade students’ knowledge about writing process and writing genres. The Elementary School Journal, 113, 565-588. https://doi.org/10.1086/669938
Graham, S. (2006). Writing. In P. Alexander & P. Winne (Eds.), Handbook of educational psychology (pp. 457-478). Erlbaum.
Graham, S., Berninger, V. W., Abbott, R. D., Abbott, S. P., & Whitaker, D. (1997). Role of mechanics in composing of elementary school students: A new methodological approach. Journal of Educational Psychology, 89, 170-182. https://doi.org/10.1037/0022-0663.89.1.170
Graham, S., Liu, X., Aitken, A., Ng, C., Bartlett, B., Harris, K. R., & Holzapfel, J. (2018). Effectiveness of literacy programs balancing reading and writing instruction: A meta‐analysis. Reading Research Quarterly, 53(3), 279-304. https://doi.org/10.1002/rrq.194
Graham, S., Schwartz, S., & MacArthur, C. A. (1993). Knowledge of writing and the composing process, attitude toward writing, and self-efficacy for students with and without learning disabilities. Journal of Learning Disabilities, 26, 237-249. https://doi.org/10.1177/002221949302600404
Hambleton, R. K. (1991). Fundamentals of item response theory. Sage.
Hammill, D. D., & Larsen, S. C. (2009). Test of written language (4th ed.). PRO-ED.
Hasan, R. (1984). The nursery tale as a genre. Nottingham Linguistic Circular, 13, 71-102. http://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&idt=12004084
Hayes, J. R. (1996). A new framework for understanding writing. In C. M. Levy & S. Randall (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp.1-27). Erlbaum.
Hayes, J. R. (2011). Kinds of knowledge-telling: Modeling early writing development. Journal of Writing Research, 3(2), 73-92. https://doi.org/10.17239/jowr-2011.03.02.1
Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191-205. https://doi.org/10.1177/1094428104263675
Hemphill, L., Feldman, H. M., Camp, L., Griffin, T., Miranda, A. B., & Wolf, D. P. (1994). Developmental changes in narrative and non-narrative discourse in children with and without brain injury. Journal of Communication Disorders, 27, 107-133. https://doi.org/10.1016/0021-9924(94)90037-X
Holliway, D. R., & McCutchen, D. (2004). Audience perspective in young writers’ composing and revising. In L. Allal, L. Chanquoy & P. Largy (Eds.), Revision cognitive and instructional processes (pp. 87-101). Springer.
Kamberelis, G. (1999). Genre development and learning: Children writing stories, science reports, and poems. Research in the Teaching of English, 33, 403-460. https://www.jstor.org/stable/40171463
Klein, P. D., & Rose, M. A. (2010). Teaching argument and explanation to prepare junior students for writing to learn. Reading Research Quarterly, 45(4), 433-461. https://doi.org/10.1598/RRQ.45.4.4
Knudson, R. E. (1989). Effects of instructional strategies on children’s informational writing. Journal of Educational Research, 83(2), 91-96. https://doi.org/10.1080/00220671.1989.10885936
Kos, R. & Maslowski, C. (2001). Second graders’ perceptions of what is important in writing. The Elementary School Journal, 101(5), 567-585. https://doi.org/10.1086/499688
Langer, J. A. (1984). The effects of available information on responses to school writing tasks. Research in the Teaching of English, 18(1), 27-44. https://www.jstor.org/stable/40170977
Langer, J. (1986). Children reading and writing: Structures and strategies. Ablex.
Lin, S. C., Monroe, B. W., & Troia, G. A. (2007). Development of writing knowledge in grades 2-8: A comparison of typically developing writers and their struggling peers. Reading and Writing Quarterly, 23, 207-230. https://doi.org/10.1080/10573560701277542
Malpique, A., & Veiga-Simão, A. M. (2016). Argumentative writing by junior high school students: Discourse knowledge and writing performance. Infancia y Aprendizaje, 39(1), 150-186. https://doi.org/10.1080/02103702.2015.1111609
McCutchen, D. (1986). Domain knowledge and linguistic knowledge in the development of writing ability. Journal of Memory and Language, 25, 431-444. https://doi.org/10.1016/0749-596X(86)90036-7
McCutchen, D. (2000). Knowledge, processing, and working memory: Implications for a theory of writing. Educational Psychologist, 35(1), 13-23. https://doi.org/10.1207/S15326985EP3501_3
McCutchen, D., Francis, M., & Kerr, S. (1997). Revising for meaning: Effects of knowledge and strategy. Journal of Educational Psychology, 89, 667-676. https://doi.org/10.1037/0022-0663.89.4.667
Monroe, M., & Sherman, E. (1966). Group diagnostic reading aptitude and achievement tests. Nevins.
Mosenthal, P., Conley, M., Colella, A., & Davidson-Mosenthal, R. (1985). The influence of prior knowledge and teacher lesson structure on children’s production of narratives. The Elementary School Journal, 85, 621-634. https://doi.org/10.1086/461425
Myhill, D. A., & Watson, A. (2014). The role of grammar in the writing curriculum: A review of the literature. Journal of Child Language Teaching & Therapy, 30(1), 41-62. https://doi.org/10.1177/0265659013514070
Olinghouse, N. G., & Graham, S. (2009). The relationship between the discourse knowledge and the writing performance of elementary-grade students. Journal of Educational Psychology, 101, 37-50. https://doi.org/10.1037/a0013462
Olinghouse, N. G., Graham, S., & Gillespie, A. (2015). The relationship of discourse and topic knowledge to fifth graders’ writing performance. Journal of Educational Psychology, 107, 391-406. https://doi.org/10.1037/a0037549
Pappas, C. C. (1993). Is narrative "primary"? Some insights form Kindergarteners' pretend reading of stories and information books. Journal of Reading Behavior, 25, 92-129. https://doi.org/10.1080/10862969309547803
Raphael, T. E., Englert, C. S., & Kirschner, B. W. (1989). Students' metacognitive knowledge about writing. Research in the Teaching of English, 23, 343-379. https://www.jstor.org/stable/40171153
Saddler, B., & Graham, S. (2007). The relationship between writing knowledge and writing performance among more and less skilled writers. Reading and Writing Quarterly, 23(3) 233–249. https://doi.org/10.1080/10573560701277575
Schallert, D. L. (1982). The significance of knowledge: A synthesis of research related to schema theory. In W. Otto & S. Frank (Eds.), Reading Expository Material (pp. 13-48). Academic Press.
Schoonen, R., & de Glopper, K. (1996). Writing performance and knowledge about writing. In G. Rijlaarsdam, H. van den Bergh, & M. Couzjin (Eds.), Theories, models, and methodology in writing research (pp. 87-106). Amsterdam University Press.
Schoonen, R., van Gelderen, A., de Glopper, K., Hulstijn, J., Simis, A., Snellings, P., & Stevenson, M. (2003). First language and second language writing: The role of linguistic knowledge, speed of processing, and metacognitive knowledge. Language Learning, 53, 165-202. https://doi.org/10.1111/1467-9922.00213
Schoonen, R., van Gelderen, A., Stoel, R. D., Hulstijn, J., & de Glopper, K. (2011). Modeling the development of L1 and EFL writing proficiency of secondary school students. Language Learning, 61, 31-79. https://doi.org/10.1111/j.1467-9922.2010.00590.x
Stein, N. L., & Glenn, C. C. (1979). An analysis of story comprehension in elementary school children. In R. O. Freedle (Ed.), New directions in discourse processing (Vol.2; pp.53-120). Ablex.
Stiggins, R. J. (1982). A comparison of direct and indirect writing assessment methods. Research in the Teaching of English, 16(2), 101-114. https://www.jstor.org/stable/40170937
Streiner, D. L., Norman, G. R., & Cairney, J. (2015). Health measurement scales: A practical guide to their development and use (5th ed.). Oxford University Press.
Torrance, M., Fidalgo, R., & García, J. N. (2007). The teachability and effectiveness of cognitive self-regulation in sixth-grade writers. Learning and Instruction, 17(3), 265-285. https://doi.org/10.1016/j.learninstruc.2007.02.003
Trapman, M., van Gelderen, A., van Schooten, E., & Hulstijn, J. (2018). Writing proficiency level and writing development of low-achieving adolescents: The roles of linguistic knowledge, fluency, and metacognitive knowledge. Reading and Writing: An Interdisciplinary Journal, 31, 893-926. https://doi.org/10.1007/s11145-018-9818-9
Troia, G. A., Harbaugh, A. G., Shankland, R. K., Wolbers, K. A., & Lawrence, A. M. (2013). Relationships between writing motivation, writing activity, and writing performance: Effects of grade, sex, and ability. Reading and Writing: An Interdisciplinary Journal, 26, 17-44. https://doi.org/10.1007/s11145-012-9379-2
Truckenmiller, A. J., McKindles, J. V., Petscher, Y., Eckert, T. L., & Tock, J. (2020). Expanding curriculum-based measurement in written expression for middle school. The Journal of Special Education, 54(3), 133-145. https://doi.org/10.1177/0022466919887150
Voss, J., Vesonder, G., & Spilich, G. (1980). Text generation and recall by high-knowledge and low-knowledge individuals. Journal of Verbal Learning and Verbal Behavior, 19, 651-667. https://doi.org/10.1016/S0022-5371(80)90343-6
Wijekumar, K., Graham, S., Harris, K. R., Lei, P. W., Barkel, A., Aitken, A. Ray, A., & Houston, J. (2019). The roles of writing knowledge, motivation, strategic behaviors, and skills in predicting elementary students’ persuasive writing from source material. Reading and Writing: An Interdisciplinary Journal, 32, 1431-1457. https://doi.org/10.1007/s11145-018-9836-7
Wong, B. Y., Butler, D. L., Ficzere, S. A., & Kuperis, S. (1996). Teaching low achievers and students with learning disabilities to plan, write, and revise opinion essays. Journal of Learning Disabilities, 29, 197-212. https://doi.org/10.1177/002221949602900209
Wong, B., Wong, R., & Blenkinsop, J. (1989). Cognitive and metacognitive aspects of learning disabled adolescents’ composing problems. Learning Disability Quarterly, 12, 300-322. https://doi.org/10.2307/1510212
Wray, D. (1993). What do children think about writing? Educational Review, 45(1), 67-77. https://doi.org/10.1080/0013191930450106
Published
How to Cite
Issue
Section
License
Copyright (c) 2023 Gary Troia, Frank Lawrence, Julie Brehmer, Kaitlin Glause, Heather Reichmuth
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 3.0 Unported License.
