Efficient measurement of writing knowledge with forced-choice tasks: Preliminary data using the student knowledge of writing tests

Authors

  • Gary Troia Michigan State University
  • Frank Lawrence Michigan State University
  • Julie Brehmer Saginaw Valley State University
  • Kaitlin Glause Millikin University
  • Heather Reichmuth University of Southern Main

DOI:

https://doi.org/10.17239/jowr-2023.15.02.06

Keywords:

writing knowledge, writing quality, assessment, late elementary

Abstract

Much of the research that has examined the writing knowledge of school-age students has relied on interviews to ascertain this information, which is problematic because interviews may underestimate breadth and depth of writing knowledge, require lengthy interactions with participants, and do not permit a direct evaluation of a prescribed array of constituent knowledge elements. For these reasons, our goal in this study is to report the development, piloting, and field testing, using a sample of 335 students from grades 4 and 5, of four alternate versions of a writing knowledge assessment—the Student Knowledge of Writing Test (SKOWT)—that uses forced-choice responses to evaluate students’ knowledge of writing processes, genre elements, and linguistic features of written language. All versions of the SKOWT demonstrated adequate internal consistency reliability and construct validity based on exploratory factor analyses following deletion of some items. In addition, there was acceptable predictive criterion validity based on associations of SKOWT scores with subtests from the Test of Written Language-4 and measures of narrative, opinion, and informative essay quality. We discuss how the SKOWT might be used in future research and educational practice. 

References

Albin, M. L., Benton, S. L., & Khramtsova, I. (1996). Individual differences in interest and narrative writing. Contemporary Educational Psychology, 21(4), 305-324. https://doi.org/10.1006/ceps.1996.0024

Alexander, P. A. (1997). Stages and phases of domain learning: The dynamics of subject-matter knowledge, strategy knowledge, and motivation. In C. E. Weinstein & B. L. McCombs (Eds.), Skill, will, and self-regulation. Erlbaum.

Alexander, P. A. (2003). The development of expertise: The journey from acclimation to proficiency. Educational Researcher, 32(8), 10-14. https://doi.org/10.3102/0013189X032008010

Apel, K., Masterson, J., & Hart, P. (2004). Integration of language components in spelling: Instruction that maximizes students’ learning. In E. R. Silliman & L. C. Wilkinson (Eds.), Language and literacy learning in schools (pp. 292-315). Guilford.

Baker, F. B., & Kim, S. (2017). The basics of item response theory using R. Springer.

Barbeiro, L. F. (2011). What happens when I write? Pupils’ writing about writing. Reading and Writing: An Interdisciplinary Journal, 24, 813-834. https://doi.org/10.1007/s11145-010-9226-2

Behizadeh, N., & Pang, M. E. (2016). Awaiting a new wave: The status of state writing assessment in the United States. Assessing Writing, 29, 25-41. https://doi.org/10.1016/j.asw.2016.05.003

Bereiter, C., & Scardamalia, M. (1984). Learning about writing from reading. Written Communication, 1, 163-88. https://doi.org/10.1177/0741088384001002001

Berman, R. A., & Slobin, D. I. (1994). Relating events in narrative: A cross-linguistic developmental study. Erlbaum.

Benton, S. L., Corkill, S. J., Sharp, J. M., Downy, R., & Khramstova, I. (1995). Knowledge, interest, and narrative writing. Journal of Educational Psychology, 87, 66¬¬¬¬¬¬¬¬-79. https://doi.org/10.1037/0022-0663.87.1.66

Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quiñonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health, social, and behavioral research: A primer. Frontiers in Public Health, 6, 149. https://doi.org/10.3389/fpubh.2018.00149

Bourassa, D., Treiman, R., & Kessler, B. (2006). Use of morphology in spelling by children with dyslexia and typically developing children. Memory and Cognition, 34, 703-714. https://doi.org/10.3758/BF03193589

Bouwer, R., Béguin, A., Sanders, T., & van den Bergh, H. (2015). Effect of genre on the generalizability of writing scores. Language Testing, 32 (1), 83-100. https://doi.org/10.1177/0265532214542994

Brown, T. (2014). Confirmatory factor analysis for applied research. Guildford Press.

Chi, M. T. H. (1978). Knowledge structures and memory development. In R. S. Siegler (Ed.), Children's thinking: What develops? (pp. 73-96). Erlbaum.

Costello, A. B., & Osborne, J. (2005). Best practices in exploratory factor analysis: Four recommendations for getting the most from your analysis. Practical Assessment, Research, and Evaluation, 10(7), 1-9. https://doi.org/10.7275/jyj1-4868

Couzijn, M. (1999). Learning to write by observation of writing and reading processes: Effects on learning and transfer. Learning and Instruction, 9, 109-142. https://doi.org/10.1016/S0959-4752(98)00040-1

DeGroff, L. C. (1987). The influence of prior knowledge on writing, conferencing, and revising. The Elementary School Journal, 88, 105-118. https://doi.org/10.1086/461527

Donovan, C. A. (2001). Children's development and control of written story and informational genres: Insights from one elementary school. Research in the Teaching of English, 35, 394-447. https://www.jstor.org/stable/40171493

Donovan, C. A., & Smolkin, L. B. (2005). Elementary school children's written scientific explanations: An examination of developing abilities. Paper presented at the annual meeting of the National Reading Conference, Miami, FL.

Duke, N. K. (1999). The scarcity of informational texts in first grade. (Report No.1-007). Center for the Improvement of Early Reading Achievement. University of Michigan.

Englert, S., Raphael, T., Fear, K., & Anderson, L. (1988). Students' metacognitive knowledge about how to write informational texts. Learning Disability Quarterly, 11, 18-46. https://doi.org/10.2307/1511035

Fidalgo, R., Torrance, M., & Garcia, J. (2008). The long-term effects of writing strategy instruction for young writers. Contemporary Educational Psychology, 33, 672-693. https://doi.org/10.1016/j.cedpsych.2007.09.001

Fitzgerald, J., & Markham, L. R. (1987). Teaching children about revision in writing. Cognition and Instruction, 4, 3-24. https://doi.org/10.1207/s1532690xci0401_1

Fitzgerald, J., & Teasley, A. (1986). Effects of instruction in narrative structure on children’s writing. Journal of Educational Psychology, 78, 424-432. https://doi.org/10.1037/0022-0663.78.6.424

Franklin, S. B., Gibson, D. J., Robertson, P. A., Pohlmann, J. T., & Fralish, J. S. (1995). Parallel analysis: A method for determining significant principal components. Journal of Vegetation Science, 6(1), 99-106. https://doi.org/10.2307/3236261

Gambrell, L. B., & Chasen, S. P. (1991). Explicit story structure instruction and the narrative writing of fourth- and fifth-grade below-average readers. Reading Research and Instruction, 31, 54-62. https://doi.org/10.1080/19388079109558071

Gillespie, A., Olinghouse, N., & Graham, S. (2013). Fifth-grade students’ knowledge about writing process and writing genres. The Elementary School Journal, 113, 565-588. https://doi.org/10.1086/669938

Graham, S. (2006). Writing. In P. Alexander & P. Winne (Eds.), Handbook of educational psychology (pp. 457-478). Erlbaum.

Graham, S., Berninger, V. W., Abbott, R. D., Abbott, S. P., & Whitaker, D. (1997). Role of mechanics in composing of elementary school students: A new methodological approach. Journal of Educational Psychology, 89, 170-182. https://doi.org/10.1037/0022-0663.89.1.170

Graham, S., Liu, X., Aitken, A., Ng, C., Bartlett, B., Harris, K. R., & Holzapfel, J. (2018). Effectiveness of literacy programs balancing reading and writing instruction: A meta‐analysis. Reading Research Quarterly, 53(3), 279-304. https://doi.org/10.1002/rrq.194

Graham, S., Schwartz, S., & MacArthur, C. A. (1993). Knowledge of writing and the composing process, attitude toward writing, and self-efficacy for students with and without learning disabilities. Journal of Learning Disabilities, 26, 237-249. https://doi.org/10.1177/002221949302600404

Hambleton, R. K. (1991). Fundamentals of item response theory. Sage.

Hammill, D. D., & Larsen, S. C. (2009). Test of written language (4th ed.). PRO-ED.

Hasan, R. (1984). The nursery tale as a genre. Nottingham Linguistic Circular, 13, 71-102. http://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&idt=12004084

Hayes, J. R. (1996). A new framework for understanding writing. In C. M. Levy & S. Randall (Eds.), The science of writing: Theories, methods, individual differences, and applications (pp.1-27). Erlbaum.

Hayes, J. R. (2011). Kinds of knowledge-telling: Modeling early writing development. Journal of Writing Research, 3(2), 73-92. https://doi.org/10.17239/jowr-2011.03.02.1

Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: A tutorial on parallel analysis. Organizational Research Methods, 7(2), 191-205. https://doi.org/10.1177/1094428104263675

Hemphill, L., Feldman, H. M., Camp, L., Griffin, T., Miranda, A. B., & Wolf, D. P. (1994). Developmental changes in narrative and non-narrative discourse in children with and without brain injury. Journal of Communication Disorders, 27, 107-133. https://doi.org/10.1016/0021-9924(94)90037-X

Holliway, D. R., & McCutchen, D. (2004). Audience perspective in young writers’ composing and revising. In L. Allal, L. Chanquoy & P. Largy (Eds.), Revision cognitive and instructional processes (pp. 87-101). Springer.

Kamberelis, G. (1999). Genre development and learning: Children writing stories, science reports, and poems. Research in the Teaching of English, 33, 403-460. https://www.jstor.org/stable/40171463

Klein, P. D., & Rose, M. A. (2010). Teaching argument and explanation to prepare junior students for writing to learn. Reading Research Quarterly, 45(4), 433-461. https://doi.org/10.1598/RRQ.45.4.4

Knudson, R. E. (1989). Effects of instructional strategies on children’s informational writing. Journal of Educational Research, 83(2), 91-96. https://doi.org/10.1080/00220671.1989.10885936

Kos, R. & Maslowski, C. (2001). Second graders’ perceptions of what is important in writing. The Elementary School Journal, 101(5), 567-585. https://doi.org/10.1086/499688

Langer, J. A. (1984). The effects of available information on responses to school writing tasks. Research in the Teaching of English, 18(1), 27-44. https://www.jstor.org/stable/40170977

Langer, J. (1986). Children reading and writing: Structures and strategies. Ablex.

Lin, S. C., Monroe, B. W., & Troia, G. A. (2007). Development of writing knowledge in grades 2-8: A comparison of typically developing writers and their struggling peers. Reading and Writing Quarterly, 23, 207-230. https://doi.org/10.1080/10573560701277542

Malpique, A., & Veiga-Simão, A. M. (2016). Argumentative writing by junior high school students: Discourse knowledge and writing performance. Infancia y Aprendizaje, 39(1), 150-186. https://doi.org/10.1080/02103702.2015.1111609

McCutchen, D. (1986). Domain knowledge and linguistic knowledge in the development of writing ability. Journal of Memory and Language, 25, 431-444. https://doi.org/10.1016/0749-596X(86)90036-7

McCutchen, D. (2000). Knowledge, processing, and working memory: Implications for a theory of writing. Educational Psychologist, 35(1), 13-23. https://doi.org/10.1207/S15326985EP3501_3

McCutchen, D., Francis, M., & Kerr, S. (1997). Revising for meaning: Effects of knowledge and strategy. Journal of Educational Psychology, 89, 667-676. https://doi.org/10.1037/0022-0663.89.4.667

Monroe, M., & Sherman, E. (1966). Group diagnostic reading aptitude and achievement tests. Nevins.

Mosenthal, P., Conley, M., Colella, A., & Davidson-Mosenthal, R. (1985). The influence of prior knowledge and teacher lesson structure on children’s production of narratives. The Elementary School Journal, 85, 621-634. https://doi.org/10.1086/461425

Myhill, D. A., & Watson, A. (2014). The role of grammar in the writing curriculum: A review of the literature. Journal of Child Language Teaching & Therapy, 30(1), 41-62. https://doi.org/10.1177/0265659013514070

Olinghouse, N. G., & Graham, S. (2009). The relationship between the discourse knowledge and the writing performance of elementary-grade students. Journal of Educational Psychology, 101, 37-50. https://doi.org/10.1037/a0013462

Olinghouse, N. G., Graham, S., & Gillespie, A. (2015). The relationship of discourse and topic knowledge to fifth graders’ writing performance. Journal of Educational Psychology, 107, 391-406. https://doi.org/10.1037/a0037549

Pappas, C. C. (1993). Is narrative "primary"? Some insights form Kindergarteners' pretend reading of stories and information books. Journal of Reading Behavior, 25, 92-129. https://doi.org/10.1080/10862969309547803

Raphael, T. E., Englert, C. S., & Kirschner, B. W. (1989). Students' metacognitive knowledge about writing. Research in the Teaching of English, 23, 343-379. https://www.jstor.org/stable/40171153

Saddler, B., & Graham, S. (2007). The relationship between writing knowledge and writing performance among more and less skilled writers. Reading and Writing Quarterly, 23(3) 233–249. https://doi.org/10.1080/10573560701277575

Schallert, D. L. (1982). The significance of knowledge: A synthesis of research related to schema theory. In W. Otto & S. Frank (Eds.), Reading Expository Material (pp. 13-48). Academic Press.

Schoonen, R., & de Glopper, K. (1996). Writing performance and knowledge about writing. In G. Rijlaarsdam, H. van den Bergh, & M. Couzjin (Eds.), Theories, models, and methodology in writing research (pp. 87-106). Amsterdam University Press.

Schoonen, R., van Gelderen, A., de Glopper, K., Hulstijn, J., Simis, A., Snellings, P., & Stevenson, M. (2003). First language and second language writing: The role of linguistic knowledge, speed of processing, and metacognitive knowledge. Language Learning, 53, 165-202. https://doi.org/10.1111/1467-9922.00213

Schoonen, R., van Gelderen, A., Stoel, R. D., Hulstijn, J., & de Glopper, K. (2011). Modeling the development of L1 and EFL writing proficiency of secondary school students. Language Learning, 61, 31-79. https://doi.org/10.1111/j.1467-9922.2010.00590.x

Stein, N. L., & Glenn, C. C. (1979). An analysis of story comprehension in elementary school children. In R. O. Freedle (Ed.), New directions in discourse processing (Vol.2; pp.53-120). Ablex.

Stiggins, R. J. (1982). A comparison of direct and indirect writing assessment methods. Research in the Teaching of English, 16(2), 101-114. https://www.jstor.org/stable/40170937

Streiner, D. L., Norman, G. R., & Cairney, J. (2015). Health measurement scales: A practical guide to their development and use (5th ed.). Oxford University Press.

Torrance, M., Fidalgo, R., & García, J. N. (2007). The teachability and effectiveness of cognitive self-regulation in sixth-grade writers. Learning and Instruction, 17(3), 265-285. https://doi.org/10.1016/j.learninstruc.2007.02.003

Trapman, M., van Gelderen, A., van Schooten, E., & Hulstijn, J. (2018). Writing proficiency level and writing development of low-achieving adolescents: The roles of linguistic knowledge, fluency, and metacognitive knowledge. Reading and Writing: An Interdisciplinary Journal, 31, 893-926. https://doi.org/10.1007/s11145-018-9818-9

Troia, G. A., Harbaugh, A. G., Shankland, R. K., Wolbers, K. A., & Lawrence, A. M. (2013). Relationships between writing motivation, writing activity, and writing performance: Effects of grade, sex, and ability. Reading and Writing: An Interdisciplinary Journal, 26, 17-44. https://doi.org/10.1007/s11145-012-9379-2

Truckenmiller, A. J., McKindles, J. V., Petscher, Y., Eckert, T. L., & Tock, J. (2020). Expanding curriculum-based measurement in written expression for middle school. The Journal of Special Education, 54(3), 133-145. https://doi.org/10.1177/0022466919887150

Voss, J., Vesonder, G., & Spilich, G. (1980). Text generation and recall by high-knowledge and low-knowledge individuals. Journal of Verbal Learning and Verbal Behavior, 19, 651-667. https://doi.org/10.1016/S0022-5371(80)90343-6

Wijekumar, K., Graham, S., Harris, K. R., Lei, P. W., Barkel, A., Aitken, A. Ray, A., & Houston, J. (2019). The roles of writing knowledge, motivation, strategic behaviors, and skills in predicting elementary students’ persuasive writing from source material. Reading and Writing: An Interdisciplinary Journal, 32, 1431-1457. https://doi.org/10.1007/s11145-018-9836-7

Wong, B. Y., Butler, D. L., Ficzere, S. A., & Kuperis, S. (1996). Teaching low achievers and students with learning disabilities to plan, write, and revise opinion essays. Journal of Learning Disabilities, 29, 197-212. https://doi.org/10.1177/002221949602900209

Wong, B., Wong, R., & Blenkinsop, J. (1989). Cognitive and metacognitive aspects of learning disabled adolescents’ composing problems. Learning Disability Quarterly, 12, 300-322. https://doi.org/10.2307/1510212

Wray, D. (1993). What do children think about writing? Educational Review, 45(1), 67-77. https://doi.org/10.1080/0013191930450106

Published

2023-03-28

Issue

Section

Methods

How to Cite

Efficient measurement of writing knowledge with forced-choice tasks: Preliminary data using the student knowledge of writing tests . (2023). Journal of Writing Research, 15(2), 395-427. https://doi.org/10.17239/jowr-2023.15.02.06

Similar Articles

101-110 of 241

You may also start an advanced similarity search for this article.