Silent Reading Comprehension Sub-test: Look-Backs and Reliability
(Download attached article as a PDF below)
To many, reading comprehension is simply an understanding of a piece of writing. However, the notion of what “understanding” really means and the process involved in grasping a piece of text have been debated in education for decades. Some say that reading comprehension starts with effortless decoding of the text. After that, it becomes more complicated to define adequate comprehension and the process a reader goes through to make meaning out of the text. Experts agree that reading comprehension is a complex socio-cognitive process affected by many domains, including (but not limited to) a person's prior knowledge, motivations for reading a piece of text, activation of schemata, information gathering and processing, and social interactions with different individuals (Lewis, 1993; Rosenblatt, 1994; Ruddell & Unrau, 1994). Because of the complexity of the factors involved in reading comprehension and the expectations of what readers will take away from a piece of text, there have been different perspectives on how reading comprehension should be assessed. This short article looks at the issue of whether test-takers should be allowed to look back at passages when answering comprehension questions on reading assessments, how this issue affects the reliability of comprehension assessments, and how Let's Go Learn treats “look-backs” in its Silent Reading Comprehension sub-test.
Look-Backs in Informal Reading Inventories
Informal reading inventories (such as the Qualitative Reading Inventory, The Basic Reading Inventory, The Flynt-Cooter Reading Inventory, and the Diagnostic Assessment of Reading) have become common tools for assessing students' reading abilities. In most reading inventories, after a student finishes orally or silently reading a passage, the text of the passage is removed before he or she is directed to answer questions. This procedure is meant to give the assessor an indication of how much information the student retained from the reading of the passage and an indication of how well the student was able to process the main ideas and details of what he or she read. The main purpose of this way of assessing reading comprehension without looking back at the passage for help is to reliably quantify and qualify how much information a student retained and understood compared to other students under similar test-taking conditions (Kender & Rubenstein, 1977).
Some educators have argued that prohibiting looking back at the text to help answer comprehension questions penalizes students for short-term memory issues. They also argue that useful qualitative information is lost when children are not allowed to display their natural sense-making strategies, such as looking back at the text. Some teacher-mediated IRIs have revised their assessment protocols to give students an option of looking back at the text for help while the teacher notes in what capacity students use look-back strategies (Leslie & Caldwell, 2001; Manzo, Manzo, & Albee, 2004).
Reliability Lost vs. Affordances Gained by Allowing Look-Backs
As argued by some reading experts, allowing students to look back at the text of the reading comprehension assessment gives teachers valuable qualitative information about the test-taking and/or meaning-making strategies of students, and it provides a more authentic picture of how readers engage with text as they examine it for information. However, allowing look-backs does not necessarily translate into better assessment information about a student's reading abilities. The issue of an assessment's value often rests on its use and the realities of test administration. For schools with trained teachers and specialists who can administer one-on-one, teacher-mediated informal inventories, it might make sense to gather more qualitative data about reading behavior, such as students' propensity to ask for assistance, sub-vocalizations, or looking back to the text to help answer comprehension questions. However, when reporting quantitative data for the purpose of assigning an instructional reading level and for monitoring student progress, allowing students to look back at the text for help in answering comprehension questions introduces another variable which affects the reliability of the quantitative data reported by the assessment. This is especially true for adaptive, non-teacher-mediated, online assessments like DORA. Variables like looking back, which cannot be controlled for, introduce a higher level of error in reporting reading levels, as not all students will look back at the passage; furthermore, some students will look back at the passage because they want to clarify how they understood the passage, as opposed to looking back in order to get the right answer (Kletzien, 1991; Kender & Rubenstein, 1992).
Allowing students to look back at passages introduces a new variable into automated diagnostic reading assessments like DORA. Rather than reliably assessing the amount of information a student absorbed by reading passages independently, look-backs make it difficult to trust that the numerical reading levels reported by the assessment are valid. It becomes unclear whether the test assessed how much information a student took away from the passage or, rather, how well he or she hunted for answers in the text. Allowing students to re-read passages introduces a new variable to the assessment that is difficult to control for: that is, some students choose to re-read the passage while others choose not to (Leslie & Caldwell, 2001). Allowing students to re-read a passage introduces a test-administration variable that is difficult to control for on the comprehension sub-test, which produces poor reliability of the information gathered by automated reading assessments like DORA. For the purposes of providing reliable and diagnostic information on how well a student understood a passage, allowing students to read the passages only once on DORA produces better assessment information.
Analyzing approximately 25,000 students’ DORA assessments at an urban school district that took DORA in year one with look-back enabled and in year two with look-back disabled, DORA Comprehension sub-test variance decreased by approximately 55%. This district taught look-back as an active test-taking strategy, which signified that students in general did use the look-back option while being assessed in year one. As a result of these results, look-back was concluded to negatively impact DORA testing and was eliminated as an option in 2012.
Kender, J., & Rubenstein, H. (1977). Recall versus reinspection in IRI comprehension tests. Reading Teacher, 30, 7, 776-79.
Kletzien, S.B. (1991). Strategy use by good and poor comprehenders reading expository text of differing levels. Reading Research Quarterly, 26, 67-86.
Lewis, C. (1993). "Give people a chance": Acknowledging social differences in reading. Language Arts, 70, 454-61.
Leslie, L., & Caldwell, J. (2001). Qualitative Reading Inventory – 3. New York, NY: Addison Wesley Longman, Inc.
Manzo, A.V., Manzo, U.C., & Albee, J.J. (2004). Reading assessment for diagnostic-prescriptive teaching. Canada: Wadsworth.
Rosenblatt, L.M. (1994). The transactional theory of reading and writing. In R. Ruddell, M. Ruddell, & H. Singer (Eds.), Theoretical models and processes of reading (4th ed.) (pp. 1057-1092). Newark, DE: International Reading Association.
Ruddell, R.B. and Unrau, N.J. (1994). Reading as a meaning-construction process: The reader, the text, and the teacher. In R. Ruddell, M. Ruddell, & H. Singer (Eds.), Theoretical models and processes of reading (4th ed.) (pp. 996-1056). Newark, DE: International Reading Association.
Zabrucky, K., & Ratner, H.H. (1992). Effects of passage type on comprehension monitoring and recall in good and poor readers. Journal of Reading Behavior, 24, 373-391.