Let's Go Learn Knowledge Base
How does look-back in comprehension testing affect reliability?
 
Silent Reading Comprehension Sub-test: Look-Backs and Reliability

(Download attached article as a PDF below)

Introduction

To many, reading comprehension is simply an understanding of a piece of writing. However, the notion of what “understanding” really means and the process involved in grasping a piece of text have been debated in education for decades. Some say that reading comprehension starts with effortless decoding of the text. After that, it becomes more complicated to define adequate comprehension and the process a reader goes through to make meaning out of the text. Experts agree that reading comprehension is a complex socio-cognitive process affected by many domains, including (but not limited to) a person's prior knowledge, motivations for reading a piece of text, activation of schemata, information gathering and processing, and social interactions with different individuals (Lewis, 1993; Rosenblatt, 1994; Ruddell & Unrau, 1994). Because of the complexity of the factors involved in reading comprehension and the expectations of what readers will take away from a piece of text, there have been different perspectives on how reading comprehension should be assessed. This short article looks at the issue of whether test-takers should be allowed to look back at passages when answering comprehension questions on reading assessments, how this issue affects the reliability of comprehension assessments, and how Let's Go Learn treats “look-backs” in its Silent Reading Comprehension sub-test.

Look-Backs in Informal Reading Inventories

Informal reading inventories (such as the Qualitative Reading Inventory, The Basic Reading Inventory, The Flynt-Cooter Reading Inventory, and the Diagnostic Assessment of Reading) have become common tools for assessing students' reading abilities. In most reading inventories, after a student finishes orally or silently reading a passage, the text of the passage is removed before he or she is directed to answer questions. This procedure is meant to give the assessor an indication of how much information the student retained from the reading of the passage and an indication of how well the student was able to process the main ideas and details of what he or she read. The main purpose of this way of assessing reading comprehension without looking back at the passage for help is to reliably quantify and qualify how much information a student retained and understood compared to other students under similar test-taking conditions (Kender & Rubenstein, 1977).

Some educators have argued that prohibiting looking back at the text to help answer comprehension questions penalizes students for short-term memory issues. They also argue that useful qualitative information is lost when children are not allowed to display their natural sense-making strategies, such as looking back at the text. Some teacher-mediated IRIs have revised their assessment protocols to give students an option of looking back at the text for help while the teacher notes in what capacity students use look-back strategies (Leslie & Caldwell, 2001; Manzo, Manzo, & Albee, 2004).

Reliability Lost vs. Affordances Gained by Allowing Look-Backs

As argued by some reading experts, allowing students to look back at the text of the reading comprehension assessment gives teachers valuable qualitative information about the test-taking and/or meaning-making strategies of students, and it provides a more authentic picture of how readers engage with text as they examine it for information. However, allowing look-backs does not necessarily translate into better assessment information about a student's reading abilities. The issue of an assessment's value often rests on its use and the realities of test administration. For schools with trained teachers and specialists who can administer one-on-one, teacher-mediated informal inventories, it might make sense to gather more qualitative data about reading behavior, such as students' propensity to ask for assistance, sub-vocalizations, or looking back to the text to help answer comprehension questions. However, when reporting quantitative data for the purpose of assigning an instructional reading level and for monitoring student progress, allowing students to look back at the text for help in answering comprehension questions introduces another variable which affects the reliability of the quantitative data reported by the assessment. This is especially true for adaptive, non-teacher-mediated, online assessments like DORA. Variables like looking back, which cannot be controlled for, introduce a higher level of error in reporting reading levels, as not all students will look back at the passage; furthermore, some students will look back at the passage because they want to clarify how they understood the passage, as opposed to looking back in order to get the right answer (Kletzien, 1991; Kender & Rubenstein, 1992).

Conclusion

Allowing students to look back at passages introduces a new variable into automated diagnostic reading assessments like DORA. Rather than reliably assessing the amount of information a student absorbed by reading passages independently, look-backs make it difficult to trust that the numerical reading levels reported by the assessment are valid. It becomes unclear whether the test assessed how much information a student took away from the passage or, rather, how well he or she hunted for answers in the text. Allowing students to re-read passages introduces a new variable to the assessment that is difficult to control for: that is, some students choose to re-read the passage while others choose not to (Leslie & Caldwell, 2001). Allowing students to re-read a passage introduces a test-administration variable that is difficult to control for on the comprehension sub-test, which produces poor reliability of the information gathered by automated reading assessments like DORA. For the purposes of providing reliable and diagnostic information on how well a student understood a passage, allowing students to read the passages only once on DORA produces better assessment information.

Analyzing approximately 25,000 students’ DORA assessments at an urban school district that took DORA in year one with look-back enabled and in year two with look-back disabled, DORA Comprehension sub-test variance decreased by approximately 55%. This district taught look-back as an active test-taking strategy, which signified that students in general did use the look-back option while being assessed in year one. As a result of these results, look-back was concluded to negatively impact DORA testing and was eliminated as an option in 2012.

References

Kender, J., & Rubenstein, H. (1977). Recall versus reinspection in IRI comprehension tests. Reading Teacher, 30, 7, 776-79.

Kletzien, S.B. (1991). Strategy use by good and poor comprehenders reading expository text of differing levels. Reading Research Quarterly, 26, 67-86.

Lewis, C. (1993). "Give people a chance": Acknowledging social differences in reading. Language Arts, 70, 454-61.

Leslie, L., & Caldwell, J. (2001). Qualitative Reading Inventory – 3. New York, NY: Addison Wesley Longman, Inc.

Manzo, A.V., Manzo, U.C., & Albee, J.J. (2004). Reading assessment for diagnostic-prescriptive teaching. Canada: Wadsworth.

Rosenblatt, L.M. (1994). The transactional theory of reading and writing. In R. Ruddell, M. Ruddell, & H. Singer (Eds.), Theoretical models and processes of reading (4th ed.) (pp. 1057-1092). Newark, DE: International Reading Association.

Ruddell, R.B. and Unrau, N.J. (1994). Reading as a meaning-construction process: The reader, the text, and the teacher. In R. Ruddell, M. Ruddell, & H. Singer (Eds.), Theoretical models and processes of reading (4th ed.) (pp. 996-1056). Newark, DE: International Reading Association.

Zabrucky, K., & Ratner, H.H. (1992). Effects of passage type on comprehension monitoring and recall in good and poor readers. Journal of Reading Behavior, 24, 373-391.
Was this Article Helpful?
Please add a quick rating! It will help us improve articles for you!
Show fields from Show fields from Show fields from a related table
Report Name *
Description
Field label
Column heading override
Justification
What does auto mean?
Fields in:

Fields to Extract:

Name for the new table:
Items in the new table are called:

When you bring additional fields into a conversion, Quick Base often finds inconsistencies. For example, say you're converting your Companies column into its own table. One company, Acme Corporation, has offices in New York, Dallas and Portland. So, when you add the City column to the conversion, Quick Base finds three different locations for Acme. A single value in the column you're converting can only match one value in any additional field. Quick Base needs you to clean up the extra cities before it can create your new table. To do so, you have one of two choices:

  • If you want to create three separate Acme records (Acme-New York, Acme-Dallas and Acme-Portland) click the Conform link at the top of the column.
  • If the dissimilar entries are mistakes (say Acme only has one office in New York and the other locations are data-entry errors) go back into your table and correct the inconsistencies—in this case, changing all locations to New York. Then try the conversion again.

Read more about converting a column into a table.

We're glad you're interested in doing more with Quick Base!

Now we need to make you official before you share apps or manage your account.

Verifying your email lets you share Quick Base with others in your company.

Your work email
Your company