The Validity of Formative Assessments

by Terry Crooks

Educational Assessment Research Unit, University of Otago, Box 56, Dunedin, New Zealand

Terry.Crooks@otago.ac.nz

Paper presented to the British Educational Research Association Annual Conference, University of Leeds, 13-15 September 2001

Note: This paper is incomplete as yet. The major points to be included in the paper are presented here, but the linking text is yet to be written. Contact the author to receive the full paper when completed.



Defining Assessment

Any process that provides information about the thinking, achievement or progress of students



Defining Validity

Messick, 1989, p.13 (Validity)

Validity is an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.



Distinguishing Between Summative and Formative Assessment

Summative assessment is intended to summarise student attainment at a particular time, whereas formative assessment is intended to promote further improvement of student attainment.

Assessment OF learning versus assessment FOR learning.



The Formative Impact of Summative Assessment

Crooks, 1988, p. 467(The impact of classroom evaluation practices on students)

Classroom evaluation affects students in many different ways. For instance, it guides their judgement of what is important to learn, affects their motivation and self-perceptions of competence, structures their approaches to and timing of personal study (e.g. spaced practice), consolidates learning, and affects the development of enduring learning strategies and skills.

It appears to be one of the most potent forces influencing education. Accordingly, it deserves very careful planning and considerable investment of time from educators. Many of the skills and attitudes that are goals of education take years to develop, and their development can be undermined by lack of consistent support for them in the educational experiences of the students.

Adapted from Crooks, 1988, p. 443 (The impact of classroom evaluation practices on students)

Short Term Effects

*

Focussing attention on important aspects of the subject;
*

Giving students opportunities to practice skills and consolidate learning;
*

Guiding further instructional or learning activities within the course;

Medium and Longer Term Effects

*

Influencing students' motivation as learners and their perceptions of their capabilities (self-efficacy);
*

Communicating and reinforcing (or in some cases undermining) teaching goals, including key performance criteria and desired standards of performance;
*

Influencing students' choice of (and development of) learning strategies, skills and study patterns;
*

Influencing students' subsequent choice of courses, activities or careers



Definitions of Formative Assessment

Ministry of Education, New Zealand, 1994 (Assessment: policy to practice)

A range of formal and informal assessment procedures (for example, the monitoring of children's writing development, anecdotal records, and observations) undertaken by teachers in the classroom as an integral part of the normal teaching and learning process in order to modify and enhance learning and understanding.

Bronwen Cowie & BeverleyBell, 1996/1999 (A model of formative assessment in science education)

The process used by teachers and students to recognise and respond to student learning in order to enhance that learning, during the learning.

Black & Wiliam, 1998, p. 2 (Inside the black box)

Assessment' refers to all those activities undertaken by teachers, and by the students in assessing themselves, which provide information to be used as feedback to modify the teaching and learning activities in which they are engaged. Such assessment become 'formative assessment' when the evidence is actually used to adapt the teaching to meet the needs.



Key Points about Formative Assessment

Sadler, 1989, p.119 (Formative assessment and the design of instructional systems)

A key premise is that for students to be able to improve, they must have the capacity to monitor the quality of their own work during actual production.

This in turn requires:

*

that students possess an appreciation of what high quality work is,
*

that they have the evaluative skill necessary for them to compare with some objectivity the quality of what they are producing in relation to the higher standard, and
*

that they develop a store of tactics or moves which can be drawn upon to modify their own work.

Instructional systems which do not make explicit provision for the acquisition of evaluative expertise are deficient, because they set up artificial but potentially removable performance ceilings for students.

Crooks, 1988, pp. 468-9 (The impact of classroom evaluation practices on students)

The approach that leads to the most valuable feedback is nicely captured by Easley and Zwoyer (1975):

"If you can both listen to children and accept their answers not as things to just be judged right or wrong but as pieces of information which may reveal what the child is thinking you will have taken a giant step toward becoming a master teacher rather than merely a disseminator of information. (p. 25)"

Crooks, 1988, p. 469 (The impact of classroom evaluation practices on students)

Feedback should be specific and related to need. Simple knowledge of results should be provided consistently (directly or implicitly), with more detailed feedback only where necessary to help the student work through misconceptions or other weaknesses in performance. Praise should be used sparingly and where used should be task-specific, whereas criticism (other than simply identifying deficiencies) is usually counterproductive.



A Summary of Findings from Research on Formative Assessment

Educational research has shown that providing high quality feedback on student work is a very powerful way of raising the standard of student work. Paul Black and Dylan Wiliam (1998), in a 10 year review of research on assessment, stated that "We know of no other way of raising standards for which such a prima facie case can be made." John Hattie (1999), in his inaugural professorial lecture as Dean of Education at the University of Auckland, summarised his wide-ranging review of research on "what works" in education with the statement "the most powerful single moderator that enhances achievement is feedback."

Royce Sadler (1989) identified three elements that are crucial to the effectiveness of formative assessment:

*

helping students to recognise clearly the desired goal (understand what is required);
*

providing students with evidence about how well their work matches that goal;
*

explaining ways to close the gap between the goal and their current performance.

Self-assessment is a vital component in learning. Feedback on assessment cannot be effective unless students accept that their work can be improved and identify important aspects of their work that they wish to improve. Self-monitoring is a key component of the work of all professionals, so if we want our students to become professional learners and professionals in their fields we should actively promote self-assessment. If students are asked and encouraged to critically examine and comment on their own work, assessment can become more dialogue than monologue, and can contribute powerfully to the educational development of students. As Wynne Harlen and Mary James (1996) put it,

"...students have to be active in their own learning (teachers cannot learn for them) and unless they come to understand their own strengths and weaknesses, and how they might deal with them, they will not make progress."

Marks or grades alone produce no learning gains. Indeed, there is some evidence that students gain the most learning value from assessment when feedback is provided without marks or grades. Where marks are provided, they often seem to predominate in students' thinking, and to be seen as the real purpose of the assessment.

Student motivation is crucial to learning. Assessment is one of the major influences on student motivation. It is important, therefore, to anticipate and try to optimise the motivational effects of feedback on assessment. The research evidence available suggests that the greatest motivational benefits will come from focussing feedback on:

*

the qualities of the student's work, and not on comparisons with other students;
*

specific ways in which the student's work could be improved;
*

improvements that the student has made compared to his or her earlier work.

Five points summarise the key lessons from research about formative assessment. Assessment that promotes learning:

*

involves learning goals understood and shared by both teachers and students;
*

helps students to understand and recognise the desired standards;
*

involves students in self-assessment;
*

provides feedback which helps students to recognise next steps and how to take them;
*

builds confidence that students can improve their work.



Some Other Summaries of Requirements for Effective Formative Assessment

Black & Wiliam, 1998 (The Black Box)

Feedback to any pupil should be about the particular qualities of his or her work, with advice on what he or she can do to improve, and should avoid comparisons with other pupils.

For formative assessment to be productive, pupils should be trained in self-assessment so that they can understand the main purposes of their learning and thereby grasp what they need to do to achieve.

Opportunities for pupils to express their understanding should be designed into any piece of teaching, for this will initiate the interaction whereby formative assessment aids learning.

The dialogue between pupils and a teacher should be thoughtful, reflective, focused to evoke and explore understanding, and conducted so that all pupils have an opportunity to think and to express their ideas.

Tests and homework exercises can be an invaluable guide to learning, but the exercises must be clear and relevant to learning aims. The feedback on them should give each pupil guidance on how to improve, and each must be given opportunity and help to work at the improvement.

Assessment Reform Group, UK, 1999, p. 7 (Assessment for learning: beyond the black box)

Assessment that promotes learning:

*

is embedded in a view of teaching and learning of which it is an essential part;
*

involves sharing learning points with pupils;
*

aims to help pupils to know and to recognise the standards they are aiming for;
*

involves pupils in self-assessment;
*

provides feedback which leads to pupils recognising their next steps and how to take them;
*

is underpinned by confidence that every student can improve;
*

involves both teacher and pupils reviewing and reflecting on assessment data.



Formative Assessment of Completed or Draft Work

Student-Centred Contextual Components

Who is this student? What cultural or other considerations require special attention?

How confident is this student about this curriculum area or type of task? (self-efficacy)

How does this student respond to praise, criticism, and guidance? What works best?

Task-Specific Contextual Components

Did the student understand the task?

*

its broad thrust
*

the specific qualities desired, and their relative importance
*

the standards that would be applied.

How much effort has the student put into completing the task?

*

were there mitigating commitments or circumstances?

Did the student receive substantial help (or distraction!) while working on the task?

*

from school staff, peers, or family members

What use did the student make of help available from others?

*

ignored or rejected it
*

learned from it, and incorporated it into his/her work
*

used it essentially verbatim, without evidence of learning

Summative Components

Overall, how well does the work meet the desired criteria and standards?

Should a mark/grade be recorded for the student? If so, where and how?

Components with Both Summative and Formative Aspects

What are the strengths of the work? (criterion referenced)

What are the weaknesses of the work? (criterion referenced)

Should the student be given "knowledge of results" (in a criterion referenced way)?

Formative Components

How does this student's performance match the teacher's expectations for that student?

*

distinctly better than expected
*

largely as expected
*

distinctly worse than expected

What evaluative feedback should be given to this student?

*

what, if anything, should be commended?
*

what, if anything, should be criticised?
*

what should be passed over without comment?

What corrective suggestions/requirements should be given to this student?

*

mainly focussed on improving this particular piece of work
*

mainly focussed on better performance on future work

Should the student be required or encouraged to resubmit the work with improvements?

*

if so, is summative assessment of the work then appropriate



Formative Assessment of Work In Progress

Student-Centred Contextual Components

Who is this student? What cultural or other considerations require special attention?

How confident is the student about this curriculum area or type of task? (self-efficacy)

How does the student respond to praise, criticism, and guidance? What works best?

Task-Specific Contextual Components

Is this the sort of task that the student usually finds motivating?

Does the student understand the task?

*

its broad thrust
*

the specific qualities desired, and their relative importance
*

the standards that will be applied.

Does the student show signs of existing or emerging beliefs or understandings, related to the task, which will obstruct the desired learning or performance?

How much effort is the student putting into completing the task well?

Are other people, events, or circumstances interfering with the student's work?

What use is the student making of help from others?

Formative Components

What questions should be asked to better understand what the student is doing?

What should be done at this point to encourage the development of self-assessment skills?

How well is the work progressing towards meeting the desired criteria and standards?

What are the strengths of the work thus far? (criterion referenced)

What are the weaknesses of the work thus far? (criterion referenced)

How is the student's performance matching the teacher's expectations for that student?

*

distinctly better than expected
*

largely as expected
*

distinctly worse than expected

What evaluative feedback on the progress made should be given to this student?

*

what, if anything, should be commended at this stage?
*

what, if anything, should be criticised at this stage?
*

what should be passed over without comment at this stage?

What corrective suggestions/requirements should be given to this student?

*

focussed on improving this particular piece of work
*

intended to enhance the student's work more generally

Are there issues here that should be raised with other students or with the class as a whole?

What is required before this work can be regarded as completed, and ready for summative assessment (if required)?



Key Factors Influencing the Validity of Formative Assessment

Affective Factors

Motivation

Teacher is devoted to helping student learn

Student cares about learning and wants to improve

Trust

Teacher is encouraging, constructive, sensitive to student's feelings

Class/peer relationships and attitudes support student's learning

Student feels safe to admit difficulties and uncertainties

Task Factors

Knowledge

Teacher understands the key aspects and difficulties of the task

Criteria

Teacher identifies and explains well the qualities sought

Student understands clearly what is needed

Standards

Teacher sets standards appropriate to student

Through descriptions and examples, the standards are explained

Student understands the standards and accepts them as appropriate

Structural Factors

Connections

Final version of task can benefit from the formative assessment

Work on subsequent tasks can benefit from the formative assessment

Purposes

Formative use of task is not undermined by parallel summative use

Process Factors

Self-assessment

Teacher helps student to develop self-assessment skills

Student takes increasing responsibility for his/her own learning

Peer involvement

Teacher encourages collaboration among students to improve work

Peers learn to be constructive and generous in offering feedback

Monitoring

Teacher monitors student's work to track both process and progress

Insight

Teacher detects misunderstandings or other obstacles to success

Teacher detects exciting possibilities in student's work

Timing

Feedback is given at times when student is most receptive to it

Balance

Feedback gives attention to strengths as well as weaknesses

Selectivity

Feedback addresses mainly the aspects likely to have biggest benefit

Wisdom

Feedback is convincing, appreciated, and useful to student



Reference List

Assessment Reform Group, U.K. (1999). Assessment for learning: beyond the black box. Cambridge, U.K: University of Cambridge School of Education.

Black, P., & Wiliam, D. (1998a). Assessment and classroom learning. Assessment in Education, 5, 7-74.

Black, P., & Wiliam,D. (1998b). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139-149.

Cowie, B., & Bell, B. (1999). A model of formative assessment in science education. Assessment in Education, 6, 101-116.

Crooks, T. J. (1988). The impact of classroom evaluation practices on students. Review of Educational Research, 58, 438-481.

Gipps, C. (1994). Beyond testing: towards a theory of educational assessment. London: Falmer Press.

Harlen, W., & James, M. (1996). Creating a positive impact of assessment on learning. Paper presented at the annual meeting of the American Educational Research Association, New York, April 1996.

Hattie, J. (1999). Influences on student learning. University of Auckland, New Zealand: Inaugural professorial lecture.

Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational Measurement (3rd edn.), pp. 13-103. New York: Macmillan.

Ministry of Education, New Zealand. (1994). Assessment: policy to practice. Wellington, New Zealand: Learning Media.

Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119-144.