Meta-Analysis in Educational Research.

by Robert L Bangert-Drowns,
SUNY

Lawrence M Rudner
ERIC Clearinghouse on Assessment and Evaluation

"I had hoped to find research to support or to conclusively oppose my belief that quality integrated education is the most promising approach. For every study that contains a recommendation, there is another, equally well documented study, challenging the conclusions of the first...No one seems to agree with anyone else's approach. But more distressing: no one seems to know what works."

Senator Fritz Mondale's quote illustrates a common plight. Educational research often produces contradictory results. Differences among studies in treatments, settings, measurement instruments, and research methods make research findings difficult to compare. Even frequent replications can prove inconclusive. Literature on a topic may be so extensive as to obscure trends with an overwhelming amount of information.

Meta-analysis is a collection of systematic techniques for resolving apparent contradictions in research findings. Meta-analysts translate results from different studies to a common metric and statistically explore relations between study characteristics and findings.

This article first describes meta-analysis as a research method. The need and general approach are discussed. We then identify some common approaches toward conducting meta-analysis in education and outline their advantages and disadvantages.

META-ANALYSIS AS A RESEARCH METHOD

Gene Glass first used the term "meta-analysis" in 1976 to refer to a philosophy, not a statistical technique. Glass argued that literature review should be as systematic as primary research and should interpret the results of individual studies in the context of distributions of findings, partially determined by study characteristics and partially random. Since that time, meta-analysis has become a widely accepted research tool, encompassing a family of procedures used in a variety of disciplines. A recent search of the ERIC database identified over 800 articles written after 1980 that use or discuss meta-analysis.

Meta-analysis responds to several problems in educational research. First, important issues are studied by numerous investigators. The amount of information on a given topic therefore is often overwhelming and not amenable to summary. Even when there are relatively few studies on a given topic, it is difficult to determine if outcome differences are attributable to chance, to methodological inadequacies, or to systematic differences in study characteristics. Informal methods of narrative review permit biases to remain easily undetected. Reviewers' biases can influence decisions about study inclusion, relative weights given to different findings, and analysis of relations between study features and outcomes. These biases can have clandestine effects when reviewers do not systematically seek to reduce them or provide sufficient information for readers to evaluate their extent.

Meta-analysis typically follows the same steps as primary research. The meta-analyst first defines the review's purpose. Organizing frameworks can be practical or theoretical questions of varying scope, but they must be clear enough to guide study selection and data collection. Second, sample selection consists of applying specified procedures for locating studies that meet specified criteria for inclusion. Typically, meta-analyses are comprehensive reviews of the full population of relevant studies. Third, data are collected from studies in two ways. Study features are coded according to the objectives of the review and as checks on threats to validity. Study outcomes are transformed to a common metric so that they can be compared. A typical metric in educational research is the effect size, the standardized difference between treatment and control group means. Finally, statistical procedures are used to investigate relations among study characteristics and findings.

Criticisms of meta-analysis tend to fall into two categories. Some complain that meta-analysis obscures important qualitative information by "averaging" simple numerical representations across studies. Other critics argue that research is best reviewed by a reflective expert who can sift kernels of insight from the confusing argumentation of a field.

META-ANALYTIC APPROACHES

VOTE-COUNTING -- Some reviews categorize findings as significantly positive (favoring the treatment group), significantly negative, or nonsignificant. The category with the most entries is considered the best representation of research in this area. This as an inexact approach to integrating research. Vote-counting confuses treatment effect and sample size because statistical significance is a function of both. Given the modest power of typical educational research to detect true effects as statistically significant, conclusions from vote-counting can be very misleading.

CLASSIC OR GLASSIAN META-ANALYSIS -- Glass' early meta-analyses set the pattern for conventional meta-analysis: define questions to be examined, collect studies, code study features and outcomes, and analyze relations between study features and outcomes. These early meta-analyses, and later ones following this tradition, share three distinguishing features. First, "classic" meta-analysis applies liberal inclusion criteria. Glass argued that one should not disregard studies on the basis of study quality a priori; a meta-analysis itself can determine if study quality is related to variance in reported treatment effect. Second, the unit of analysis is the study finding. A single study can report many comparisons between groups and subgroups on different criteria. Effect sizes are calculated for each comparison. Third, meta-analysts using this approach may average effects from different dependent variables, even when these measure different constructs.

Glassian meta-analysis has proven quite robust when submitted to critical re-analysis. Its use of conventional statistical tests render the method and its results accessible to most educational researchers. However, using study findings as the units of analysis produces nonindependent data and gives greater weight to studies with many comparisons. Averaging across constructs and including studies with obvious methodological flaws can confuse the reliability of findings.

STUDY EFFECT META-ANALYSIS -- Study effect meta-analysis alters the Glassian form in two ways. First, inclusion rules are more selective. Studies with serious methodological flaws are excluded. Second, the study is the unit of analysis. One effect size is computed for each study. This preserves the independence of the data and gives equal weight to all included studies. Unfortunately, it also reduces the number of data points analyzed in the review. And, of course, a reviewer's biases may operate in decisions to exclude studies.

TESTS OF HOMOGENEITY -- Some reviewers argue that conventional statistical tests are inappropriate for meta-analysis. Homogeneity tests were developed to determine the likelihood that variance among effect sizes is due only to sampling error. If the homogeneity statistic is significant for a group of studies, a procedure analogous to analysis of variance can be used. Studies are repeatedly divided into subgroups according to study features until within-group variation is nonsignificant.

Numerous factors can cause variation in effect sizes: measurement unreliability, range restrictions, reporting errors, within-study statistical adjustments, unreported factors, etc. Homogeneity tests are very likely to indicate heterogeneity among effect sizes even when the variation is of no practical or theoretical importance. Successively dividing subgroups according to these tests can capitalize on chance and cause the incorrect identification of moderators. Kulik and Kulik defend conventional analysis of variance for meta-analysis and suggest that homogeneity tests may ignore an important nesting factor.

PSYCHOMETRIC META-ANALYSIS -- Hunter and Schmidt's approach to meta-analysis combines some of the best features of other approaches. All studies related to a given topic are gathered, regardless of quality. The distribution of effect sizes is corrected for sampling error, measurement error, range restriction, and other systematic artifacts. If the remaining variance is still large, effect sizes are grouped into subsets according to preselected study features, and each subset is meta-analyzed separately. Ideally, the meta-analysis should estimate true treatment effects under conditions typical of those represented in the studies and predict treatment effects under conditions determined by the reviewer. Unfortunately, this technique requires substantial information from individual studies for accurate correction of effect sizes. This information is not always available in research reports.

author: Bangert-Drowns, Robert L. & Rudner, Lawrence M. (1991). Meta-analysis in educational research. Practical Assessment, Research & Evaluation, 2(8). Retrieved October 4, 2011 from http://PAREonline.net/getvn.asp?v=2&n=8 . This paper has been viewed 44,141 times since 11/13/1999.


SUGGESTED READING

Bangert-Drowns, R.L. (1986). Review of developments in meta-analytic methods. Psychological Bulletin, 99, 388-399.

Glass, G.V, McGaw, B., & M.L. Smith (1981). Meta-analysis in social research. Beverly Hills, CA: Sage.

Hunter, J.E., & F.L. Schmidt (1990). Methods of meta-analysis. Newbury Park, CA: Sage.

Kulik, J.A., & C.-L.C. Kulik (1989). Meta-analysis in education. International Journal of Educational Research, 13, 221-340.

The Case for Validity Generalization

by Rafilson, Fred
Wonderlick, Inc.

An important issue in educational and employment settings is the degree to which evidence of validity obtained in one situation can be generalized to another situation without further study of validity in the new situation. The issue of Validity Generalization is discussed in this article. Theory, procedures, and applications are addressed.

The extent to which predictive or concurrent evidence of validity can be used as criterion-related evidence in new situations is, in large measure, a function of accumulated research. In the past, judgments about the generalization or transportability of validity were often based on nonquantitative reviews of the literature. Today, quantitative techniques have been more frequently employed to study the generalization of validity (Schmidt, Hunter, Pearlman, & Hirsh, 1985). Both approaches have been used to support inferences about the degree to which the validity of a given predictor variable can generalize from one situation or setting to another similar set of circumstances.

If validity generalization evidence is limited, then local criterion-related evidence of validity may be necessary to justify the use of a test. If, on the other hand, validity generalization evidence is extensive, then situation-specific evidence of validity may not be required.

THEORY

A major limitation to local validation studies is that they can readily suffer from unseen local methodological problems. By comparing validation and fairness findings across multiple studies, however, it is possible to determine if the criterion-related validity of a test is relatively stable or if the test is valid only in certain situations. Drawing on meta-analysis techniques, this comparative procedure is called validity generalization in the personnel selection and psychometric literature.

Several types of measures lend themselves particularly well to validity generalization. Meta-analyses of the plethora of validity studies conducted on general cognitive ability (g) have repeatedly shown that the validity of g for predicting success in a given job differs little from one setting to another (Schmidt & Hunter, 1981). Thus, there is significant evidence that the validation results for general cognitive ability measures are generalizable across settings. It is not necessary, therefore, to conduct a validity study for a given job at every business location in America. The validity of 'general cognitive ability' for predicting clerical performance in one setting, for example, can be inferred from the validity found in the hundreds of previous studies.

Another limitation of specific local validation studies is the accuracy of the generated statistics (Schmidt, Hunter & Urry, 1976). Accurate statistics require large sample sizes. The criterion related validity of a test in a local validation study is usually inferred only if the findings reach a certain level of magnitude called 'statistical significance'. The smaller the sample of subjects, the higher the observed validity coefficient would need to be in order to infer an acceptable level of validity.

You would not expect, for example, to draw accurate predictions of a national election by polling a sample of only 15 voters. Most polls interview 1,000 voters or more. The same is true of the statistics produced by a local validation study; there is huge sampling error in individual validation studies conducted with small samples. Unless there are hundreds of subjects at a particular location, the data cannot be used to draw accurate conclusions in isolation. Rather, the data from small local samples can only be used cumulatively by combining them with the results from other local studies as is done in a validity generalization study.

PROCEDURE

In conducting validity generalization studies, data used from local studies may vary according to several situational facets. These may include:

* differences in the way the predictor construct are measured;
* the type of job or curriculum involved;
* the type of criterion measure;
* the type of test takers; and
* the time period in which the study was conducted.

In any particular validity generalization study, any number of these facets may vary. A major objective of the study is to determine whether variation in these facets affects the generalizability of validity evidence.

A common procedure for conducting a meta-analysis to determine the degree to which validity findings can be generalized is to

a) estimate the population validity by computing the mean of the observed sample validities,

b) correct the observed validities by removing the effects of statistical artifacts (Four readily quantifiable artifacts which can be controlled statistically are: sampling error, criterion unreliability, range restriction, and predictor unreliability),

c) find the variance of the corrected observed validities (the residual variance of the observed correlations after removing the statistical artifacts).

If the variance of the corrected observed validity is nearly zero, then validity generalizes and can be transported to other situations or locations.

MODELS

At present there are three different models for assessing Validity Generalization:

* the correlation model,
* the covariance model, and
* the regression slope model.

A recent empirical Monte Carlo study (Raju, Williams, & Pappas, 1989), conducted with an extremely large database (N=84,808), showed that all three models perform similarly. The regression slope model, however, may be more robust in some situations when the metrics for the predictor and the criterion can be considered comparable across studies.

APPLICATIONS

There are two main uses of validity generalization studies. First, the results of generalization studies can serve to draw scientific conclusions about the relationships between variables. A good example of this application is the conclusion drawn by Hunter and Schmidt (1981) that "the most frequently used cognitive ability tests are valid for all jobs and all job families...that the validity of the cognitive tests studied is neit her specific to situations or specific to jobs." In turn, these findings can improve our understanding of the true test/criterion relationships, allowing for a more useful application of predictor scores.

Second, the evidence of criterion related validity obtained from prior studies can be used to support the use of a test in a new situation. This application of validity generalization theory has enormous potential for educators and employers who lack sufficient sample sizes or resources in a given organization, yet would like to implement a proven valid testing program. This 'transference' of a test from one situation in which the test has been proven valid to another similar situation or location is often referred to as the 'transportability' of validity from one situation to another.

author: Rafilson, Fred (1991). The case for validity generalization. Practical Assessment, Research & Evaluation, 2(13). Retrieved October 4, 2011 from http://PAREonline.net/getvn.asp?v=2&n=13 . This paper has been viewed 27,025 times since 11/13/1999.


REFERENCES

Raju, N.S., Williams, C.P., & Pappas, S., (1989), An empirical monte carlo test of the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization. Journal of Applied Psychology, 74, 901911.

Schmidt, F.L., & Hunter, J.E. (1981), Employment testing: Old theories and new research findings. American Psychologist, 36, 1128-1137.

Schmidt, F.L., Hunter, J.E., Pearlman, K., & Hirsh, H.R. (1985). Forty questions about validity generalization and meta-analysis. Personnel Psychology, 38, 697-798.

Schmidt, F.L., Hunter, J.E., & Urry, V.W. (1976), Statistical power in criterion-related validity studies. Journal of Applied Psychology, 61, 473-485.

Reliability and Validity

In order for assessments to be sound, they must be free of bias and distortion. Reliability and validity are two concepts that are important for defining and measuring bias and distortion.


Reliability refers to the extent to which assessments are consistent. Just as we enjoy having reliable cars (cars that start every time we need them), we strive to have reliable, consistent instruments to measure student achievement. Another way to think of reliability is to imagine a kitchen scale. If you weigh five pounds of potatoes in the morning, and the scale is reliable, the same scale should register five pounds for the potatoes an hour later (unless, of course, you peeled and cooked them). Likewise, instruments such as classroom tests and national standardized exams should be reliable – it should not make any difference whether a student takes the assessment in the morning or afternoon; one day or the next.


Another measure of reliability is the internal consistency of the items. For example, if you create a quiz to measure students’ ability to solve quadratic equations, you should be able to assume that if a student gets an item correct, he or she will also get other, similar items correct. The following table outlines three common reliability measures.




Type of Reliability

How to Measure
Stability or Test-Retest Give the same assessment twice, separated by days, weeks, or months. Reliability is stated as the correlation between scores at Time 1 and Time 2.
Alternate Form Create two forms of the same test (vary the items slightly). Reliability is stated as correlation between scores of Test 1 and Test 2.
Internal Consistency (Alpha, a) Compare one half of the test to the other half. Or, use methods such as Kuder-Richardson Formula 20 (KR20) or Cronbach's Alpha.




The values for reliability coefficients range from 0 to 1.0. A coefficient of 0 means no reliability and 1.0 means perfect reliability. Since all tests have some error, reliability coefficients never reach 1.0. Generally, if the reliability of a standardized test is above .80, it is said to have very good reliability; if it is below .50, it would not be considered a very reliable test.


Validity refers to the accuracy of an assessment -- whether or not it measures what it is supposed to measure. Even if a test is reliable, it may not provide a valid measure. Let’s imagine a bathroom scale that consistently tells you that you weigh 130 pounds. The reliability (consistency) of this scale is very good, but it is not accurate (valid) because you actually weigh 145 pounds (perhaps you re-set the scale in a weak moment)! Since teachers, parents, and school districts make decisions about students based on assessments (such as grades, promotions, and graduation), the validity inferred from the assessments is essential -- even more crucial than the reliability. Also, if a test is valid, it is almost always reliable.


There are three ways in which validity can be measured. In order to have confidence that a test is valid (and therefore the inferences we make based on the test scores are valid), all three kinds of validity evidence should be considered.
Type of Validity

Definition

Example/Non-Example
Content The extent to which the content of the test matches the instructional objectives. A semester or quarter exam that only includes content covered during the last six weeks is not a valid measure of the course's overall objectives -- it has very low content validity.
Criterion The extent to which scores on the test are in agreement with (concurrent validity) or predict (predictive validity) an external criterion. If the end-of-year math tests in 4th grade correlate highly with the statewide math tests, they would have high concurrent validity.
Construct The extent to which an assessment corresponds to other variables, as predicted by some rationale or theory. If you can correctly hypothesize that ESOL students will perform differently on a reading test than English-speaking students (because of theory), the assessment may have construct validity.

So, does all this talk about validity and reliability mean you need to conduct statistical analyses on your classroom quizzes? No, it doesn't. (Although you may, on occasion, want to ask one of your peers to verify the content validity of your major assessments.) However, you should be aware of the basic tenets of validity and reliability as you construct your classroom assessments, and you should be able to help parents interpret scores for the standardized exams.

The Validity of Formative Assessments

by Terry Crooks

Educational Assessment Research Unit, University of Otago, Box 56, Dunedin, New Zealand

Terry.Crooks@otago.ac.nz

Paper presented to the British Educational Research Association Annual Conference, University of Leeds, 13-15 September 2001

Note: This paper is incomplete as yet. The major points to be included in the paper are presented here, but the linking text is yet to be written. Contact the author to receive the full paper when completed.



Defining Assessment

Any process that provides information about the thinking, achievement or progress of students



Defining Validity

Messick, 1989, p.13 (Validity)

Validity is an integrated evaluative judgement of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.



Distinguishing Between Summative and Formative Assessment

Summative assessment is intended to summarise student attainment at a particular time, whereas formative assessment is intended to promote further improvement of student attainment.

Assessment OF learning versus assessment FOR learning.



The Formative Impact of Summative Assessment

Crooks, 1988, p. 467(The impact of classroom evaluation practices on students)

Classroom evaluation affects students in many different ways. For instance, it guides their judgement of what is important to learn, affects their motivation and self-perceptions of competence, structures their approaches to and timing of personal study (e.g. spaced practice), consolidates learning, and affects the development of enduring learning strategies and skills.

It appears to be one of the most potent forces influencing education. Accordingly, it deserves very careful planning and considerable investment of time from educators. Many of the skills and attitudes that are goals of education take years to develop, and their development can be undermined by lack of consistent support for them in the educational experiences of the students.

Adapted from Crooks, 1988, p. 443 (The impact of classroom evaluation practices on students)

Short Term Effects

*

Focussing attention on important aspects of the subject;
*

Giving students opportunities to practice skills and consolidate learning;
*

Guiding further instructional or learning activities within the course;

Medium and Longer Term Effects

*

Influencing students' motivation as learners and their perceptions of their capabilities (self-efficacy);
*

Communicating and reinforcing (or in some cases undermining) teaching goals, including key performance criteria and desired standards of performance;
*

Influencing students' choice of (and development of) learning strategies, skills and study patterns;
*

Influencing students' subsequent choice of courses, activities or careers



Definitions of Formative Assessment

Ministry of Education, New Zealand, 1994 (Assessment: policy to practice)

A range of formal and informal assessment procedures (for example, the monitoring of children's writing development, anecdotal records, and observations) undertaken by teachers in the classroom as an integral part of the normal teaching and learning process in order to modify and enhance learning and understanding.

Bronwen Cowie & BeverleyBell, 1996/1999 (A model of formative assessment in science education)

The process used by teachers and students to recognise and respond to student learning in order to enhance that learning, during the learning.

Black & Wiliam, 1998, p. 2 (Inside the black box)

Assessment' refers to all those activities undertaken by teachers, and by the students in assessing themselves, which provide information to be used as feedback to modify the teaching and learning activities in which they are engaged. Such assessment become 'formative assessment' when the evidence is actually used to adapt the teaching to meet the needs.



Key Points about Formative Assessment

Sadler, 1989, p.119 (Formative assessment and the design of instructional systems)

A key premise is that for students to be able to improve, they must have the capacity to monitor the quality of their own work during actual production.

This in turn requires:

*

that students possess an appreciation of what high quality work is,
*

that they have the evaluative skill necessary for them to compare with some objectivity the quality of what they are producing in relation to the higher standard, and
*

that they develop a store of tactics or moves which can be drawn upon to modify their own work.

Instructional systems which do not make explicit provision for the acquisition of evaluative expertise are deficient, because they set up artificial but potentially removable performance ceilings for students.

Crooks, 1988, pp. 468-9 (The impact of classroom evaluation practices on students)

The approach that leads to the most valuable feedback is nicely captured by Easley and Zwoyer (1975):

"If you can both listen to children and accept their answers not as things to just be judged right or wrong but as pieces of information which may reveal what the child is thinking you will have taken a giant step toward becoming a master teacher rather than merely a disseminator of information. (p. 25)"

Crooks, 1988, p. 469 (The impact of classroom evaluation practices on students)

Feedback should be specific and related to need. Simple knowledge of results should be provided consistently (directly or implicitly), with more detailed feedback only where necessary to help the student work through misconceptions or other weaknesses in performance. Praise should be used sparingly and where used should be task-specific, whereas criticism (other than simply identifying deficiencies) is usually counterproductive.



A Summary of Findings from Research on Formative Assessment

Educational research has shown that providing high quality feedback on student work is a very powerful way of raising the standard of student work. Paul Black and Dylan Wiliam (1998), in a 10 year review of research on assessment, stated that "We know of no other way of raising standards for which such a prima facie case can be made." John Hattie (1999), in his inaugural professorial lecture as Dean of Education at the University of Auckland, summarised his wide-ranging review of research on "what works" in education with the statement "the most powerful single moderator that enhances achievement is feedback."

Royce Sadler (1989) identified three elements that are crucial to the effectiveness of formative assessment:

*

helping students to recognise clearly the desired goal (understand what is required);
*

providing students with evidence about how well their work matches that goal;
*

explaining ways to close the gap between the goal and their current performance.

Self-assessment is a vital component in learning. Feedback on assessment cannot be effective unless students accept that their work can be improved and identify important aspects of their work that they wish to improve. Self-monitoring is a key component of the work of all professionals, so if we want our students to become professional learners and professionals in their fields we should actively promote self-assessment. If students are asked and encouraged to critically examine and comment on their own work, assessment can become more dialogue than monologue, and can contribute powerfully to the educational development of students. As Wynne Harlen and Mary James (1996) put it,

"...students have to be active in their own learning (teachers cannot learn for them) and unless they come to understand their own strengths and weaknesses, and how they might deal with them, they will not make progress."

Marks or grades alone produce no learning gains. Indeed, there is some evidence that students gain the most learning value from assessment when feedback is provided without marks or grades. Where marks are provided, they often seem to predominate in students' thinking, and to be seen as the real purpose of the assessment.

Student motivation is crucial to learning. Assessment is one of the major influences on student motivation. It is important, therefore, to anticipate and try to optimise the motivational effects of feedback on assessment. The research evidence available suggests that the greatest motivational benefits will come from focussing feedback on:

*

the qualities of the student's work, and not on comparisons with other students;
*

specific ways in which the student's work could be improved;
*

improvements that the student has made compared to his or her earlier work.

Five points summarise the key lessons from research about formative assessment. Assessment that promotes learning:

*

involves learning goals understood and shared by both teachers and students;
*

helps students to understand and recognise the desired standards;
*

involves students in self-assessment;
*

provides feedback which helps students to recognise next steps and how to take them;
*

builds confidence that students can improve their work.



Some Other Summaries of Requirements for Effective Formative Assessment

Black & Wiliam, 1998 (The Black Box)

Feedback to any pupil should be about the particular qualities of his or her work, with advice on what he or she can do to improve, and should avoid comparisons with other pupils.

For formative assessment to be productive, pupils should be trained in self-assessment so that they can understand the main purposes of their learning and thereby grasp what they need to do to achieve.

Opportunities for pupils to express their understanding should be designed into any piece of teaching, for this will initiate the interaction whereby formative assessment aids learning.

The dialogue between pupils and a teacher should be thoughtful, reflective, focused to evoke and explore understanding, and conducted so that all pupils have an opportunity to think and to express their ideas.

Tests and homework exercises can be an invaluable guide to learning, but the exercises must be clear and relevant to learning aims. The feedback on them should give each pupil guidance on how to improve, and each must be given opportunity and help to work at the improvement.

Assessment Reform Group, UK, 1999, p. 7 (Assessment for learning: beyond the black box)

Assessment that promotes learning:

*

is embedded in a view of teaching and learning of which it is an essential part;
*

involves sharing learning points with pupils;
*

aims to help pupils to know and to recognise the standards they are aiming for;
*

involves pupils in self-assessment;
*

provides feedback which leads to pupils recognising their next steps and how to take them;
*

is underpinned by confidence that every student can improve;
*

involves both teacher and pupils reviewing and reflecting on assessment data.



Formative Assessment of Completed or Draft Work

Student-Centred Contextual Components

Who is this student? What cultural or other considerations require special attention?

How confident is this student about this curriculum area or type of task? (self-efficacy)

How does this student respond to praise, criticism, and guidance? What works best?

Task-Specific Contextual Components

Did the student understand the task?

*

its broad thrust
*

the specific qualities desired, and their relative importance
*

the standards that would be applied.

How much effort has the student put into completing the task?

*

were there mitigating commitments or circumstances?

Did the student receive substantial help (or distraction!) while working on the task?

*

from school staff, peers, or family members

What use did the student make of help available from others?

*

ignored or rejected it
*

learned from it, and incorporated it into his/her work
*

used it essentially verbatim, without evidence of learning

Summative Components

Overall, how well does the work meet the desired criteria and standards?

Should a mark/grade be recorded for the student? If so, where and how?

Components with Both Summative and Formative Aspects

What are the strengths of the work? (criterion referenced)

What are the weaknesses of the work? (criterion referenced)

Should the student be given "knowledge of results" (in a criterion referenced way)?

Formative Components

How does this student's performance match the teacher's expectations for that student?

*

distinctly better than expected
*

largely as expected
*

distinctly worse than expected

What evaluative feedback should be given to this student?

*

what, if anything, should be commended?
*

what, if anything, should be criticised?
*

what should be passed over without comment?

What corrective suggestions/requirements should be given to this student?

*

mainly focussed on improving this particular piece of work
*

mainly focussed on better performance on future work

Should the student be required or encouraged to resubmit the work with improvements?

*

if so, is summative assessment of the work then appropriate



Formative Assessment of Work In Progress

Student-Centred Contextual Components

Who is this student? What cultural or other considerations require special attention?

How confident is the student about this curriculum area or type of task? (self-efficacy)

How does the student respond to praise, criticism, and guidance? What works best?

Task-Specific Contextual Components

Is this the sort of task that the student usually finds motivating?

Does the student understand the task?

*

its broad thrust
*

the specific qualities desired, and their relative importance
*

the standards that will be applied.

Does the student show signs of existing or emerging beliefs or understandings, related to the task, which will obstruct the desired learning or performance?

How much effort is the student putting into completing the task well?

Are other people, events, or circumstances interfering with the student's work?

What use is the student making of help from others?

Formative Components

What questions should be asked to better understand what the student is doing?

What should be done at this point to encourage the development of self-assessment skills?

How well is the work progressing towards meeting the desired criteria and standards?

What are the strengths of the work thus far? (criterion referenced)

What are the weaknesses of the work thus far? (criterion referenced)

How is the student's performance matching the teacher's expectations for that student?

*

distinctly better than expected
*

largely as expected
*

distinctly worse than expected

What evaluative feedback on the progress made should be given to this student?

*

what, if anything, should be commended at this stage?
*

what, if anything, should be criticised at this stage?
*

what should be passed over without comment at this stage?

What corrective suggestions/requirements should be given to this student?

*

focussed on improving this particular piece of work
*

intended to enhance the student's work more generally

Are there issues here that should be raised with other students or with the class as a whole?

What is required before this work can be regarded as completed, and ready for summative assessment (if required)?



Key Factors Influencing the Validity of Formative Assessment

Affective Factors

Motivation

Teacher is devoted to helping student learn

Student cares about learning and wants to improve

Trust

Teacher is encouraging, constructive, sensitive to student's feelings

Class/peer relationships and attitudes support student's learning

Student feels safe to admit difficulties and uncertainties

Task Factors

Knowledge

Teacher understands the key aspects and difficulties of the task

Criteria

Teacher identifies and explains well the qualities sought

Student understands clearly what is needed

Standards

Teacher sets standards appropriate to student

Through descriptions and examples, the standards are explained

Student understands the standards and accepts them as appropriate

Structural Factors

Connections

Final version of task can benefit from the formative assessment

Work on subsequent tasks can benefit from the formative assessment

Purposes

Formative use of task is not undermined by parallel summative use

Process Factors

Self-assessment

Teacher helps student to develop self-assessment skills

Student takes increasing responsibility for his/her own learning

Peer involvement

Teacher encourages collaboration among students to improve work

Peers learn to be constructive and generous in offering feedback

Monitoring

Teacher monitors student's work to track both process and progress

Insight

Teacher detects misunderstandings or other obstacles to success

Teacher detects exciting possibilities in student's work

Timing

Feedback is given at times when student is most receptive to it

Balance

Feedback gives attention to strengths as well as weaknesses

Selectivity

Feedback addresses mainly the aspects likely to have biggest benefit

Wisdom

Feedback is convincing, appreciated, and useful to student



Reference List

Assessment Reform Group, U.K. (1999). Assessment for learning: beyond the black box. Cambridge, U.K: University of Cambridge School of Education.

Black, P., & Wiliam, D. (1998a). Assessment and classroom learning. Assessment in Education, 5, 7-74.

Black, P., & Wiliam,D. (1998b). Inside the black box: Raising standards through classroom assessment. Phi Delta Kappan, 80, 139-149.

Cowie, B., & Bell, B. (1999). A model of formative assessment in science education. Assessment in Education, 6, 101-116.

Crooks, T. J. (1988). The impact of classroom evaluation practices on students. Review of Educational Research, 58, 438-481.

Gipps, C. (1994). Beyond testing: towards a theory of educational assessment. London: Falmer Press.

Harlen, W., & James, M. (1996). Creating a positive impact of assessment on learning. Paper presented at the annual meeting of the American Educational Research Association, New York, April 1996.

Hattie, J. (1999). Influences on student learning. University of Auckland, New Zealand: Inaugural professorial lecture.

Messick, S. (1989). Validity. In R.L. Linn (Ed.), Educational Measurement (3rd edn.), pp. 13-103. New York: Macmillan.

Ministry of Education, New Zealand. (1994). Assessment: policy to practice. Wellington, New Zealand: Learning Media.

Sadler, D. R. (1989). Formative assessment and the design of instructional systems. Instructional Science, 18, 119-144.

Consequences of (mis)use of the Texas Assessment of Academic Skills (TAAS) for high-stakes decisions: A comment on Haney and the Texas miracle in education

by J. Thomas Kellow, University of Houston
Victor L. Willson, Texas A&M University

Abstract This brief paper explores the consequences of failing to incorporate measurement error in the development of cut-scores in criterion-referenced measures.  The authors use the case of Texas and the Texas Assessment of Academic Skills (TAAS) to illustrate the impact of measurement error on “false negative” decisions.  These results serve as further evidence to support Haney’s (2000) contentions regarding the (mis)use of high-stakes testing in the state of Texas.

Walt Haney’s (2000) treatise on The Myth of the Texas Miracle in Education highlights a number of concerns related to high-stakes decision-making in K-12 settings.  His elaboration of the history and development of the Texas Assessment of Academic Skills (TAAS) was illuminating, especially for those unfamiliar with the often capricious fashion in which standards are ultimately set for high-stakes decisions.  The evidence Haney presents in evaluating the TAAS fits well with Samuel Messick’s (1994) argument for considering the consequences of test use.  Although sometimes referred to as consequential validity, Messick considered this aspect of test interpretation and use to be one of many forms of construct validity.  His view of the evaluative role of validity is summarized nicely in the following paragraph:

When assessing any of these constructs – whether attributes of persons or groups, of objects or situations – validity needs to be systematically addressed, as do other basic measurement issues such as reliability, comparability, and fairness.  This is so because validity, comparability, and fairness are not just measurement principles, they are social values that have meaning and force outside of measurement whenever evaluative judgment and decisions are made.  As such, validity assumes both a scientific and political role that can by no means be fulfilled by a simple correlation coefficient between test scores and a purported criterion or by expert judgments that test content is relevant to the proposed test use (p. 1).

One technical aspect of the TAAS that Haney takes to task is the reliability of scores yielded by the test and the potential for misclassification as a function of measurement error.  As the author notes, “The reason the setting of passing scores on a high-stakes test such as the TAAS is so important is that the passing score divides a continuum of scores into just two categories, pass and fail.  Doing so is hazardous because all standardized test scores contain some degree of measurement error”  (Haney 2000, p. 10).  Haney also points out that measures of test-retest (or alternate-form) reliability and not internal consistency should ideally be used to inform judgment as to the potential for misclassfication due to measurement error.  In light of the conspicuous absence of test-retest data on the TAAS, he attempts to use extant data to approximate this reliability estimate as compared to the internal consistency (KR20) estimates provided by the Texas Education Agency (TEA).  Although the approach he uses is somewhat problematic (see Wainer, 1999), it is clear that these test-retest estimates are lower than the KR20 estimates.

The thrust of Haney’s argument is that measurement error inherent in the TAAS (or any other measure, for that matter) contributes appreciably to the rate of “false negatives,” or misclassifying passing students as non-passing.  His focus, however, is exclusively on the tenth-grade Exit-Level TAAS since this is the test high school students in the state of Texas must pass in order to graduate from high school (although it is not the sole criterion).  But in some districts, the TAAS is also being used at other grade levels for high-stakes decisions – namely, promotion and retention of students.  Waco Independent School District and Houston Independent School District, among others, require students in grades 3 through 8 to pass all portions of the TAAS in order to be considered for promotion to grade level.  Students who fail any subtest during the statewide spring administration can expect to attend mandatory summer school, after which they are tested on a “released” version of the TAAS.  Those students who fail are held back in grade.  This stands in contrast to the Exit-Level testing schedule where, as Haney notes, students have as many as eight opportunities to pass the exam.  In addition, the TEA has announced plans to use the TAAS for promotion/retention decisions in the third, fifth, and eighth grades beginning in the fall of 2001 (Texas Education Agency, 1999).
Given the current political climate and the clarion calls for school district accountability throughout the nation, the number of districts in Texas using the TAAS for promotion and retention decisions in all tested grades no doubt will increase.  Indeed, the Houston model has been lauded for its strict criteria for student grade promotion, which is believed to increase student motivation and achievement while reducing the student dropout rate (Markley, 1999).  Our purpose in this response is to elaborate on the potential social consequences of using the TAAS for high-stakes decisions, specifically grade promotion and retention.  The question we asked is rather simple:  “Given the imperfect reliability of the TAAS test, how many students with a true passing score are potentially misclassified as failing across grades and subtests in a given year”?   Fortunately, sufficient summary statistics and frequency distributions were available from the TEA to estimate these numbers. 

Method

The most recent reliability estimates for the TAAS were reported for the 1998-99 school year; therefore we used means, standard deviations, and frequency distributions for this same year.  We want to emphasize that there are a number of ways to go about this estimation process. The method presented emerged because it is fairly intuitive and required minimal summary statistics from the data. 

First, we should note that the 70% passing standard on the TAAS mentioned by Haney is not fixed across grades and subtests.  Because of differential item and thus test difficulty, the proportion of items correct needed to pass a given subtest ranges from .64 to .75, according to the TEA.  The Texas Learning Index (TLI) was developed by the TEA for the purpose of, among other things, providing a consistent passing standard across test forms.  The TLI is a linear standardized scoring transformation with a standard deviation of 15  and an anchor (rather than mean) of 70, which represents the passing standard for a given subtest at a given grade.  The TLI is calculated in z-score form as:

TLI= [(z observed - z passing)*15] + 70

(1)

Although the TLI observed score of 70 is the passing standard, this standard fails to incorporate measurement error in determining the appropriate cut score.  Specifically, the process of modifying cut-scores involves determining the domain score in proportion-correct form that constitutes mastery (t_o ) and then adjusting this value to estimate a new cut score (X₀) in number-correct form that reflects the measurement error in the test data (Crocker & Algina, 1986).  Huynh and Sanders (1980) provide an approximate procedure for this purpose that works well when a test consists of 20 or more items and the observed proportion correct cut score falls within .50 to .80. The TAAS subtests meet both criteria.  This formula is given as:

(2)

Where n is the number of items on the test, t_o is the observed proportion-correct cut score, and m_x is the mean number of items correct.  As noted by Crocker and Algina (1986), as KR21 approaches 1.0, X₀  approaches n t_o + .5 irrespective of the value of m_x. 

Our question focuses on the estimate X₀  when transformed to a TLI score metric.  Put simply: What is the passing TLI adjusted for measurement error for a given subtest in a given grade, and what percent and number of students met this adjusted criterion but not the standard cut score of 70?  The following steps were employed to determine X₀ in TLI form:

1. calculate X₀ in raw score form;
2. transform X₀ into a z-score;
3. substitute z X₀ for z observed in Formula 1.

Results

Table 1 provides the adjusted cut score X₀  in the TLI metric across grades for both the mathematics and reading subtests.

Table 1 TLI passing cut-scores adjusted for measurement error by subtest and grade
Grade	Reading	Mathematics
3rd	67.5	67.4
4th	67.2	67.6
5th	67.2	67.0
6th	67.6	68.2
7th	67.9	68.4
8th	67.7	68.3
10th	66.9	68.7

   

We then used TLI frequency distributions for the 1998-99 administration of the TAAS to determine the percent and number of students who received a TLI score of X₀  or higher but less that 70.  Because the TLI frequency distribution tables obtained from TEA report whole number values, we rounded the obtained X₀ estimates to the closest whole number.  These data are presented in Table 2 disaggregated by subtest and grade.

Table 2 Percent and number of potentially misclassified students by subtest and grade
Grade	Reading	Mathematics
3rd	1.7 (n=4,243)	2.9 (n=7,151)
4th	1.7 (n=4,105)	2.1 (n=5,239)
5th	2.2 (n=5,509)	2.4 (n=6,007)
6th	2.2 (n=5,725)	2.5 (n=6,653)
7th	2.0 (n=5,410)	2.8 (n=7,474)
8th	1.4 (n=3,620)	2.6 (n=6,903)
10th	2.9 (n=6,570)	1.6 (n=3,650)

Because of the rounding procedure mentioned earlier, these percentages and student numbers are approximations.  Roughly 35,182 students who took the reading subtest in the 1998-99 school year were classified as failing, despite having an observed score that would have met (or exceeded) the passing criterion had the presence of measurement error been incorporated into the cut score.  On the mathematics subtest, 43,077 students who failed met (or exceeded) the adjusted observed criterion score. 

Discussion

     Because all tests are inherently unreliable to some degree, measurement errors must be accommodated in the development of cut-scores for criterion-referenced tests, particularly when these instruments are used to make high-stakes decisions for student placement.  Our analysis focused exclusively on the impact of false negative classification errors.  There exists, of course, a second type of misclassification termed a “false positive,” or misclassifying non-passing students as passing.  Although both types of misclassification are serious, a survey of Texas educators conducted by Haney (2000) as part of his investigation of the TAAS  indicated that respondents viewed the consequences of denying a high school diploma to a qualified student based on a classification error (false negative) as considerably more serious that granting a diploma to an unqualified student (false positive).  Indeed, only the consequences to society of granting a license to an unqualified pilot, physician, or teacher, respectively, were viewed as more serious.  Additionally, the literature on grade retention is fairly consistent in noting the deleterious effects of these policies (e.g., increased dropout rates), particularly when strong individualized remediation procedures are not in place (McCoy & Reynolds, 1999).  Put simply, based on Haney’s (2000) survey and the empirical findings on the consequences of retaining students, we feel it seems reasonable to place greater emphasis on the occurrence of false negatives -- at least in the context of education and student promotion decisions.

     The estimation process we employed produced results suggesting that about 2% of students who take the state-mandated TAAS exam will be scored as false negatives on one or more of the subtests.  The consequences of misclassification will become more evident as the TAAS is increasingly used for promotion and retention decisions in Texas.  There is, however, a much larger picture emerging at the national level regarding the (mis)use of standardized assessment tools.  To put this in a broader perspective, we extended our analysis to include an estimate of how many students nationally would be misclassified as false negatives if a testing program such as the TAAS were in place.  We assumed that testing would include the same grade levels as the Texas accountability model, and assumed also a national testing instrument with the same technical adequacy (reliability) as the TAAS.  National data were obtained for the 1998-99 school year disaggregated by grade level.   Combining both reading and mathematics error rates results in approximately  1.1 million students that would potentially be misclassified as false negatives each year across the country.  Two percent clearly is no small number in the national context.        

     The recently installed Bush administration has issued school accountability reform measures that rely almost exclusively on standardized achievement tests.  Many questions remain, however, regarding the structure and implementation of the testing program that will serve as a measure of student performance across states.  What these tests will look like, the extent to which they yield scores that are psychometrically meaningful, and the importance of the scores in guiding student-level decisions are issues that have not been addressed to date.   It is notable that both the American Psychological Association (APA) and, more recently, the American Educational Research Association (AERA) have issued position statements advising against the use of a single assessment for high-stakes decisions at the individual level.  It seems probable, however, that the national appetite for school accountability in the form of student achievement scores will overwhelm any concerns over the ethical consequences of high-stakes testing.

the author: Kellow, J. Thomas & Victor L. Willson (2001). Consequences of (mis)use of the texas assessment of academic skills (taas) for high-stakes decisions: a comment on haney and the texas miracle in education. Practical Assessment, Research & Evaluation, 7(24). Retrieved October 4, 2011 from http://PAREonline.net/getvn.asp?v=7&n=24 . This paper has been viewed 16,436 times since 12/7/2001.

 

References

Crocker, L., & Algina, J.  (1986).  Introduction to classical and modern test theory.  Orlando:  Harcourt Brace.

Haney, W. (2000).   The myth of the Texas Miracle in education.  Education Policy Analysis Archives [On-line serial], 8 (41).  Available:  http://epaa.asu.edu/epaa/v8n41/.

Huynh, H., & Saunders, J. C.  (1980).  Accuracy of two procedures for estimating reliability of mastery tests.  Journal of Educational Measurement, 17, 351-358.

Markley, M.  . (1999, September 8). HISD rules holding more students back:  Expanded standards cut social promotions.  Houston Chronicle, p. A1.

Messick, S. A.  (1994).  Foundations of validity:  Meaning and consequences in psychological assessment.  European Journal of Psychological Assessment, 10, 1-9.

McCoy, A. R., & Reynolds, A. J.  (1999).  Grade retention and school performance:  An extended investigation.  Journal of School Psychology, 37, 273-298.

Texas Education Agency (1999, June 22).  Briefing book:  Legislation affecting public education [On-line].  Available:  http://www.tea.state.tx.us/brief/doc2.html.

Wainer, H. (1999).   Comments on the ad hoc committee’s critique of the Massachusetts Teacher Tests.   Education Policy Analysis Archives [On-line serial], 7 (5).  Available:  http://epaa.asu.edu/epaa/v7n5.html.

 

Designing Scoring Rubrics for Your Classroom

Mertler, Craig A. (2001). Designing scoring rubrics for your classroom. Practical Assessment, Research & Evaluation, 7(25). Retrieved October 4, 2011 from http://PAREonline.net/getvn.asp?v=7&n=25 . This paper has been viewed 256,680 times since 12/11/2001.

by Craig A. Mertler
Bowling Green State University

Rubrics are rating scales-as opposed to checklists-that are used with performance assessments. They are formally defined as scoring guides, consisting of specific pre-established performance criteria, used in evaluating student work on performance assessments. Rubrics are typically the specific form of scoring instrument used when evaluating student performances or products resulting from a performance task.

There are two types of rubrics: holistic and analytic (see Figure 1). A holistic rubric requires the teacher to score the overall process or product as a whole, without judging the component parts separately (Nitko, 2001). In contrast, with an analytic rubric, the teacher scores separate, individual parts of the product or performance first, then sums the individual scores to obtain a total score (Moskal, 2000; Nitko, 2001).

Figure 1: Types of scoring instruments for performance assessments

Holistic rubrics are customarily utilized when errors in some part of the process can be tolerated provided the overall quality is high (Chase, 1999). Nitko (2001) further states that use of holistic rubrics is probably more appropriate when performance tasks require students to create some sort of response and where there is no definitive correct answer. The focus of a score reported using a holistic rubric is on the overall quality, proficiency, or understanding of the specific content and skills-it involves assessment on a unidimensional level (Mertler, 2001). Use of holistic rubrics can result in a somewhat quicker scoring process than use of analytic rubrics (Nitko, 2001). This is basically due to the fact that the teacher is required to read through or otherwise examine the student product or performance only once, in order to get an "overall" sense of what the student was able to accomplish (Mertler, 2001). Since assessment of the overall performance is the key, holistic rubrics are also typically, though not exclusively, used when the purpose of the performance assessment is summative in nature. At most, only limited feedback is provided to the student as a result of scoring performance tasks in this manner. A template for holistic scoring rubrics is presented in Table 1.

Table 1: Template for Holistic Rubrics
Score	Description
5	Demonstrates complete understanding of the problem. All requirements of task are included in response.
4	Demonstrates considerable understanding of the problem. All requirements of task are included.
3	Demonstrates partial understanding of the problem. Most requirements of task are included.
2	Demonstrates little understanding of the problem. Many requirements of task are missing.
1	Demonstrates no understanding of the problem.
0	No response/task not attempted.

Analytic rubrics are usually preferred when a fairly focused type of response is required (Nitko, 2001); that is, for performance tasks in which there may be one or two acceptable responses and creativity is not an essential feature of the students' responses. Furthermore, analytic rubrics result initially in several scores, followed by a summed total score-their use represents assessment on a multidimensional level (Mertler, 2001). As previously mentioned, the use of analytic rubrics can cause the scoring process to be substantially slower, mainly because assessing several different skills or characteristics individually requires a teacher to examine the product several times. Both their construction and use can be quite time-consuming. A general rule of thumb is that an individual's work should be examined a separate time for each of the specific performance tasks or scoring criteria (Mertler, 2001). However, the advantage to the use of analytic rubrics is quite substantial. The degree of feedback offered to students-and to teachers-is significant. Students receive specific feedback on their performance with respect to each of the individual scoring criteria-something that does not happen when using holistic rubrics (Nitko, 2001). It is possible to then create a "profile" of specific student strengths and weaknesses (Mertler, 2001). A template for analytic scoring rubrics is presented in Table 2.

Table 2: Template for analytic rubrics
	Beginning 1	Developing 2	Accomplished 3	Exemplary 4	Score
Criteria #1	Description reflecting beginning level of performance	Description reflecting movement toward mastery level of performance	Description reflecting achievement of mastery level of performance	Description reflecting highest level of performance
Criteria #2	Description reflecting beginning level of performance	Description reflecting movement toward mastery level of performance	Description reflecting achievement of mastery level of performance	Description reflecting highest level of performance
Criteria #3	Description reflecting beginning level of performance	Description reflecting movement toward mastery level of performance	Description reflecting achievement of mastery level of performance	Description reflecting highest level of performance
Criteria #4	Description reflecting beginning level of performance	Description reflecting movement toward mastery level of performance	Description reflecting achievement of mastery level of performance	Description reflecting highest level of performance

Prior to designing a specific rubric, a teacher must decide whether the performance or product will be scored holistically or analytically (Airasian, 2000 & 2001). Regardless of which type of rubric is selected, specific performance criteria and observable indicators must be identified as an initial step to development. The decision regarding the use of a holistic or analytic approach to scoring has several possible implications. The most important of these is that teachers must consider first how they intend to use the results. If an overall, summative score is desired, a holistic scoring approach would be more desirable. In contrast, if formative feedback is the goal, an analytic scoring rubric should be used. It is important to note that one type of rubric is not inherently better than the other-you must find a format that works best for your purposes (Montgomery, 2001). Other implications include the time requirements, the nature of the task itself, and the specific performance criteria being observed. 
As you saw demonstrated in the templates (Tables 1 and 2), the various levels of student performance can be defined using either quantitative (i.e., numerical) or qualitative (i.e., descriptive) labels. In some instances, teachers might want to utilize both quantitative and qualitative labels. If a rubric contains four levels of proficiency or understanding on a continuum, quantitative labels would typically range from "1" to "4." When using qualitative labels, teachers have much more flexibility, and can be more creative. A common type of qualitative scale might include the following labels: master, expert, apprentice, and novice. Nearly any type of qualitative scale will suffice, provided it "fits" with the task. 
One potentially frustrating aspect of scoring student work with rubrics is the issue of somehow converting them to "grades." It is not a good idea to think of rubrics in terms of percentages (Trice, 2000). For example, if a rubric has six levels (or "points"), a score of 3 should not be equated to 50% (an "F" in most letter grading systems). The process of converting rubric scores to grades or categories is more a process of logic than it is a mathematical one. Trice (2000) suggests that in a rubric scoring system, there are typically more scores at the average and above average categories (i.e., equating to grades of "C" or better) than there are below average categories. For instance, if a rubric consisted of nine score categories, the equivalent grades and categories might look like this:

Table 3: Sample grades and categories
Rubric Score	Grade	Category
8	A+	Excellent
7	A	Excellent
6	B+	Good
5	B	Good
4	C+	Fair
3	C	Fair
2	U	Unsatisfactory
1	U	Unsatisfactory
0	U	Unsatisfactory

When converting rubric scores to grades (typical at the secondary level) or descriptive feedback (typical at the elementary level), it is important to remember that there is not necessarily one correct way to accomplish this. The bottom line for classroom teachers is that they must find a system of conversion that works for them and fits comfortably into their individual system of reporting student performance.
Steps in the Design of Scoring Rubrics
A step-by-step process for designing scoring rubrics for classroom use is presented below. Information for these procedures was compiled from various sources (Airasian, 2000 & 2001; Mertler, 2001; Montgomery, 2001; Nitko, 2001; Tombari & Borich, 1999). The steps will be summarized and discussed, followed by presentations of two sample scoring rubrics.

Step 1:	Re-examine the learning objectives to be addressed by the task. This allows you to match your scoring guide with your objectives and actual instruction.
Step 2:	Identify specific observable attributes that you want to see (as well as those you don’t want to see) your students demonstrate in their product, process, or performance. Specify the characteristics, skills, or behaviors that you will be looking for, as well as common mistakes you do not want to see.
Step 3:	Brainstorm characteristics that describe each attribute. Identify ways to describe above average, average, and below average performance for each observable attribute identified in Step 2.
Step 4a:	For holistic rubrics, write thorough narrative descriptions for excellent work and poor work incorporating each attribute into the description. Describe the highest and lowest levels of performance combining the descriptors for all attributes.
Step 4b:	For analytic rubrics, write thorough narrative descriptions for excellent work and poor work for each individual attribute. Describe the highest and lowest levels of performance using the descriptors for each attribute separately.
Step 5a:	For holistic rubrics, complete the rubric by describing other levels on the continuum that ranges from excellent to poor work for the collective attributes. Write descriptions for all intermediate levels of performance.
Step 5b:	For analytic rubrics, complete the rubric by describing other levels on the continuum that ranges from excellent to poor work for each attribute. Write descriptions for all intermediate levels of performance for each attribute separately.
Step 6:	Collect samples of student work that exemplify each level. These will help you score in the future by serving as benchmarks.
Step 7:	Revise the rubric, as necessary. Be prepared to reflect on the effectiveness of the rubric and revise it prior to its next implementation.

These steps involved in the design of rubrics have been summarized in Figure 2.

Figure 2: Designing Scoring Rubrics: Step-by-step procedures

Two Examples
Two sample scoring rubrics corresponding to specific performance assessment tasks are presented next. Brief discussions precede the actual rubrics. For illustrative purposes, a holistic rubric is presented for the first task and an analytic rubric for the second. It should be noted that either a holistic or an analytic rubric could have been designed for either task.

Example 1:
Subject - Mathematics
Grade Level(s) - Upper Elementary

Mr. Harris, a fourth-grade teacher, is planning a unit on the topic of data analysis, focusing primarily on the skills of estimation and interpretation of graphs. Specifically, at the end of this unit, he wants to be able to assess his students' mastery of the following instructional objectives:

• Students will properly interpret a bar graph.
• Students will accurately estimate values from within a bar graph. (step 1)

Since the purpose of his performance task is summative in nature - the results will be incorporated into the students' grades, he decides to develop a holistic rubric. He identifies the following four attributes on which to focus his rubric: estimation, mathematical computation, conclusions, and communication of explanations (steps 2 & 3). Finally, he begins drafting descriptions of the various levels of performance for the observable attributes (steps 4 & 5). The final rubric for his task appears in Table 4.

Table 4: Math Performance Task – Scoring Rubric Data Analysis
Name _____________________________		Date ___________
Score	Description
4	Makes accurate estimations. Uses appropriate mathematical operations with no mistakes. Draws logical conclusions supported by graph. Sound explanations of thinking.
3	Makes good estimations. Uses appropriate mathematical operations with few mistakes. Draws logical conclusions supported by graph. Good explanations of thinking.
2	Attempts estimations, although many inaccurate. Uses inappropriate mathematical operations, but with no mistakes. Draws conclusions not supported by graph. Offers little explanation.
1	Makes inaccurate estimations. Uses inappropriate mathematical operations. Draws no conclusions related to graph. Offers no explanations of thinking.
0	No response/task not attempted.

Example 2:
Subjects - Social Studies; Probability & Statistics
              Grade Level(s) - 9 - 12            

Mrs. Wolfe is a high school American government teacher. She is beginning a unit on the electoral process and knows from past years that her students sometimes have difficulty with the concepts of sampling and election polling. She decides to give her students a performance assessment so they can demonstrate their levels of understanding of these concepts. The main idea that she wants to focus on is that samples (surveys) can accurately predict the viewpoints of an entire population. Specifically, she wants to be able to assess her students on the following instructional objectives:

• Students will collect data using appropriate methods.
• Students will accurately analyze and summarize their data.
• Students will effectively communicate their results. (step 1)

Since the purpose of this performance task is formative in nature, she decides to develop an analytic rubric focusing on the following attributes: sampling technique, data collection, statistical analyses, and communication of results (steps 2 & 3). She drafts descriptions of the various levels of performance for the observable attributes (steps 4 & 5). The final rubric for this task appears in Table 5. 

Table 5: Performance Task – Scoring Rubric Population Sampling
Name ____________________________			Date ________________
	Beginning 1	Developing 2	Accomplished 3	Exemplary 4	Score
Sampling Technique	Inappropriate sampling technique used	Appropriate technique used to select sample; major errors in execution	Appropriate technique used to select sample; minor errors in execution	Appropriate technique used to select sample; no errors in procedures
Survey/ Interview Questions	Inappropriate questions asked to gather needed information	Few pertinent questions asked; data on sample is inadequate	Most pertinent questions asked; data on sample is adequate	All pertinent questions asked; data on sample is complete
Statistical Analyses	No attempt at summarizing collected data	Attempts analysis of data, but inappropriate procedures	Proper analytical procedures used, but analysis incomplete	All proper analytical procedures used to summarize data
Communication of Results	Communication of results is incomplete, unorganized, and difficult to follow	Communicates some important information; not organized well enough to support decision	Communicates most of important information; shows support for decision	Communication of results is very thorough; shows insight into how data predicted outcome
Total Score = ____

Resources for Rubrics on the Web
The following is just a partial list of some Web resources for information about and samples of scoring rubrics.

• "Scoring Rubrics: What, When, & How?" (http://pareonline.net/getvn.asp?v=7&n=3). This article appears in Practical Assessment, Research, & Evaluation and is authored by Barbara M. Moskal. The article discusses what rubrics are, and distinguishes between holistic and analytic types. Examples and additional resources are provided.
  
• "Performance Assessment-Scoring" (http://www.pgcps.pg.k12.md.us/~elc/scoringtasks.html). Staff in the Prince George's County (MD) Public Schools have developed a series of pages that provide descriptions of the steps involved in the design of performance tasks. This particular page provides several rubric samples.
  
• "Rubrics from the Staff Room for Ontario Teachers" ( http://www.quadro.net/~ecoxon/Reporting/rubrics.htm ) This site is a collection of literally hundreds of teacher-developed rubrics for scoring performance tasks. The rubrics are categorized by subject area and type of task. This is a fantastic resource…check it out!
  
• "Rubistar Rubric Generator" (http://rubistar.4teachers.org/)
  
• "Teacher Rubric Maker" (http://www.teach-nology.com/web_tools/rubrics/) These two sites house Web-based rubric generators for teachers. Teachers can customize their own rubrics based on templates on each site. In both cases, rubric templates are organized by subject area and/or type of performance task. These are wonderful resources for teachers!

References

Airasian, P. W. (2000). Assessment in the classroom: A concise approach (2nd ed.). Boston: McGraw-Hill.
Airasian, P. W. (2001). Classroom assessment: Concepts and applications (4th ed.). Boston: McGraw-Hill.
Chase, C. I. (1999). Contemporary assessment for educators. New York: Longman.
Mertler, C. A. (2001). Using performance assessment in your classroom. Unpublished manuscript, Bowling Green State University.
Montgomery, K. (2001). Authentic assessment: A guide for elementary teachers. New York: Longman.
Moskal, B. M. (2000). Scoring rubrics: what, when, and how?. Practical Assessment, Research, & Evaluation, 7(3). Available online: http://pareonline.net/getvn.asp?v=7&n=3
Nitko, A. J. (2001). Educational assessment of students (3rd ed.). Upper Saddle River, NJ: Merrill.
Tombari, M. & Borich, G. (1999). Authentic assessment in the classroom: Applications and practice. Upper Saddle River, NJ: Merrill.
Trice, A. D. (2000). A handbook of classroom assessment. New York: Longman.

Contact information

Craig A. Mertler
Educational Foundations & Inquiry Program
College of Education & Human Development
Bowling Green State University
Bowling Green, OH 43403
mertler@bgnet.bgsu.edu
Phone: 419-372-9357 Fax: 419-372-8265