Appendix 5 – Activities 4 & 5: Building and Using a Grading Rubric 29
Appendix 6 – The Case for Authentic Assessment by Grant Wiggins 30
Appendix 7 -- State and National Standards, Academic & Vocational Competencies 32
Appendix 8 – Assessment Report Examples 36
Appendix 9 – Assessment Plan Examples Internet Sites 40
Appendix 10 – Activity 5 – Program SLOs from Competency Statements 41
Appendix 11 – Examples of Program Assessment Reports 42
Appendix 12 – General Education Student Learning Outcomes 44
Appendix 13—Resources and References for Student Learning Outcomes Assessment 52
URL for this document: http://cai.cc.ca.us/workshops/SLOFocusOnResults.doc
For further information contact: Bill Scroggins
Modesto Junior College
The Teaching-Learning Cycle
Using Student Learning Outcome Results to Improve Teaching & Learning Since the Accrediting Commission identified “measuring student learning outcomes” as the focus of the latest revision of the WASC standards, many of us have been struggling with what we are expected to do differently.i Whatever we do to implement Student Learning Outcomes, this initiative must be seen to add value to the teaching and learning process—value that clearly outweighs the task of constructing SLOs. Those of us who have taught for years consider that we already measure student learning. However, I have come to believe that SLOs really do have a new and useful emphasis that can be best captured by one word: results—collecting them, sharing them, and using them to improve both learning and the operation of our colleges. This series of reflections are intended to address getting useful results from the SLO process—maximizing utility and minimizing futility. (That little “f” really makes a difference, doesn’t it?)
Results of SLOs at the Lesson Level
Most of us have criteria for grading student learning for the individual objectives of each lesson we teach—but we may not write these down or share them with students. Here’s an example:
Lesson Learning Objective: Describe and draw the four vibrations of carbon dioxide and show how IR light is absorbed by CO2.
Sample Graded Question: What two types of motion are caused by the absorbance of IR light by CO2? Draw one of these motions.
Full credit: Student names “bending” and “stretching” and draws ball-and-stick models with arrows up-and-down for bending and side-to-side for stretching.
Deductions: 25% for one name missing; 50% for both; 25% for wrong or missing arrows; 50% for no drawing.
Results of Grading Student Work:
82% earned full credit; 5% confused arrows; 13% had no drawing.
Action to Improve Learning:
The greatest deficiency seems to be drawing, so do in-class drawing exercise.
Student Learning Outcomes at the Lesson Level
As we teach each lesson and grade the related student assignments, we typically have a clear concept of the results expected, and we have defined methods for assessing student work and assigning grades. However, there are several things that we typically don’t do that can potentially improve student learning. While many of us do give students written learning objectives for each lesson, we usually do not write down criteria for grading nor share them with students—other than how total points relate to the final grade in the course.
In listening to practitioners of SLOs such as Lisa Brewsterii, a Speech teacher at San Diego Miramar College, and Janet Fulksiii, a Microbiology teacher at Bakersfield College, it is clear that SLOs can become a powerful pedagogical tool by:
sharing grading criteria with students,
getting students to use these criteria as a way to better understand the material, and
having students evaluate their own and each other’s work.
In small groups by discipline or cluster of related disciplines, discuss how you develop grading criteria.
Do you write down your grading criteria for each assignment?
How consistent are you in applying your grading criteria?
Do you use the results of student assessment to improve your grading criteria?
Do you communicate your grading criteria to students? Before or after the assignment?
Do you encourage students to apply the grading criteria to their own work?
Do you involve students in developing or modifying your grading criteria?
Do you share your grading criteria with other faculty?
Aggregating the feedback from grading student assignments can provide valuable insight into areas in need of improvement. With all the demands on our time, we may not give adequate attention to mining this valuable source of information for improvement of the teaching and learning process. One of the challenges that the new accreditation standards present is creating an assessment plan that outlines objectives, grading criteria, results of assessing student work, and how we use those results to improve student learning.
Of course, part of improving student learning is improving the way we teach. This inevitable outcome can potentially be threatening to faculty members. However, when these issues have been raised in workshops with faculty, the result has generally been a serious engagement in discussions of teaching methods to improve authentic, deep learning.iv It is extremely important to build environments for discussing the improvement of student learning which are positive and reinforcing. Several colleges have made explicit commitments to this principle.v (The endnote references include approaches by Palomar College in California, College of DuPage in Illinois, and the American Association of Higher Education.)
Read the following resource documents (see Appendix 1) and join in the group discussion on “Good Practices for Assessment of Student Learning Outcomes.”
“An Assessment Manifesto” by College of DuPage (IL)
“9 Principles of Good Practice for Assessing Student Learning” by AAHE
“Palomar College Statement of Principles on Assessment” from Palomar College (CA)
“Closing the Loop—Seven Common (Mis)Perceptions About Outcomes Assessment” by Tom Angelo
“Five Myths of ‘Assessment’” by David Clement, faculty member, Monterey Peninsula College
Student Learning Outcomes at the Course Level: From Course Objectives to SLOs
Beyond the lesson level, we must address results of student learning at the course level. Moreover, we should do so for all sections of each course, meaning collaboration among the full- and part-time faculty teaching the course. In stating the desired student learning outcomes, we have the advantage of agreed-upon student objectives in the course outline.
A great deal of energy has been expended in discussing the difference between a course objective and a student learning outcome. The difference may be clearer when viewed in the context of producing assessment results that 1) provide useful feedback to improve the teaching and learning process and 2) provide useful information to improve college practices. SLOs more clearly connect with how the instructor will evaluate student work to determine if the objective has been met. When we write an assignment, we provide a context in which the student will respond and we evaluate the response based on criteria we use to judge if the student has met the objective—usually we have at least a mental construct of minimum acceptable performance standards. These are the two additional pieces that transform an objective into an SLO. Here’s how it might work.
If course objectives have been written well, they will be complete, measurable, and rigorous. In practice, as faculty look more closely at the criteria and methods to assess these objectives, changes often result. To “operationalize” an objective for assessment purposes, that is, to transform it into a statement of desired student learning outcomes, typically we must address:
the stated objectives in terms of acquired knowledge, skill or values (hopefully, the existing course objectives),
the context or conditions under which the student will be expected to apply the knowledge, skill or values, and
the primary traits which will be used in assessing student performance.
Below are some examples of “robust course objectives” or “statements of desired student learning outcomes.” (Note that this difference is largely semantic. Some colleges have chosen to put SLO statements in course outlines as an enhancement of the objectives, while others have built statements of desired SLOs into a departmental assessment plan, typically related to program review.) Whatever vehicle the college uses to operationalize course objectives to SLOs, it must be done collaboratively among faculty who teach the course.
Examples of Course Objectives Transformed Into Student Learning Outcomes
Statement of Desired SLO
Write well-organized, accurate and significant content. (English)
Context: Given an in-class writing task based on an assigned reading,
Objective: demonstrate appropriate and competent writing which
Traits: states a thesis, supports assertions, maintains unity of thought and purpose, is organized, and is technically correct in paragraph composition, sentence structure, grammar, spelling, and word use.
Analyze behavior following the major accepted theories. (Psychology)
Context: Given a particular behavior and its context (e.g., playing incessantly with one’s hair when under pressure in the presence of the opposite sex),
Objective: describe how the perspectives of behaviorism, humanistic, psychoanalytic, and biological psychology would interpret that behavior and what methods might each use to alter that behavior.
Traits: Include theoretical basis, description of causality, and treatment regimen.
Understand and apply the scientific method. (Biology)
Context: Given a hypothesis,
Objective: design experiments and interpret data according to the scientific method in order to evaluate the hypothesis.
Traits: Include the ability to approach the scientific method in a variety of ways, formulate questions, design experiments that answer the questions; and manipulate and evaluate the experimental data to reach conclusions.
Compare and contrast the text and film versions of a literary work. (Film)
Context: After viewing an assigned film based on a literary text,
Objective: write a review of the film.
Traits: Include an appraisal of the director’s selection and effective translation of content from the literary text and the dominant tone the director seems to be trying to achieve, supporting each statement with detail from the text and film and your personal reaction to the cited scenes.
Perform the “Writing Student Learning Outcomes” exercise in Appendix 3. Review the first example. Then for the second course objective, complete the Performance Context, Measurable Objective, and Primary Traits. Finally, select an objective from a course in your discipline and construct the three-part SLO statement.
Primary Trait Analysis: Statements of Grading Criteria
Primary traits are the characteristics that are evaluated in assessing student work. Identifying primary traits for a given assignment involved listing those specific components that, taken together, make up a complete piece of work. They are the collection of things that we as teachers look for when we grade student work.
Definition of Primary Trait Assessment
Primary trait assessment is a method of explicitly stating the criteria and standards for evaluation of student performance of an assignment or test. The professor identifies the traits that will be evaluated, and ranks the student's performance of each trait on a scale of "most effective" to "least effective" realization of the assignment goals. On this scale, the level of the student's performance is explicitly ranked so that the student knows how she is being evaluated. The instructor has created the scale for direct application to the assignment the student is performing so that if the entire class does poorly on the assignment, it is clear to the instructor what difficulties the class may share with one another. This recursive feedback of primary trait assessment can be used to inform classroom and departmental improvement. vi
While “primary traits” are the categories into which we can sort competencies when we evaluate student work, we look for specific levels of performance in each of these areas. For example, an essay might be rated on development, organization, style, and mechanics. These primary traits are then rated on some sort of a scale—as simple as A/B/C/D/F or more descriptive as excellent/superior/satisfactory/poor/unsatisfactory. Occasionally, points are given based on this scale. The challenge presented by the Student Learning Outcomes process is to write down those observable student performance characteristics in an explicit way for each of the primary traits we have identified. This system, known as a “grading rubric,” can be used to grade student work collected through all manner of assessment methods.vii
Building a Rubric
Start with expectations for satisfactory work for each trait such as Organization in the table to the left:
Then stretch up to excellent and down to unsatisfactory.
Template for a Grading Rubric:
Primary Traits and Observable Characteristics
Rubrics can be applied in total by specifically rating each primary trait (an “analytic” grading rubric) or holistically (using the rubric as a guide to determine the overall rating of excellent, satisfactory, or unsatisfactory—or whatever performance levels have been agreed upon). An example is given below.
Primary Trait Grading of Math Problem Solvingviii
complete understanding of the problem in the problem statement section as well as in the development of the plan and interpretation of the solution
good understanding of the problem in the problem statement section. Some minor point(s) of the problem may be overlooked in the problem statement, the development of the plan, or the interpretation of the solution
minimal understanding of the problem; the problem statement may be unclear to the reader. The plan and/or interpretation of the solution overlooks significant parts of the problem
no understanding of the problem; the problem statement section does not address the problem or may even be missing. The plan and discussion of the solution have nothing to do with the problem
plan is clearly articulated AND will lead to a correct solution
plan is articulated reasonably well and correct OR may contain a minor flaw based on a correct interpretation of the problem
explanation lacks clarity or is incomplete but does indicate some correct and relevant reasoning;
plan is partially implemented and no solution is provided
All of the following characteristics must be present:
answer is incorrect;
explanation, if any, uses irrelevant arguments;
no plan for solution is attempted beyond just copying data given in the problem statement
Grading rubrics can be applied to a wide variety of subjects and used in association with a range of assessment techniques. (See the endnote on rubrics for references to good practices for using rubrics and for a range of examples of rubrics at a variety of colleges and across several disciplines.)
Before doing these two activities on rubrics, read “Developing and Applying Rubrics” by Mary Allen in Appendix 3. If possible, review some of the sample rubrics listed in Appendix 4.
Activity 4: Building a Rubric
Using the grid in Appendix 5A, select or write an SLO, identify Primary Traits, and then decide on “observables” for each assessment level
Activity 5: Using a Grading Rubric and Norming the Results
Use the English rubric in Appendix 5B to grade the sample student essay in Appendix 5C. Compare your results with colleagues who graded the same paper. Where were your assessments different? Can you come to agreement on the overall rating of the paper?