The Teaching-Learning Cycle: Using Student Learning Outcome Results to Improve Teaching and Learning



Download 0.72 Mb.
Page1/11
Date conversion01.08.2017
Size0.72 Mb.
  1   2   3   4   5   6   7   8   9   10   11



The Teaching-Learning Cycle:

Using Student Learning Outcome Results

to Improve Teaching and Learning







Workshop Activities & Resource Materials



Bill Scroggins

November 2004

Table of Contents
Student Learning Outcomes at the Lesson Level 1
Student Learning Outcomes at the Course Level: From Course Objectives to SLOs 2
Primary Trait Analysis: Statements of Grading Criteria 4

Selecting the Assessment Method: Authentic Assessment and Deep Learning 6

Norming or Inter-Rater Reliability: Assuring Consistency of Grading Among Faculty 8


The Assessment Report: Sharing the Results of Student Learning Outcomes 8
Program Level Student Learning Outcomes 9

Direct and Indirect Measures of Student Learning Outcomes 10

Identifying Program Competencies—External and Internal Sources 11
Strategies for Direct Assessment of Program SLOs: Mosaic and Capstone Approaches 11
General Education Student Learning Outcomes 13
Conclusion 14


Appendices
Appendix 1 – Good Practices in Assessing Student Learning Outcomes 15

Appendix 2 – Activity 3: Writing Student Learning Outcomes 22


Appendix 3 – Developing and Applying Rubrics 23

Appendix 4 – Examples of Scoring Rubrics 28


Appendix 5 – Activities 4 & 5: Building and Using a Grading Rubric 29

Appendix 6 – The Case for Authentic Assessment by Grant Wiggins 30

Appendix 7 -- State and National Standards, Academic & Vocational Competencies 32


Appendix 8 – Assessment Report Examples 36

Appendix 9 – Assessment Plan Examples Internet Sites 40


Appendix 10 – Activity 5 – Program SLOs from Competency Statements 41

Appendix 11 – Examples of Program Assessment Reports 42

Appendix 12 – General Education Student Learning Outcomes 44

Appendix 13—Resources and References for Student Learning Outcomes Assessment 52

Endnotes 57
URL for this document: http://cai.cc.ca.us/workshops/SLOFocusOnResults.doc

For further information contact: Bill Scroggins

Interim President

Modesto Junior College

scrogginsb@yosemite.cc.ca.us


The Teaching-Learning Cycle


Using Student Learning Outcome Results to Improve Teaching & Learning
Since the Accrediting Commission identified “measuring student learning outcomes” as the focus of the latest revision of the WASC standards, many of us have been struggling with what we are expected to do differently.i Whatever we do to implement Student Learning Outcomes, this initiative must be seen to add value to the teaching and learning process—value that clearly outweighs the task of constructing SLOs. Those of us who have taught for years consider that we already measure student learning. However, I have come to believe that SLOs really do have a new and useful emphasis that can be best captured by one word: results—collecting them, sharing them, and using them to improve both learning and the operation of our colleges. This series of reflections are intended to address getting useful results from the SLO process—maximizing utility and minimizing futility. (That little “f” really makes a difference, doesn’t it?)


Results of SLOs at the Lesson Level

Most of us have criteria for grading student learning for the individual objectives of each lesson we teach—but we may not write these down or share them with students. Here’s an example:



Lesson Learning Objective: Describe and draw the four vibrations of carbon dioxide and show how IR light is absorbed by CO2.

Sample Graded Question: What two types of motion are caused by the absorbance of IR light by CO2? Draw one of these motions.

Grading Criteria:

Full credit: Student names “bending” and “stretching” and draws ball-and-stick models with arrows up-and-down for bending and side-to-side for stretching.

Deductions: 25% for one name missing; 50% for both; 25% for wrong or missing arrows; 50% for no drawing.


Results of Grading Student Work:

82% earned full credit; 5% confused arrows; 13% had no drawing.



Action to Improve Learning:

The greatest deficiency seems to be drawing, so do in-class drawing exercise.





Student Learning Outcomes at the Lesson Level

As we teach each lesson and grade the related student assignments, we typically have a clear concept of the results expected, and we have defined methods for assessing student work and assigning grades. However, there are several things that we typically don’t do that can potentially improve student learning. While many of us do give students written learning objectives for each lesson, we usually do not write down criteria for grading nor share them with students—other than how total points relate to the final grade in the course.


In listening to practitioners of SLOs such as Lisa Brewsterii, a Speech teacher at San Diego Miramar College, and Janet Fulksiii, a Microbiology teacher at Bakersfield College, it is clear that SLOs can become a powerful pedagogical tool by:

  • sharing grading criteria with students,

  • getting students to use these criteria as a way to better understand the material, and

  • having students evaluate their own and each other’s work.




Activity 1

In small groups by discipline or cluster of related disciplines, discuss how you develop grading criteria.


  • Do you write down your grading criteria for each assignment?


  • How consistent are you in applying your grading criteria?

  • Do you use the results of student assessment to improve your grading criteria?

  • Do you communicate your grading criteria to students? Before or after the assignment?

  • Do you encourage students to apply the grading criteria to their own work?

  • Do you involve students in developing or modifying your grading criteria?

  • Do you share your grading criteria with other faculty?

Aggregating the feedback from grading student assignments can provide valuable insight into areas in need of improvement. With all the demands on our time, we may not give adequate attention to mining this valuable source of information for improvement of the teaching and learning process. One of the challenges that the new accreditation standards present is creating an assessment plan that outlines objectives, grading criteria, results of assessing student work, and how we use those results to improve student learning.
Of course, part of improving student learning is improving the way we teach. This inevitable outcome can potentially be threatening to faculty members. However, when these issues have been raised in workshops with faculty, the result has generally been a serious engagement in discussions of teaching methods to improve authentic, deep learning.iv It is extremely important to build environments for discussing the improvement of student learning which are positive and reinforcing. Several colleges have made explicit commitments to this principle.v (The endnote references include approaches by Palomar College in California, College of DuPage in Illinois, and the American Association of Higher Education.)

Activity 2

Read the following resource documents (see Appendix 1) and join in the group discussion on “Good Practices for Assessment of Student Learning Outcomes.”

“An Assessment Manifesto” by College of DuPage (IL)

“9 Principles of Good Practice for Assessing Student Learning” by AAHE

“Palomar College Statement of Principles on Assessment” from Palomar College (CA)

“Closing the Loop—Seven Common (Mis)Perceptions About Outcomes Assessment” by Tom Angelo

“Five Myths of ‘Assessment’” by David Clement, faculty member, Monterey Peninsula College




Student Learning Outcomes at the Course Level: From Course Objectives to SLOs

Beyond the lesson level, we must address results of student learning at the course level. Moreover, we should do so for all sections of each course, meaning collaboration among the full- and part-time faculty teaching the course. In stating the desired student learning outcomes, we have the advantage of agreed-upon student objectives in the course outline.

A great deal of energy has been expended in discussing the difference between a course objective and a student learning outcome. The difference may be clearer when viewed in the context of producing assessment results that 1) provide useful feedback to improve the teaching and learning process and 2) provide useful information to improve college practices. SLOs more clearly connect with how the instructor will evaluate student work to determine if the objective has been met. When we write an assignment, we provide a context in which the student will respond and we evaluate the response based on criteria we use to judge if the student has met the objective—usually we have at least a mental construct of minimum acceptable performance standards. These are the two additional pieces that transform an objective into an SLO. Here’s how it might work.

If course objectives have been written well, they will be complete, measurable, and rigorous. In practice, as faculty look more closely at the criteria and methods to assess these objectives, changes often result. To “operationalize” an objective for assessment purposes, that is, to transform it into a statement of desired student learning outcomes, typically we must address:


  1. the stated objectives in terms of acquired knowledge, skill or values (hopefully, the existing course objectives),




  1. the context or conditions under which the student will be expected to apply the knowledge, skill or values, and




  1. the primary traits which will be used in assessing student performance.

Below are some examples of “robust course objectives” or “statements of desired student learning outcomes.” (Note that this difference is largely semantic. Some colleges have chosen to put SLO statements in course outlines as an enhancement of the objectives, while others have built statements of desired SLOs into a departmental assessment plan, typically related to program review.) Whatever vehicle the college uses to operationalize course objectives to SLOs, it must be done collaboratively among faculty who teach the course.



Examples of Course Objectives Transformed Into Student Learning Outcomes





Course Objective

Statement of Desired SLO

Write well-organized, accurate and significant content. (English)


Context: Given an in-class writing task based on an assigned reading,

Objective: demonstrate appropriate and competent writing which

Traits: states a thesis, supports assertions, maintains unity of thought and purpose, is organized, and is technically correct in paragraph composition, sentence structure, grammar, spelling, and word use.


Analyze behavior following the major accepted theories. (Psychology)

Context: Given a particular behavior and its context (e.g., playing incessantly with one’s hair when under pressure in the presence of the opposite sex),

Objective: describe how the perspectives of behaviorism, humanistic, psychoanalytic, and biological psychology would interpret that behavior and what methods might each use to alter that behavior.

Traits: Include theoretical basis, description of causality, and treatment regimen.


Understand and apply the scientific method. (Biology)

Context: Given a hypothesis,

Objective: design experiments and interpret data according to the scientific method in order to evaluate the hypothesis.

Traits: Include the ability to approach the scientific method in a variety of ways, formulate questions, design experiments that answer the questions; and manipulate and evaluate the experimental data to reach conclusions.


Compare and contrast the text and film versions of a literary work. (Film)

Context: After viewing an assigned film based on a literary text,

Objective: write a review of the film.

Traits: Include an appraisal of the director’s selection and effective translation of content from the literary text and the dominant tone the director seems to be trying to achieve, supporting each statement with detail from the text and film and your personal reaction to the cited scenes.




Activity 3

Perform the “Writing Student Learning Outcomes” exercise in Appendix 3. Review the first example. Then for the second course objective, complete the Performance Context, Measurable Objective, and Primary Traits. Finally, select an objective from a course in your discipline and construct the three-part SLO statement.




Primary Trait Analysis: Statements of Grading Criteria

Primary traits are the characteristics that are evaluated in assessing student work. Identifying primary traits for a given assignment involved listing those specific components that, taken together, make up a complete piece of work. They are the collection of things that we as teachers look for when we grade student work.



Definition of Primary Trait Assessment

Primary trait assessment is a method of explicitly stating the criteria and standards for evaluation of student performance of an assignment or test. The professor identifies the traits that will be evaluated, and ranks the student's performance of each trait on a scale of "most effective" to "least effective" realization of the assignment goals. On this scale, the level of the student's performance is explicitly ranked so that the student knows how she is being evaluated. The instructor has created the scale for direct application to the assignment the student is performing so that if the entire class does poorly on the assignment, it is clear to the instructor what difficulties the class may share with one another. This recursive feedback of primary trait assessment can be used to inform classroom and departmental improvement. vi

While “primary traits” are the categories into which we can sort competencies when we evaluate student work, we look for specific levels of performance in each of these areas. For example, an essay might be rated on development, organization, style, and mechanics. These primary traits are then rated on some sort of a scale—as simple as A/B/C/D/F or more descriptive as excellent/superior/satisfactory/poor/unsatisfactory. Occasionally, points are given based on this scale. The challenge presented by the Student Learning Outcomes process is to write down those observable student performance characteristics in an explicit way for each of the primary traits we have identified. This system, known as a “grading rubric,” can be used to grade student work collected through all manner of assessment methods.vii

Building a Rubric

Start with expectations for satisfactory work for each trait such as Organization in the table to the left:



  • Ideas generally related to one another and to the focus, but may have some unrelated material

  • Adequate introduction and conclusion

  • Some attempt at transitions

Then stretch up to excellent and down to unsatisfactory.
Template for a Grading Rubric:

Primary Traits and Observable Characteristics

Trait

Excellent

Superior

Satisfactory

Poor

Unsatisfactory

Development

















Organization
















Style
















Mechanics















Rubrics can be applied in total by specifically rating each primary trait (an “analytic” grading rubric) or holistically (using the rubric as a guide to determine the overall rating of excellent, satisfactory, or unsatisfactory—or whatever performance levels have been agreed upon). An example is given below.




Primary Trait Grading of Math Problem Solvingviii

Trait

3 points

2 points

1 point

0 points

Understanding

complete understanding of the problem in the problem statement section as well as in the development of the plan and interpretation of the solution


good understanding of the problem in the problem statement section. Some minor point(s) of the problem may be overlooked in the problem statement, the development of the plan, or the interpretation of the solution

minimal understanding of the problem; the problem statement may be unclear to the reader. The plan and/or interpretation of the solution overlooks significant parts of the problem

no understanding of the problem; the problem statement section does not address the problem or may even be missing. The plan and discussion of the solution have nothing to do with the problem

Plan

plan is clearly articulated AND will lead to a correct solution


plan is articulated reasonably well and correct OR may contain a minor flaw based on a correct interpretation of the problem

plan is not clearly presented OR only partially correct based on a correct/partially correct understanding of the problem

no plan OR the plan is completely incorrect

Solution

solution is correct AND clearly labeled OR though the solution is incorrect it is the expected outcome of a slightly flawed plan that is correctly implemented

solution is incorrect due to a minor error in implementation of either a correct or incorrect plan OR solution is not clearly labeled

solution is incorrect due to a significant error in implementation of either a correct or incorrect plan

no solution is given

Presentation








overall appearance of the paper is neat and easy to read, and all pertinent information can be readily found

paper is hard to read OR pertinent information is hard to find




Holistic Grading of Math Problem Solving viii

Trait

3 points

2 points

1 point

0 points

Analyzed holistically



All of the following characteristics must be present:

  • answer is correct; explanation is clear and complete;

  • explanation includes complete implementation of a mathematically correct plan

Exactly one of the following characteristics is present:

  • answer is incorrect due to a minor flaw in plan or an algebraic error;

  • explanation lacks clarity;

  • explanation is incomplete

Exactly two of the characteristics in the 2-point section are present OR

One or more of the following characteristics are present.

  • answer is incorrect due to a major flaw in the plan;
  • explanation lacks clarity or is incomplete but does indicate some correct and relevant reasoning;


  • plan is partially implemented and no solution is provided

All of the following characteristics must be present:

  • answer is incorrect;

  • explanation, if any, uses irrelevant arguments;

  • no plan for solution is attempted beyond just copying data given in the problem statement

Grading rubrics can be applied to a wide variety of subjects and used in association with a range of assessment techniques. (See the endnote on rubrics for references to good practices for using rubrics and for a range of examples of rubrics at a variety of colleges and across several disciplines.)




Before doing these two activities on rubrics, read “Developing and Applying Rubrics” by Mary Allen in Appendix 3. If possible, review some of the sample rubrics listed in Appendix 4.

Activity 4: Building a Rubric

Using the grid in Appendix 5A, select or write an SLO, identify Primary Traits, and then decide on “observables” for each assessment level



Activity 5: Using a Grading Rubric and Norming the Results

Use the English rubric in Appendix 5B to grade the sample student essay in Appendix 5C. Compare your results with colleagues who graded the same paper. Where were your assessments different? Can you come to agreement on the overall rating of the paper?


T

  1   2   3   4   5   6   7   8   9   10   11


The database is protected by copyright ©hestories.info 2017
send message

    Main page