Using nrs data for Program Management and Improvement This guide was prepared for the project

Download 0.55 Mb.
Size0.55 Mb.
  1   2   3   4   5   6   7   8   9   ...   21

Using NRS Data for Program

Management and Improvement

This guide was prepared for the project:
Promoting the Quality and Use of

National Reporting System (NRS) Data

Contract # ED-01-CO-0026


U.S. Department of Education

Office of Vocational and Adult Education,

Division of Adult Education and Literacy

Carol D’Amico, Assistant Secretary for

Vocational and Adult Education

Cheryl Keenan, Director

Division of Adult Education and Literacy
Mike Dean, Program Specialist
Division of Adult Education and Literacy


Larry Condelli, Project Director

Aashti Zaidi
Cover and Graphics By:

Matthew Green

American Institutes for Research

1000 Thomas Jefferson, Street, N.W.

Washington, D.C. 20007
Revised May 2003


In preparing this guide we received assistance from local and state adult education staff, who provided us with information, advice, data tables and other assistance on how to use data for program management and improvement. We have incorporated their experience and data into this guide and wish to thank them for their help.

Donna Cornellier, Massachusetts Department of Education

Diane Glass, Kansas Board of Regents

John Hartwig, Iowa Department of Education

Joe Macaluso, Alabama Department of Education

Judy McCoy, Eastside Union Adult School, San Jose, California

Ron Russell, Alabama Department of Education

Shirley Spencer, Idaho Department of Education

Sandy Strunk, Lancaster-Lebanon Intermediate Adult School, Pennsylvania

Fran Tracy-Mumford, Delaware Department of Education

Other data and examples in this guide are from NRS Annual Statistical Report tables for program year 2000, Office of Vocational and Adult Education, Division of Adult Education and Literacy, U.S. Department of Education and the student database of the What Works Study for Adult ESL Literacy Students (2003), Planning and Evaluation Service, Office of the Under Secretary, U.S. Department of Education.

Table of Contents


Acknowledgements i

Chapter I. Information, Data and the National Reporting System 1

Why Care About Data? 2

Data Clarify and Explain 2

Data For Program Management and Improvement 2

Data For Clarification and Improvement: An Example 3

Purpose of This Guide 4

Overview of the Guide 4

Chapter II. Data and Its Uses: Accountability and Marketing 7

The Dimensions of Data 7

Types of Data 7

Qualitative and Quantitative Data in Adult Literacy

Programs 8

Data Quality 9

Data Use for Adult Literacy Programs 10

What Data Can Do 12

Data to Demonstrate and Describe 12

Using Data for Accountability 12

Adult Education Report Cards 14

Using Data for Marketing 15

Guidance for Using Data for Marketing 18

Chapter III. Data for Decision-Making: Program Management and Improvement 21

Data For Program Management and Improvement: A Model 21

Identify Issues or Topics 22

Performance Requirements 23

Research and Practice 23

Curiosity 23

Develop Measurable Questions 23

Focus the Question: Reasonable and Measurable 24

Break Down Questions: Educational Inputs and Outputs 24

Ask What Will Lead to Program Improvement 25

Summary: Refining Questions 26

Develop An Analysis Plan 27

Data Needed 27

Data Presentation 29

Analyze and Interpret Data 33

Interpreting Data: Examples 35

Develop A Plan for Change 39

Clearly Define the Change 40

Identify Staff Responsible for Changes 40

Identify Needed Resources 41

Implement Changes 41

Staff Buy-in 41

Leadership 42

Managing and Stabilizing Change 42

Evaluate the Change 43

Using Data For Program Management and Improvement:

Examples 43

Placement and Class Levels 43

Managing Enrollment 44

High Intensity Instruction 46

Chapter IV. Using Data Effectively: What Every Program Needs 49

Using Data: Systems and Culture 49

Technical Systems 49

Database Reporting Abilities 49

Quality Data Collection 51

Human Factors and Data Use Culture 52

Attitudes and Leadership 52

Collaboration and Training 53

State Examples 54

Delaware 54

Kansas 55

Iowa 56

Idaho 60

Massachusetts 62

Bibliography 65

Glossary 69

List of Exhibits


Exhibit 1.1
Student Native Language and Age
For ESL and ESL Citizenship Classes 3

Exhibit 2.1
Qualitative and Quantitative Data 7

Exhibit 2.2
NRS and Other Data Available in Local Program Databases
By Topic Area 11

Exhibit 2.3
Accountability Data to Compare Performance: A State Example 14

Exhibit 2.4
Alabama State Report Card (Excerpt) 16

Exhibit 2.4 (Continued)
Alabama State Report Card (Excerpt) 17

Exhibit 2.5
Iowa Employability Study:
Highest Degree and 2001 Earnings 18

Exhibit 3.1
Model for Using Data for Program Management and Improvement 22

Exhibit 3.2
Developing Questions for Data-Driven Decisions 26

Exhibit 3.3
Identifying Data Needs: Teachers, Students, Retention and Learning 28

Exhibit 3.4
Frequency Tables of Ethnicity 29

Exhibit 3.5
Examples of Graphic Data Displays:
Pie Chart, Bar Chart, Line Chart 31

Exhibit 3.6
Instructional Hours per Student for Selected States:
Mean, Median, Mode and Variance 33

Exhibit 3.7
Percent of Students Completing Levels by Age 36

Exhibit 3.8
Percent of Students Completing Levels by Ethnicity 36

Exhibit 3.9
Attendance Hours, Rate of Attendance and Test Gains for ESL Students 38

Exhibit 3.10

Percent of Students Pre- and Post-tested by Site and Instructional Hours 39

Exhibit 3.11
Level Completions and TABE Reading Scores by Level 44

Exhibit 3.12
Level Completions and TABE Reading Scores by Years of Education 44

Exhibit 3.13
Retention and Attendance at
Two Sites With Open-Entry/Exit 45

Exhibit 3.14
Retention and Attendance at Sites With
Managed Enrollment and Open-Entry 45

Exhibit 3.15
Enrollment, Attendance and TABE Gains
For Night and Day Classes 46

Exhibit 3.16
Enrollment, Attendance and TABE Gains
For Day Classes and Higher Intensity Night Classes 47

Exhibit 4.1
Components of a Quality Data Collection System 51

Exhibit 4.2
Data Quality Circle 52

Exhibit 4.3
Five Stages of NRS Implementation 53

Exhibit 4.4
Educational Functioning Level Completions by Gender 55

Exhibit 4.5
Sample Activity
Kansas Board of Regents – Adult Education
Data Quality Workshop** 57

Exhibit 4.6
Iowa Benchmark Development Process 59

Exhibit 4.8

Sample Program Report From Cognos 63

Chapter I. Information, Data and the National Reporting System
We live in a world in which information is everywhere. Every day, a multitude of sources – books, newspapers, the broadcast media, the Internet – exposes us to facts and figures on dozens of topics that demand our attention. For many of us, the challenge is to filter and evaluate from this information overload, the knowledge we need to run our lives or to make decisions and judgments important to us.
Ironically, adult educators often do not use one important source of information: the data they have available on their students, programs and instruction. Adult education teachers and program staff spend much time and resources collecting and reporting data to meet requirements of the National Reporting System (NRS) and their state accountability systems, but frequently do not see these data as a valuable source of information about their program. Yet, these data can help demonstrate a program’s accomplishments to the general public and funders, help manage the program and identify areas in need of improvement.
Program staff often sees data collection as paperwork that is done for someone else – the “feds,” the state, their bosses, bureaucrats – that has little value to them and what they do. Misperceptions abound about data and its use, including:

  • Data are “just numbers.” They do not capture what is important about students or instruction.

  • I’m not quantitative and can’t deal with math.” Data require complex technical skills to understand and use.

  • It’s not my job.” Teachers and other staff often do not perceive collecting and interpreting data to be part of their job.

  • You can prove anything with lies and statistics.” Data are untrustworthy– they can be used to show anything.

In addition, program staff often lack direct access to data and may not be able to review the information they collect. Poor data collection procedures that may exist in programs also work against data use. Staff recognizes data of low quality are of little use, reinforcing perceptions that data are not valuable.

The purpose of this guide is to change these perceptions– specifically to change how you think about data collected for the NRS. Besides explaining basic concepts and issues about data, the guide also includes a framework and model for using NRS data for program management and improvement.

Why Care About Data?

No one becomes an adult educator for fame or fortune. Adult education is a helping profession and the major reason people become involved is to help others improve their lives. The goals of adult education, as stated in legislation such as the Workforce Investment Act (WIA) and other frameworks, clearly articulate these reasons: to help people improve their literacy skills, become lifelong learners, succeed in employment and become better parents and citizens. To achieve these goals, programs must meet the needs of learners by designing effective instruction and services. Only through information about who is attending your program, how long they attend, what they achieve and why – in other words, data – can you design a quality program that helps students. It is almost impossible to provide quality instruction without using data as a guide.

Unlike in business and many other professions, adult educators do not typically see data in this way or use data to make decisions. In fact, in many educational settings data have often been used punitively, to demonstrate why schools are failing or ineffective, for example. Most teachers can relate at least one negative experience they have had due to something in “the data.” But once they begin thinking about how they can improve practice, teachers and other adult educators usually begin to see how data can clarify, explain and help in program management and improvement efforts.

Directory: docs
docs -> New directions newsletter the domestic abuse shelter of knox county
docs -> As a child Collins studied classical piano with Antonia Brico, making her public debut at age 13 performing Mozart's
docs -> The Archetype
docs -> Straight From the Heart by Dr. Linda Boen
docs -> Early Childhood iPad App Recommendations note
docs -> This romantic story is about two college students, Jenny and Oliver, who meet when Oliver visits the library to buy a book. Although Oliver’s family is poor and Jenny’s is rich, the two young people fall in love
docs -> Snow White Interactive Story
docs -> Frankenstein sfx questions By Stephen Jewell
docs -> -
docs -> It’s Your Story—Tell It!: A world of Girls Audience: This series is for Brownie Girl Scouts and is suitable for in-school and after-school troops. Purpose

Share with your friends:
  1   2   3   4   5   6   7   8   9   ...   21

The database is protected by copyright © 2019
send message

    Main page