Step 1. Assessing Currently Used Performance Measures
How does your court or court system currently measure its performance? How are the current measures distributed across inputs, outputs, outcomes, key success areas, perspectives, and core performance areas? Are they sufficient in terms of number, type, and balance of perspectives? Which of the measures seem more important than others? From what data sources are they drawn – administrative databases, surveys or third-party reports? By what methods are the measures taken and by whom? What required or desired performance information currently is not available to the court? What specific performance measures would provide that information? Answering these questions is the first important step in building an effective court performance measurement system (CPMS).
This first step identifies existing data sources, performance measures and methodologies that can be used to facilitate the identification and development of desired measures in Step 2. It helps in identifying the simplest and least expensive measures.
The choice of measures inevitably involves a balance of quality and cost. Taking a critical look at the performance measures your court and justice system partners (e.g., probation departments, city and county planning departments, and law enforcement agencies) currently are using is both a logical and practical starting point. Rather than trying to tackle the question of what your court ought to be measuring with a blank slate or with the adoption of a model set of performance measures (which may be unfamiliar or controversial) this first step draws on knowledge already available to the court.
Step 1 Provides A Learning Opportunity
Step 1 provides a familiar context and opportunity to learn about concepts and principles that should inform the choice of measures. It allows members of the court’s design team to explore, for example, the importance of a balanced scorecard of performance measures, the preference for outcome measures and other design principles or guidelines in an operational environment familiar to them.
Step 1 typically will dispel the usually inaccurate contention that “we do not measure anything.” For example, as part of its SMAART Program, the 19th Circuit Court of Lake County, Illinois, initially identified a total of 130 performance measures and indicators in use in May 2002. The Provincial Court of British Columbia produced an inventory of 42 court performance measures in use in January 2004 as part of its early efforts to build a comprehensive CPMS. The Municipal Court of Seattle identified a total of 55 performance measures in use in April 2004 for processing cases and hearings alone. Finally, the Yuma County (Arizona) Superior Court produced an inventory of existing performance measures that included 64 performance measures in use in June 2006 loosely categorized by performance success factors (domains or key areas) instead of divisions or units of the court system (Clerk’s Office, Probation Department).
Quick Successes, Small Wins
This first step produces results in a relatively short time. Even if a court postpones or abandons the building of a CPMS, both the process and the results of this first step can serve as valuable reference points for planning and development of various court improvement projects. Strategic planning and performance-based budgeting, for example, require the identification of performance measures and indicators. (See Martin, John A. An Approach to Long Range Planning for the Courts. Denver, CO: Center for Public Policy Studies, 1992.)
Three Tasks of Step 1
The products of this first step are an inventory of input, output and outcome measures currently used by the court and its divisions, departments and units, and an evaluation of the adequacy of the measures as a basis for building a CPMS. Three tasks are required to complete this step:
(1) cataloguing the court’s performance measures and indicators, as well those of the court’s justice system partners;
(2) categorizing the measures; and,
(3) assessing their completeness and balance.
Task 1 – Catalogue Current Performance Measures
The first task of this step is a simple listing of measures and indicators currently used by the court and its justice system partners (e.g., county agencies, state administrative offices of the courts, law enforcement agencies, and other courts). Because most courts use dozens, even hundreds, of input, output and outcome measures and indicators that may or may not be used routinely throughout the court, the initial listing should be compiled by court staff knowledgeable about the operations of all divisions, departments and units of the court.
Most of the measures identified are likely to be measures of resources or inputs (e.g., number of hearings and cases) and program or service outputs (e.g., number of clients served or training sessions conducted), not necessarily measures of outcomes or accomplishments. Some of the measures, especially case processing and financial measures, may be routinely taken and used across the entire court. Others may be used only by a particular division or unit of the court (e.g., the number of public service hours completed by probationers, the rate of successful and unsuccessful probation completion, or the percent change in number of psychological evaluations completed). Others may be unique to a particular court program within a court division (e.g., the number of hours logged by volunteers of a CASA (Court Appointed Special Advocate) program). Still others may be taken to satisfy federal regulation or grant requirements (e.g., the percent of dependency cases in which a "permanency hearing" is held within the 14-month target set by the Adoption and Safe Families Act (ASFA)).
A court’s justice system partners may be using measures that the court is not. For example, at the local level, county executive offices may be conducting employee opinion surveys, and another court in the court’s geographical area (e.g., a municipal court) may be measuring its collection of fines and fees. Likewise, at the state level, performance measurement initiatives may have identified measures relevant for the design and development of court performance measurement. For example, the Oregon Benchmarks of the Oregon Progress Board measure progress towards the state’s strategic vision, Oregon Shines. Its goals are three-fold: 1) quality jobs for all Oregonians, 2) safe, caring and engaged communities, and 3) healthy, sustainable surroundings. Benchmarks are organized into seven categories: economy, education, civic engagement, social support, public safety, community development and environment. Oregon state agencies are required to link their key performance measures to them. Knowledge of these existing local and state measures and the methodologies of their collection will benefit courts building CPMSs.
The effectiveness of a CPMS is enhanced when it reflects a balanced scorecard -- different views or perspectives of what is important. The initial catalogue of existing performance measures and indicators should be a brainstorming exercise, a process that does not systematically or inadvertently exclude identification of measures or indicators that are used on a limited basis or that are not well known. It should be free of criticism and judgment, encourage open thinking, and avoid premature analyses of identified measures. The initial catalogue will include labeled or unlabeled groupings of measures by court division (e.g., probation) and justice system partner (e.g., state administrative office of the courts), functions (e.g., civil case processing), or by automated case management systems as reported by court staff familiar with them.
Task 2 - Categorize Performance Measures Currently Used by the Court
The second task of this step includes a review and classification of the measures in the catalogue or inventory. Because slightly different names of two measures may, upon further definition, reveal subtle and important differences, only virtually identical measures should be excluded from the catalogue. Also, similar but not identical measures -- for example, the number of cases continued due to a lack of an interpreter and the average wait time experienced by an interpreter for case to be called -- may represent different aspects of the same core measure and may be included in a hierarchy of logically related performance measures in a CPMS.
After the initial review of the catalogue of measures, including the removal of redundant measures and the definition of measures that are unclear, the design team classifies the measures according to the focus on performance inputs, outputs and outcomes. Most courts are likely to find that the bulk of their measures focus on level of resources and frequency of tasks or activities (input and output measures), not on progress toward strategic goals and purposes (outcome measures).
Categorizing the measures according to their focus on inputs, outputs and outcomes is a convenient and effective method of introducing the court to a logic model, an abstract simplification of what is being measured from initial inputs through end outcomes, as well as providing an early look at the how much work may need to be done to build a CPMS. Court managers benefit from constructing a logic model for almost every strategic initiative, program, service or specific strategy -- including their inputs, outputs, activities, and initial, intermediate and long-term outcomes -- that describes how they theoretically work to achieve the desired results. Logic models are a useful framework not only for examining how a program or strategy may bring about results that benefit citizens and the community, but also for identifying the program components that should be measured to assess its efficiency and effectiveness.
Task 3 - Assess the Current Performance Measures
The final task of this initial step of building an effective CPMS is an assessment of the completeness and balance of the court’s current performance measures across factors that drive the court’s success. How many measures focus on the highest policy levels that indicate progress toward strategic goals and purpose (i.e., outcomes)? Are most of the court’s measures focused on case processing and little else? How many of the catalogued measures focus on such key success factors as access to justice, court user satisfaction, procedural fairness, independence and accountability, and public trust and confidence?
Perhaps, most of the court’s current performance measures currently track information that would not be considered important (perhaps even trivial) by court leaders and managers. Think about the car dashboard metaphor. Few would consider a dashboard instrument indicating the number of dings on a car's body as very helpful. Apart from unnecessary, most of us would consider such information distracting to the main task of driving.
Whether or not the court's key success factors already are defined as result of goal-setting or strategic planning initiatives, this task requires the court to think deeply not only about what measures should be tracked but also more fundamentally about what are the most important things for the court to do and accomplish. This linkage of performance measures to the mission, values and vision of a court is an important attribute of an effective CPMS.
Previous postings in this series:
Introduction to the Six-Step Process for the Design of an Effective Performance Measurement System (Part 1), Made2Measure, June 6, 2006
Introduction to the Six-Step Process (Part 2), Made2Measure, June 12, 2006
For the latest posts and archives of Made2Measure click here.
Copyright CourtMetrics 2006. All rights reserved.
Comments
Post a Comment