Step 3 – Creating Measurement Hierarchies

This is the seventh posting in a multi-part series exploring the Six-Step Process for Building an Effective Court Performance Measurement System (CPMS) first summarized in Made2Measure (M2M) in October 2005. See below for a listing of the parts in this series to date.

Introduction

The third step of building a court performance measurement system (CPMS) is the construction of hierarchies (some call these “families” or “cascades”) that include a court’s core performance measures (identified in Step 2) and related subordinate measures (many of which may have been identified in Step 1). Step 3 involves three overlapping sub-steps: (3.1) breaking out (disaggregating) a court’s core measures; (3.2) identifying and defining subordinate measures that are related to core measures but are not necessarily breakouts of core measures; and (3.3) creating a hierarchy of performance measures for each of the core measure.

Beyond Core Measures


Measures are almost always proxies of outcomes or concepts – they are not the thing itself. A particular cholesterol level is not health. Case clearance is not court productivity or efficiency. To varying degrees, measures are removed and highly simplified version of the outcome of interest. We use these proxies in order to make it possible to measure the outcome easily, routinely and at a reasonable cost. The value of the proxy measure is that it is expected to correlate with the desired outcome. Because the correlation is rarely perfect, it is best that a single measure is not used on its own but together with a group of related measures in a hierarchy that provide a more valid, reliable, and rounded view of progress toward a strategic goal (e.g., access to justice).

Connecting the Dots

From a management perspective, the great value of measurement hierarchies is that they identify opportunities for collaboration. They put the court’s top management in much greater contact with every level of staff by defining the connection between high-level strategic goals and performance measures with lower-level departmental or divisional objectives and measures. Driving the CPMS to every level of the court signals to everyone what the drivers of success are and provides them with the concrete knowledge of how they contribute to that success.

Creating measurement hierarchies for each core measures ensures that important information at the level of court divisions, units, and programs is not masked by exclusive reliance on aggregated core measures. Measurement hierarchies identify opportunities for teamwork and collaboration. They put the court’s top management in much greater contact with every level of staff by defining the connection – a line of sight -- between high-level strategic goals and performance measures with lower-level departmental or divisional objectives and measures. Driving the CPMS to every level of the court signals to everyone what the drivers of success are and provides them with the concrete knowledge of how they contribute to that success.Measurement hierarchies in which lower-level subordinate measures “cascade” down from core measures help align the overall goals of the court with the goals and objectives of its divisions, units, and programs. They help make it clear to all court employees precisely how their actions help fulfill the court’s mission and strategic goals. For example, a clerk in a court’s jury commissioner’s office, will recognize that improvement in the percent of undeliverable mail – a lower-level subordinate performance measure -- drives juror qualification yield, which in turn drives the overall juror yield, which drives jury representativeness, a high-level core measure of fairness and equality.Or, for example, a core measure of cost per case (Measure 10 of the CourTools) aggregated across all case types may show no change over time. This is possible if cost per case of general civil cases rises sharply while that of all other cases drops slightly. Those responsible for improving case processing will want to see the breakouts (disaggregations) of cost per case for all the major case types rather than just the aggregate. The sharp rise in the cost of general civil cases may pinpoint problems or issues requiring immediate attention.

Technology Makes It Feasible

The advent of electronic databases and performance management and business intelligence software has all but eliminated the need for court managers to decide whether or not the problem of obscuring important information outweighs the value of using a vital few core performance measures. They need not choose between broad overview and rich detail. By organizing other important performance measures related to the core measure in a hierarchical format, court managers quickly will be able to drill down (descend) to lower levels of the hierarchy and access increasingly more detailed performance information. At the same time, court staff at lower levels will be able to ascend the hierarchy and align their day-to-day results with the court’s high-level core performance measures and broad strategic goals.

Breakouts by Individual Judicial Officers Not Recommended

The focus of CPMSs is organizational performance and not individual performance. Court performance measurement and individual performance evaluation differ in purpose, methodology, interpretation and use. Reporting breakouts by individual court employees, especially by elected judges as part of judicial performance evaluation, is likely to cause animosity and contention and carries the danger of destroying a CPMS. If a court wants to track breakouts by individual performance, such breakouts should not be made part of the measurement hierarchies of a CPMS and should be used for internal purposes only (e.g., motivating personnel).

Step 3.1 – Identify Simple Breakouts of Core Measures

The first sub- step is relatively straightforward. Each of the core measures identified in Step 2 is broken out (disaggregated) into its component data elements. For example, the first-level breakouts for a measure of jury representativeness may be the parities for each of three race/national origin groups. Other high-level core measures such as on-time case processing can be broken down by case type. The measure of employee opinion can be broken down into the various units or departments of a court.

Basic types of breakouts include: (1) organizational unit, department or project (e.g., civil division or drug court); (2) workload or difficulty of workload (e.g., case types); (3) geography (e.g., the different locations of a court); (4) court user category (e.g., attorneys, litigants, witnesses, jurors, title companies, and court observers); and (5) reason for outcome or rating (e.g., in surveys of citizens and court users, expressed overall dissatisfaction can be broken down into dissatisfaction with access, timeliness, courtesy and responsiveness of the court, and so forth). Though not focused on courts, Chapter 8, “Making Outcome Information Useful: Providing Indicator Breakouts,” in the Urban Institute’s Harry P. Hatry’s 1999 book, Performance Measurement: Getting Results (Washington, DC: Urban Institute Press, 103-105) is a helpful reference.

Measures derived from surveys are relatively easy to disaggregate by question, survey location, or respondent characteristics. For example, particular groups of respondents like victims of crime can be isolated from others to determine their unique perception of the survey as a whole or of specific questions (e.g., equal treatment or awareness of rights to assistance).

Step 3.2 – Identify Other Subordinate Measures

The requirement of the second sub- step is a little more subtle and more difficult. It involves another look at the inventory of performance measures produced in Step 1 -- all of the input, output, and outcome measures currently used by the court and its divisions, units and programs – and alignment of these measures in the hierarchies of core measures. For example, performance measures of a jurisdiction’s drug court (e.g., number of days between referral to drug court and admission to a treatment program) or a dependency court (e.g., average or median time form filing of the original petition to permanent child placement) that may not contribute to the aggregated core measure of on-time case processing should nonetheless be aligned in the hierarchy of this core measure. The alignment required of this sub-step helps to connect the performance of various court divisions and bridge the gap between day-to-day activities and strategic goals and mission.

It is important to remember that the relationship of the measures in a hierarchy may not simply be mathematical as is the case when the core measure is an average, a composite or index created by mathematically combining several related measures. The relationship can be logical as is the case when the subordinate measures are input and output measures, and the core measure is an outcome measure in the same key performance area, or when the factors measured by the subordinate measures are thought to be causes of the factors indicated by a core or superior measure. Or, the relationship simply can be based on a meaningful and appealing assemblage of performance measures that associated with a particular success factor like fairness, equality and integrity.

Step 3. 3 – Construct Measurement Hierarchies

The third and final sub-step of Step 3 is to combine and to order – cascade -- the core measure, the breakouts of the core measure, and the other subordinate measure in hierarchical fashion. Subordinate performance measures at lower levels in a hierarchy will be less comprehensive and aggregated, and more likely to be focused on inputs and outputs.

In Summary

Hierarchies of measures keep the number of core performance measures to a vital few at the top of the hierarchy while at the same time allowing the measurement of different perspectives, levels or aspects of the core measures from top to bottom of the court. Developing measurement hierarchies helps ensure that the right levels of measures are collected, distributed, and used at the right level of the court. While a court manager and other court leaders may focus on the high-level core measure of on time case processing aggregated across case types and case processing events, department or unit managers may focus their attentions on lower levels of hierarchy of this core measure such as on-time case processing of various case types, the elapsed time from filing to judge assignment, assignment to first hearing, disposition to post-disposition event, and so forth. Effective measurement hierarchies ensure that everyone in a court can gain a solid understanding of the subordinate measures and their objectives that make up the high-level core measures.

Next in this series: Step 4. Testing and Demonstrating the Measures.

Previous postings in this series:

  1. Introduction to the Six-Step Process for the Design of an Effective Performance Measurement System (Part 1), Made2Measure, June 6, 2006
  2. Introduction to the Six-Step Process (Part 2), Made2Measure, June 12, 2006
  3. Step 1 -- Assessing Currently Used Performance Measures, Made2Measure, June 17, 2006
  4. Q & A: Can Step 1 Be Taken At the State-Level, Made2Measure, June 23, 2006
  5. Step 2 -- Identifying Desired Performance Measures (Part 1), Made2Measure, July 2, 2006
  6. Step 2 – Identifying Desired Performance Measures (Sub-Steps 2.1 and 2.2), Made2Meassure, July 10, 2006

    For the latest posts and archives of Made2Measure click here.

Copyright CourtMetrics 2006. All rights reserved.

Comments

Post a Comment

Popular posts from this blog

Top 10 Reasons for Performance Measurement

There is a growing demand of “show me the data” from citizens and taxpayers, legislators, executive agencies, oversight boards and communities – all of whom exchange money and other support (e.g., trust and confidence) for services -- that courts can ill afford to ignore. Courts need to measure their performance to meet the demand for public accountability. But the benefits of court performance extend beyond accountability including improved prediction, better understanding and control, influence, a sharpened focus on what is important, and discovery of better practices. Objective performance data is central to good management practices like strategic planning, performance-based budgeting, Balanced Scorecards, Total Quality Management (TQM), and Six Sigma. Performance measures allow court leaders and managers, policymakers, legislators and the public to evaluate court programs’ inputs (the resources allocated), outputs (direct results of program or service activities) and outcomes (bro
Read more

The “What Ifs” Along the Road in the Quest of a Justice Index

What if we could draw an indicator of the “health” of the justice systems throughout the United States – a Justice Index - from readily available public data that is impartial, reliable, objective and comparable, let’s say the amount of time that criminal defendants spend in custody without a trial, or even without any access to legal or justice services? Or the bail amounts imposed on misdemeanor defendants?   This kind of a measure would not be perfect – no measure is – but it would be something, better than nothing. It would address questions of fairness and efficient use of public resources. That is, a criminal defendant languishing in jail awaiting trial suffers, at the least, a delay in justice. He or she is also occupying a jail bed and spending taxpayers’ money. What if we then join this measure with another meaningful indicator of the health of our justice systems in counties and states, let’s say the degree to which jurisdictions prosecute domestic violence cases.
Read more

Q & A: Outcome vs. Measure vs. Target vs. Standard

Q: Those of who write and speak about court performance measurement rely on their share of jargon -- and it's often confusing. What are the differences among the terms “outcome,” “measure,” “target” and “standard”? A: Socrates said that the beginning of wisdom is definition. We’ll see. Outcomes are the benefits or changes for the intended beneficiaries of a court’s programs and services. Outcomes may relate to knowledge, skills, attitudes, values, behavior, condition or status of the program participants or recipients of services. Examples include litigants' satisfaction with a court's courtesy and responsiveness, success of probation, time to case disposition, clarity of orders, integrity of case files, percent of mediation agreements, enforcement of orders, and perceived fairness of proceedings, as well as percent of expected expenditures and percent of revenues received. Outcomes should not be confused with the activities of those who run the court. (See the October 1
Read more