Chapter 13 — Measuring the Impact of Systemic Trial-Monitoring Activities 13.2 Choosing indicators to measure impact In selecting appropriate indicators for measuring impact, it is important to try to identify indicators that are specific, measurable, accurate, reliable and time-bound.192 These should relate to the achievement of a specific task or objective, or to progress toward resolving an identified issue or problem. The indicators can relate to whether a specific recommendation of a trial-monitoring programme has been implemented in practice. For example, measurable indicators in trial-monitoring programmes might include: • the number of trials or other judicial proceedings open to the public, compared to the number in the period preceding the monitoring; • the number and quality of steps that are being taken to make court schedules and dockets more transparent and easily accessible; • the categories of documents (previously unavailable) to which the public has been given access; • recommended changes that are made to laws or regulations to bring them into line with international standards; • quantifiable improvements in specific practices, e.g., defence attorneys are more readily available to indigent defendants; • increased resources and personnel available to deal with overwhelming court workloads; and • statistics showing that delays in hearing cases or issuing judgments have been reduced. This approach typifies a “learning model”, as described above, by testing project impact rather than project output. For instance, it would be short-sighted to assume that a training session on a particular international standard would, in itself, achieve compliance with that standard. Such a conclusion would involve making a number of possibly flawed assumptions, including that the training has been effective in terms of quality; that it has been attended by all those whose practices are problematic; that its messages were understood; that the individuals trained have changed their practices; and that non-compliance with a standard was due to a lack of knowledge, rather than to some other factor. While an “audit model” might record that the project met its goal by providing training, a “learning model” would need to assess whether the training resulted in improved implementation of the standard in question. This could be assessed through further monitoring. Quantitative measures can also be useful indicators, if the data they measure are carefully selected and if it actually measures impact. Some of the bullet points above, for example, could be measured quantitatively by tracking the prevalence of specific procedures in court cases over a period of time, and using the data to assess whether particular concerns have been resolved or reduced. However, it is important that the indicators be designed to capture the necessary level of detail to ensure their validity. As an illustration, in designing an indicator aimed at measuring the efficiency of processing cases through minor offence courts, it would be misleading merely to compare the statistics reflecting the overall caseload over two consecutive years if, during that time, traffic violations were removed from the jurisdiction of these courts. Even when well designed indicators show progress, it is advisable to urge monitors and analysts to be alert for any signs that a problem may persist, e.g., public surveys or recurring media reports that demonstrate continuing low confidence in the judicial process. These observations could prompt programmes to review the root causes of problems more closely and adapt their activities accordingly, or to review their analytical methodology. Conducting surveys and holding focus groups – either among justice-system actors or the general public – can also be ways to help determine whether projects are having the desired impact. 192  Ibid, pp. 7 and 12 ff. 127