Document Text (Pages 81-90) Back to Document

Community Capacity and Governance – New Approaches to Development and Evaluation

by Banyai, Cindy Lyn, PhD


Page 81

formulation in response to them more effective, thus making decentralization and participatory
governance more effective.

Decentralization policy needs to be appropriately designed and effectively implemented and
complemented by officials who see devolution and empowerment as a benefit rather than a threat
(Cheema & Rondinelli, 2007, p. 9). Without true decentralization of fiscal and political
authority, clientelism will prevail (Grindle, 2007, p. 71). As with most concepts in the social
sciences, the failures of decentralization are not necessarily due to weaknesses in the concept, but
rather due to ineffective or problematic implementation of frameworks based on the concept
(Cheema & Rondinelli, 2007, p. 9).

In response to the difficulties faced by decentralization, localization is considered as an
important complement in Chapter Six. Using cases of training for evaluation and the localization
of evaluation, some ways in which localization to improve decentralization can be fostered.

2. Public Administration

Public administration is the management of public policy. Public administration describes
the structure of governance, including formal civil service, political and government systems, as
well the relationship between government actors and private contractors delivering public
services. According to NDI (2009), public policy can be defined as “the result of interactions
between different groups and parties about the course of action that should be taken (p. 46).”
Public policy is usually established by governments then implemented by the administrative
structure and monitored by civil society (NDI, 2009, p. 46).

Most governments around the world have undergone some sort of reform over the past
several decades, with mixed success (Peters, 2001, p. 16). Peters (2001) identifies four models
of government: 1) market government, 2) participative government, 3) flexible government, and
4) deregulated government (p. 21). Most public administration reforms fall into one of these
four categories. When it comes to public administration reforms, context is crucially important
(Peters, 2001, p. 180); meaning that there is no one model of governance that is suitable in every
circumstance and time.

Since the middle of the twentieth century, public administration reforms have moved toward
a more entrepreneurial style of governance (Ocampo, 2008, p. 248). These reforms are based on
the concept of the market and the idea that competition between service providers leads to better
and more efficient service to the customer, the citizen. Furthermore, market-based public
administration reforms are thought to empower citizens by localizing control of service delivery
to the community (Ocampo, 2008, p. 248). Other benefits of the market approach to public
administration include: a) a focus on outcomes instead of inputs; b) direction guided by goals
and vision, rather than rules and regulations; c) pre-emptive problem solving; d) economic
efficiency; e) decentralized authority; f) participatory management; and g) being a catalyst for
improvement of the community at large (Ocampo, 2008, p. 248). These are the touted benefits
of adopting an entrepreneurial style of governance, but they may not pan out in real life.
However, this does not mean that there is not an incentive to attempt to reform public
administration to meet these benefits.

New Public Management (NPM) adopts a market approach to governance and seeks to
improve public administration by making it function in ways similar to business. Initially, NPM
had two meanings: 1) managerialism – the introduction of private business methods into public

81


Page 82

administration, and 2) new institutional economics – the introduction of incentive structures,
such as market competition, into public service provision (Rhodes, 1997, p. 48).

NPM involves less government but more governance (Ocampo, 2008, p. 253; Rhodes, 1997,
p. 49). NPM makes government more mission-driven than rule bound, asserts that government
should meet the needs of the people, not the bureaucracy, and incorporates theories of
decentralization into its practice (Cheema & Rondinelli, 2007, p. 4). While, traditional systems
of public administration focused mainly on compliance, NPM places more emphasis on
accountability, performance, program, and policy focusing more highly on outputs and outcomes
(Stewart, 1985 as cited in Morita, 2009).

NPM is a group of administrative doctrines that emerged in the 1970s out of the Organization
for Economic Cooperation and Development (OECD) countries (Ocampo, 2008. p. 249). There
are eight basic trends in public management:
1. strengthening steering functions at the center;
2. devolving authority, providing flexibility;
3. ensuring performance, control, accountability;
4. improving the management of human resources;
5. optimizing information technology;
6. developing competition and choice;
7. improving the quality of regulation; and
8. providing responsive service (Kickert, 1997 as cited in Ocampo, 2008. p.
249).

NPM can also be described as managing for results (Rhodes, 1997, p. 49), which has been
adopted by several countries and by the OECD’s DAC (see also Lavergne & Alba, 2003).
Results-based management improves management effectiveness and accountability through
involving key stakeholders to define realistic results, assess risk, monitor progress, integrate
lessons learned into management decisions, and report on performance (Lavergne & Alba, 2003,
p. 47).

Sachs (2005) describes the necessary components of effective public management strategy
for development as decentralization, public sector capacity building, information technologies,
audits, and continuous monitoring and evaluation with measurable benchmarks (p. 278). There
are many similarities between Sach’s recommendation for public management for development
and the concepts involved in NPM.

NPM has been criticized because government cannot be consistently run like a business, may
not be global applicability, as well as the paradigm itself not being properly implemented in
many circumstances (Ocampo, 2008, p. 252). Many of the criticisms of and obstacles of
decentralization apply to NMP because it holds decentralization as one of its main tenets.
Although there have been critiques of NPM, it still continues to inspire new reforms in public
administration in an attempt to improve governance (Ocampo, 2008, p. 253).

Another problem with NPM is its tendency to lead to a “hollowing out of the state” (Rhodes,
1997, p. 53). The hollow state refers to an approach to policy implementation that relies on
private or non-profit organizations to deliver public goods (Fredericksen & London, 2000, p.
230). The basic contention of the hollow state is that it allows for more localized and adaptive
public service (Fredericksen & London, 2000, p. 230).

However, there are also some negative aspects of the hollow state. The five basic negative
points of hollowing out are:
1. privatization and limiting the scope and forms of public intervention;

82


Page 83

2. loss of functions by central and local government departments to
alternative delivery systems (such as agencies);
3. loss of functions by governments and institutions;
4. limited discretion of public servants; and
5. more political control thorough sharper distinction between politics and
administration (Rhodes, 1997, pp. 53-54).

Hollowing out of the state erodes accountability because a) institutional complexity obscures
personal accountability, b) contractors and specialized departments have replaced central
departments and local councils in service delivery and are not subject to the same public scrutiny
and accountability, c) transparency and consumer responsiveness do not equate to public
accountability because the user has limited choice thus reduced power to hold the agencies
accountable, and d) multitudes of service providers leave the administration’s position
ambiguous without proper arrangement (Rhodes, 1997, p. 54). Another point of contention with
NPM is that is creates difficulty in determining which agency or department is responsible for an
outcome, which causes problems for evaluation (Rhodes, 1997, p. 55).

Many already developed countries are convinced that an entrepreneurial approach to public
administration is the best form and employ a derivation of NPM. Decentralization is consistent
with these approaches. However, often times the challenges of developing countries are not
consistent with the ideals set for in NPM. Instead, these countries focus on establishing a clear
rule of law and balancing power between the state and business first (Peters, 2001, pp. 8-9). In
these cases, public administration reform should take a different form. Consistent with the
concepts that were presented earlier in this chapter and in Chapter Two, which demonstrate that
participatory mechanisms and governance are the most suitable for achieving equitable
development, a model of participative government may be advisable for developing countries, as
well as already developed ones.

Participative governments are useful to reform governments that have a strong hierarchy
(Peters, 2001, p. 21). Participative reforms include a flatter organization, management such as
Total Quality Management (TQM) and management in teams, consultative policy making, with
the public also involved in policy making and consultation (Peters, 2001, p. 21). The
participation model sees coordination as being driven from the bottom up, which requires a
population of citizens that can articulate their demands effectively and administrative
organizations that are oriented toward delivering holistic services (Peters, 2001, p. 185).
Furthermore, not all government functions would work well utilizing participatory methods,
calling for it to be applied in contexts that are suited for participation and where participation is
demanded (Peters, 2001, p. 193).

2.1. Public Administration Tools

The previous sections of this chapter discussed concepts of governance and methods of
public administration and this section will look at a few key tools that are of vital importance to
those types of governance and public administration. These tools include the management cycle,
the logic framework, and evaluation.

2.1.1. The Management Cycle

83


Page 84

The management cycle is a way to articulate the necessary and continuous steps in policy
management. Figure 5 is a visual representation of the policy management cycle (Miyoshi,
2008a; JICA, 2004; see also Wolf, 1986, pp. 91-93 regarding the preliminary stages of policy
management).

Figure - Policy Management Cycle

Source: Miyoshi 2008a, based on JICA, 2004

The cycle starts with the ex-post evaluations (evaluation at the completion of a policy,
program, or project (Miyoshi, 2008b, p. 9)) and the feedback on previous policies, programs, and
projects related to an initiative. The initial stages of policy planning come next and then the
cycle moves through a series of preliminary assessments of the proposed policy, culminating in a
full ex-ante evaluation (evaluation commenced before the implementation of a policy, program,
or project (Miyoshi, 2008b, p, 9)). Based on the results of the ex-ante evaluation, a policy is
implemented, abandoned, or reformulated. After a policy has been implemented, a schedule of
mid-term and terminal evaluations (also known as monitoring (Miyoshi, 2008b, p. 9)) for its
various projects and programs is decided upon followed by an ex-post evaluation, which then
leads back to the continuation of the policy, as well as the creation of new policies.

This policy management cycle should proceed similarly at all levels of governance and for
the programs and projects affiliated with each policy to create a fully functioning and assessed
policy structure in both vertical and horizontal terms. Following the policy management cycle
leads to increased accountability of the policy, especially if the evaluations are appropriately
reported to the public and mechanisms for feedback are in place.

84


Page 85

2.2. The Logic Framework

The logic framework is an analytical and management tool for policy managers and planners.
The six main functions of the logic framework are:
1. to analyze the existing situation during project preparation;
2. to establish a logical hierarchy of means by which objectives will be
reached;
3. to identify the potential risks to achieving objectives and to sustain
outcomes;
4. to establish how outputs and outcomes might best be monitored and
evaluated;
5. to present a summary of the project in a standard format; and
6. to monitor and review projects during implementation (AusGUIDElines,
2003, p. 1).

Overall, the logic framework serves to provide consistency and organization for management
and evaluation of a policy structure. The logic framework complements the program theory
matrix (PTM), which is another tool for policy management that outlines the specific plan for
implementation of a program or project.

Figure 6 shows the basic outline of the logic framework, as well as how the logic framework
fits in with common policy management parlance.

Figure - Logic Framework

Source: Miyoshi, 2008a, based on JICA, 2004

Reading from left to right, the logic framework starts with end outcomes, which represent the
desired change in society (Miyoshi, 2008b, p. 3). The framework then moves on to intermediate
outcomes, which is the desired change in the target groups of the policy, program, or project
(Miyoshi, 2008b, p. 3). The end outcomes and the intermediate outcomes (the pink boxes in
Figure 4) combine for the overall outcomes for the policy. The outcomes cannot be controlled
directly through the plan, but are the logical results of the plan outlined on the right of the
framework (the green boxes in Figure 6).

85


Page 86

The outputs are the goods and services resulting from the activities that are executed using
inputs such as human and material resources, operating funds, facilities, capital, expertise, and
time (Miyoshi, 2008b, p. 3). These green boxes can be described as the implementation or
administrative activities of a policy, program, or project. The implementation of a policy,
program, or project can be directly controlled through administrative activities and should
logically lead to the desired outcomes. The outputs and the outcomes collectively can be
described as the results of a policy, program, or project.

A policy, program, or project has vertical logic when it does what it is intended to do,
clarifies causal relationships, and specifies the important assumptions and uncertainties beyond
managers’ control (AusGUIDElines, 2003, p. 15). There is horizontal logic when the specified
objectives of the policy, program, or project are measured and the means by which they are
measured are verified (AusGUIDElines, 2003, p. 15).

While the logic framework is widely used and provides a common ground for policy
management between levels of governance and extra-national entities, there are still some
discrepancies between the various understandings and vocabulary associated with the
framework. For instance, in Australia, the logic framework commonly used for international
development initiatives uses the word goal instead of end outcomes and adds extra outcomes
categories (see AusGUIDElines, 2003, p. 2) in comparison to the model used in Japan, which is
described here. Despite some inconsistencies in the use of the logic framework around the
world, the basic intent and structure of the framework are understood and implemented similarly,
which creates common ground for administrators and practitioners, but allows for flexibility
based on context, circumstance, and understanding.

The logic framework can be used as guide to better understand the plan and expectations of a
policy, program, or project, as well as a framework for both quantitative and qualitative
evaluation (Gardener & Geierstranger, 2007, p. 9). Indicators can be crafted for each portion of
the logic framework and evaluated accordingly. The logic framework is used to analyze the
policy structures of the cases of rural revitalization in Japan in Chapter Five, as well as to
analyze the local policy structure of Pagudpud in Chapter Seven.

Evaluation is a key component of the policy management cycle and the logic framework can
be used to guide evaluation. The next subsection will delve into the necessity of evaluation and
some accompanying tools for it.

2.3. Evaluation

Surprisingly, government policies often do not have clear objectives or benefits that are
astutely analyzed (Miyoshi, 2008b, p. 1). While some policies may seem beneficial in concept, it
does not necessarily mean that they are effectively crafted or implemented. It is for this reason
that it is necessary to conduct evaluations. This need was first recognized in the United States by
New York Senator Daniel Patrick Moynihan through a study that he commissioned on poverty in
the United States, which identified the necessity of using social science and evaluation to
improve public policy effectiveness (Morehouse, 1972, p. 870). The need for monitoring and
evaluation is now widely recognized, especially as people demand more and better results from
their governance systems (UNDP, 1997, p. 29).

Evaluation is more than just a way to look critically at the planning and processes in a policy,
program, or project. It employs social research to see how well public policy works (Morehouse,

86


Page 87

1972, p. 868). Basically evaluation is employed to determine what works and what does not
(Morehouse, 1972, p. 868; WB, 2003, p. 17).

Morehouse (1972) offers an early definition of evaluation: “the means of determining the
extent to which a program is achieving its objectives, using methods that yield evidence that is
objective, systematic, and comprehensive (p. 869).” Later, Carol Weiss (1998) expands on the
concept of evaluation defining it as “the systematic assessment of the operation and/or the
outcomes of a project, program, or policy, compared to a set of explicit or implicit standards, as a
means of contributing to the improvement of the project, program or policy (p.4).” This
definition offers a straightforward explanation of evaluation in relation to a policy structure.
Michael Q. Patton (2002) offers a similar definition of program evaluation, describing it as “the
systematic collection of information about the activities, characteristics, and outcomes of
programs to make judgments about the program, improve program effectiveness, and/or inform
decision about future programming (p. 10).” While Patton’s definition is specifically referring to
program evaluation, it can easily be related to the evaluation of any part of a policy structure, be
it a project, program or overall policy.

Patton (2002) also defines process evaluation as a process that “aim(s) at elucidating and
understanding internal dynamics of how a program, organization, or relationship operates (p.
159).” This definition extends evaluation to include the descriptions of how people engage with
one another and the description of their experiences in a dynamic environment (Patton 2002, p.
159), which applies more aptly to the aim of participatory methods in evaluating community
situations and providing voice to various community groups, in addition to commenting on the
local policy structure.

With these definitions, a working definition of evaluation for this work is rendered:
evaluation is the systematic collection and assessment of information related to the outcomes,
operation, or process of a policy structure, organization or relationship.

There are two main purposes of evaluation: 1) improving administrative operations through
obtaining a better understanding of operations – learning; and 2) providing results to
stakeholders – accountability (Miyoshi, 2008b, p. 2; see also High & Nemes, 2007, p. 104;
Weiss, 1998, pp. 27-28). Both of these purposes of evaluation emphasize the importance of
feedback. Evaluation is focused on practically improving policy and is therefore more applied
than academic, focusing on practical methods (Miyoshi, 2008b, p. 2).

Good evaluation has the following requirements:
1. usefulness;
2. impartiality and independence;
3. credibility; and
4. participation of stakeholders (JICA, 2004).
These requirements ensure that evaluation is undertaken appropriately and serves the
purposes for which it is being undertaken.
Evaluation is most effective when these five basic steps are taken:
1. understand the context and policy environment;
2. develop a theory of change to explain how and why a project's activities
are expected to lead to desired policy changes;
3. define benchmarks;
4. collect data; and
5. use findings (Louie & Guthrie, 2007, p. 5).

87


Page 88

By following these steps it is more likely that a good evaluation will be conducted. After
there is an understanding of the context in which the evaluation is to take place, developing a
theory using the logic framework is the next step. Once a policy has been outlined in the logic
framework, the specific steps of evaluation – defining benchmarks, collecting data, and then
giving feedback can be conducted.

Monitoring and evaluation are a part of the policy management cycle and should be
undertaken on an ongoing basis (Louie & Guthrie, 2007, p. 5; Miyoshi, 2008b, p. 9; Weiss, 1998,
pp.25-26). Coffman (2007) suggests using the logic framework to create a continuous evaluation
plan that is most useful for the stakeholders involved (p. 4). An important part of the
incorporation of evaluation into the management cycle is providing feedback from the
evaluations to the public and other stakeholders. Another factor in integrating evaluation into the
policy management cycle is the need to link the results of evaluations to improvements to an
existing policy or the creation of a new policy, particularly in terms of altering or eliminating
underperforming portions in budgeting considerations (Kaufmann et al., 2002, p. 22).

Providing feedback to the public will help to inspire public discourse on the policy in
question, which is a reinforcing principle of evaluation and data collection (Kaufmann et al.,
2002, p. 22). Feedback is important to ensure that evaluation serves the purpose of improving
policy and public relations. However, the information rendered by an evaluation must be useful
to the stakeholders (Kaufmann et al., 2002, p. 22), which suggests the need for careful
consideration of feedback usage in crafting an evaluation framework.

Evaluation is useful both in terms of the results that it produces and the process involved in
the evaluation (Patton, 1988 as cited in Go, 2009). Broadening the scope of understanding and
potential use of evaluation will help to make it a more useful tool for policy and policy making
(Go, 2009).

There is a need for a consistent understanding of evaluation terminology and overall stability
to ensure that all stakeholders involved in evaluation are discussing the same things (Morehouse,
1972, p. 871). The logic framework can be used to build evaluations to provide consistency and
stability. Across the logic framework, the relationship between the ends (outcomes) and
methods (inputs/activities/outcomes) is analyzed to address causal relationships (Miyoshi,
2008b, p. 3). Using the logic framework also helps to clarify a policy, which is an important step
in evaluation (Miyoshi, 2008b, p. 3).

Policy evaluation usually starts with the end outcomes and program evaluation begins with
intermediate outcomes, and project evaluation start from the outputs (Miyoshi, 2008b, p. 6).
This means that there is slightly different approach to evaluation taken at each level of a policy
(Miyoshi, 2008b, p. 6). However, this does not mean that each level of evaluation only follows a
potion of the logic framework. Rather, each policy, program, and project should have a fully
operational logic model to guide and describe their role within the larger policy, yet each logic
model should be consistently nested within the largest policy framework. Figure 7 is a diagram
showing the integration of the logic framework into a policy structure.

88


Page 89

Figure – Integrated Policy Structure

Source: Miyoshi, 2008a

In order to construct an evaluation, questions must be formulated. The formulation of
questions for an evaluation can most effectively be done in a participatory way. There are three
broad categories of questions: 1) measuring performance; 2) examining implementation
processes and clarifying; and 3) causal relations (United States General Accounting Office 1991
as cited in Miyoshi, 2008b, p. 4). Questions help to design the overall evaluation and decide on
indicators (Weiss, 1998, p. 87).

There are three types of evaluation questions:
1. descriptive questions – focus on the current situation, examine
implementation process;
2. normative questions – compare with targets, measure performance; and
3. cause-effect questions – measure effect of implementation, impact
evaluation (Miyoshi, 2008a).

Descriptive questions search for implementation failure and normative questions looks for
the degree of achievement, while cause-effect questions examine design failure (Miyoshi,
2008a).

Indicators are concepts that help explain and provide understanding of phenomena (Miyoshi,
2008a). They provide the information necessary to determine progress toward stated policy
objectives (AusGUIDElines, 2003, p. 17). Indicators are operationalized through measurement
(Miyoshi, 2008a) that clearly defines the quantity, quality, and timing of expected results
(AusGUIDElines, 2003, p. 17).

Each evaluation will have indicators that are specifically tailored to gauge the success and
progress of that policy, program, or project. However, there are some general guidelines for the
creation and selection of indicators. Australia uses the acronym SMART to describe indicator
criteria: Specific, Measureable, Attainable, Relevant, and Timely (AusGUIDElines, 2003, pp.
25-26). JICA (2004) breaks up the guidelines further into indicators and measurement.
Indicators need to be direct and valid, operational, and adequate, and measurement of indicators

89


Page 90

should be practical and reliable (Miyoshi, 2008a). Following these criteria will ensure that the
selected indicators will render data that will be useful for producing coherent reports on the
status of a policy, program, or project.

To help guide evaluation of development projects by various donors, the DAC established
indicator criteria for evaluation. The DAC-5 evaluation criteria are: 1) efficiency, 2)
effectiveness, 3) impact, 4) relevance, and 5) sustainability (Miyoshi, 2008a). These categories
of indicators are useful in the design of evaluation, as well as for rendering reports to gauge the
importance of the data collected during the evaluation.

Figure 8 shows the integration of the logic framework with the criteria for evaluation and
evaluation questions. The components of the logic framework are in the middle of the diagram
and the appropriately corresponding categories of evaluation questions are on the left of the
framework. Process is examined through analyzing the steps between inputs and outputs.
Performance is measured by gauging the effects that the outputs have on the intermediate
outcomes. The relationship between the logic framework and the DAC-5 evaluation criteria can
be found on the right of the framework.

Figure - Integrated logic framework and evaluation criteria

Source: Miyoshi, 2008a

Efficiency is examined by analyzing the inputs through the outputs, as in examining the
process. Effectiveness is similar to measuring performance in that it highlights the relationship
between the outputs and the intermediate outcomes. The DAC-5 criteria go further then to
examine impact, which is analyzed by looking at the relationship between the intermediate
outcomes and the end outcomes. The relevance of a policy is also examined through analyzing
the link between intermediate outcomes and end outcomes. Sustainability is determined by
analyzing the entire logical framework in comparison to the desired outcomes in the long run.

90

© 2009 OpenThesis.org. All Rights Reserved.