Document Text (Pages 171-180) Back to Document

Community Capacity and Governance – New Approaches to Development and Evaluation

by Banyai, Cindy Lyn, PhD


Page 171

The evaluation lectures noted that in addition to this localization of policy structure,
evaluation can itself be a tool for change within an organization by examining the roles and
functions of the organization and building capacity through a participatory evaluation process
(Miyoshi & Stenning, 2008). It was pointed out during the August 2007 training that the main
weakness of the evaluation system in Nepal was that the system did not incorporate evaluation
into the planning process, indicating a lapse in the policy management cycle. Furthermore, it
was noted that the information from evaluation was collected by the central government but
often was not utilized afterwards for planning or for feedback to the project or community in
question; this also problematic.

Through the course of the Tokyo training, the participants were encouraged to discuss issues
and ideas in regards to evaluation. Many recurrent issues came up, such as the difficulty of
coordinating evaluation with multiple stakeholders, including donors, issues with participatory
evaluation, and how to go about effective localization of evaluations. The participants seemed
eager to improve their evaluation system and incorporate localization into evaluation; however
there was a significant amount of discussion about how to have effective localization of
evaluation and involve the local level in the policy management cycle up through national level
policy planning. It was clear from both the topics of the training and the discussions of the
participants that localization of evaluation is of key importance for evaluation reform in Nepal.

According to the results of a survey conducted by JICA as a part of the mid-term progress
report for fiscal year 2007, several issues and challenges to the institutionalization of the
evaluation system in Nepal were described including the utilization of results, quality of reports,
human resources in M&E, updating and using report formats and guidelines, and the planning of
M&E (JICA & NPCS, 2007b, p. 10).

The utilization of results and the planning of M&E activities reflect lapses in the proper
functioning of the evaluation cycle. The low quality of the reports, inadequate human resources
in monitoring and evaluation, and the lack of updating and utilizing report forms and guidelines
demonstrate low levels of evaluation capacity. These issues are consistent with the issues found
in the institutionalization of evaluation systems in other developing countries (discussed later in
this chapter) and the lack of evaluation capacity and evaluation cycle described by Biju Shrestra
(2009). There should be an increased awareness of these issues and the development of
strategies for overcoming them. There should also be recognition of the importance of the
evaluation process and a functioning evaluation cycle for effective evaluation and its
institutionalization.

Although this survey describes some of the complications with evaluation reform in Nepal
with the continued localization of evaluation and the evaluation capacity building trainings
included in the SMES project, progress with evaluation institutionalization has been made.
However, the additional potential benefits of the SMES project should not be overlooked by
either JICA or Nepal. Such benefits include social and institutional development through the
utilization of evaluation as an information sharing and capacity building tool.

The strategies of the SMES project facilitate the functioning of the evaluation cycle through
the introduction of cyclical policy structure management methods. Ultimately the training of core
trainers and the other trainings conducted through the project will serve to increase the
evaluation capacity in Nepal. It is also hoped that a culture of evaluation can be developed
through the trainings. Discussions on pervasive issues in evaluation, the promotion of a
productive atmosphere of evaluation, as well as on participation and ownership of evaluation are
encouraged during the trainings, which were observed in the August 2007 training of core

171


Page 172

trainers in Tokyo. The SMES project has strategies to address the issues with evaluation through
localization, human resource development in evaluation skills, manual and reporting
improvements, and has the potential to be quite effective in improving the functioning,
effectiveness, usefulness, and accountability of the evaluation system in Nepal.

The SMES project concluded in 2009. By the end of the program, it is expected that there
will be enough growth of evaluation capacity, a perpetuation of the evaluation cycle, and the
preliminary process of effective localization of evaluation that the evaluation reform and good
governance can take hold in Nepal.

1.1.2. The Localization of Evaluation of MDGs in the Philippines

The MDGs are broad goals, often presented in generic terms. However, there is a need for
the MDGs to be contextualized, not only to the country level, but disaggregated further to
specific regions and communities (Jahan, 2005, p. 4). The Localized Monitoring System on the
Millennium Development Goals (LMS) is a pilot project supported by the EU to “operationalize
the millennium goals” through localization to “facilitate monitoring, planning, resource
mobilization and program implementation (Stanta Ana, Abano, & Paredes, 2006, p. 12).” The
pilot project included three municipalities in Bohol, Philippines (Tubigon, Bilar, Jagna) and
coordinated Local Government Units (LGUs) with partner NGOs to create a locally specific
system to identify, assess, and monitor information and situations related to the MDGs and their
associated development plans.

The project took one year to complete and finished in March 2006. After the primary
commitment of the participating municipalities was confirmed, a Technical Working Group
(TWG) was created and charged with the task of executing the project. The TWG is an
interagency group made up of mostly senior staff from the municipal departments and
representatives from the NGOs (Santa Ana et al., 2006, p. 13). Members of the TWG
participated in a training that focused on the localization and contextualization of the MDGs,
tools for MDG monitoring, localization and advocacy, participatory MDG reporting, as well as
additional courses in monitoring software including Geographic Information System (GIS),
DevInfo5, and open source MDG Planning Matrix (Santa Ana et al., 2006, p. 14).

One of the most important outputs of the TWG trainings was the design development of the
local MDG monitoring system in each municipality based on the local situation, needs, and
capacity. The local framework has the goal of addressing poverty and improving people's well
being and quality of life. The localized MDG monitoring system keeps track of the progress on
the MDGs based on locally appropriate indicators. The system is also key to the policy
management cycle overall, as it provides information for planning and evaluating the
effectiveness of development programs in terms of real local impact on poverty and people's well
being, as well as to improve service delivery and local governance (Santa Ana et al., 2006, p.
14).

The municipality of Jagna provides an example of a fully functioning system of evaluation
and an incorporation of local monitoring and assessment into the policy management cycle.
Under the guidance of Mayor Exuperio Lloren and cooperating agencies, Jagna gathered data

5

DevInfo is a database developed by UNICEF for the international monitoring of the MDGs. Each participating
municipality can collate and analyze the data that they collect on the MDG and produce related visual graphics. The
data is also incorporated for provincial, national, and international data comparisons.

172


Page 173

and conducted surveys and household interviews to gathering information about problems,
access and necessity regarding the components of the MDGs. After the initial assessment,
Mayor Lloren then had the municipal department chiefs propose strategies to address their
respective development issues (Santa Ana et al., 2006, p. 6). This data was then used to develop
local targets under the eight MDG categories and incorporated into the 10-year Municipal
Development Plan of Jagna (Lloren, 2007).

The LMS project provided some interesting lessons for the development of localized
evaluation systems. The first lesson was that localization is possible, even in places with limited
resources and capacity. Next, political will and the participation, commitment, and cooperation
of top local officials is critical for the success of local monitoring and its sustainability.
Partnerships and networking between governments, NGOs, and civil society is beneficial for
garnering support for the MDGs and can help facilitate the creation of an evaluation culture.
Another lesson from this project indicated that gender-sensitive monitoring is still challenging,
especially in terms of identifying indicators, setting targets, and gathering information for gender
and development planning. The localization of evaluation helps to build understanding and
international partnerships on development issues. The final lesson from this project was that
there is no single, comprehensive path to development and local consideration and approaches to
monitoring and evaluation are the keys to making progress on development targets (Santa Ana et
al., 2006, p. 19). The findings from this project confirm what other authors and practitioners
have discussed on the localization of evaluation, particularly in terms of collaboration with
various organizations and government administrations and the importance of a contextualized
approach to evaluation.

The LMS localization pilot provides an interesting framework for the localization of
international indicators for development. It is a useful tool, particularly in terms of
complementing local planning and policy structure development for poverty reduction. The
project definitely contributes to building evaluation capacity, especially in terms of data
gathering techniques and software usage.

The case of Jagna provides an example of the localization and specification of aggregate
poverty indicators and the harmonization of outcomes throughout the various levels of a policy
structure from the local level to the international level. The development of an evaluation culture
can be seen in through LMS. The training of the local working group in seeks to bring local
stakeholders into the process of the evaluation, which develops a culture of evaluation.
Involving local stakeholders provides a comprehensive and conducive environment for
information exchange and benefits all parties. The results gathered from the evaluation must be
cycled back into the planning of the policy structure at all levels, including the local level,
otherwise the evaluation remains merely an exercise rather than an informational tool. Although
the information from the pilot was used in the planning of the local policy structure, according to
Mayor Lloren, it is unclear if other stakeholders in the evaluation were fully involved in the
process or were mere information givers; thus potentially stunting the positive effects of this
project.

1.1.3. Recap of Localization

Localization is important for the improvement of evaluation systems and the attainment of
localized and aggregate development goals. The importance of localizing evaluation has been
recognized, most notably in the cases of Nepal and the Philippines, and measures are being taken

173


Page 174

to incorporate the local level in the evaluation process. Although the reform toward localization
is underway, there are still doubts and hesitations in its implementation. However, through
trainings like the one that officials from Nepal are involved in and pilot localization projects, like
the one in Bohol, Philippines, effective localization of evaluation can be achieved and
improvements in the evaluation system can be seen.

More emphasis should be placed on the localization of policy structure and evaluation and
this can be done through decentralization policies and practices. However, in order for the real
practice and effectiveness of the localization of evaluation and policy structure to be realized,
better intervention strategies to address issues with evaluation in developing countries need to be
implemented. Furthermore, more investigation into the situation behind the issues in evaluation
needs to be done in order to gain a clearer picture of the issues so better policy structures can be
developed. Additionally, a better understanding of the process and functions of the localization
of evaluation and policy structure is necessary to improve localization and reap all the benefits
that it can offer.

The implementation of localized evaluation is pivotal to the implementation of a fully
functioning evaluation system having a complete policy management cycle, a productive
atmosphere of evaluation (evaluation culture), and appropriate policy structure and
organizational infrastructure. By localizing evaluation, local administrations build evaluation
capacity through the process of evaluation, which, in turn, also promotes ownership of
development projects and provides voice for local stakeholders. Localized evaluation increases
the amount of transparency, responsibility, and accountability in all levels of the policy structure,
which are some of the primary objectives of evaluation. The localization of evaluation is truly a
key component for the success of projects and ultimately national policy. Its impact on the
evaluation process should not be downplayed or overlooked.

The SMES project in Nepal emphasizes the localization of evaluation in the reform and
institutionalization of its evaluation system. The project tackles the issues of evaluation capacity
through trainings and the localization of evaluation itself. Additionally, the SMES project
attempts to clarify the organizational structure of evaluation through localization and helps to
build a culture of evaluation through bringing local stakeholders into the process of evaluation.
The SMES project is an interesting example of evaluation reform through the institutionalization
of the localization of evaluation.

In Jagna, an example of donor-driven localization of evaluation to improve international
policy effectiveness can be seen. Through operationalizing international and national
development indicators at the local level, better targeted policy structures can be developed.
Localizing large aggregate indicators helps to bring relevance to the local level, improves the
structure for coordination and organization of evaluation, and provides voice to the local
stakeholders in international policy. These advantages then increase the local evaluation
capacity and the impact of the policy.

These examples of efforts being made in evaluation improvement and localization
demonstrate that there is responsiveness to the call for improving governance and poverty
alleviation efforts. Furthermore, the SEMES project of Nepal and the Jagna case highlight some
ways in which localization can be facilitated and evaluation can be improved through multi-level
collaboration, coordination, and capacity building.

2. Crucial Issues with Evaluation in Developing Countries

174


Page 175

Evaluation is, at best, complex and arduous in developed countries. Developing countries
face special challenges when it comes to evaluation and the development of evaluation systems.
Although each country has its own specific issues when it comes to their evaluation systems,
across many countries the primary lapses in evaluation stem from a general lack of human
resource capacity to execute the evaluations properly. In addition to this overall lack of
evaluation capacity, and usually as a result of it, many developing countries face issues with the
full implementation of the policy management cycle. These lapses lend themselves to a critical
view on evaluation altogether; thus making the establishment of a culture of evaluation difficult
and causing problems with accurate and proactive evaluation execution and procurement of
support and funds.

Since many developing countries rely on donors for development projects, there must also be
a special consideration for the delicate balancing process that donor-influenced policy structures
have to cope with in regards to their external stakeholders. The external stakeholder issue goes
along with the difficulties in incorporating all relevant stakeholders into the evaluation process
that is faced by many countries.

One last major challenge developing countries face in terms of reform of evaluation systems
is the problematic institutional structure regarding evaluation and the legal framework. These
issues will be looked at in detail in this section, using information from the participants of the
Forum for the Institutionalization of Evaluation System (herein after referred to as the Forum),
that took place in March 2007 in Tokyo. This JICA training course for the promotion of
practical evaluation skills, as well as useable and effective evaluations systems, focused on
building the evaluation capacity of central government officials from 12 countries6. By
reviewing the inception reports from the participants of the Forum some clear issues with
evaluation begin to emerge. These issues include evaluation capacity, the use of the evaluation
in the policy management cycle, the cultivation of a culture of evaluation, the coordination and
involvement of stakeholders, and the development of legal frameworks and organizational
structures.

2.1. Evaluation Capacity

A lack of evaluation capacity is by far the leading challenge for reform and institutionalizing
evaluation systems in developing countries. Although there is a lot of talk in the fields of
development and public administration regarding evaluation capacity building (Cling et al.,
2002b; Kaufmann et al., 2002, p. 23; Razafindrakoto & Roubaud, 2002b, p. 291), it is difficult
ascertain a concrete definition of evaluation capacity itself, except that it deals with the skills
involved in conducing evaluations. Evaluation capacity is defined here as the ability for
individuals, agencies, or organizations to perform effective evaluations. Components of
evaluation capacity include the human resource capacity to execute an evaluation, appropriate
evaluation methods and associated skills, a general understanding of the importance and process
of evaluation, the selection and analysis of indicators, and efficient and reliable data collection
skills and methods. Evaluation capacity is crucial for the development of an evaluation culture
and the successful application of the policy management cycle; thus, effective evaluations
overall.

6

The countries include Cambodia, Ghana, Indonesia, Laos, Mongolia, Myanmar, Nepal, Pakistan, the Philippines,
Sri Lanka, Thailand, and Vietnam.

175


Page 176

Furthermore, the capacity to evaluate becomes increasingly more important as the shift in
evaluation paradigms goes from top-down monitoring to participatory evaluation processes
involving stakeholders at all levels. Thus, building evaluation capacity is pivotal to the success of
a policy structure and helps to improve the process horizontally and vertically through reporting
mechanisms (Cling et al., 2002b, p. 175; Vernooy et al., 2003, p. 25). Local stakeholders have
the ability to evaluate and report on the minute functioning and execution of the program or
project that stakeholders at higher levels may overlook. This makes the contribution that local
stakeholders valuable and is consistent with the concept of delegated discretion (see Fukuyama,
2004).

All participants of the Forum described all or some aspects of evaluation capacity as lacking
in their country. Cambodia reported specifically a lack of human resource capacity for
evaluations (Chou, 2007). In Indonesia, there is a general lack of evaluation capacity and
understanding of evaluation methods (Purwanto, 2007). There is a lack of an accepted evaluation
framework in Mongolia, and this leads to overall low evaluation capacity, especially in terms of
understanding, knowledge, and skills in evaluation (Munkhbat, 2007). Furthermore, there is
difficulty in proper indicator selection leading to time consuming and costly evaluation, as well
as low quality information reported in Mongolia (Dashbal, 2007). The current evaluation system
in Nepal is problematic in terms of consistency of evaluation methods, reporting techniques,
cyclical use of evaluation information and feedback mechanisms (Shrestha & Basant, 2007). In
Pakistan, the low level of evaluation capacity can be seen in terms of problematic indicator
selection (Akhtar, 2007). In Sri Lanka, the key factor in evaluation reform is finding the “right
balance between 'Monitoring' and 'Evaluation' (Siriwardana, 2007).” In the Philippines, there is
limited knowledge of the programs on the part of implementers and beneficiaries, as well as
problems with the reliability of data collected, the selection of critical indicators, and the
completeness of many reports (Carino, 2007). There is an overall lack of evaluation capacity in
Myanmar, which can be identified by the low amount of trained staff in evaluation and the lack
of use of appropriate evaluation methods and techniques, including indicator selection, data
collection and use, as well as relying on mostly monitoring methods instead of comprehensive
evaluations (Myint, 2007). The participant from Vietnam notes that there is generally low
evaluation capacity, particularly at the local level (Nguyen, 2007).

Evaluation capacity encompasses all relevant skills, forms, and processes involved in
evaluation and all participants of the Forum reported issues reported issues in this area, making it
a top concern. It is clear that a necessary step in improving evaluation is fortifying human
resources, technical skills, and other areas of evaluation capacity.

2.2. Evaluation and the Policy Management Cycle

Integrating evaluation into the management of a policy structure from the planning phase
through completion and beyond is described as an integral part of the policy management cycle
(Cling et al., 2002b, p. 151; Vernooy et al., 2003, p. 25; JICA, 2004; see also Chapter Three).
Evaluation results are used to improve the effectiveness of a policy structure; this can only be
done with the understanding of policy structure formulation and implementation that is
articulated through evaluation. Incorporating evaluation into the cycle makes the purpose of
evaluation and evaluation results clear throughout the evaluation process, which contributes to
more focused and efficient evaluations and better policy overall (Vernooy et al., 2003 p. 149,
Cling et al., 2002b, p. 174). Inclusion of evaluation in policy management leads to increased

176


Page 177

accountability, greater progress on poverty reduction outcomes, and higher levels of well-being
in a population (Razafindrakoto & Roubaud, 2002b, p. 269)

Analysis on the intended use of information and the actual use of information is important to
improve the function of the evaluation in the policy management cycle. Effective evaluation
systems depend on many factors and constraints, but the design and the mechanism should match
the intended uses of evaluation information, such as budget allocation process, policy planning
process, or project management and delivery of government services (Mackay, 2006). Therefore,
defining the purpose of the evaluation system and analyzing the current system against this
purpose is an essential part of the diagnosis.

Although the full implementation and utilization of the evaluation in the policy management
cycle is indicative of high levels of evaluation capacity, some countries specifically mentioned
lapses in this area. In the case of Mongolia, information gathered in evaluations is not often used
in the planning and implementation process (Munkhbat, 2007), which is indicative of the
absence of a cyclical process of evaluation. For Sri Lanka, the donor-driven nature of evaluation
there also leads to problems in the functioning of the policy management cycle, leaving gaps of
information and inadequate feedback mechanisms (Siriwardana, 2007). The Philippines report
that there is an inefficient feedback mechanism for evaluation, which undermines the usefulness
of the information gathered from evaluations and highlights a blip in the policy management
cycle (Carino, 2007). The participant from Myanmar states that there is “no long run evaluation
for project performance according to the National Plan (Myint, 2007).” This statement
demonstrates that there are notable gaps in the policy management cycle in Myanmar because
there needs to be continuous reporting and synthesis of information on the policy structure with
the intention of improving the current and future policy structure. Ghana has issues with the
implementation and utilization of the policy management cycle. The participant from Ghana also
notes that there is more of a focus on monitoring rather than full evaluation and results of
evaluation are rarely used in planning or budgetary allocations (Abu-Bonsrah, 2007).

Incorporating evaluation into policy management is crucial to ensuring accountability,
transparency, and effectiveness in a policy structure. Most participants of the Forum reported
lapses in their evaluation cycle, which inherently affects the usefulness of evaluation, as well as
the effectiveness of policy and poverty alleviation efforts.

2.3. Evaluation Culture

In addition to building the evaluation capacity of stakeholders and incorporating evaluation
into policy management, it is also important to create an atmosphere of evaluation that seeks to
bring all stakeholders into the evaluation process, rather than keeping some stakeholders,
especially lower level stakeholders, weary or fearful of evaluation. Donors and administrators
have used evaluation to determine whether or not the funding for polices should continue, often
times causing implementers and lower level administrators to be disillusioned with the practice
(Cling et al., 2002b, pp. 161-162; Gariba, 1998, p. 67). Sometimes this weariness spreads
between administrative levels due to a misalignment of values and poor communication on what
is expected of the evaluation (High & Nemes, 2007, p. 104; Mohan & Sullivan, 2006, pp. 8, 9).
Some lower level stakeholders may skew evaluation results to appease higher level
administrators or to preserve their positions (Morehouse, 1972, p. 870). The localization of
evaluation can facilitate the creation evaluation culture.

177


Page 178

The concept of evaluation culture draws attention to the necessity of active participation, a
positive environment, and a positive perception of evaluation in order for it to be useful. Shifting
the view of evaluation from a negative, punitive process to an information-gathering,
participatory process will create an atmosphere of evaluation that will increase evaluation and
program effectiveness (Cling et al., 2002b, p. 174). Enabling all stakeholders to evaluate and
think critically about their roles and outcomes through evaluation capacity building also creates
comfort with evaluation and creates ownership (Cling et al., 2002b, p. 174; Patton, 2002, p. 184;
WB, 2002, p. 2), thus adding to the atmosphere of evaluation. An established culture of
evaluation combines a positive atmosphere of evaluation, evaluation capacity, and constructive
participation of all stakeholders.

Many participants of the Forum recognized specifically that a culture of evaluation had yet to
develop in their country, while some countries described factors that are indicative of a lack of
evaluation culture. In Cambodia, there is a lack of budget to conduct proper evaluation, which
could also be indicative of a lack of recognition of the importance of evaluation (Chou, 2007).
The Mongolian participant says that there is no positive atmosphere of evaluation, no culture of
evaluation and evaluation is often considered a threat by those parties being evaluated (Dashbal,
2007). A pronounced lack of evaluation culture negatively affects several aspects of evaluation
and the establishment of a productive evaluation system in Nepal (Shrestha & Basant, 2007).
One of the reasons for the stunted evaluation culture in Nepal is the political ownership of
evaluation (Shrestha & Basant, 2007), which deters proactive evaluation processes. There is a
significant lack of evaluation culture in Pakistan due to the generally negative view of evaluation
in Pakistan and the difficulty of getting all necessary stakeholders to actively participate in the
evaluation process (Akhter, 2007). Sri Lanka also faces significant challenges to the
establishment of an evaluation culture. This is largely due to the perception that evaluation is
donor-driven and outside of the control and realm of importance of local agencies (Siriwardana,
2007). In terms of problematic evaluation culture, there is a traditional mindset of local officials
that hinders the evaluation process, a basic lack of political will for evaluation, uncooperative
beneficiaries, and there is little policy support for the implementation of evaluation, which leads
to few resources being allocated for evaluation purposes in the Philippines (Carino, 2007). A
participant from Ghana also notes similar problems in terms of securing adequate funding for
evaluation and the establishment of an evaluation culture (Mends, 2007). Many officers in the
Ministry of Finance and Economic Planning in Ghana seem to undervalue the importance of
evaluation and are weary of the credibility of internal evaluation (Mends, 2007). There is little in
the way of an established evaluation culture in Vietnam; therefore evaluation itself is
undervalued, making securing funding difficult with little follow through on evaluation activities
(Nguyen, 2007).

Promoting a culture of evaluation is necessary to facilitate participation of stakeholders and
ensure that evaluations are used to improve policy. While a discussion on establishing a culture
of evaluation is not necessarily on the forefront of improvement practice, noting that most of the
Forum participants expressed issues related to a productive atmosphere of evaluation, building a
positive culture of evaluation needs to be addressed.

2.4. Coordination with and Involvement of Stakeholders

It is necessary to involve relevant stakeholders in the evaluation process. However, it is
difficult to incorporate many stakeholders in evaluation, especially when there is low evaluation

178


Page 179

capacity and no culture of evaluation. Furthermore, developing countries have the added aspect
of the involvement of international donors, which further complicates the process.

The Cambodian participant cites the main problems with evaluation includes a lack of
coordination of obtained data between the National Institute of Statistics and those responsible
for evaluation for specific projects, which means that the projects must collect such data
independently; thus leading to potential redundancy and inefficiency (Chou, 2007). In
Indonesia,the most pressing issues in evaluation are the coordination of evaluation with multiple
stakeholders and this is affected by the lack of evaluation capacity and general understanding of
evaluation methods (Purwanto, 2007). While seeking to reform evaluation within the civil
service, the participants from Laos note that there is a lack of consistent and comprehensive
evaluation systems and mechanisms there (Xaovana, 2007; Dalavong, 2007). Now, international
donors, ASEAN member states, the international and the local business communities are
expecting that the civil servants of Laos will perform their jobs in a timely and efficient manner
with transparent methods and predictable systems, and this expectation is also echoed by the
increasingly aware Laotian public (Xaovana, 2007; Dalavong, 2007)., but with the lack of
consistency between the institutions and donors fulfilling this expectation is difficult.

Evaluation in Nepal is also complicated by the prevalence of varied evaluation systems of
donor organizations and the lack of coordination of evaluation systems and information use
between various ministries and organizations (Shrestha & Basant, 2007). Due to the strong sway
of donors in Ghana there is little coordination with all stakeholders for all projects implemented
by the Ministry of Finance and Economic Planning (Mends, 2007). Also in Ghana, evaluation is
often neglected when the projects are solely state administered and those projects with external
donors have difficulty integrating evaluation methods and techniques with national ones (Mends,
2007). Interestingly, there are some participatory practices for evaluation in place in Vietnam;
however, it is difficult to interest all stakeholders in fully participating in evaluation, leaving
open the possibility for abuse of programs and funds (Nguyen, 2007). Like many other countries
with strong external donors, Vietnam faces issues with coordinating evaluations with donor
practices. The discrepancy is particularly noticeable in terms of indicator selection and overall
evaluation practices (Nguyen, 2007).

Between the internal layers of decentralization and donors, coordinating policy and its
accompanying evaluation is difficult. This issue is particularly poignant in developing countries
that typically have multitudes of external donors in addition to their own levels of governance.
Although the coordination of stakeholders was only specifically mentioned by half of the Forum
participants, harmonization in policy and evaluation is clearly an area that needs to be tackled.

2.5. Legal Framework and Organizational Structures

To fortify the institutionalization of an evaluation system, it is necessary to have a sound
legal framework to legitimate evaluation systems and organizational structure. This ensures
credibility and objectivity of evaluations and the utilization of evaluation results, which was
reinforced by the DAC Hanoi Roundtable (Managing for Development Results, 2007). This issue
is faced by most developing countries in terms of establishing a fully usable and proactive
evaluation system. A few participants of the Forum made specific comments on this matter.

In Nepal, consistency is further compounded by difficulties in evaluation data collection,
complexity of reporting format, lack of a management information system, and an overall weak
organizational structure (Shrestha & Basant, 2007). Sri Lanka discusses the lack of legal

179


Page 180

foundation for evaluation and defunct evaluation institutional arrangements and the negative
effects that these have on obtaining funding for evaluation (Siriwardana, 2007). A participant
from Ghana notes that there are problems with the methods of evaluation, particularly in terms of
reliable data collection and clarification of roles and functions in evaluation (Abu-Bonsrah,
2007). In Vietnam, there are problematic evaluation methods in the current legal framework for
evaluation, noting that there are weak reporting mechanisms in place, inappropriate items in
policies, laws, and regulations in regards to evaluation outcomes, and that the results of
evaluations are not appropriately disclosed (Nguyen, 2007). With an appropriate legal
framework and organizational structure, roles of evaluation would be better defined and many
problems could be overcome.

Formalizing the processes and structure involved with evaluation is a vital step in
institutionalizing evaluation systems (Managing for Development Results, 2007). Furthermore,
provisions for the appropriate legal framework and organizational structures for evaluation
becomes increasingly important when coordinating between multiple levels of governance and
stakeholders, which is a necessary part of decentralization and localization.

The participants of the JICA Forum on the Institutionalization of Evaluation Systems pointed
out several cross-cutting issues of evaluation including the need for building evaluation capacity,
the necessity of continuity in the evaluation cycle, the development of a culture of evaluation, the
need for coordination and involvement of all stakeholders, particularly local stakeholders, in
evaluation and the creation of legal and organizational structures for evaluation. Although there
is no grand prescription for ameliorating these issues, the following section will look at various
types of evaluation and how they can help overcome these and other issues.

3. Evaluation Improvements

Problems with evaluation have not only been noticed by administrators in developing
countries, but have been observed in developed nations and international organizations, as well.
As evaluation began to be recognized as a valuable administrative tool, Morehouse (1972)
acknowledged that many evaluations were unsuccessful because the research methodology
employed did not match the situation which they were being applied, often being too simplistic
in comparison to the complex reality (p. 873). While evaluation methods have matured, there is
a continuing need for the development of adequate and contextual evaluations, particularly in
terms of policy, community organization (Figueira-McDonough, 2001, p. xii), and poverty
alleviation initiatives (Cling, Razafindrakoto, & Roubaud, 2002a, p. 14).

In recent years, with the development of the conceptualization of community capacity,
several indices and assessments of community capacity have come about (see Bush, Dower, &
Mutch, 2002; McKnight & Kretzmann, 1996; Mendis-Millard & Reed, 2007). Community
capacity assessments incorporate many of the approaches to evaluation described in this section,
as well as providing a more contextualized and holistic view of the community in the evaluation.

The last sub-section described the major issues with the establishment of evaluation systems
in developing countries as a lack of evaluation capacity and culture, little to no integration of
evaluation into the policy management cycle, poor coordination and involvement of
stakeholders, and ill-formed legal frameworks and organizational structures. With these issues in
mind, this sub-section proceeds to delve into some approaches to evaluation that will contribute
to the improvement of evaluation overall and help ease the issues presented in the previous sub-

180

© 2009 OpenThesis.org. All Rights Reserved.