[Chapter 9, page 138]

Cluster evaluations as a method for assessing progress

In Chapter 9, we discuss randomized controlled studies and observational or quasi-experimental studies as methods for assessing the progress of a program or initiative. Another method is the cluster evaluation that examines a group of grants with common or similar objectives, with the aim of learning whether the grants are achieving the funder’s goals for a program or initiative. Although it may incidentally surface problems with particular grantees, a cluster evaluation is directed toward discovering whether the set of grants is effectively implementing the funder’s strategy and, if not, determining why not, with the goals of helping the organizations improve their effectiveness and helping the funder improve its strategy.

As an example, consider the Hewlett Foundation’s 2005 evaluation of ten non-governmental organizations (NGOs) in Europe and elsewhere that advocate with their governments and internationally, mainly to encourage bilateral and multilateral official development assistance for sexual and reproductive health and rights in developing countries. Most of these organizations are in Europe and are called EuroNGOs (pronounced “eurongos”). The cluster evaluation was done by two outside expert consultants.[1]

EuroNGOs came into being as a consequence of the 1994 International Conference on Population and Development’s (ICPD) in Cairo, in which UN member countries agreed on a Programme of Action that stressed the rights of individuals—particularly women—to determine their sexual and reproductive lives and health. Their work has been funded by the Gates, Hewlett, MacArthur, Packard, and Rockefeller foundations.

The Hewlett Foundation thought it was timely to learn about the effectiveness of the EuroNGOs in light of many changes in the ten years since the Cairo conference. These changes included:

• The United Nation’s adoption of the Millennium Development Goals (MGDs) in 2000. The MGDs largely shape overseas development assistance (ODA) priorities, with the overall goal of reducing extreme poverty. While some of the MDGs are concerned with gender equality and aspects of health, including maternal health, none explicitly refers to sexual and reproductive health or rights.
• A change in the modalities of ODA, including sector-wide approaches and categorical special funds, the increasing role of The World Bank and other multilateral donors, and increased devolution of decisionmaking to country offices, have led to a focus on the exclusion of sexual and reproductive health and rights (SRHR).
• The creation of the Global Fund for HIV/AIDS, Malaria and Tuberculosis, and the Global Initiative for Vaccines and Immunization, which target particular health interventions not necessarily focused on SRHR. They operate independently of the individual donor governments that had been the focus EuroNGO advocacy.
• The expansion of the European Union and the growth of the European Commission as a multilateral donor organization. In addition to the complexities of influencing the European Commission, some of its new Member States are not sympathetic to SRHR.
• The US government’s opposition to the Cairo agenda, which has made some donor governments reluctant to request ODA for SRHR in order not to alienate the U.S.

As the consultants summarized the situation, “The EuroNGOs today are operating in a very different environment from the one in which they were created. SRHR issues are slipping off the international agenda. They are either being crowded out by other priorities such as poverty reduction, debt relief and combating HIV/AIDS; or are being marginalized intentionally by anti-SRHR forces in the US and the newly enlarged European Union.” A business-as-usual model will not work. The bulk of the report focuses on how the EuroNGOs can meet these challenges.

The report suggests how the EuroNGOs can improve advocacy, both in national and international spheres, through personal contact with decisionmakers; work with cross-party groups of parliamentarians; alliances with other organizations (including nontraditional allies, such as church-based groups); and messages focused on connections between SRHR and the MDGs' poverty-alleviation goals. Many of the messages that worked a decade ago are not relevant today. In particular, “the EuroNGOs need to develop fresh, new approaches to language, rather than relying on the defensive approaches” taken in recent years. They must especially emphasize the links between SRHR and meeting the MDGs, especially the poverty and the health goals—for example, how women’s economic empowerment contributes to economic growth and poverty reduction, and the essential role of women’s ability to control their sexual and reproductive lives in their empowerment. And they must establish the importance of SRHR for HIV/AIDS prevention.

The report argues that the EuroNGOs need to develop alliances with strong development NGOs that are concerned with health, education, social development, rights and humanitarian assistance. It argues for increased work in developing countries themselves, helping build the capacity of indigenous NGOs. The report also suggests how the individual EuroNGOs can strengthen their management and fundraising, and argues for the need to improve collaboration among the EuroNGOs themselves—particularly in advocacy within the European Union.

Discussions of the consultants’ report by the Hewlett Foundation and its EuroNGO grantees have led to increased attention to the European Union and the World Bank. In addition to thwarting plans to eliminate loans for SRHR programs, some grantees have actively collaborated with the World Bank to improve contraceptive supplies in the developing world. Many of the EuroNGOs, especially those affiliated with International Planned Parenthood Federation, have improved their coordination and are undertaking advocacy work with Federation affiliates within the developing world. And several of the organizations are participating in an initiative to measure the impact of SRHR and population growth on economic growth and poverty alleviation, and are planning communications activities based on the emerging knowledge about these connections.

Other foundations have played a role in implementing the recommendations of the cluster evaluation. For example, the Gates Foundation now provides funds specifically for advocacy with the European Union, and the MacArthur Foundation supports the EuroNGOs secretariat, a body that shares information and coordinates policy responses within the group.