Dear all,

Several of you have brought up the issue of measuring the impact of “system level” results on target enterprises and households. There are two strategies that I know of:

For some programs, the impact of each individual intervention they implement can be assessed separately. They then add up these impacts, taking into account any overlap in the enterprises and households reached.

However, as several of you have pointed out, it may be challenging to separate the impacts of individual interventions in a value chain. Therefore, some programs are assessing impact (quantitatively and qualitatively) at the level of a whole value chain – for example by assessing the overall change in that value chain in terms of sales, productivity etc. and in terms of qualitative changes such as new relationships and innovations. The programs then estimate how much of that overall change is due to program activities by assessing changes produced by individual interventions and using methods such as comparing the development of a value chain in a targeted area against national trends or other areas. Finally they estimate the impacts of these changes on enterprises and household through a combination of careful measurement of the impact of individual interventions or groups of interventions and assessing changes in enterprises (and sometimes households) that are likely to have been affected by changes in the overall value chain.

For both these assessment strategies, some programs also assess the extent to which other enterprises copy those that have been more directly affected by a program. For example, how many other farmers are copying the farming techniques of those trained as a result of a programs’ work with trainers or input suppliers. Some also look at the “market uptake” of new business models or types of relationships that have been introduced or promoted by the program. For example, if the program promoted outgrower contracts for small farmers, how many small farmers throughout the value chain have got outgrower contracts for the first time since program activities? And, when interviewing small farmers or the contractors, what is the likelihood that these new contracts resulted, at least partially, from program activities? This type of “triangulation” (gathering and comparing information from different sources) enables these programs to arrive at a reasonable estimate of their impact on enterprises or households as a result of “system level” change.

Best,

Aly

Dear Participants

The DCED supported system has a few features that make is attractive:

Internal versus external?
Impact assessment should ideally be external, but in the complex economies where most projects operate, it is almost impossible for external M&E experts to really understand which and how changes take place. Project staff has at least some idea about this. In many cases it will take a separate similar size M&E project to understand the reality. It is rare that donors are willing to finance this. Conducting M&E only in house on the other hand reduces credibility. This system of auditing is a perfect compromise.

Comparing:
Good programs should not have many reasons to be scared to share impact information. One reason however is, that their honest data is compared to that is “crooks”
The Auditing assures that the real crooks are exposed. More important, the standard makes the data comparable. (How long after the project to measure, where to put boundaries, measuring crowding in, displacement etc.)
Apples will be compared with apples and if there are still differences in for example the assumptions, these become explicit.

Project owned development
The standards have emerged from projects and are being tested by projects before any donor can get its hand on it. This is a serious improvement from some past experiences where donors or external specialists design systems that are not executable. Each project that is involved in early stage can influence the system and can assure it is realistically executable.

Improving or proving
Not many projects like to spend serious efforts to produce a figure at the end of the project. Projects want to create impact. Working with Impact logics is a great management tool to produce impact, the proving almost comes for free as a side product.

Small and large projects
The standards make a distinction between “must haves” and “recommended”. This allows project to discuss with their donors how much money they would like to spend on the system. A trade off can be made between costs and accuracy. This will allow projects with low budgets to join. But also allows larger project to build a solid system around the standards.

My own opinion:
The field has been calling for data for a long time. With this system we have at least half an answer. It is clear that this approach still has a way to go to become the market standard. Already in this shape however it is a good tool.
If the system becomes the standard in the field, there will be projects that are in trouble. Failing projects will be exposed and successful ones will be rewarded. This way M&E could not only be a way to justify investments and to improve projects, it should also force bad projects to become better. For me that is one of the main arguments to support this.

Peter