Dear Eric,

Let me make an attempt to answer your questions:

Integrating impact assessment to an existing programme. As long as the goal of the project is related to additional income of enterprises in the passion fruit value chain this should not be a problem. Your first problem is knowing how are you going to do this! Once you have a good idea of how you and your partners will achieve this goal then impact assessment is all about measuring the indicators that take you along this path. You may be worried about having baselines to compare the situation before and after. Baselines in my experience do not have to be large and frightening but they do need to be targeted to giving you the data that will answer the right questions. If you want to compare incomes before and after you intervention then it would be good to do this sooner than later, but only after you are clear on your attribution path. This comes from the clear results chain that you develop for each of the intervention that you think will actually create a change on income for your target group. This brings me to your second question, tools.

Tools for M+E. For me the best tool is the Results Chain. This is sometimes called the Impact Logic or Causal Model. Whatever you call it the tool helps you to think through how you will stimulate some change in your target group and then how this change will make this world a better place! The tool however is only as good as the user, so ultimately you have to come up with the contents of what is in the Results Chain. You can get more information on this from the GTZ website if you look for “Results Based Monitoring”. Once you get this logic right everything else flows from this, including attribution.

Involvement of farmers or the target group. Actually they do not have to be involved in your system or how it functions. Their concern is that what you and your partners are doing is making a difference to their incomes. This is where you and your target group have a common objective. The measurement of this income change is good for you because you can learn from this and it is good for them for obvious reasons. This information is important to other farmers who have not been affected by your project because you can use this information to convince them that what you are doing is likely to have the same impact on their incomes too.

With regard to attribution. If you have a good results chain or causal model you should be able to know what the main influences are affecting the issue that you are trying to change. If you prioritise these in terms of their likely attribution, you can use this information to a) select the most important issues or b) select the areas on which you are able to concentrate or c) involve some other partners who are better able to address this issue. This analysis help you to determine how significant your intervention is to the change. This same process will be repeated at each important stage of your results chain and though this process you will be able to see how significant your attribution is to changes at each level.

With best regards,

Jim Tomecko
Advisor on Tri-Lateral Cooperation

GTZ Office Thailand
193/63 16th Floor Lake Rajada Office Complex
New Ratchadaphisek Road
Klongtoey, Bangkok, 10110
Thailand
Tel: ++66 2 661 9273
Fax: ++ 66 2 661 9281/2
Mobile ++66 81 829 5210

——-Original Message——-
From: communities@seepnetwork.org [mailto:communities@seepnetwork.org]
Sent: Thursday, September 03, 2009 8:31 PM
To: Tomecko James GTZ TH
Subject: Comment for Discussion: What makes valid and useful results assessment?

An earlier comment by Shivani Manaktala, reclassified to this discussion:

Hello All,

I am Shivani Manaktala working with Ashoka:Innovators for the Public’s Hybrid Value Chain Initiative – which for us at Ashoka implies building business-social collaborations as a way of reaching low income markets. Its been a while since I worked on typical VC /enterprise development type of projects but am happy to share partially my past experience and much of my current one which too is highly relevant I think.

For me the important thing in measuring effectiveness is to look at impact at 2 levels:
1. System level impact
2. Client level impact

More often than not we end up looking largely at client level (both quantitative and qualitative)
I would be very keen to understand how people look at the former i.e. the system level impact. For me this would imply looking at questions around:

1. are there new ways of doing business that are emerging
2. are business and social sector organisations collaborating within value chains
3. are new forms of organisation structures emerging
4. are commercial returns being used to leverage donors funds for enhanced impact

Looking forward to thoughts around this.

Shivani
Director India
Full Economic Citizenship

Hi Ally,
The case study is very interesting and clear and helps understanding the concept of impact given the clear links with a planned intervention and its impact. However since this measures the impact of a single intervention or primarily at the enterprise level( which is then multiplied by the number) , I am interested to know

1. How impact can be calculated at the systemic level of the value chain

2. What is the big picture framework that one needs to keep in mind from an M and E point of view while designing complex projects as most value chain development projects would have a maze of interventions where one intervention sucess may effect another and so on and some of the impact may be indirect as well. Basically what are the main steps in designing impact assement for a complex program?

3. What are the prerequisites for a good results Assement/ M and E design?

Dear Patrick,

Thanks for your very interesting message. You mention that your regularly include in your assessments “producers, intermediaries , collectors , traders etc operating along the value chain. They are enterprises of different characteristics and all these can see a change in different terms -Income and wages , value of produce, increased sales etc .” How do you gathering this information? What tools to you use? How often?

You also discuss various new relationships and business models that emerge from your work in value chains. How do you know that these new ways of doing things are emerging? And how do you know that they are likely to keep expanding throughout the value chain system? Specifically, what information do you gather from whom? And how do you interpret that information?

Thanks!

Aly

Jane Madumadu-Tanui
Export Promotion Council – Regional Coordinator Western, Kenya

How can we gauge we are on the right truck of enabling the targeted communities or value chain players to a sustainable efficient and reliable way of life as we aim to alleviate poverty and create jobs.

How can we ensure harmonisation of the various clusters within a value chain to ensure sustainability of the business models

Nyabinda Naman O.
Value chain Sluggish connection
High participants, I have been reading through the contributions of different participants and got carried with wonderful ideas that come from difference experiences. I would like to ask one question about the value chain actors/ players/ SE’s;
How best can value chain facilitators perfect the flow of information among the actors/players/SE’s to improve market development???
Naman Nyabinda
AMPATH/FPI-Kenya

Hi,

I wanted to respond to Gabriela Prudencio’s questions on the role of qualitative methods for understanding the results of VC projects.

There is general consensus that combining quantitative and qualitative approaches in evaluations improves the depth of understanding and the strength of the findings.

What is the difference between quantitative and qualitative approaches? Quantitative approaches generally involve collecting standardized data through a survey questionnaire and applying statistical analysis to the numerical findings. It data generally comes from a sampling frame that is representative of a broader population so the findings can be generalized.

Qualitative approaches typically use narrative analysis, and focus on understanding why things have changed, or not. Data collection methods fall into two groups (1) Ethnographic methods include semi-structured interviews with clients or key informants, review of documents or diaries, oral histories, discussions, and observations; (2) Participatory methods include focus group discussion, mapping exercises, ranking exercises, and other participatory rapid assessment methods. Qualitative approaches usually have smaller sample sizes that are not representative of larger populations, so the findings cannot be generalized. I think it is important to note that both types of qualitative methods can be used to study client level changes (enterprises and households) and to understand systemic level changes (the ‘dynamics’ of a value chain).

A useful qualitative method for monitoring VC programs is ‘process documentation’ which is a method that charts the evolution of an intervention at regular intervals. It documents the starting point in terms of actual activities, what works, what does not, what changes, and why. It can identify constraints that arise from qualitative factors and be part of learning process that helps value chain actors respond to these constraints. This relates to the ‘pathways’ approach that Mike Field will be sharing.

Usefulness? I don’t think we have evolved to the point of standardized questions for qualitative studies of VC projects (someone may want to correct me here). But qualitative research can help to understand why change does or does not take place in VCs at both the client and systemic levels. Qualitative studies can explore ‘impact processes’, for example, incentives and risks for value chain actors associated with adopting new technologies, selling through new market channels, or using new inputs or services or other ‘behavior changes’. It can help to understand horizontal and vertical relationships within value chains – how information flows, whether there are relationships of trust, and what factors may influences these dynamics (gender, socio-cultural factors, geography, etc). It can also help to understand the distribution of benefits across actors in the value chain – and how this affects its overall functioning. All of these factors affect competitiveness in terms of the ability of the value chain, as a whole, to grow and respond to changes in market demand. They are important to look at because they affect the sustainability of the change. These may be the types of general questions that can be applied across projects or communities. A similar set of questions have been used in the AMAP value chain impact assessments in Guatemala, Kenya, Tanzania, and Zambia.

Validity? In terms of ensuring the validity and credibility of qualitative studies, in general, it is important to state the specific questions being addressed, and document the methods used, who collected the data, who was interviewed, where and when the data was collected and how the findings were recorded and analyzed. Describing any problems encountered in collecting or analyzing the data and how they were overcome also helps to establish the credibility of the findings.

Regarding Gabriela’s question about comparing results across programs or communities, it is very difficult methodologically because of differences in contextual factors and interventions, and the need for strict and common protocols. Experience to date suggests that this would be very challenging (for both quantitative and qualitative studies) and not very practical or cost effective. That said there is no reason not to address the same questions in more than one place and to look for common patterns.

Jennefer Sebstad

Hi Aly and Jim,

The Longan case has long been a favorite in such discussions and there are reasons why. It follows a very good program approach- its nuanced in a sector wide problem- low productivity, builds on an opportunity- a local university has developed a service that can improve productivity, facilitates a favorable market demand- the partnership between GTZ and the university, and finally builds on developing a system- creating service providers once the demand for the service appeared favorable. M&E becomes easy when the program design is simple and methodical. But unfortunately such cases are not common in practice. In the Longan case productivity increases due to the service provision that develops pruning skill- in contrast increasing productivity of the vegetable farmers in Bangladesh requires us to intervene in creating access to quality seeds, creating demand for and access to soil testing services, improving farmers knowledge on appropriate pesticide and insecticide use and many. This arises the attribution problem- how much should we attribute to which intervention? Does it than make more sense to do a combine M&E instead of looking at the impacts of each interventions? Going beyond, seasonal patterns, for instance late rainfall or early rainfall can have significant impact on both production and productivity within the project tenure. This means the impact can even be negative though the interventions are working quite appropriately. Some projects have tried to address this problem by accounting for net gain in loss (the interventions have worked as safety nets thereby reducing the amount of loss) which makes good sense to me. Interventions also have varied results based on cultural and demographics contexts. For instance, farmers in some areas show higher tendency to adapt new skills while farmers in other areas fall short though the intervention followed the exact routine. These are issues that external auditors would find difficult to comprehend. But how to account for such issues in measuring results? I usually suggest my staffs to continuously assess impacts at field level and write simple cases based on their findings. These individual case studies work wonder to show a time series in the situation and we can also find externalities that are having positive or negative influences. From these we can determine proxies that can be accounted for while making a comprehensive study on the impact- as you have done in the Longan case by deflating the impact by 25%.
Now coming to measuring or documenting systems change- I would like to draw your attention to the fact that markets are dynamic and an intervention promoted in year 1 is more likely to become obsolete in year 3 because of newer challenges or simply market competition that is driving changes. I have usually seen projects trying hard to find out the same systems change in year 3 as was promoted in year 1 and commenting that things did not work out simply because the same logical framework can’t be traced. While actually the fact is that people built on the system that was promoted and changed it to fit to the changing market conditions. I suggest that projects try to assess the current market conditions to trace back to preconditions that resulted in the current improvements. If the intervention was appropriate it should by now start to regenerate and this is what sustainability should mean to us. But it has been argued that this might lead to attribution of anything and everything and projects should try to document only the immediate impact. While i do agree that we should not go onto researching for quantitative data to prove this I do believe that we should still have qualitative surveys that hint onto such development. Would love to hear from you.

Regards,

Rubaiyath

Dear Rubaiyath,

Thanks for the interesting and detailed message. It’s very helpful to discuss examples when trying to puzzle these issues out.

As you point out, sometimes interventions depend on and build on each other to achieve a particular change at the level of enterprises/farms, such as an increase in yields or a decrease in costs. There is no requirement in the DCED Standard to isolate the effects of individual interventions. The requirement is to use reasonable methods to estimate the extent to which measured changes can be attributed to program activities. Some programs choose to focus measurement on individual interventions. Others focus measurement on all interventions in a specific sector. Others, like GTZ, cluster activities into “interventions” that aim to achieve a particular change at the enterprise level, such as reaching new markets, increasing yields or decreasing costs. This not only helps with assessing interventions but also helps with the clarity of design.

As you point out, dealing with attribution is very important in agricultural initiatives because farm performance can vary greatly per year due to the weather (or other factors), so before and after measurements only can give a false picture of program performance. Programs are handling attribution in different ways. Some are using participant and control groups, usually in the early stages of interventions or work in a sector so that it is possible to have separate groups without the control group being affected by interventions. Others compare, areas where interventions are being implemented against national trends or other areas – but this doesn’t always handle the weather issue. Others are taking an approach which, I think, is similar to yours. They are conducting careful case studies and qualitative information gathering to assess the extent to which program activities actually led, through a chain of changes, to observed changes among enterprises or target households. This can be done starting with program activities and going “forward” through the causal chain or starting from observed changes and going “backward” through the causal chain (or both). Experience in evaluation indicates that these studies, whether formal or informal, are most effective when they are based on results chains that hypothesize the various changes leading from program activities to impacts in enterprises or households. However, the studies also stay open to discovering links that were not hypothesized at the start. It is harder to estimate attribution through these qualitative methods, but there is useful experimentation going on, for example asking participants to rank the reasons for changes and uses the ranking to estimate attribution.

At the level of sustainable changes to sectors which keep them competitive and pro-poor: I agree with you that this is challenging. The “industry pathways” used by the PROFIT program which we will be looking at on 1-2 September are a very useful tool for hypothesizing and then assessing the extent to which sectors are becoming more dynamic and pro-poor, rather than only looking at the immediate changes produced by interventions in the short run. Here is where indicators of sustainability become important, including indicators such as the development of “industry drivers” which are associations, lead firms, groups of enterprises or others that, formally or informally, lead sectors, stimulating innovation and on-going upgrading.

However, while looking at this qualitatively is very helpful for program improvement, there is still a need to generate reasonable and credible estimates of attributable impact to satisfy the needs of donors, parliaments and others for accountability. There are many that say that if you can’t measure impact, then it’s not there. Measuring relatively short term impacts can also point towards larger system change and can be quite manageable within programs. For example, part of sector growth happens as farmers or enterprises invest in expanding. It is quite possible to assess the extent of this expansion by interviewing a random sample of enterprises/farmers who will explain that they have increased their area of maize cultivation or added another fish pond. Sector growth is also occurs with new entrants. This is more difficult to assess quantitatively but it is possible to find clear pointers. For example, you might visit a village where there was very little maize grown before program activities in the maize sector and then visit after program activities have had time to generate change to see how many farmers are now cultivating maize. This would be followed by interviews with farmers to see the extent to which the reasons they started cultivating maize can be linked to changes in the sector that assessment has shown result from program activities.

Ultimately, to make these types of measurements or assessments credible, there must be some outside oversight. It used to be that only externally conducted evaluations were considered credible. The DCED Standard proposes that impact assessment can be conducted internally but that external auditors will assess the quality of the assessment to avoid, as you say, that programs take credit for anything and everything. I think that external auditors can grasp quite complex programs if they are clearly designed and well documented provided that they have some time to interact with the program managers and staff. These internal assessments can be complimented by periodic or post-program external impact evaluations if necessary.

Best,

Aly

GCSE Coursework | Dissertations | Essays

Research Papers | Buy Thesis