Our Observation: Monitoring and Evaluation tools and approaches focus too heavily on programmatic impact—leaving managers without effective means whereby to track the ongoing progress of their interventions. Facilitators require tools that enable them to monitor progress in strengthening of trust and coordination among market actors, as well as ensuring the ongoing relevance of their facilitation given evolving market trends.
Do you agree with these observations?—If so, please provide anecdotes of how you:
(1) determine ongoing relevance of your programs in rapidly evolving market systems, as well as any mid-course corrections based on your M&E, including exit.
(2) Use M&E as a programmatic tool—including pushing out your monitoring to program actors as a resource for them as well.
—If you do not have examples, what types of information would enable you to take on the above activities, if they could be easily and timely tracked?
—What else would you like to use your M&E for in terms of management?—outside of measuring impact.
10 Comments
M&E as a Means of Identifying When to Exit
Eric Derks of AFE kicked off this discussion yesterday when in discussing exit strategies, he noted the importance of a good M&E framework and performance indicators to identifying when a facilitator should exit.
He noted that indicators could look at factors such as:
-growth in commerical relations facilitated by the project
-the “commercial strength” of market actors
(Tracy’s addition to this last indicator—presumably this would measure competitveness and actors ability to sustaina competitiveness over time, which could look at a host of indicators, depending on the type of firms such as governance and access to finance for cooperatives, or diversity of linkages to final markets and access to input services upon which to improve efficiency for small enterprises)
Lucho and I look forward to other examples of how you use M&E in the management of your programs and the types of indicators you use.
Regards,
Tracy
Some thoughts
It’s quite frustrating when you don’t see eye to eye with your donor on the M&E approach and emphasis. There are still many donors who are overly fond of quantitative measurement which are quite inappropriate when it comes to the sort of work we do i.e fostering greater communication and trust between market actors. Donors need to be made aware of newer ,more qualitative techniques of understanding the transformations taking place in the market system. One good example of such a tool is the relationship matrix which was tested by Practical Action Bangladesh. If Abdur Rob is online he would be able to explain this to us better. It is interesting to know what difficulties he faced in implementing this technique.
I am quite keen on participatory M&E approaches (e.g stories of change). The extent to which these would provide useful and open feedback is dependent upon the trust relations between the facilitating organization and the community. Internalizing these new approaches can also be very challenge as we experienced in Sri Lanka. It takes time for the staff to be convinced and adopt these techniques which are different from the traditional quantitative approaches they are used to.
The periodic team meeting can be a very useful forum for monitoring, provided they are used constructively. A truly open learning culture needs to be in place if it is to work.
Communicating to the communities on what action will be taken once they have given their feedback is an important step which is often not given much importance. This often undermines the whole participatory M&E process.
Quantitative and qualitative
Hi
Sonali makes a good point about qualitative tools. She refers to the Relationship Matrix (as it is the Eidd Holiday I think Rob is off-line now). This is a qualitative tool that practitioners can use to systematically measure changes in relationships between key sets of market actors around key processes they have identified as important. Practical Action first heard about this at the 2005 ILO BDS Seminar, presented by Marian of SDCSouth Asia. We adapted it and Rob tried it out in a dairy project in Bangladesh. The team used it as a way to get market actors to talk about their current situation, and where they want to be in the future, and then periodically they came back to it to look at what progress they were making. So it was a very participatory M&E and actually built further communication and trust between the actors. I am sure he could add more about the specific challenges but I know the teams have gone on to use it in other market systems like groundnut. We are trying to learn more about this way of doing things so please contact us if you want further info on the tool.
Eric from AFE also makes good points yesterday about quantitative indicators forming part of the designing for exit process. Suffice to say that I liked his list because it was short!! We have been developing an Indicators Guideline which is getting scarily long – that’s not to say that projects use all of them, but there is a danger that the sheer number can be overwhelming. Again we would be happy to share it with others and would LOVE to get feedback!
Thanks
Alison
Alison Griffith
Head of International Markets and Livelihoods Programme
Practical Action
Direct +44 (0)1926 634467
Skype ali.griff
Relation between Exit Strategies and M&E
Hi all:
I notice that there is a very strong link between Exit Strategies (one of the hottest topics of the week) and M&E.
Some of you talked about the need to have indicators to tell us when is time to leave; and the need to periodically monitor how other stakeholders are reacting to the facilitation process (e.g. are they taking over? are they hesitant?)
It is not just about measuring the impacts of the market transformation itself (how many farmers, how much profit, how much product, how good the product is) but also the process of transformation in itself.
In this sense M&E of the facilitation process is fundamental for a good exit.
It would be great if you all fleshed out a bit more that dimension of M&E. Examples, challenges, guidelines, practices, methods, tools… ?
The day is finishing for me in the UK so, I will disconnect after I send the summary of day 4. Please feel free to keep on sharing your experiences. The conference formally ends today but the interaction continues! Tracy and I will be sharing ideas very soon on how to take the Market Facilitation Initiative forward after this amazing event. Your contributions are very important to move in the right direction.
All the best,
Lucho
M&E and facilitation
The reason why we do market facilitation is because we have a strong theoretical and empircial belief that markets can be transformed to work for the poor. Unless we have good and solid M&E systems in place, we will not be able to speak authoritatively about the change that our facilitation is making.
My own experience is that M&E ia an integral part of the facilitation process. Again, it is difficult to pinpoint M&E tools but participants have highlighted some of the common practices that we have found effective in our work. These include Relationship Matrices and Stories of Change. Relationship matrices allow us to trace the changes in the relationships between market actors and asses if real empowerment is taking place for mutual benefit. Stories of exchange allow market actors to tell us the changes taking place in ther businesses and livelihoods as a result of market facilitation.
A conclusion we can make is that well developed and implemented M&E tools allow facilitators and market actors to continuously assess if the market is working for them and to make adjustment where limited or no progress is being made.
Relationship Matrix (RM)
Thanks Sonali for your comments and interest for additional information on RM
Thanks Alison for your valuable inputs on our experiences.
In my message posted on day 3, I mentioned about few positive things happened because of using relationship matrix in our dairy work in Dinajpur- northern part of Bangladesh. I am attaching a power point presentation that we made in a seminar on our experience of using the relationship matrix. I hope this will give our colleagues some understanding on how we applied it on our dairy project. Apart from the points included in the presentation, I would like to mention few more positive results that the relationship matrix group yielded. I will also add up some new challenges in response to Sonali’s request for information.
The group observed that relationship matrix facilitates effective solutions to technical problems; allows quick transfer of resources and speeds up response to demand of services. It enhances personal communication skills. However, the relationship matrix group (in the dairy project it is known as market working group) faced a number of challenges due to lack of vision, unclear role and responsibilities and ambiguous authority. The group also faced initial resistance to changes as some members feared the loss of status, authority, income and control over business. The main issue is still unanswered i.e., whether the working group can function as matrix guardian without external facilitation. I think the group will survive (if not in the present size and structure) as it increases flow of information through the creation of horizontal communication channels. Besides, the relationship matrix group has created at least two following offshoots:
1. Para-vet Association in Birampur Upazila (sub-district) and
2. Processors Association in Birampur and Dinajpur
Both the associations are showing positive sign of making progress in terms of supporting members and contributing to market development system especially in the area of capacity of individual para-vets involved in providing service to farmers and quality of milk based products.
Abdur Rob
Practical Action Bangladesh
M&E
Thanks Rob for sharing that presentation. Very insightful. Any additional information you may have on the matrix that you can share is appreciated! It looks like a very valuable tool.
I second Sonali’s comment on ensuring a solid mix between qualitative and quantitative indicators. As part of MEDA’s Behind the Veil project in Pakistan, we conducted bi-monthly questionnaires with a set number of project clients to track changes in the household decision making as well as other standard of living dimensions. These figures were extremely enlightening and helped highlight key results beyond the income and sales increases. In one of our projects in Zambia, we are using case studies to track the impact on target clients over the life of the project. I would be interested in hearing other practitioners challenges in using these quantative tools (such as most significant change methodology) etc.
I would definitely agree on the relation between indicators and exit strategies. When thinking about exit strategies, it is important to have indicators that not only signal the shift from intervention to exit strategy but also indicators to signal that your exit strategy have been successful!
Thanks
Alex
Market failure, failing markets, and M+E
Hi everybody,
I honestly regret that I’ve only now been able to catch up on the whole discussion, since it was obviously very stimulating. Regarding M+E, my main experience has been that a purely formal approach, and in particular an approach that uses mostly quantitative indicators and operates at arm’s length, is unlikely to work well. When we facilitate a market, there are two, and I think only two, possible constellations:
a) A given market so far has not worked due to market failure (information asymmetry, public goods, indivisibility, etc.)
b) A given market does not work because there is somebody who benefits from it’s not working (“failing market”)
In situation (a), M+E shouldn’t be much of an issue — either one finds a way to overcome the market failure, and that shouldn’t take too long, or one finds that it cannot be overcome so that hierarchy or network is the solution, in which case we are facing either the usual principal-agent problems (like: how do I assure that the government lab that promised to do fresh fruit testing actually does that, and to the conditions agreed, without demanding bribes?) or the usual problems of collective action.
In my experience, situation (b) is more complicated. There’s again two possible constellations. (b1) is a constellation where we’ve involved that powerful player early on and found a win-win solution so that s/he is not going to fight our intervention. Problem solved. (b2) is what I’ve observed more often, namely a situation where we initially didn’t properly understand why a given market didn’t work, and came up with a proposed intervention that was unlikely to work since the problem wasn’t understood properly. Only through the failure of our intervention we began to understand what we were up against, and began to revise the design of our intervention. In this kind of situation, M+E is not so much a separate activity but at the heart of our activities, which involve a rapid (and sometimes painful) learning process. I would tend to argue that the most appropriate way of addressing this kind of situation is not to introduce more sophisticated M+E tools, but rather to “fail faster”, firstly by not wasting too much time on research that is done by some external researcher or consultant (who is likely to miss the “failing market” reality since neither the perpetrator nor the victims like to talk about it), and secondly by moving quickly into rapid appraisal / action / reflection / adaptation iterations.
Monitoring and Evaluation
Thanks Alexandra for your kind words. Sure, I will share additional information asap.
Hi every body,
In my opinion, monitoring and evaluation is certainly intended for continuous improvement and decision making. A systematic process that includes information collection, analysis, sharing it for stakeholders’ feedback and decision making is the key to have a strong mechanism for monitoring and evaluation. The question is how far shall we go for monitoring i.e. up to what level of value chain actors and service providers we need to monitor and evaluate? Practical Action Bangladesh considers staff, beneficiaries, and target people for whom or target institutions for which the market development project is committed, should be the part of this monitoring and evaluation system. It requires proper planning of the monitoring and evaluation process which has to be participatory, qualitative and quantitative. By reviewing the monitoring and evaluation plans we can quickly identify the exit strategy (if there is any) of an intervention.
I am attaching the monitoring and evaluation plan of our Making Markets Work for Small-holder Farmer and Rural Producers project as an example of the monitoring planning tool we use.
Abdur Rob
Practical Action Bangladesh
Peer Learning Groups
Sorry for almost completely missing this very interesting boat. I have no idea whether this comment will post or not. My excuse is being ill when I could have contributed. I am writing to represent an alliance called Forest Connect – that is a bunch of agencies dedicated to facilitating better outcomes for small forest enterprise in Burkina Faso, China, Ethiopia, Ghana, Guatemala, Guyana, India, Laos, Mali, Mozambique and Nepal. The central idea is to facilitate better linkages: to each other, to BDS and FS service providers, to markets (of course) and to decision makers. There is not a lot of money involved, so subsidies don’t really come into our work much. We do try to improve information about SMEs, service providers, policy bottlenecks. We also try to expose unhelpful alliances that rig markets against the poor. PArtners work directly with small forest enterprise groups and try to help them assess how to move forward. Following an international meeting early this year – we are trying to develop a toolkit on facilitation of support for small forest enterprises. The areas where practitioners felt further guidance was needed included the following:
*Communication platforms and strategies to reach remote enterprises? *Organisation strengthening – governance and structural tips for groups? *Market literacy and information – how to increase exposure? *Product development steps? *Business and financial planning guidance (lots out there on this) *Benchmarking service providers – how to do this? *How to build in sustainability from the start? *How to do and present policy analysis that results in change?
Any help on these areas would be much appreciated.
Our collective M&E involves international meetings to share progress and best practice. We have four main areas of M&E but only do three…
*Project perspective – any cash needs to be accounted for against clear objectives *Client perspective – before and after analysis of enterprise groups to see whether we are meeting their needs *Uptake pathways / stories – the degree to which facilitation leads to others pursuing training, group advocacy etc versus what happens *Evidence – detailed analysis of the sub-sectors involved which is far beyond what we have finance for.
As an alliance our main indicators of success are the degree to which members of the alliance understand the concepts and suite of potential tools to help them in the practical work which they all have independent mandates to pursue.
We would love to see your state of practice review and any other guidance you have.
Check us out at http://forestconnect.ning.com
Duncan