Lean QuIP: Testing qualitative methods for impact attribution

Since 2022, 60dB has been working in partnership with the Busara Center for Behavioral Economics and the Bill & Melinda Gates Foundation to improve methodologies for measuring the access to and impact of digital solutions aimed at improving smallholder farmer welfare. As a part of this broad initiative, we are testing the lean QuIP methodology to measure the impact of digital tools on farmer wellbeing.

What is a QuIP?

The Qualitative Impact Protocol (QuIP), originally developed at Bath University, is an approach used to help establish impact by tracing causal relationships between activities and outcomes. It is a qualitative approach that focuses on what individuals perceive to have changed and their explanations for why and how. It avoids direct questions about specific inputs, allowing participants to freely express what they consider important in relation to specific outcomes, thereby helping to mitigate confirmation bias. QuIP uses a small sample, typically 20-30 respondents, to confirm or challenge theories about causal links in an intervention (theory of change).

“Lean QuIPs” and Digital Farmer Services

In 2022-2023, 60 Decibels conducted “Lean QuIPs” with two digital agriculture advisory services—Ignitia and DigiCow—to assess the utility of QuIP when conducted remotely with farmer users of digital solutions. These pilot studies evaluated the feasibility and practicality of implementing QuIP and how well it could capture meaningful impact data for DFS.

We asked farmers to talk about changes within a specific time period related to selected outcome domains. For example, “has anything changed in the way you farm since Easter?” We then probed farmers to share (1) the main driver of that change, and (2) to whom/what they attribute that change.

What did the companies learn?

Lean QuIPs provided insights that DigiCow could use right away. Many farmers independently attributed positive changes in their farming methods and productivity to the DigiCow app. DigiCow could easily see which new practices were most salient to farmers, because the farmers mentioned them completely organically, without a list of multiple choice options. Qualitative questioning allowed us to capture unexpected insights — for example, sometimes there were external factors like feed prices and climate shocks that prevented farmers from realizing the impact of DigiCow’s training. Similarly, Ignitia learned that farmers had improved their planting practices, attributing that change to Ignitia’s weather-informed advisory services. They learned which practices were most salient to farmers (more effective fertilizer and chemical application and timely planting).

What did we learn about Lean QuIPs?

• Lean QuIPs are most helpful in validating causal pathways or strengthening/challenging a theory of change. The goal is not to quantify the impact of an intervention, but rather understand how it works and validate the causal link. Therefore, having a clear understanding of the expected causal pathway proves highly beneficial when designing a QuIP, as it allows for more easily anticipated prompts and provides a structured approach.
• Lean QuIPs are useful for measuring digital information services with short causal pathways. QuIPs are effective when there is a short gap between the intervention and the outcome, making it easier for respondents to establish a direct link. When there are multiple middle linkages between the two, drawing conclusions can be challenging. QuIPs are especially well-suited for digital information services, as they can isolate the specific source of information that led to the observed outcome**.**
• The specificity of the behaviour change is helpful for a successful Lean QuIP. The intervention should be narrow and focused, targeting specific behaviours or outcomes for easier recall and analysis. Framing questions according to the intervention's key element ensures a more effective evaluation process.
• Lean QuIPs can use partial blindfolding if you want to capture or validate customer experience. For one of our studies, interviewers knew the name of the company so that we could “unblind” the respondent after the open-ended QuIP and ask direct questions about customer experience. The staging of questions is important, asking open-ended questions first ensures that you gather less directed evidence on wider potential drivers of change before you narrow down the questions to the intervention in question. We closely monitored the data collection process and found that effective training and monitoring can mitigate bias in responses.

Download the full report for all our insights and lessons. We’re testing other lean methodologies for impact measurement and look forward to sharing those results with you! Sign up for our ag sector updates to make sure you don’t miss them.