Nearly 500 million smallholder farmers cultivate a third of the world’s food, but face significant challenges like remote locations, limited market access, climate risks, and scarce resources. Digital Farming Solutions (DFS) offer a way to bridge these gaps, providing farmers access to information, markets, and credit without in-person interactions. But how do we know if these tools are truly impactful?
Since 2022, 60dB, in partnership with the Busara Center for Behavioral Economics and the Bill & Melinda Gates Foundation, has been refining methods to measure the impact of DFS on smallholder farmer welfare. This blog summarizes our findings and offers guidance on how to select a method.
The most important step is defining the question you want to answer, as the methodology must fit the question at hand. Measuring the impact of a DFS first requires sustained adoption by farmers, so focusing on adoption may be a good place to start. Below, we’ve summarized some types of questions DFS stakeholders might have, and the appropriate methods for answering them.
Busara’s behavior change study examines the psychological, social, and economic factors that influence DFS adoption. Farmers decide whether and how to adopt new technologies based on factors like cost, availability, risk, value, and social norms. Behavioral science reveals how farmers think, act, and make decisions, using insights from psychology, sociology, and economics. Understanding these behaviors is essential for improving digital farming services.
A 60dB Lean Data study can capture this by conducting short, one-time, phone-based interviews, gathering firsthand insights into farmers’ satisfaction, perceived benefits, and likelihood of continued DFS use.
The Qualitative Impact Protocol (QuIP) is a method designed to capture farmers' explanations of changes on their farm without directly mentioning the DFS intervention. This approach helps reduce confirmation bias and identifies causal links based on the farmers' own perspectives.
A cross-sectional study can help determine this by capturing data from DFS users and non-users at a single point in time. It highlights differences in both subjective and objective outcomes between these groups. Although it doesn’t establish causality, this approach provides a robust look at potential impacts of DFS on farming practices across users and non-users.
Lean Evaluations, using a difference-in-differences (DiD) approach and short phone interviews, compare outcomes in a treatment and a comparison group over time, allowing for causal inference with a lower resource requirement than RCTs.
Read on for a closer look at each method and learn how to select the most relevant approach for your DFS impact study!
Behaviour Change Study
The Busara Behaviour Change Study enhances digital farming services by following a three-phase methodology: diagnosis, co-design, and testing. In the diagnostic phase, we identified barriers to digital tool adoption by gathering insights into farmers' experiences. The co-design phase involved developing targeted solutions with farmers and stakeholders, creating practical, locally relevant interventions. Finally, the testing phase evaluated these interventions through controlled trials, ensuring that solutions were user-centered and effective in addressing adoption challenges.
60dB Lean Data Study
The 60 Decibels’ Lean Data methodology is an approach to impact measurement built on short, phone-based interviews, it captures self-reported insights directly from those who matter most: the end beneficiaries.
Lean Data surveys are brief and focused, typically engaging around 275 respondents per project. Through a targeted series of questions, beneficiaries share how much a service has improved key areas of their lives, whether it fills a gap in the market, and if it reaches underserved communities.
This approach relies on a farmer’s own attribution of impacts to the DFS.
Lean QuIP
The Qualitative Impact Protocol (QuIP) , developed at Bath University, is an innovative approach for understanding impact by capturing people’s own explanations of what has changed in their lives and why. By allowing respondents to speak freely, QuIPs reduce confirmation bias and highlight the factors they find most meaningful. The methodology relies on a small sample, typically 20-30 respondents, to confirm or challenge theories about causal links in an intervention (theory of change).
Broadly, QuIP interviews use open-ended questions to explore change without mentioning the specific intervention, a process called ‘blindfolding.’ Respondents identify the main driver of change and who or what caused it, with findings categorised as ‘direct’ or ‘indirect’ attribution depending on whether they name the intervention.
In 2022-2023, 60 Decibels conducted “Lean QuIPs” with two digital agriculture advisory services—Ignitia and DigiCow—to assess the utility of QuIP when conducted remotely with farmer users of digital solutions. These pilot studies evaluated the feasibility and practicality of implementing QuIP and how well it could.
Cross-sectional Comparison Study
A cross-sectional study compares groups at a single point in time to identify differences in outcomes. In 2022-2023, we worked with TomorrowNow and KALRO to evaluate the impact of digital farming services on Kenyan maize farmers, using both self-reported (subjective) and measurable (objective) data for a well-rounded perspective.
In this study, the treatment group received advisory services enhanced by TomorrowNow’s hyperlocal information, while the comparison group received standard advisory alone. We explored differences in satisfaction, likelihood of continued use, and perceived benefits like improved crop quality and market access, along with objective outcomes such as productivity and income.
Cross-sectional studies are effective for capturing a snapshot of differences between groups; however, they do not establish causation and may be influenced by short-term factors or self-reporting biases. Compared to other methods, this approach provides a more robust measurement of DFS impact, allowing for a quantitative assessment of how DFS may influence specific outcomes for farmers.
Lean Evaluation
A lean evaluation is an approach to causal inference that avoids the high costs and time demands of traditional Randomized Control Trials (RCTs) or other quasi-experimental designs. This method uses a Difference-in-Differences (DiD) approach, which compares changes in outcomes over time between a treatment group (using the DFS) and a comparison group (not using the DFS). By analysing these “before and after” differences across groups, lean evaluations help attribute observed changes to the intervention more confidently than observational studies.
We piloted this lean evaluation approach with two DFS companies in Sub-Saharan Africa, gaining valuable insights into how it performs in real-world DFS settings. Our findings reveal important lessons about applying the method effectively with DFS, which you can read more about here.
Each method offers different advantages, and selecting the right one depends on the specific context, questions, and goals of the study. For some, a quick, cost-effective Lean Data study may be best, while others may benefit from the nuanced causality insights offered by Lean QuIP or Lean Evaluation. The key is finding the right balance between operational constraints and the level of rigor needed to uncover meaningful insights. By selecting the right approach, DFS providers and stakeholders can better understand how their services are truly impacting smallholder farmers and driving value.
© 2024 All Rights Reserved
