Lean Evaluations

To measure the impact of DFS at the provider-level, we tested two methodologies: Lean evaluations and Lean Qualitative Impact Protocol (QuIP).

Lean evaluations: For lean evaluations we utilised two methodologies:

• A difference-in-difference (DiD) methodology, which compares the difference in outcomes between two groups before and after using a digital tool to determine the causal effect of that tool.
• A cross-sectional study comparing users of a digital service to non-users, employing regression analysis to control for observable factors and estimate the difference in practices and outcomes between users and non-users.
  

Lean QuIP: The Qualitative Impact Protocol (QuIP) is an impact evaluation approach established by the Centre for Development Studies at University of Bath and licensed by Bath Social & Development Research Ltd. QuIPs gather evidence of an organization's impact through narrative causal statements, meaning they do not require a comparison group to establish causality.

Instead, respondents are asked to talk about a change over a specific period and to share both (1) the main driver of that change, and (2) to whom/what they attribute that change. QuIPs have small sample sizes, typically 20-30 respondents. This is because the QuIP does not aim to be representative, but instead seeks evidence to strengthen or challenge an intervention’s theory of change i.e. understanding how an intervention worked. This means that a bigger sample will not yield different results.

Interviews can be either single or double blinded. In a single-blind study, the farmer does not know the organisation being evaluated, but the enumerator does, while in a double-blind study, neither the farmer nor the enumerator know the organization that is being evaluated.

A summary of the lean evaluations and QuIPs undertaken is as follows: