Challenges

The full value of the project would only be achieved in the long term: For example, the interviews revealed that the genesis of the project went back to when the government was preparing its 10-year Soil Health and Fertility Improvement Strategy. In particular, the interviewees identified three main issues that needed to be addressed:

• absence of comprehensive datasets to develop evidence-based strategy

• data collected through public funds remained with researchers and were not necessarily shared

• data were collected in different formats, for different purposes, and quality was not consistent.

The inference is that, at least from the government perspective, the full value of the project will only be achieved once all three issues are resolved.

Value is multifaceted and means different things to different people: The interviews revealed that there was no agreed definition of value and therefore how it should be measured. What was clear, however, was that stakeholders interpreted this concept to be multifaceted and much wider than just financial and economic.

Much of the value is generated by intangible mechanisms that do not lend themselves to quantification: For example, interviewees believed that considerable value was being generated due to a change in culture. The new culture that the project was encouraging individuals and organizations to exchange data and consider a more formal process for doing so.

It is challenging to set the boundaries of where value is created appropriately: The clearest example of this was when stakeholders identified spill-overs beyond the scope of the original project, particularly in relation to the wider agriculture sector. The inference is that the indirect value could be considerably larger than the direct value of the project itself.

Institutional challenges: Political and institutional instability, delays caused by institutional arrangements, and a stakeholder engagement plan that was too narrow were all cited as potential challenges to maximizing value. For example, there was some evidence of dissatisfaction in planning delays as a result of debates about whether to include data for the whole of the agricultural sector. Eventually, it was agreed that policies were to be limited to the agronomy sector, allowing for lessons learned to be applied later to the broader agricultural sector.

Value and ethics: In order to maximize the value of this project, while ensuring the project adheres to the highest ethical standards, it is important that there is engagement with a diverse set of stakeholders. The challenge with stakeholder engagement is that it is time consuming and potentially costly. The evidence here, however, is that with more time and with wider engagement, particularly in relation to regional policymakers, practical and ethical concerns can be identified and addressed in order to maximize value.

Best practice

Legacy effects: An unexpected spill-over effect was that the informal coalition created had a life beyond the project and was able to contribute to the evidence base needed to address other contemporary challenges such as the Covid 19 pandemic. This illustrates the difficulty of ascertaining value; a solution to improve data governance in the agronomy sector of Ethiopia found a substantial secondary purpose supporting the wider economy.

Building a diverse coalition: The project was able to build a coalition of international, national, regional, and local stakeholders which safeguarded that the project adhered to the highest ethical standards. Experienced government and non-government stakeholders ensured that the correct skills, knowledge, and levels of influence were in place to deliver the project, while combining experienced international and local institutions ensured that learning from previous projects was also taken into consideration.

Implications for quantitative assessment

• Recognition that any quantitative assessment will be limited as only certain aspects of value can be quantitatively measured.

• The direct impact of value will need to be tightly defined.

• Indirect impacts should be included while recognizing they may be inaccurately estimated.

• The timing of the evaluation means that only the short-term value can be accurately measured.

• Attempts should be made to quantify the contribution of both tangible (e.g. training events/workshops) and intangible (e.g. trust) mechanisms that contribute to overall value. This observation is only listed here but is equally supported by the findings from other domains (safe people, safe settings, safe data).

Challenges

Targeted engagement: Interviewees commented on the need to identify and work with high ranking officials to champion the project but also highlighted the importance of identifying key “blockers.” These had the potential to become “powerful advocates for the policy itself” once their issues were addressed. Attendance of the “right people” from donor organizations and international project partners was important in building momentum and attracting a diverse set of local stakeholders.

National and regional governance: The issue of federalism was a recurring issue within the interviews. The regional structure in Ethiopia is very important both institutionally and culturally. Training would not only be needed at the national level, but there would also need to be training focused on upskilling researchers in managing local datasets; these can then be combined to piece together the regional and national pictures.

Best practice

Local ownership and empowerment: One significant development in the project was the formation of the “Coalition of the Willing” (CoW). Initially, this included 26 participants from 17 major research institutions in Ethiopia; over time, this grew to approaching 100 members. This was set up by researchers to inform the development of the data sharing policy, demonstrating the value of sharing by doing so. This had the direct effect of creating local ownership of the project, improving communication between stakeholders, ensuring buy-in for the project goals, and providing a forum for momentum about the project to be generated.

Top-down, bottom-up: The project also drove forward the “top-down” development of the Soil and Agronomy Data Sharing Policy in partnership with the Ethiopian government; tight feedback loops between these approaches benefited the project. It also helped that the CoW fed directly into the government taskforce. Indeed, several individuals were members of both. This “bottom-up” and “top-down” approach is mutually reinforcing, in a dual approach of incentive-and-sanction working in tandem. The incentive demonstrates what can be achieved when working together; this gives impetus for the sanction (national data sharing policy) to be developed, to encourage those not actively participating. The CoW in particular contributed to the “bottom-up” approach by driving forward localized data sharing initiatives. It worked because there was a clear incentive for individuals to take part, inasmuch as they would be given access to other members’ data.

Building trust: The necessity of building trust (especially between international organizations and local stakeholders), forming strong relationships, and empowering stakeholders was clearly key to developing and potentially implementing the policy. To address long-held cultural and social norms (e.g. in Ethiopia, historically information had been shared as a result of personal relationships rather than through any systematic process), it was important to empower stakeholders to develop their own solutions. Co-created solutions were enabled through an agile and flexible approach to managing the project. The project team also had a significant role in building capacity through the identification of skills gaps and then co-developing materials and workshops to fill such gaps.

Upskilling: Much of the focus on training was for the long-term sustainability of the project which would ensure that individuals were trained to use data effectively and securely. In order to do this, the project identified key individuals who could be trained up, who in turn would be able to share the knowledge within their institutions and with future generations.

Implications for quantitative assessment

• Potential to estimate the value regionally as oppose to just nationally: given the importance of the regional structure, additional benefit could be generated if regional estimates of the value were provided (but given the limited data this was judged to be beyond the scope of this study).

Challenges

Development of culture and systems: The principle of appropriate access was central to the development of the policy. Safe settings are often considered a technical or process matter. However, the evidence here pointed to the need for this to be accompanied by a cultural shift in relation to data access in Ethiopia. Prior to the commencement of the project, data sharing was often only advanced by the use of official letters to request that data be made available for a particular purpose. This process potentially created bottlenecks, particularly for those datasets where this level of formality may not be required. The evidence was that this project had shifted perceptions of at least some organizations.

Supporting framework conditions: The interviewees commented that investment in technology could automate data sharing efficiently and securely. However, some questioned the scope for automation and suggested the need for further investment in people and skills to build capacity to support and administer the technical solutions. Within the Ethiopian setting, the evidence suggested that investment in technology was not the problem, but the relevant framework conditions (e.g. legal, institutional, procedural) require adaptation.

Best practice

Technology: Within the project, interviewees suggested that the technology was either already available within Ethiopia or easy to acquire. As such, they stressed the need to look beyond the technical solutions. The suggestion was that in future, the role for CABI in Ethiopia would be to provide advice around infrastructure, connectivity, and the minimum requirements to enable data sharing in line with policy and procedures.

Knowledge transfer: One way to develop a safe model of delivery is to learn from other institutions who have similar data handling needs. In support of this aim, the discussion pointed to Members of CoW visiting the Bangladesh Soil Resource Development Institute to learn how it uses a safe setting to manage restricted-access data.

Implications for quantitative assessment

• Technology is a considerable cost and can be directly measured, but complementary human skills may be harder to assess.

• There are potentially significant indirect costs in relation to developing the supporting framework conditions; however, these have been judged to be beyond the scope of this analysis.

Challenges

Expertise in data governance: The interviewees highlighted the fact that much of the success of the project was driven through local empowerment; however, much of this expertise was in soil and data science, rather than data governance and access. As such, it was highlighted that it was important to bring in additional (international) expertise to help upskill capacity in relation to the safe management of data.

Supporting framework conditions: The supporting framework conditions were highlighted as requiring modification. For example, the lack of consistency in data standards was considered a barrier to sharing data. However, there was some progress on developing harmonized standards and protocols; an example was cited of core treatments being standardized across a number of individual experiments. Some highlighted a lack of trust in institutions, people, and systems as additional barriers; others identified legal and technical issues, while the federal institutional setup posed its own additional challenges. The interviewees recognized the need to shift toward a culture of more open data, and it was suggested that, as a result of the project and its wide engagement, there had been progress in this area.

Shared understanding: Lack of clarity over terminology was identified as a limitation, particularly around the different dimensions of data (e.g. accuracy, completeness, consistency, timeliness, validity, etc.). This prompted suggestions to develop a clear language and clear definitions. This would include key concepts such as what is an identifier and what variables have statistical value. Using internationally accepted frameworks, such as findable, accessible, interoperable, re-usable data, can help develop a shared understanding that can be applied in multiple contexts. Interviewees also pointed to the lack of clarity over those datasets that should be funded, collected, and maintained nationally, compared to those that are only required and collected locally. Being able to clearly identify the former would be helpful.

Best practice

Local ownership and empowerment: Local empowerment and ownership were highlighted as important factors. For example, a data sharing task force was formed with the primary objective of developing “informal” guidelines for data access and sharing. It was hoped that the task force could add value to the project by advising on issues around data quality, accessibility, and governance procedures, while the group was also seen as being an important influence on the national soil and agronomy data sharing policy.

Implications for quantitative assessment

• Reinforcing the need for clear definition of terms and agreement on what is/is not measurable.

Challenges

Human capital development: Experience in confidentiality protection was mixed; some reported work on implementation being well advanced, while others suggested that progress was less developed, as they were considered second-order issues. This is potentially an issue, as ensuring that outputs are safe and confidentiality maintained is hugely important to build trust in the process.

Given the sensitivities of working with different types of data, it is important for clear data standard controls to be set and enforced. Training in disclosure checking was not practiced, but there was support for training to maximize the value of statistical analysis. Much of this was “on the job” rather than formal training.

Best practice

Top-down, bottom-up: Once again, the top-down, bottom-up approach was recognized as being important for this domain. The interviews discussed a dual approach using both incentives and sanctions. Promoting the value of safe outputs can act as the incentive to complement the legal sanction to enforce compliance from those yet to be convinced by the incentive. Interviewees stressed the importance of being able to demonstrate the value of how data collected at a local level can be combined into important national datasets. For example, workshops were used to present the results of using large datasets.

Implications for quantitative assessment

• Research outputs (i.e. papers) are directly measurable.

• Where possible, it is important to measure activities in relation to developing human capital (e.g. training events/workshops).

Input costs: data gathered from financial reports

The costs presented in Table 4 provide a high-level overview of the inputs for: (1) the CoW and (2) activities undertaken to support the establishment of a national IT host for a centralized data repository (SIRM). Given the uncertainty with the coverage of the data, these input costs should be seen as indicative.

Table 4. High level breakdown of input costs, November 2018–July 2020

Source: GIZ.

Input costs of activities in relation to improving data governance and data access in the SSHI project were approximately US$640 000. To return a positive net benefit, total benefits would need to exceed this figure.

Output: direct internal value

Estimates of direct internal value (DIV) are calculated by multiplying the number of activities (A) undertaken (e.g. workshops) by the number of people from each sector (broken down by six sectors) who have engaged with the event (P_s), hourly wage rate for the sector (HR_s), and length of time in engagement, measured in hours (H). The assumption that underlies this value is that the expected value from the event is at least as high as the cost of the individual attending the event (measured in this case by paid-for hours). This may be an overestimate as: (1) individuals may be under pressure to attend, (2) individuals are attending to claim “per diem” expenses, (3) individuals may not consider the cost to their employer of attendance, and (4) the real value gained may be systematically lower/higher than the direct measure calculated. For example, using this methodology activities with large numbers of participants are always recorded as being of greater value than events with small numbers of participants, regardless of the outcome.

Equation 1: Direct Internal Value Footnote ²

$$DIV = \mathop \sum \limits_{s = 1}^6 P{S_s} \times H{R_s} \times A \times H$$

Benefits arise from added value over and above that directly funded by the project. The total DIV was therefore allocated between those on the project payroll (labeled as funded) and those external to the project (labeled as unfunded).

Figure 3 records the DIV of nine activities measured. In total, it shows that they generated over US$353 000 DIV. Over 95% of the DIV was added value to the project as it accrued to individuals external to the project – this is shown by the orange (unfunded) segment of the graph, while 54% of the total overall benefit was generated by just one activity – CoW.

Figure 3. Direct internal value by activity. Source: authors’ calculations.

Output: indirect internal value

Indirect internal value (IIV) was estimated through asking project experts (including some, but not limited to, those individuals interviewed as part of the qualitative framework) to judge how an activity affected the delivery of subsequent activities within the project. They were asked to estimate the time saved and/or extra time needed of individuals working on the project as a result of each activity. For example, the experts judged that if the inception workshop had not been held, other work packages would have taken longer to complete – this was estimated to be the equivalent of one person working 141 days.

Respondents were then asked to attribute this increase or decrease to one of seven potential reasons for the time saving and/or extra time needed (i.e. trust, champions, blockers, culture, data governance knowledge and skills, consistency of data and other), identified in the qualitative research.

Equation 2: Indirect Internal Value Footnote ³

$$NTS = T{S_w} - ET{N_w}$$

$$AHW = \mathop \sum \limits_{s = 1}^6 H{W_s} \div 6$$

$$IIV = NTS \times AHW$$

Due to a lack of responses to this particular question, there were insufficient data to generate IIV estimates for International Exposure Visits; estimates for all other activities are presented in Table 5.

Table 5. Indirect internal value

Source: authors’ calculations.

Every category contributed to an indirect time saving, in total equivalent to 719 person-days. The single largest net contributor was the inception workshop, demonstrating the importance of this event in engaging the local community to support the project.

The experts were asked to apportion their estimates of the time saved to the mechanism through which it was most likely to occur. Figure 4 provides proportions of the time saved (719 days) attributed to the different mechanisms. As such, the total for all activities totals 100%.

Figure 4. Weighted average of the proportion of time saved allocated to the reason for the time saving. Source: authors’ calculations.

In support of the importance suggested in the qualitative interviews, working with “Champions” was judged to accounted for over a quarter (28%) of the IIV time saving estimated in Table 5 – this equated to nearly 200 days. Change in the culture toward more open data and an increase in trust between stakeholders were estimated to account for 22% (156 days) and 15% (106 days) of the IIV, respectively. What is of particular interest about these two indicators is that any positive change in culture and trust would be expected to extend past the length of the project and therefore generate longer-term impacts. This illustrates one of the difficulties of assessment where structural change is concerned.

Intermediate outcomes: external tangible value

Given the need for the evaluation to be proportionate and the limited availability of data, the (tangible) outcomes were limited to just measuring publications produced as a result of these activities (some of which will be published in the special edition of Experimental Agriculture). The strength of this approach is that the output is tangible and measurable, and can be directly attributed to the activity. A further step could attempt to monetize the value of the publications, but, given the limited information available on the quality of each publication at the time the evaluation was completed, the authors judged that any such estimation approach would lack validity.

As well as not directly monetizing the value of the output, the main weakness with this approach is that it does not capture the full indirect outcomes (e.g., publications leading to changes in policy and practice which result in the generation of additional value). Therefore in terms of a traditional CBA, this approach is an underestimate of the true benefit (assuming outcomes are generally positive).

Table 6 lists the 21 publications produced as identified by the experts.

Table 6. Publications produced

Source: authors’ calculations.

The challenges in defining and accounting for data governance and access elements of the SSHI projects, and the difficulties of quantifying and monetizing what potentially is an unquantifiable concept, mean that the estimates here should only be considered as illustrative. Given these caveats, Table 7 summarizes the following results.

Table 7. Total cost and benefits

Source: authors’ calculations.

The case study results suggested that, on a direct cost measure, the SSHI data governance activities yielded a negative net return of US$234 542 (column 3). These figures reflect the monetized direct measurable net benefit for the project.

The net return, however, does not include non-monetized benefits (column 4–5), including indirect benefits and longer term spill-over effects. In fact, Whittard et al. (Reference Whittard, Ritchie and Nyengani2021) in the full SSHI evaluation report that inclusion of relatively few un-monetized benefits is sufficient to generate a positive return, for example, valuing academic publications (column 4) on cost basis.

During the qualitative interviews, the respondents indicated that the overall impact of improved data governance and access was likely to generate wide-ranging spill-over impacts, potentially way beyond the soil and agronomy sector. Given the challenges of estimating a monetary value for these, the research team limited their analysis to estimating a long-term impact score based on likelihood and potential impact (column 5) – for more detail see Whittard et al. (Reference Whittard, Ritchie and Nyengani2021). Any score over 6.25 indicates a positive spill-over is likely and/or will be impactful, and the strength of the likelihood and/or impact is related to the size of the estimate (25 being the maximum score). Therefore, the results reported in column 5 indicate substantial positive spill-over effects, both within the soil and agronomy sector, but also on the wider economy itself. This would be a pure gain for Ethiopian society from the project if the direct benefit were positive or close to zero.