5.1 Methodology of the 40 Year Evaluation
The methodology employed for the evaluation is based on the range of Research Impact Assessment (RIA) approaches reported in literature across different disciplines, and specifically on previously implemented and proven Agricultural RIA (ARIA) methodologies.[1]
The chosen methodology is derived from methods implemented by INRA (French National Institute for Agricultural Research)[2], CSIRO (Commonwealth Scientific and Industrial Research Organization)[3], CGIAR[4] (Consultative Group on International Agricultural Research) and EMBRAPA[5] (Brazilian Agricultural Research Corporation), among others. These bodies use case study impact analysis as a means of assessing impact of the organization. In the case studies, the pathway between the research activities and outcomes and the eventual impacts are analyzed, most often according to categories defined in the OECD glossary of terms for evaluation[6]. For the 40-year evaluation analysis, BARD adopted impact categories and indicators used by the institutions and organizations cited above, as well as those from the impact statements of the USDA-NIFA. The main categories of impact for which indicators have been selected are academic, environmental, social, and economic.
The evaluation was conducted by two complementary pathways: a survey and case studies. Both applied mutual indicators for assessment of impact.
5.1.1 Survey
This component of the evaluation carried out a broad assessment of research studies conducted by Principal Investigators (PIs) who were awarded funding and began their research between the years 1994 –2014. This 20-year timeframe provides an effective window for evaluation. It allows inclusion of researchers who are currently still active in research, while also providing a time lag during which the mid-to-long term impact of research outcomes has had time to evolve and manifest. The survey was conducted in a self-reporting format, in which the researchers were provided an online questionnaire containing a set of key impact indicators. The questionnaire is presented in the Evaluation Compendium.
5.1.2 Case Studies
Twenty case studies were selected for an in-depth research impact analysis that provides both a narrative and quantitative analysis of selected indicators. The materials were retrieved from multiple sources: personal interviews with the researchers, stakeholders and beneficiaries of any practical outcome, as well as publications, written reports, databases and websites. The information collected from the different sources was triangulated and cross-referenced. The following describes the process of selecting the case studies for analysis.
Step 1: BARD approached 13 Israeli academic and research institutes and requested that their respective research authorities nominate projects with outstanding research outcomes. Following the provided guidelines (detailed in the Evaluation Compendium, available online on BARD's website) as well as their respective internal process, the research authorities returned a list totaling 120 Israeli PIs, whose joint projects with US researchers were considered the most outstanding and impactful.
Step 2: The 120 nominated projects were screened by the BARD evaluation team, which subsequently compiled a short-list of 60 projects for case study analysis, based on (i) the research authorities’ selection (ii) input from experts in the fields of the proposed projects (iii) the self-reported survey evaluations, and (iv) prior knowledge.
The two main categories used to assess the projects were (i) academic advances and impact and (ii) practical agricultural application.
Step 3: The short-list of 60 projects was narrowed down by the steering committee to 20 projects chosen to serve as case studies. Most of the projects selected had begun within a time window that enabled a measurable outcome to evolve. Several others were conducted in more recent years and were included for scientific advances that have already achieved substantial international impact, despite not yet attaining full impact potential. While diversity of disciplines was not in itself a criterion for selection, the number of projects from any specific discipline was limited.
5.1.3 Economic Analysis
Using 2018 Dollar-Terms
The economic calculations go back to 1979, the first active year of BARD. Since each project has a different starting point between 1979 and the present, the calculations in this review are expressed in present value terms. Therefore, all flows of money were adjusted to real dollars using the US Consumer Price Index (CPI). For past flows of money, we calculated the real rate of return of 10-year treasury bonds for the years 1979 – 2017, which is 3%. However, in the last 20 years this measure is lower, and we also calculated the long-term real GDP growth rate for the years 1979 – 2017, which is 2.7%. Therefore, past flows are compounded to 2018 dollar-terms using an interest rate of 2.7%.[7]
Regarding future projections, our working assumption is that most projects will generate additional competitive practical agricultural solutions within the coming decade. Therefore, we limited calculations to 2028 and capped the benefit calculation period for all projects at 30 years from first implementation. An exception to this rule was case study 7 (see Appendix 1), the research outcomes of which provided the basis for the emergence of a new industry. As with the other projects, the benefit for this case study was calculated through 2028 but sums to a total of 38 years of benefit since the project began.
Future flows of money were discounted to 2018 dollar-terms using the same rate of 2.7%. All results and calculations in this document are expressed in 2018 discounted dollar-terms, unless otherwise stated.
The Investment
For all projects, BARD’s investment is calculated according to the sums of the awards connected to the research project, and the specific years of the grants. Any additional investments from academic, governmental and private funds are also included in the calculation. The total amount is adjusted to 2018 dollar-terms.
In most cases, BARD invested in the initial and hence most risky stage of the project. BARD provided funding for both fundamental as well as translational research, positioned at a stage where the research was able to demonstrate a preliminary applicative character. This early stage precedes scale-up and entrance of the private sector, since proof of concept has yet to be demonstrated and uncertainty is still high, making BARD’s investment at this point critical.
We estimated the risk premium attributed to BARD and to the other foundational investors by bringing their investments to 2018 dollar-terms using a real interest rate of 7.7% per annum. This figure combines the 2.7% rate and a 5% risk premium to reflect the risk interest rate in the last decade.[8] Any additional investments by commercial and industry players that followed up on the BARD-funded research were calculated using the standard 2.7% rate as the interest rate. These calculations were conducted only for the purpose of quantifying BARD’s share in the overall economic benefit from the projects it funded. We limited the results of this calculation to attribute no more than an extra 15% to BARD’s share in the benefit.
In several case studies, (6, 7, 16 and 18; see Appendix 1) we were unable to obtain information on the additional investment made by academic and governmental funds that were active in the foundational research phase. In these cases, we evaluated BARD’s share through personal interviews with the researchers and relevant experts, and by extrapolating from previous evaluations.
The Benefit
Certain projects generated value added by improving productivity and yield, while others did so by reducing damage and loss to agricultural production and products. Others yet generated value added through developing a new product.
The benefit was calculated according to farmgate prices plus its effect on the retail price. In order to calculate the value added, we compounded the farmgate prices with the impact on retail prices. The two components of the supply chain that are relevant to this calculation are the revenue of the wholesaler and that of the retailer, which are a set percentage of the ultimate retail price of a given product. Our working assumption is that the cost of other components throughout the value chain are not affected by fluctuations in farmgate price[9]. For example, if the price of an agricultural product was reduced by $4/ton because of BARD’s project, and we know that the retail and wholesale revenue is 30% of the consumer price, then we calculated the benefit of the project as $4/(1-0.3) = $5.7/ton.
We estimated the benefit to the relevant global activity and to all players throughout the value chain. We did not address the breakdown of the benefit throughout the value chain, such as how it is distributed between producers and consumers. However, a number of projects benefited the US or Israel as consumer parties. For example, the US and Israel consumption of tilapia is mainly generated from imports. In this case, the US and Israel do not enjoy the producer’s benefit, but do enjoy the consumer’s surplus. For these case studies, we estimated the consumer surplus in both countries in order to estimate their share of the overall benefit.[10]
While we described environmental and social benefits, these are excluded from the calculation.
____________________
1. A comparison between some of the implemented ARIAs can be found in Joly, P. et al. (2016), OECD Food, Agriculture and Fisheries Papers, No. 98, http://dx.doi.org/10.1787/5339e165-en
2. ASIRPA guide to analyzing the impacts of research, 2013, INRA; can be found at http://inra-dam-front-resources-cdn.wedia-group.com/ressources/afile/238248-a43fa-resource-asirpa.html
3. (a) CSIRO. (2014). How CSIRO ensures it delivers impact (Vol. 9) and (b) CSIRO. (2015). Impact Evaluation Guide. Recent impact evaluation reports for specific projects can be viewed and downloaded from https://www.csiro.au/en/About/Our-impact/Our-impact-in-action
4. (a) Stevenson, J. et al., 2018. The Rigor Revolution in Impact Assessment: Implications for CGIAR. (b) Raitzer et al., Benefit–cost meta-analysis of investment in the International Agricultural Research Centers of the CGIAR, Agricultural Systems 96 (2008) 108–123.
5. Avila, F. et al., (2016). EMBRAPA Experience on the Impact Assessment of Agricultural R&D: 15 Years Using a Multidimensional Approach.
6. OECD. (2002). Glossary of key terms in evaluation and results-based management. Development Assistance Committee (DAC), Working Party on Aid Evaluation.
7. See the internet site of the US government Bureau of Economic Analysis, Alston et al used a discount rate of 3% as a base scenario
8. A. Damodaran analyzed financial indicators and found that since 2008, the expected return on stocks has stagnated at about 8%.
We calculated expected inflation according to the average US-CPI in the last three years: 1.2%
9. The information about retail and wholesale dollar share in the supply chain
10. Main source for the demand and supply elasticity, And the book: A Database for Trade Liberalization Studies, published in 1989 by ERS, USDA