An Assessment of ARPA-E (2017) / Chapter Skim
Currently Skimming:
4 Technical Assessment
4 Technical Assessment
Pages 91-136
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 91... ...
With respect to ARPA-E's stated goals, this chapter begins by explaining two terms that are key to those goals -- transformational technologies and white space. The chapter then describes the external metrics used to assess impacts of ARPA-E programs and projects.
|
From page 92... ...
. For this study, transformational technologies and white space were defined as described below.
|
From page 93... ...
Figure 4-1 provides illustrative examples of the unpredictability of technology disruption in terms of the performance/price ratio over time for common information storage technologies. In the mid-1980s, looking at the curve for magnetic tape since 1950, it would not have been immediately clear that its performance to price ratio would overtake the incumbent technology, printed material.
|
From page 94... ...
. For this assessment, then, the committee developed a set of questions to use in determining whether a program or project is potentially transformational: • Does the program/project hold potential to create "forks" in the technology roadmap that could change the conventional understanding of what is possible, practical, and profitable in applications with large potential impact?
|
From page 95... ...
As discussed in Chapter 2, searching for white space for ARPA-E entails pursuing energy technology innovation with two distinct but related objectives: the search for technological approaches that are truly novel or greatly underexplored, and the search to fill gaps left in other research or funding programs. Figure 4-2 shows how the Full-spectrum Optimized Conversion and Utilization of Sunlight (FOCUS)
|
From page 96... ...
Rather, keeping in mind the distinction drawn above, funding projects that fill a white space may also produce results consistent with the agency's objectives and help enable transformational energy technologies. EXTERNAL METRICS USED TO ASSESS OUTCOMES The statement of task for this study directs the committee to examine processes, deliverables, and metrics used to assess the short- and long-term success of ARPA-E programs.
|
From page 97... ...
. However, federal funding of energy technologies receives more scrutiny than defense-related funding, and any technical evaluation of ARPA-E must rely to some extent on metrics.
|
From page 98... ...
The committee was interested in developing systematic metrics for the social, economic, and environmental impacts of ARPA-E projects, but given the relatively short timeframe being assessed and the small number of projects that have achieved downstream market engagement, the committee did not believe there would be measurable evidence with which to conduct assessment against the program's long-term objectives. As noted in Chapter 2, the goals for ARPA-E include reducing imports of fossil fuel, reducing energy-related emissions, and ensuring that the United States maintains a technological lead in Copyright National Academy of Sciences.
|
From page 99... ...
EVIDENCE OF OUTCOMES FROM ANALYSIS OF KEY EXTERNAL METRICS: SCIENTIFIC PUBLICATIONS, PATENTS, AND FOLLOW ON FUNDING AND NEW FIRM FOUNDATION As discussed above, the committee's central task was to assess the impact of projects funded by ARPA-E given the relatively short timeframe that has elapsed since the agency began operating. ARPA-E provided the committee with aggregate information on some intermediate outcomes of the work it has funded.
|
From page 100... ...
For this latter analysis, it was possible not only to look at publications and patents but also to leverage ARPA-E's own public data reporting on the external market engagement of its projects. While the committee recognizes that these offices have different missions both from each other and from ARPA-E, these analyses help demonstrate ARPA-E's productivity to date and how the agency can contribute to DOE's overall mission to develop and deploy innovative energy technologies.
|
From page 101... ...
As with scientific publications, it is possible to compare the patenting activities of projects funded by ARPA-E with those of projects funded by the Office of Science and EERE, once again controlling for such factors as the size and 3 For this analysis, a top journal refers to one of the 40 journals from the Thompson Reuters Energy and Fuels category with the highest number of citations for the time period 2005–2015. Copyright National Academy of Sciences.
|
From page 102... ...
The author evaluated this award-level variance with respect both to patents and publications and to additional metrics related to market engagement. Specifically, in addition to patents and publications, Goldstein leveraged ARPA E's publication of a series of market engagement metrics focused on three types Copyright National Academy of Sciences.
|
From page 103... ...
While established firms may not perform as well as other organization types with respect to these metrics, then, more research is needed to identify an appropriate set of outcome metrics for evaluating ARPA-E projects of established companies. Team composition appears to have an effect on project outcomes.
|
From page 104... ...
Most programs have yet to produce a patent, but 60 percent of projects in the Batteries for Electrical Energy Storage in Transportation (BEEST) program and more than one-third of projects in the OPEN 2009 program have produced patents.
|
From page 105... ...
projects; and the third includes 10 individual projects grouped into three categories -- successful, cancelled, and other. The first two types allowed the committee to understand the implementation and impacts of a broad set of projects within both a single program (SWITCHES)
|
From page 106... ...
Because most projects are funded for approximately 3 years, there was at most 3 years of data available on completed projects for use in this assessment. Therefore, the focus was on evaluating ARPA-E's operations and providing a technical assessment of whether the selected programs and projects dealt with potentially transformational technologies and/or filled white spaces not being pursued by other public and private entities because of technical and financial uncertainty.
|
From page 107... ...
. Currently, industrial power electronics circuits are overwhelmingly (>95 percent)
|
From page 108... ...
ARPA-E also considers the SWITCHES program a complement to the New Generation Power Electronics Innovation Institute, which is working to help create and manufacture WBG semiconductor-based power electronics in the United States (ARPA-E, 2014e)
|
From page 109... ...
The corollary is that once scientific discoveries are brought into the design space, engineers will find applications not fully conceived of before. Thus, ARPA-E's efforts in the SWITCHES program are consistent with its goal of bringing potential transformational technologies beyond discovery to the demonstration phase.
|
From page 110... ...
The committee did not consider projects focused on thermal storage or on improvements to battery management systems, sensors, or testing. Program Descriptions ARPA-E has funded its electricity storage projects through three focused programs (BEEST, GRIDS, and Robust Affordable Next Generation Energy storage systems [RANGE]
|
From page 111... ...
Funding of White Space and Project Storyline As discussed previously, ARPA-E has been mandated to seek out and fund transformational ideas and to find technical white space in which no significant work has been done. In about half of the projects, there is clear evidence of such white space being addressed (Figure 4-5)
|
From page 112... ...
these the prior work was done at the institution of the project's principal investigator. These findings show that ARPA-E uses its funding to move early stage innovations to the demonstration stage.
|
From page 113... ...
) , technologies for improving battery management and sensor systems (Advanced Management and Protection of Energy storage Devices [AMPED]
|
From page 114... ...
Further, the committee notes that the technical background of the program directors has been appropriate for most projects, with most having had significant entrepreneurial experience. Given the high-risk/high-payoff approach both DARPA and ARPA-E take in creating programs and funding projects, the degree of follow-on support resulting from company founding appears to be reasonable for these projects, a significant portion of which found support after the ARPA-E funding ended.
|
From page 115... ...
NOTES: The full case studies are in Appendix D ADEPT = Agile Delivery of Electrical Power Technology; BEEST = Batteries for Electrical Energy Storage in Transportation; GENI = Green Electricity Network Integration; SWITCHES = Strategies for Wide bandgap, Inexpensive Transistors for Controlling High-Efficiency Systems.
|
From page 116... ...
It should be noted at the outset that, although these projects are categorized as successful in that they show potential for success, none of them have as yet been transformational in the energy sector -- as would be expected given the extended period of time in the market required for transformational technologies to become apparent. Cost-Effective Silicon Wafers for Solar Cells 1366 Technologies is developing a potentially disruptive solar technology, a process aimed at reducing the cost of solar wafer manufacturing by 50 percent by 2020.
|
From page 117... ...
. 1366 Technologies received venture capital funding prior to ARPA-E funding; it received $4 million from March 1, 2010, to June 30, 2012; and it received more than $50 million of follow-on funding.
|
From page 118... ...
The project term was from April 26, 2013, to July 25, 2016. A year and a half into the project, BASF Venture Capital raised $3 million in the initial financing round in October 2014 to create a start-up company, SLIPS Technologies, Inc.
|
From page 119... ...
Since receiving the funding from ARPA-E, Smart Wires has undertaken several rounds of successful fundraising, including $30.8 million in 2015 to bring its PowerLine Guardian product to commercial production. Summary of Evidence from Case Studies The case studies evaluated by the committee help demonstrate how the agency is fulfilling its mandate to fund transformational technologies or fill white space opportunities.
|
From page 120... ...
(radiative cooling) to technologies that are emerging into commercial products (e.g., cost-effective silicon wafers for solar cells, SLIPS, Smart Wires)
|
From page 121... ...
TRANSFORMING ENERGY INDUSTRY ATTITUDES ARPA-E projects develop technologies for applications in the energy sector, a complex, extensively regulated sector that is capital-intensive and risk adverse and adopts innovations over long time horizons. ARPA-E is "developing technologies to alter the status quo of the existing energy economy, where much of the end product is a uniform commodity (liquid fuel or delivered electricity)
|
From page 122... ...
Most of the project case studies examined for this study involved prototyping, data gathering in operational environments, scale-up, and the development of pathways to commercialization rather than research into entirely new scientific concepts. As noted in the full Smart Wires write-up in Appendix D, for example, the program director predicted that utility executives' realization that they can modulate power flows on command would have transformational effects on their decision making, ultimately leading to far more efficient, lower-cost transmission.
|
From page 123... ...
provides a comprehensive study of the operations of DARPA's Microsystems Technology Office between 1992 and 2008 that highlights the key role played by that agency's program managers in convening and managing social networks to achieve technological objectives. According to Fuchs (2010)
|
From page 124... ...
The committee was unable to identify instances in which ARPA-E programs are attempting to play a longer-term role in their evolution into new, lasting research communities. Advanced Management and Protection of Energy storage Devices (AMPED)
|
From page 125... ...
IMPROVING PUBLIC AWARENESS OF ARPA-E ACHIEVEMENTS While the committee noted significant outcomes emerging from projects funded by ARPA-E, these achievements may not be sufficiently appreciated outside of energy research communities within companies, government agencies, and academic institutions. While many exhibitors at the ARPA-E Energy Innovation Summit and participants in program meetings attended by the committee appeared to have the capability to discuss the technical aspects of Copyright National Academy of Sciences.
|
From page 126... ...
Their efforts could clarify that diverse metrics are necessary to understand the success of individual projects and programs and help dispel the idea that there may be a single metric suitable for evaluating all projects and programs. ADDITIONAL METRICS THAT MAY BE USED FOR TECHNOLOGY ASSESSMENTS As discussed earlier in this chapter, an important part of this study was developing systematic metrics for the social, economic, and environmental impacts of ARPA-E projects, but given the relatively short timeframe since ARPA-E commenced operations, there was little measurable evidence to use for assessing performance against the agency's long-term objectives.
|
From page 127... ...
ARPA-E has in place an extensive data gathering and recordkeeping system at the project level that can track and monitor internal metrics and facilitate active program management. It has a less extensive system that can collect, track, and report publicly available high-level innovation metrics such as publications, funding from other sources, and intellectual property information, Copyright National Academy of Sciences.
|
From page 128... ...
What really is needed is a framework that maps the linkages of technical goals of each program through intermediate outcomes, such as traditional innovation metrics, to the agency's statutory mission, means, and goals. Developing and implementing such a framework for impact evaluation would be very valuable and important for ARPA-E to undertake as soon as practicable, providing the agency greater ability to assess and demonstrate its value and impact.
|
From page 129... ...
Several supported technologies have received follow-on private funding and appear to be on a trajectory toward commercialization of products -- with at least one already in the marketplace -- that will impact the energy sector and the environment. The committee's review of programs and selected projects indicated that the agency created some funding programs for technology areas that the private sector or other federal agencies would have been unlikely to pursue.
|
From page 130... ...
By comparison, as noted above, truly transformational technologies emerge over much longer time periods. The committee emphasizes the most fundamental tension to be managed by ARPA-E between having a short-term impact on a technology within the 3-year funding timeframe while producing transformational technologies.
|
From page 131... ...
Despite these limitations on an early assessment, currently available qualitative and quantitative data suggest that ARPA-E is pointed in a positive direction -- it has funded projects that support its statutory mission and goals -- and there are signals that indicate potential for future success. A review of programs and selected projects suggests that the agency has identified technology areas that neither the private sector nor the federal government would likely have pursued alone, funded projects when no other entity would, and provided crucial early-stage funding for some projects.
|
From page 132... ...
The agency may best be served by a balanced portfolio of projects aimed at attaining both large and small successes, not only "home runs." Funding white space can be orthogonal to funding transformational technologies, including "filling in" where other agencies are not working. And driving down costs (or the cost-to-performance ratio)
|
From page 133... ...
That said, quarterly reporting in terms of required written documentation is currently challenging, depending on the technical context. Given that quarterly written reports are offset in time with site visits from program directors and their teams, a project performer may end up having 8–10 direct interactions with ARPA-E per year.
|
From page 134... ...
Recommendation 4-6: ARPA-E program directors should compile a document or other repository of lessons learned on all projects, including both positive and negative outcomes. Improving Awareness of ARPA-E's Role among the Public and Policy Makers Finding 4-8: ARPA-E is in many cases successfully enhancing the economic and energy security of the United States by funding transformational activities, white space, and feasibility studies to open up new technological directions and evaluate the technical merit of potential directions.
|
From page 135... ...
The agency is in a good position, though, to create a mapping framework that links the technical goals of each program through intermediate outcomes, such as traditional innovation metrics, to the agency's mission, means, and goals. The agency could link data from its robust internal database of project level metrics to program-level goals, including indicators of commercial and noncommercial outcomes over the short and long terms; connect those goals to standard, observable innovation metrics; and then translate those metrics into progress toward achieving the agency-level mission and goals.
|
From page 136... ...
An Assessment of ARPA-E 136 AN ASSESSMENT OF ARPA-E empowered to develop and implement such a framework in a way that best serves its mission and goals. Recommendation 4-8: The ARPA-E director and program directors should develop and implement a framework for measuring and assessing the agency's impact in achieving its mission and goals.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.