An Assessment of ARPA-E (2017) / Chapter Skim
Currently Skimming:
3 ARPA-E's Internal Operations: Culture, People, and Processes
3 ARPA-E's Internal Operations: Culture, People, and Processes
Pages 47-90
The Chapter Skim interface presents what we've algorithmically identified as the most significant single chunk of text within every page in the chapter.
Select key terms on the right to highlight them within pages of the chapter.
From page 47... ...
An Assessment of ARPA-E 3 ARPA-E's Internal Operations: Culture, People, and Processes In accordance with its statement of task (Box 1-1 in Chapter 1) , the committee undertook an operational assessment to "appraise the appropriateness and effectiveness of the agency's structure to position it to achieve its mission and goals." This chapter describes and assesses ARPA-E's internal operations, addressing the following specific charges in the statement of task: • Evaluate ARPA-E's methods and procedures to develop and evaluate its portfolio of activities; • Examine appropriateness and effectiveness of ARPA-E programs to provide awardees with non-technical assistance such as practical financial, business, and marketing skills; • Assess ARPA-E's recruiting and hiring procedures to attract and retain qualified key personnel; • Examine the process, deliverables, and metrics used to assess the short and long term success of ARPA-E programs; • Assess ARPA-E's coordination with other Federal agencies and alignment with long-term DOE objectives; • Provide guidance for strengthening the agency's structure, operations, and procedures; • Evaluate, to the extent possible and appropriate, ARPA-E's success at implementing successful practices and ideas utilized by the Defense Advanced Research Projects Agency (DARPA)
|
From page 48... ...
The agency selects projects to fund through a multifaceted process that entails evaluating each project's potential, if successful, to help achieve the agency's goals instead of adhering strictly to ranking of external peer reviewer scores. This chapter presents the evidence supporting the importance of these organizational features, together with the committee's findings and recom mendations on ARPA-E's internal operations.
|
From page 49... ...
The chapter ends with a summary of the committee's findings and its recommendations with respect to ARPA-E's internal operations. ARPA-E'S CULTURE, PEOPLE, AND PROCESSES: A QUALITATIVE AND QUANTITATIVE ANALYSIS ARPA-E exhibits a number of features that suggest the organization's policies and practices, internal structure, and culture are interlinked and interdependent in a way that are essential for strategic management of innovation (Kfir, 2000)
|
From page 50... ...
. A number of ARPA-E's activities and procedures indicate the agency features a culture focused on talent, openness, and empowerment that encourages risk taking, and hence a high tolerance for flexibility to learn what does not work on a path producing innovative outcomes.
|
From page 51... ...
Exploration, on the other hand, seeks to discover entirely new technology development curves. Exploitation often entails the use of known research tools and activities, while exploration involves potentially wasted effort, inferior actions, and many early failures.
|
From page 52... ...
Program directors all demand rigorous critiques of their pitches for new focused technology projects. Several former program directors expressed the sentiment that the worst thing that can happen when an idea is first 3 Interview with Le Minh.
|
From page 53... ...
ARPA-E has described its processes as the search for "white space." Searching for white space for ARPA-E entails pursuing two distinct but related areas of research with the objective of producing energy technology innovations: the first is the pursuit of technological ideas or approaches that are truly novel or greatly underexplored; the second is a deliberate attempt to fill gaps left in other research or funding programs. In addition to this search, future orientation means keeping an influx of talented program directors empowered to seek white space, construct new programs, and actively guide their projects.
|
From page 54... ...
While such a short time series does not allow for drawing conclusions as robust as policy makers may want, the data still clearly suggest that ARPA-E has done well in this regard. Leadership and Vision of the ARPA-E Director: Qualitative Analysis As noted above, the committee identified the director's key role in ARPA-E as a defining organizational feature of the agency.
|
From page 55... ...
The director must be widely recognized and respected as an accomplished scientist or engineer. This requirement was anticipated by ARPA-E's authorizing legislation, which stipulates that the agency's director is to be an individual who, "by reason of professional background and experience, is especially qualified to advise" the secretary of energy on "matters pertaining to long-term and high-risk barriers to the development of energy technologies." 6 Acknowledged technical competency and skill serve as a foundation for building trust that in turn enables empowerment and autonomy among program directors and other key personnel.
|
From page 56... ...
Using a professional background taxonomy, the prior careers of program directors who had held only academic jobs following attainment of their final degree were categorized as Academia, those with careers only in industry as Industry, and those that had held positions in both academic and industry settings as Blended. The Academia program directors accounted for 28 percent of the total, Industry program directors accounted for 34 percent, and Blended program directors for 38 percent.
|
From page 57... ...
7 Several former program directors presented information to the committee during open data gathering sessions. The committee also corresponded with several and consulted with others, and met with then-current program directors at ARPA-E events such as kickoff and annual meetings.
|
From page 58... ...
They work with performer teams to create technology-to market milestones as part of award negotiation, and they work with the program directors to identify and encourage commercialization pathways during the award timeframe. Given the long life spans of incumbent energy technologies, the relatively long timeframe and large amounts of capital required to adequately verify and validate new energy technologies and move them to commercial product development, there is an inherent tension between funding early-stage transformational technologies and bringing products to market quickly.
|
From page 59... ...
There is also a risk of ARPA-E's T2M team and the program directors becoming institutionally isolated from one another. The committee was unable to conduct any quantitative analysis for T2M personnel as it did for program directors, but nonetheless was able to gather some qualitative data on these personnel, including their roles, functions, and activities.
|
From page 60... ...
One challenge is a concern that formally interviewing SETA contractors as part of an assessment may place them at risk of violating the prohibition on non governmental personnel representing the government. Autonomy's Importance for Effective Active Project Management Providing autonomy for rapid learning and adaptation is a key element of active management within the DARPA tradition.
|
From page 61... ...
The committee sought evidence that ARPA-E followed such practices, and directed its consultants to construct a large set of ARPA-E data from anonymized information on all concept paper applications; full-proposal applications; review scores for all full proposals; binary project selection; quarterly progress reports for each project; and outcome metrics associated with each project, including patent applications, publications, and a series of indicators for market engagement (e.g., follow-on funding) (the outcome metrics are presented in Chapter 4)
|
From page 62... ...
Ranking the applications in order of mean overall score within a program shows that ARPA-E frequently selected applications from across the full range of scores rather than systematically selecting the projects with the highest reviewer scores. The selection rate varies significantly by program.
|
From page 63... ...
For 20 percent of applications, the standard deviation for the overall score is greater than 1. Perhaps most interesting, the mean overall review score (or any other review category, for that FIGURE 3-3 Percent of concept paper applications that result in selection for award negotiation.
|
From page 64... ...
Moreover, holding mean overall review score constant, projects with a larger range of review scores were more likely to be selected. Given a constant perceived project capability, as shown by the average of reviewer scores, program directors were more likely to select those projects whose technical merit reviewers disagreed on, as shown by a larger spread between highest and lowest scores.
|
From page 65... ...
BOX 3-1 Creating the Counterfactual Project Selection System Using data on review scores for each selected application, the committee created an indicator variable to identify whether an application was selected based on program director discretion rather than a simple ranking of reviewer scores. The general method for identifying applications that were "discretionarily selected" from a low score was to create a counterfactual score cutoff, which would be used to determine selection if there were no program director discretion in making recommendations or agency discretion in making funding decisions.
|
From page 66... ...
FIGURE 3-6 Program director discretion in project selection, by focused program: Portion of projects with a low score that were discretionarily selected. NOTES: Discretionarily selected projects are those that would not have been chosen using a simple ranking by reviewer score (see Box 3-1 for more detail)
|
From page 67... ...
The agency's merit review process appears to account for possible biases in reviewer scores that can actually help with making the most use of them as one possible guide to locating truly novel ideas. Performance of Discretionarily Selected Projects Having determined that this discretion exists, a natural follow-up question arose regarding the performance of projects on the lower and upper ends of the review score distribution.
|
From page 68... ...
Conversations with performer teams and with ARPA-E technical staff, including program directors and systems engineering and technical assistance (SETA) support staff, strongly suggest that a common component of active project management is to modify milestones in response to data generated through the course of a project.
|
From page 69... ...
Project Assessment to Inform Active Management: Quantitative Assessment Each quarter program directors rate projects according to how well they are meeting their milestones for technical, cost, schedule, and overall performance. In 2015 an additional milestone was added to track technology-to market performance.
|
From page 70... ...
For example, if ARPA-E had a bright line rule, say, requiring termination of any project with two consecutive red quarters, then it is possible that program directors would be required to terminate projects for nontechnical reasons such as a principal investigator's suffering from an acute but short-term medical condition or a family emergency that temporarily slowed or halted work. But while the project management dashboard rating system is based on much information, it is still not the total picture of a given project.
|
From page 71... ...
This is evidence that active program management is taking place and satisfying Congress's requirement that program directors recommend restructuring or termination as needed. 14 It also points to the third defining organizational feature of ARPA-E listed earlier: that active project management is important to ARPA-E.
|
From page 72... ...
In other words, after controlling for organization type, initial award amount, and initial project length, there is no relationship between a project's having had at least one program director change and the project's having been terminated early, ending up with a status of green (meeting all milestones) , or ending up with a status of red (failed to meet milestones)
|
From page 73... ...
One former program director observed that the intrinsic purpose of cost-sharing requirements is to leverage federal funds and to ensure that industry participants have genuine interest in conducting research in a given technology area. In contrast, another former program director stated the belief that cost sharing yields few benefits, has various unintended negative consequences, and may already be seen by private companies as a disincentive to applying for ARPA-E funding.
|
From page 74... ...
Other key areas of similarity include an organizational culture of risk taking, a focus on hiring highly qualified technical staff with academic and industrial backgrounds, and providing broad autonomy for program managers/directors to identify and support relevant technologies for specific purposes. There also are some important differences, though, in the agencies' attributes and how they undertake their work.
|
From page 75... ...
DARPA, however, does not utilize peer review in its process because of concerns that peer reviewer comments or scores tend to discount truly novel and potentially breakthrough ideas. ARPA-E's merit review process, on the other hand, makes use of information from peer reviewers, especially written comments, although evidence presented in Figures 3-5 and 3-6 show that many projects are "discretionarily selected." Another area of commonality between ARPA-E and DARPA is making use of two different types of programs to fund research, ones focused on a specific technology or technology area, and so-called open announcements seeking any good idea that fits within the agency's mission.
|
From page 76... ...
TABLE 3-2 Comparison of Attributes of DARPA and ARPA-E Attribute DARPA ARPA-E Agency Organization Direct Report to Yes Yes Department Secretary Flat Organizational Yes Yes Structure Agency Features Focused Mission Yes: initiate rather than be the Yes, technologically narrower: victim of strategic overcome the long-term and technological surprises high-risk technological barriers in the development of energy technologies Budget Size Approximately $3 billion/year Approximately $280 million/ year Concentrated Yes: Department of Defense No Primary Market and prime contractors, but for many sectors no longer the dominant source of demand Number of Program Nearly 100 Approximately 15 Managers Term Appointments Yes, 3-5 years Yes, 3 years for Program Mangers Culture an Yes Yes Important Feature to Support Mission Success Technical Offices Yes: office managers No: agency budget too small Focused on orchestrate a "pyramid of Relevant technologies" across programs Technology Areas Copyright National Academy of Sciences. All rights reserved.
|
From page 77... ...
An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 77 TABLE 3-2 Continued Attribute DARPA ARPA-E Set Aside Full No: annual appropriations Yes: to mitigate uncertainty of Multiyear Project sufficiently certain to enable future budgets Funding in Year annual allocations One Procedures and Processes Orchestration of Yes: specific strategy for all Aspirational, but not yet a Technology office directors and program specific strategy Directions managers in support of achieving technical outcomes and agency mission Funding of Entire Yes No: budget too small Platform of Technologies Necessary to Achieve Particular Goals Program Yes: no peer review Yes: program directors utilize Managers/Directors information from peer review, Have Autonomy in but are not obligated to follow Funding score cutoff or "rack and stack" Recommendations Effort to Build New Yes: specific strategy for all Unclear: at present, ad hoc with Research office directors and program anecdotal evidence Communities managers in support of achieving technical outcomes and agency mission; aim is to connect previously disconnected researchers to enable otherwise infeasible research Active Yes Yes: regular site visits, review Management of of data with performers, Projects suggestions regarding technical directions of research and project team Go/No-Go Yes, but how much has varied Yes: legislatively required; Technical by decade and program quarterly review of progress Milestones for toward achieving milestones Projects and feasibility (Continued) Copyright National Academy of Sciences.
|
From page 78... ...
An Assessment of ARPA-E 78 AN ASSESSMENT OF ARPA-E TABLE 3-2 Continued Attribute DARPA ARPA-E Program or Yes: focused on military Yes: specific program and Procedures to application; Adaptive personnel to assist awardees Ensure Awardees Execution Office seeks to with orienting to eventual Plan/Prepare for accelerate transition from market entry, identifying Eventual DARPA project to Defense commercial applications, and Commercialization Department capability; focus finding sources of funding to on commercializing for defense continue past ARPA-E award with increasing interest in dual- period use potential; most applications envision some application of technology; expectation that good technologies and teams will find new funding sources to continue past DARPA granting period since "DARPA is not in the business of sustaining the technology" Intensive Yes: intensive problem-focused No Gatherings of gatherings of thought leaders to Entire Research and determine specific problems to Innovation solve Community to Identify Priorities and Directions Inducement Prizes Yes No, presumably because of and Competitions, budget limitations and Grand Challenges Program Features Programs Focused Yes Yes on Specific Technical Outcomes with Measurable Goals Broad Area or Yes Restricted: OPEN program is Open Programs to made available roughly every Capture Promising 3 years, with an application Ideas That Do Not window of approximately Fit within Particular 120 days Focused Programs Copyright National Academy of Sciences. All rights reserved.
|
From page 79... ...
The applied offices make their applied research investments hew closely to technology roadmaps that appear less risky, with the expectation of continuing their investments in particular technologies or programs over long periods of time across many stages of the innovation cycle, from pilot projects through large scale demonstrations and on to late-stage deployment, although they also fund applied research (EERE, 2015)
|
From page 80... ...
. As seen in Figure 3-9, there is evidence that ARPA-E has funded projects mainly at technology readiness levels 2 to 4, as reported by the performer teams in their full applications during early FOAs.
|
From page 81... ...
They learn of potential leads for both private funding and potential DOE applied office funding for performer teams from coordination with other DOE offices, although the bulk of that knowledge comes from their own direct engagement and networking with private funders. Much as with program creation and funding decisions, when it comes to technology transfer, there is coordination with the rest of DOE but through informal channels, and thus far in ad hoc ways closely associated with individual program directors.
|
From page 82... ...
using FOAs and similar processes for selecting and funding proposals, such as pre-proposals similar to ARPA-E's concept papers; (2) holding workshops to convene experts; (3)
|
From page 83... ...
It is more difficult, however, for OE to adopt 19 Consultation with David Danielson, Assistant Secretary for Energy Efficiency and Renewable Energy, DOE, 2015. 20 Based on answers to written questions from the committee obtained from Christopher Smith, Assistant Secretary for Fossil Energy, DOE, 2016.
|
From page 84... ...
Specifically, ARPA-E program directors exert significant autonomy and discretion in program creation and project selection and remain actively engaged in managing projects on a quarterly basis through milestone revisions, budget modifications, and timeline adjustments. ARPA-E also appears to have positive spillover effects on other offices in DOE, especially EERE.
|
From page 85... ...
activities to be an ongoing experiment, and the challenge of developing such a program may be greater than originally thought. As discussed previously, given the long life spans of incumbent energy technologies and the relatively long timeframe and large amounts of capital required to adequately verify and validate new energy technologies and move them to commercial product development, there is an inherent tension between funding early-stage transformational technologies and bringing products to market quickly.
|
From page 86... ...
This was seen when, even after controlling for performers' organization type, initial award amount, and initial project length, the likelihood of a project's filing a patent application appeared to increase if the program director changed. These findings will become more specific as the sample size increases in subsequent assessments.
|
From page 87... ...
use cooperative agreements that can be terminated; (5) engage in active program management, including go/no-go decisions (ability to terminate projects)
|
From page 88... ...
Finding 3-9: ARPA-E program directors actively manage projects through technical research guidance and feedback, regular and frequent assessments of progress made toward stated technical milestones, and revision of milestones in response to new findings and research discoveries. Throughout a program's life cycle -- from review of applications, through award negotiations, to project completion -- program directors engage in active program management.
|
From page 89... ...
An Assessment of ARPA-E ARPA-E'S INTERNAL OPERATIONS 89 personnel or subcontractors. They regularly engage with performers to discuss a project's technical approach and collaborate to revise it based on results to date.
|
From page 90... ...
An Assessment of ARPA-E Copyright National Academy of Sciences. All rights reserved.
|
Key Terms
This material may be derived from roughly machine-read images, and so is provided only to facilitate research.
More
information on Chapter Skim is available.