Innovation Benefits Realization for Industrial Research (Part-8)

Technology Innovation Management
Framework for Industrial Research
Part-8
Dr. Iain Sanders

January 2005

WHAT R&D METRICS?

Platform 3:
(Stages VIII, Part 3)

Integrating Technology Innovation with Business Function – Part I: Laying the Foundation

PLATFORM 3: VIII (3)

What are our R&D Metrics?

 R&D Metrics
 Set standards for measuring new concept R&D performance
and evaluating progress / outcomes achieved against
milestones / targets set.

6

Top 39 Ranking R&D metrics
11.5 Strategic Alignment 5.5 Development Cycle Time 3 Product Quality and Reliability

11.5 a. Corporate & business unit 5.5 a. Market cycle time 3 a. Customer Evaluation
11.5 b. Goal coverage 5.5 b. Project management cycle 3 b. Reliability/Defects
time
10 Financial Return 5.5 Customer Satisfaction 3 Market Share
10 a. New Sales Ratio 5 Number of Ways Technology is 2.3 Development Pipeline Milestone
Exploited Achieved
10 b. Cost Savings Ratio 5 Projects with Business/ 2 Goal Clarity
Marketing Approval
10 c. R&D Yield 5 a. Project has approval 1.8 Comparative Manufacturing Cost
10 d. R&D Return 5 b. Ratio of projects 1.8 Gross Margin
9 Projected Value of the R&D 3.3 Quality of Personnel 1.8 Use of Cross-Functional Teams
Pipeline
9 a. Projected Sales 3.3 a. Internal Customer Rating 1.8 Rating of Technology Features
and Benefits
9 b. Projected Income 3.3 b. External Customer Rating 1.8 Response Time to Competitors
moves
7.8 Distribution of Technology 3.3 c. External Recognition 1.5 Comparative Technology
Investment Investment
6.8 Use of Project Milestone/ 3.3 Efficiency of Internal Technical 1.5 Employee Morale
Stage-Gate Process Processes
7


Strategic Alignment Metric (11.5)

 Definition
 This R&D metric assess the degree of alignment of an R&D project or an
R&D portfolio with the strategic intents of the company or a division of the
company. The strategic intents are often the corporate goals embodied in
its business plan.

 Why it is used
 This metric is used to gauge the degree of relevance of the R&D program
to the corporate goals. The strategic intents of a company may change
more rapidly than the R&D program can respond to those changes
creating various degrees of misalignment. The misalignment can be with
regard to work area, long term versus short term needs, or degree of risk.

8



 How to use metric
 There are a number of ways this metric can be applied, including both prospective and
retrospective views. It can be applied by R&D management, general management, or by both
working in partnership. An alignment index would be assigned to each project; a linear scale
of 1-5, for example, would work well. these scores may also have value when considering
relative merit of individual projects in the portfolio. A composite score for the entire portfolio
would then be determined. This could be a weighted average reflecting sizes of projects with
regard to technical head count, project budget or some other appropriate weighting factor.
Once a baseline for alignment has been established, R&D management can then decide if
and how this index should migrate to greater or lesser degrees of alignment through
modifications of the portfolio. Applied to individual projects, there could be a cut-off point for
the alignment parameter below which projects are not supported.

 Consideration must be given to the degree of alignment desired. Although the desired state
in may cases is toward greater degrees of alignment, you can envision situations where that
may not be the case. A research organization charged with taking the company in new
directions may not want its project portfolio highly aligned with the current business plan. A
more visionary business plan may capture new directions as well as current businesses, but
many organizations find that some degree of decoupling is desirable.

9



 Options
 Both retrospective and prospective views are options of this metric. The
retrospective view entails applying the metric to an existing portfolio of
R&D projects to determine the degree of the portfolio with corporate goals.

 The prospective view is to apply the metric to a proposed project or slate
of projects. If the R&D organization is trying to increase its alignment
index, then the management will be less likely to initiate projects that move
the composite score in the wrong direction. Similarly, an alignment index
cut-off may be instituted. Projects falling below some minimum value of
alignment would not be supported.

10


Financial Return (10)
 New Sales Ratio (a)
 The New Sales Ratio is the % of current sales originating from new products.
There are two sub-definitions that are required. What is a new product? and, How
old is new?

 The most frequently used and simplest definition of a new product is any new SKU
(inventory code: Stock Keeping Unit) that has required R&D support to implement.
This avoids counting new SKUs which are only packaging changes or other
modifications made easily by marketing or manufacturing. When is a new product
old is different for each business and technology. In rapidly changing and evolving
fields, such as electronic chips and software, new might only be one year, but
certainly not more than three. Three years is more likely the norm for businesses
that are a mix of fashion and formulation, such as cosmetics & toiletries. And, for
more intense capital and industrial products, three to seven is a more likely range
to select a number that is right for you.

11


 New Cost Savings Ratio (b)
The Cost Savings Ratio is the % reduction in cost of goods or cost of operations
(including depreciation charges) that are realized in a year to year comparison that
have originated from technology changes that are new. Again, the same issues must be
resolved as sub-definitions as required for the New Sales Ratio. In other words, what is
being attributed to R&D and how long is new?

 Since SKUs are not used to catalogue changes made in operations an alternative must
be found that will work for your firm. A simple solution can often be created based on
the capital approval process. Most companies require a specific approval for all
individual capital projects. These can be coded and tracked for R&D involvement and
for realized cost savings. However, it is recognized that the accounting involved in
examining the impacts on cost savings may be more difficult than that for new sales. It
is therefore a more common metric only in those cases which are more significantly
impacted by cost savings than by new product sales. And, since new technology for
operations or manufacturing has a different useful life than a new product per se, it
must be tracked for a different length of time that is industry specific. In some cases, it
may be very linked to the product life in other cases, it may go on much longer. In any
case, it is not likely that 'new' will reach beyond 7-10 years.

12


 R&D Yield (c)
 R&D Yield is the contribution of R&D to current financial performance. It is a metric
that is composed of definitions from New Sales Ratio and Cost Savings Ratio, plus
an evaluation of gross profit from the new sales.

 It is the annual combined financial benefit that is derived from the annual gross
profit of new products and the annual cost savings of new processes. This is the
current contribution that the company receives that is associated with its past
stream of R&D investments, i.e. the part of the 'bottom line' that is relatively 'new'
and derived from R&D.

13


 R&D Return (d)
 R&D Return is the relative ROI measure that relates to R&D. It is composed of the
R&D Yield divided by the annual investment in R&D. Hopefully, this is a large
number that is proportional to the risks and variances that are part of R&D.

14


 Advantages and Limitations
 The advantages of these financial metrics are that they relate directly to the financial
benefits to the company, they are quantitative and they are comparable to metrics that
can be used in different parts of the same firm or between firms. They capture the
degree to which R&D is truly making a financial contribution to the value of the
enterprise. They answer the question: What have you [R&D] done for me [the
Business] lately? However, they only represent the tip of a process that takes place
over a number of years and that involves other functions besides R&D. This means that
the numbers reflected by these metrics are associated with activities that are in the
past. These metrics are lagging indicators. They are a nice track record, but they may
not be reflecting accurately a current level of effectiveness.

15


 How to Use the Metric
 The metrics should be tracked at least on a once a year basis. Because of measurement and definition
problems, a baseline of two years or more of historical data is needed before accurate judgments can be
made about trends and ratio efficiencies.

 The metrics should be examined carefully for consistency with business strategies and the results required
vs. the investments in R&D. In situations where the metrics, requirements and available resources are not
in balance, there will be a difficulty in executing the overall business and technology strategies. One or the
other must be shifted, and variations in how R&D is conducted need to be examined.

 If the Financial Return metrics are being maintained or going up: the corporation has the likely raw material
to extend a technology-based or innovation-based growth program; the investors have the possibility of an
extended stream of positive returns from the accumulation of financial pay-offs from technology-based
innovations; and the R&D units enjoy the likelihood of consistent funding to reinvest in various aspects of
technology application for the near term and base building for the future.

 The key words here are likely or possible. Positive Financial Returns are a necessary but not sufficient
condition for growth. It is also only a measure of the moment, whether it is looking to the past or to the
future. And, any downward movements will predict the difficulties the business will have in achieving solid
gains against the competition. These indicators are crucial to assessing the total returns from R&D
investments, whether enough is being spent on R&D, and what is the likely future value of the company
from a technology perspective.

16



 Options and Variations
 There is always something new and innovative that is contributing to revenue or profit.
These financial return metrics are intended to capture the new portion of these changes
in the business that are related to R&D. They require definitions of what is to be
considered new and for how long. These elements can then be tracked separately or
together, in ratio or absolute form, by themselves as benefits or as an investment return
vs R&D. The options and variations fall into place based on each company's views of
these items.

 The most common variations are based on the length of time that is new. the most
frequent categories are three years, five years and seven years. Another variation is to
use these same metrics in a prospective, future mode.

17


Projected Value of the R&D Pipeline (9)

 Projected Sales (a)
 Projected Sales is the calculated sum of future sales from
current R&D projects. This metric may be expressed in
absolute terms or a % of future sales.

 Definitions must be provided for how this is to be evaluated.
Normally, it is the forward side of the newness range, i.e. if
new products are those introduced within five years, projected
sales will be calculated for five years after commercialization.
A probability of attainment is usually figured into this metric.

18



 Projected Income (b)
 Projected Income is the income stream associated with the
Projected Sales. Similar definitions apply. This metric may be
expressed as an absolute number or as a fraction of net
income.

19



 This metric provides an ongoing anticipation of the expected
results from R&D. Because it is projected, it provides an
evaluation of the benefits that are being created with today's
R&D investments.

 The limitations are due to the intrinsic difficulties of obtaining
estimates about the likely commercial benefits if the
technology is successful.

20


 How to use the Metric
 This is perhaps the single most important and least used metrics. It is the singular
indication of the future business that is to be developed as a result of successful
outcomes from R&D. Are the sales impacts large enough, are there enough new
products, is the timing of elements in the pipeline adequate, are the overall returns
related to R&D adequate? Are these metrics on a year to year basis showing
constancy, increase or decline?

 This metric provides ongoing guidance to the company regarding the future gain to
be expected in the business due to R&D. This should be used as a check that both
the strategy and the resource allocations are correct.

 If this metric is staying constant or increasing, particularly with respect to the R&D
resources, then the effectiveness of R&D is being maintained or increasing.

 If, alternatively, this metric is declining, then further diagnostics should be
examined to understand the reasons and to take corrective action.
21



 Common options are few because this is an underused
metric. One is to look at absolute sales or net income over a
five year horizon on the presumption that some projects will
finish and be productive within the next two-three years and
that others will impact a bit later, i.e. in the fourth and fifth
year. A related option is not to adjust the commercial impact
by any probabilities.

22


Distribution of Technology
Investment (7.8)
 Definition
 This metric provides a means of assessing how well an R&D
program is protecting the technology investment and technical
position of the company. It forces consideration of how the technical
assets should be distributed, setting directions for modifying the R&D
portfolio.

 The portfolio of an R&D organization may not be protecting the
strategic interests of the company for any number of reasons such as
skill set mismatch, slow response to changes in the company‟s
mission and markets, and a rapidly changing competitive
environment. There can be an over-emphasis on certain business
units and products. This metric causes the management to examine
how well the R&D effort is protecting and expanding the technical
position of the company in areas of greatest importance.
23


Investment (7.8)
 This metric is applied by first determining how the technology investment should be distributed. As an
example, consider a company with six business units. The R&D portfolio can be distributed among these
six business according to a number of models. Six examples of distribution models are listed below:

 The revenue that each business generates.
 The opportunity market share (potential market growth).
 The impact that technology can make in the different business units.
 Competitive Impact (Base, Key, Pacing)
 Some combination of the above distribution models.
 The profitability that each business demonstrates.

 These considerations often involve the concept of the technological basis of competition, that is how does
technology provide a sustainable competitive advantage in a particular product or market. The discussion
could also consider the distribution of the competitive impact of the company‟s technology investment by
categorizing them as Base, Key or Pacing . Base technologies are essential to the business but widely
exploited by competitors. Key technologies are highly differentiating to the company‟s current products.
Pacing technologies are new technologies where the competitive impact is less certain but likely to be
high. All three of the impact categories require protection of the competitive position, but the distribution of
resources among the three categories may vary based on the company‟s business plans.

24


Investment (7.8)

 How to Use the Metric (continued)
 Once a distribution model has been agreed upon, the portfolio is measured against that model. The
distribution could be with regard to number of R&D projects in each segment of the distribution, head count
devoted to each segment, or expected value of projects in each segment. Modifications are made to the
portfolio to move the distribution toward the desired state.

25


Investment (7.8)
 Options
 This Both prospective and retrospective views are supported by this
metric. In the retrospective view, the R&D projects are categorized
according to the corporate investments or markets they are intended
to protect or create, or the competitive impact that they offer. This
does not have to address the entire R&D effort of the company, since
the metric can be applied to any subset of the portfolio. The projects
are appropriately weighted to reflect their size and cost. A distribution
of the R&D efforts supporting each of the categories is determined.
This current state distribution is then compared with the desired
state. The degree of misappropriation can then be quantified as the
fraction or percentage of the R&D effort that is improperly distributed.

 The prospective view for this metric involves consideration of how a
proposed project shifts the distribution of technology investment
toward or away from the desired state.
26


Use of Project Milestones /
Stage-Gate Process (6.8)
 Definition
 Percent of projects in the total portfolio going through a
defined project management system with defined milestones.

 Percent of R&D expenditure on projects using a defined
project management system with defined milestones.

27


 Percent Project management systems including milestones can provide a way of reducing
cycle time and providing R&D and business management with a sense of the health of
projects. These systems also can improve the linkage between R&D, marketing and business
management by enabling coordination with manufacturing and marketing to hit key windows.
When these systems include documentation through status reports, they can be used to
promote organizational learning. Driving behaviors which increase the numerical value of this
metric should therefore improve both the linkage of R&D to the business and improve the
effectiveness of R&D.

 When used with a formal stage-gate process this metric provides a measure of compliance
with that system. Since companies will generally use a defined project management system
and establish milestones in the later phases of innovation, this metric may also be an
indicator of the distribution of projects in the innovation pipeline (see metric: "Distribution of
Technology Investment" ).

28


 Advantages and Limitations (continued)
 The metric may be limited by the difficulty of counting projects outside the project
management system. Further since project management systems may not be appropriate
early in the innovation process, the ideal value for this metric will depend on the firm's
desired balance of early and late stage projects. For short term projects such as minor
product or process variations, use of formal project management systems and this metric
may create unnecessary red tape and potential delays.

29


 How to Use the metric
 As a concurrent metric, the total number of projects with defined (written) project plans
including definite milestone dates can be divided by the total number of identifiable R&D
projects to calculate the metric.

 When used with the appropriate accounting system this metric can be calculated from the
cost of projects divided by the total R&D cost. In this case the budgeted projects should be
audited to determine compliance with requirements for plans and milestones.

 The metric can be used as a concurrent metric (a snapshot of the current R&D activities) or
as a retrospective measure to determine how many R&D projects used a defined process. It
should be equally suitable for service and manufacturing companies.

30


Development Cycle Time (5.5)
 Market Cycle Time
 This metric measures the elapsed time from identification of a customer
product need until commercial sales commence.

 Project Management Cycle Time
 This metric measures the elapsed time from establishment of a discrete
project to address an identified customer product need until commercial
sales commence.

 For both metrics described, the end point can be time when
manufacturing feasibility is established for those cases where
no commercialization occurs. Compare to historical values and
benchmark vs. competition, if possible. Group by categories of
projects (e.g. major new product, minor product variation, etc.)
Can also be used to track milestone attainment rate for firms
using a stage gate management process.
31


 Advantages and Limitations (Market Cycle Time)
 The advantages of this measure is that it is quantitative and can be used to measure the
entire process or various parts of the process if stage gates are examined. The process can
be analyzed to determine what parts are driving the overall cycle time so that improvements
to the process can be made.

 The limitations of this metric could include R&D's position that it does not adequately
influence the process until after a need has been more defined. Another limitation is that a
strong documentation system is helpful to make the cycle time metric as accurate as
possible. An additional limitation is that defining the commencement of sales as the end of
the cycle does not account for post start-up issues such as efficiency, waste, % of
manufactured product within specification, etc.; this could lead to focusing on shortening the
cycle time at the expense of later, non-measured parts of the cycle. One must also keep in
mind that for breakthrough or paradigm shifting projects,, cycle time measurements. The
advantage of this metric is that it supports having the clarity of when a project is initiated
based on approvals, assignment of resources, start of spending, etc. The metric can be used
to measure the entire process or various parts of the process if stage gates are examined.
The process can be analyzed to determine what parts are driving the overall cycle time so
that improvements to the process can be made.

32


 How to Use this Metric
 Both metrics should preferably be used in combination with a project reporting system that
can track the project initiation date (based on approval and assignment of resources), the
length of time in each stage gate of the innovation process, and date of sales
commencement. For the first metric, the initiation date could be the date the customer need
was determined (i.e. marketing request or date of customer research results). Cycle times for
different types of projects (new products, cost savings, product improvements, etc.) should
be compared to help predict and manage resource allocation. Cycle times for different
divisions could also be compared (with caution) to identify practices driving lower cycle time
to adapt where possible.

 Variations could exist regarding looking at cycle times for only certain parts of the process for
which R&D feels it has most control or influence. Cycle time could be extended past the
commencement of sales based on what is important to the organization and R&D's
involvement, i.e. when target efficiency is achieved, target manufacturing cost, % of product
within spec, etc.

33


Customer Satisfaction (5.5)

 Definition
 The customer satisfaction metric has two variations:

1. Measures of external (end-customer) satisfaction. These may be
such metrics as ratings of quality of technical personnel or
technical capabilities, or technology benefits within products or
processes.
2. Internal customer satisfaction. Since the immediate customer of
R&D is normally the businesses within the corporation that R&D
serves, measures such as customer satisfaction in
engineering, marketing, or manufacturing may be appropriate.
Typical metrics might include on-time technology
delivery, competitiveness or appropriateness of the technology
solutions delivered, and overall satisfaction with the R&D track
record of technological support.

34


 There are few disadvantages to good customer satisfaction metrics. In
the case of external, or end-customer satisfaction metrics, one
complication may be that the entire innovation cycle is under review by
the end customer. A bad grade by the customer, while a valid rating of
the corporate innovation process, may not be merely an indictment of the
R&D operation, but a judgment of the overall product development
process within the company, involving manufacturing and product
engineering, market forecasts, consumer needs and attitudes, and
competence of corporate management. One the other hand, a well-
thought-out customer satisfaction metric (or set of metrics) for the
internal or immediate customer within the corporation -- normally the
corporate businesses and their various R&D-related organizations -- may
be the key diagnostic to indicate that R&D processes are lacking and
need adjustment or redesign. The R&D organization itself is probably
better served with well thought-out internal metrics than with external
metrics that complicate the diagnostic process for R&D when problems
are indicated.
35


 External customer satisfaction metrics. These metrics will normally be marketing-
related or implemented. One variation would be to use a marketing survey, in
which various aspects of technology benefits are rated on a five -point scale

 There are two dimensions for internal customer satisfaction metrics: strategic and
tactical.

 Strategic metrics deal specifically with whether the R&D function is meeting the
strategic needs of the customer. The review process might involve matching
technology and product roadmaps in a joint meeting, in which technology timing
mismatches are resolved. Information can be exchanged; technology previews by
R&D to alert the businesses to possible market-creating or market share
increasing discontinuities, and the businesses to share future market window and
product definitions with R&D. Various metrics can be used, including the five-point
rating system mentioned above, or a metric which highlights number or percent of
mismatches between product and technology roadmaps.

36


 Tactical metrics deal with whether specific projects are meeting the goal or delivery
requirements of the internal customer base. For technology projects in the latter
stages of development (near to or entering product development), regular project
reviews with the intended customer(s) is important. At quarterly or perhaps
semiannual reviews, customer and R&D representatives join in a review of project
progress. A useful metric in this case is a report card which each customer
representative is required to fill out in the review meeting. This is a very simple
questionnaire which has 3-5 survey questions on project progress, and suitability
of both the project and the technological approach to satisfy the customer(s)
needs. Typical questions might be:

 Does this project meet your product technology needs?
 Do project milestone dates meet your market window?
 Have any strategy changes on your part not been addressed?
 What is the overall project score (typical scale 1-5)?
 As a related metric, trend analyses can be made both by project and organizationally.

37


 One variation on the customer satisfaction process is to establish steering teams
to address technology, business and market issues and provide guidance to the
R&D organization on strategic issues. For external issues, the teams might consist
of focus groups that meet periodically, or focus teams that convene groups which
are diverse either geographically, ethnically, or with respect to age group, for
example. The focus might be on functional needs that technology capabilities
address. The metric would be meetings held, or issues addressed and settled.
Another metric might be problems surfaced to be addressed and reported on by
the R&D team.

 Internal issues might be addressed by customer teams composed of
manufacturing, engineering, marketing, financial, and related personnel (including
even external consultants) who provide guidance and assist in forming cross-
functional project teams. Appropriate metrics are meetings held, issues settled, or
problems surfaced. An ongoing metric can also keep track of the % of problems
addressed and resolved versus those surfaced.

38


Number of Ways Technology is
Exploited (5)
 Definition
 This metric assess the number of ways a technical asset can
bring value to the corporation.

 This metric is applied to gauge project attractiveness, or to
understand the value of a technical asset already developed.
It is generally agreed that a larger number of potential
uses, both within the company's current markets and in
markets not yet developed, makes a technical asset more
valuable. The metric is a bit arbitrary and can be misleading in
cases where there are few, though very large and/or lucrative
commercializations.

39


Exploited (5)
 How to Apply the Metric
 This metric is applied by taking an existing or potential
technical asset, such as a project to develop a new type of
lower cost, light weight composite material, and conducting a
thoughtful analysis of how many ways this asset can be
exploited commercially. The count could consider:

 Number of business units in the corporation that could make
use of the asset
 Number of markets the company serves that could be
impacted by this technology
 Total number of markets served by the corporation and other
companies where the technology may have an impact.
 Number of products that could utilize the technical asset.
40


Exploited (5)
 How to Apply the Metric (continued)
 This Used in this way, the metric is a single numerical value. A larger number of
potential uses means that the corporation is not depending on a single or small
number of products to succeed in order for the technical asset to deliver value. The
risks associated with the exploitation of the technical asset are spread over a
larger number of potential uses. A larger number also provides greater opportunity
for unforeseen benefits, like taking the company into new markets and new
products.

 Using the low cost, light weight composites as an example, the primary market for
the company may be the automotive market, with four different auto parts that
could use the strong, light weight tubes produced by the new process. In
addition, there could be a market for the technology in the aeronautical
industry, served by another business unit of the company. The third exploitation
could be in high performance bicycle frames, a market that is new to the company.

41


Exploited (5)
 Options
 An option in the implementation of this metric can involve adjusting the number for
relative importance of the commercializations, or keeping sub-metrics of the
number of exploitations with certain value ranges as demonstrated with the
continuing example of the light weight composites.

42


Projects with Business /
Marketing Approval (5)

 Definition
 Percent of projects in the total R&D portfolio with explicit
business unit and or corporate business management sign-off.

 The intent of this metric is to provide an indicator of the degree
of alignment with business and corporate strategy and tactics.
The metric is closely related in some corporate structures to
metric: "percent Funding by the Business“.

43


 Advantages: Several studies have suggested that close alignment of R&D to marketing and
to business and corporate strategies increases the odds of success for new products and
processes. Thus actions which drive this metric to higher values can be expected to improve
the amount of R&D spent on successful projects and the predictability of the outcome from
R&D efforts.

 Limitations: The metric will be valuable to the extent business/marketing management and
R&D management jointly develop strategy and plans. Use of the metric to drive R&D without
such teamwork will likely lead to short term projects and suboptimal use of R&D resources. In
those companies where R&D is corporately funded, business/marketing management may
also be tempted to give approval to projects in their market segments to insure that they
receive "their share" of R&D resources. Finally if the corporation uses a formal innovation
process which requires business/marketing approval at some stage, the metric runs the risk
of becoming a measure of compliance with use of the process or a measure of the percent of
project past the approval stage.

44


 Explicit approval may be sought at any point in the innovation process.
Seeking approval early in the innovation process probably provides
maximum value. One form of approval is the provision of a sales forecast
from marketing management for each new product and agreement to
commercialize if the product meets technical requirements in a timely
manner.

 The level of approval from the marketing/business management and the
point where approval should be sought should be explicitly defined if the
firm uses a formal innovation process. If not, the level should be
commensurate with the amount of R&D resources and commercialization
resources which will be required.
45


 For projects having broad corporate strategic value, approval of a director
of corporate planning or director of corporate business development might
be an appropriate substitute for the business/marketing management
approval. For corporations where out-licensing of technology is a major
thrust, approval of a director or vice president of licensing may be an
appropriate substitution.

46


Quality of Personnel (3.3)
 Definition
 This is a measurement of the skills and ability of the R&D staff to execute strategic programs.

 Internal Customer Ratings (a)
 Internal customers rate the quality of the R&D staff on their ability to execute programs.
Measures such as percentage of mileposts met versus project plans, novelty of
concepts, patentability of concepts, and competitive advantage of the technology are
parameters that can be considered.

 External Customer Ratings (b)
 External customers rate the quality of the R&D staff of their ability to meet customer
expectations. Problem solving, novelty of approach, responsiveness, knowledge of
customer's operations are parameters that can be considered.

 External Recognition (c)
 Publications in refereed or industry trade journals, external presentations, citations in the
literature, invited lectures and patents are parameters to be considered.

47



 The internal and external customer ratings measure the ability of R&D to
meet customer expectations and contribute to the growth of the
corporation or enhance competitiveness. They are largely objective
measures that can be tied to tangible value. The value of external
recognition via patents, publications and presentations is more difficult to
measure objectively. The numbers of different subjects covered by public
disclosures should be evaluated rather than the total number of all
disclosures. It is easy to become subverted to a self servicing metric if only
numbers are considered. Maintenance of technology as trade secrets
must be considered in this evaluation.

48



 Internal customer surveys can be conducted using a 1 to 4 scale for rating. A 1
represents below standard execution on a given project. The causes for this poor
performance have to be determined since they could arise from inadequate
skills, poor judgment, lack of responsiveness, poor planning, etc. The causes may
not be related to the quality of the personnel but poor management practices.
Having multiple internal customers (marketing, manufacturing, sales, etc.), conduct
the evaluation is a form of 360o review. Superior performance by reaching targets
ahead of schedule, lower than expected costs, developing a significant competitive
advantage, etc., should be rated as a 4. In establishing this survey
system, agreement should be reached on the different levels of performance. The
survey should be applied to different projects with the same population of the
internal customers as raters. This rating should be conducted on a regular basis
and over time trends will emerge.

49



 External customer surveys should be conducted using the same 1 to 4 scale. A 1 rating
would indicate that the customer was not satisfied with the parameter being
measured, while a 4 would indicate that expectations were exceeded. Key parameters
should be selected beforehand and could include timeliness of response, knowledge of
products, knowledge of customer s operations and knowledge of customer needs. The
parameters will vary by industry. Several levels of the customer s organization should
be sampled such as plant operators, first-line supervisors and management. A simple
postcard type of survey instrument mailed after customer contact can be used. A
database can be developed over time and trends will emerge.

 The subjects covered in public disclosure should be tabulated and compared to
strategic technology goals. A subjective 1 to 4 rating system can be created by R&D
management to determine fit with the goals. A rate of 1 corresponds to a poor fit and a
need to enhance skills, while a 4 implies that all areas are being addressed.

50



 Service and consumer product companies may find the external customer survey
to be a valuable tool in assessing the effectiveness of their R&D organization. One
may also measure the quality by the number and type of external awards from
recognized organizations (ACS, AIChE, IRI ...).

51


Efficiency of Internal Technical
Processes (3.3)
 Definition
 This set of metrics seeks to provide a measure of both the efficiency and effectiveness of the
operation of R&D processes within the firm.

 Project Assessment
 The total cost of all commercially successful projects divided by the number of commercially
successful projects. (Useful when tracked over time with similar projects with similar scopes).

 The ratio of actual to projected costs (and timing) for all projects.

 Percentage of costs devoted to commercially successful projects.

 Portfolio Assessment
 The total R&D budget divided by the number of projects with commercial output. Subdivide
by projects of similar type (technical service, short term, long term) and used in conjunction
with project value assessment.

52


Processes (3.3)
 This set of metrics needs to be adapted to the needs of each firm -- considering
the goals, objectives, and priorities for the firm. Assessments need to be made for
individual projects (stage-gate , PACE, or similar processes), and for the collection
of projects (Portfolios).

53


Processes (3.3)

 Each firm will need to set the metrics relative to its specific goals and
objectives. An assessment of the selection termination and
management of projects can be made using the following four stages
for “Technical projects: Selection, termination and project
management”.

54


Processes (3.3)
 Stage 1 Technical projects:
Selection, termination and project management
A. favors short term projects
B. politically driven selection
C. no project monitoring or pre-project planning
D. little inter-functional participation in project teams
E. erratic turnover of team staffing
F. project leader roles not defined
G. no training for project leaders
H. unclear charters for project teams

55


Processes (3.3)
 Stage 2 Technical projects: Selection, termination
and project management
A. mix of short and medium-term projects
B. no inter-product-line analysis
C. priorities set erratically
D. project tracking
E. some inter-functional participation but not all key
functions represented.
F. formal release process for new products
G. some project team stability but conflicts over work
H. priorities
I. project leaders given only minimum guidance or training
56


Processes (3.3)
A. selection based on multiple inputs from internal and external
sources
B. balance of short-, medium- and long-term projects
C. risk analysis incorporated at key phases
D. projects still schedule driven
E. Inter-functional teams wherever needed
F. clear allocation of project and functional responsibilities
G. training for project leaders

57


Processes (3.3)
A. clear links between selection criteria and business and product-line
strategy
B. disciplined process for project termination
C. cross-functional planning and execution
D. continual improvement- postmortems, quality measures of both project
process and product
E. performance
F. projects are milestone driven
G. differentiated project management procedures for different types of
projects
H. scheduling and capacity planning avoid resource
contention by competing projects
58


Product Quality and Reliability (3)

 Definition

 Customer or Consumer Evaluation (a)
 Relative quality and reliability compared to competitive
products through evaluation by customers or consumers.

 Reliability/Defect Rate Assessment (b)
 Fraction of a firm's output, either by individual product or by
sum of all products, that meets or exceeds the established
quality standards.

59



 The advantage of this metric is that product benefits resulting from
R&D activities are directly evaluated by the customer or consumer.
Comparison with the competitors' products is usually the basis for
evaluation. Limitations of the metric chiefly are related to the
reliability and accuracy of the survey techniques chosen as
appropriate for the industry, though firms usually gain confidence in
their preferred methods through repeated use and incremental
improvement.

 Similar to the preceding discussion, the advantage to this metric is
that the direct benefit from R&D activities can be obtained through
specific measurements made by the firm.

60


 For the Customer or Consumer Evaluation metric, each firm will generally have a
preferred technique for directly or indirectly obtaining data showing how well the firm's
products perform in comparison to competitive products. Data for the Reliability/Defect
Rate.

 Assessment result from internal quality measurements. When both metrics are
utilized, the impact of product quality improvements on customer satisfaction should be
demonstrated. Product Quality and Reliability metrics are retrospective, showing the
results of past technology or product introduction to the market.

 This metric fits well with the trend toward greater input to R&D planning from customers
and consumers and with the "Quality" protocols that have adopted by firms of all types.
Though basically a retrospective measure, product needs that arise during the data
collection can be used for prospective purposes.
61


Market Share (3)
 Definition
 Firm (or business unit) market share in various product
categories measured as appropriate for the industry or
category, expressed as a percentage of the total market.

 This metric is meant to reflect value creation for the firm and
the value of the firm's technology. Similar to Gross Profit
Margin metrics, caution must be taken in the interpretation of
Market Share data from the perspective of measurement of
the contribution of R&D activities to the whole. There can be
many confounding factors in a market share
determination, such as the size and quality of the marketing
effort, the competitive response, the relative state of the
economy, etc.
62


Market Share (3)
 Changes in Market Share should be assessed at least annually to
determine the rate of progress or decline. The expectation is that
improvement in a firm's technologies and products will result in a
greater share of the market. Market Share is a retrospective
metric, showing the results of past technology or product introduction
to the market. Since competitors' market share data is usually also
available, this is a metric that can be used as a benchmark with the
competition.

 As an indication of threats or opportunities, share data in markets
related to a firm's products can be followed. This "Related Market
Share" metric can serve as a component of a strategy to anticipate
the potential application of similar technology into a firm's
marketplace.

63


Development Pipeline Milestones
Achieved (2.3)
 Definition
 Development Pipeline Milestones Achieved is a metric which is useful in
grading the effectiveness of management and planning of each R&D
project. It may also be used as an indicator of performance problems on a
given project which can be used to initiate diagnostic and recovery
procedures to give each project the best possible chance of success.
There are two possible variations of the metric which may be used:
1. Percent of project milestones achieved -- the percent, by project or overall by
sub-organization or laboratory, of all project milestones completed on schedule
or within some acceptable time window (90 days, for example) of the forecast
date. Trend studies can then be established for organizational performance
based on analysis of the data collected by quarter or for whatever other time
period is appropriate for the particular industry group.
2. Performance level at each milestone -- on a project basis, percent of all
expected objectives met at the milestone date by which they are forecast to be
completed.

64


Achieved (2.3)
 The primary advantages of this metric are for project diagnostics and as an
indicator of the overall planning and management health of the organization.
There are several possible limitations. First, a consistent management system
must be in place to assure that the variability of the number/quality/difficulty of
the milestones does not cause random fluctuations in the metric. Secondly, the
calibration of the metric is important. Since both the stage of the R&D project and
the industry of the business involved can greatly affect the trend analyses, the
metric user must be careful to identify what the real danger signals are in terms
of management and planning deficiencies, and what really constitutes an
indicator of project problems for this metric to be useful.

65


Achieved (2.3)
 Percent of project milestones achieved -- a simple approach is to tabulate
milestones met on schedule, 1-90 days late (which may constitute on schedule in
some cases), 90-180 days late, and so forth. The tabulation can be done
quarterly, and the trend information publicized to the organization to focus
attention on performance issues.

 Performance level at each milestone -- This is more useful as a project
diagnostic. One approach is to require project managers to use this metric to
summarize performance quarterly or semiannually at an operations review and
be prepared to cover diagnostic or recovery steps if a given project shows trends
over two or more time periods of failing to meet an acceptable performance level
(which will depend on the R&D discipline, the industry, and the stage of the
innovation process).

66


Achieved (2.3)
 One problem with using milestone achievement as a key metric is that
pressure is exerted to make all the milestones. With strong pressure to meet
milestone dates, the project manager may be tempted to populate the
project plan with easy milestones, to assure that his milestone hit rate is
good. It is important therefore to make sure that milestones are realistic and
aggressive. Especially in the later stages of the innovation cycle, in product
design and productization or manufacturing process development, it is
important to assure that the business customer, marketing
representative, etc. approve the project milestones to assure that the product
development cycle meets the market window requirements of the business.
In this case, a much closer tracking of milestone achievement may be
necessary than in the earlier stages of the innovation cycle (for
example, monthly or in some cases even more frequently).

67


Achieved (2.3)
 Options and Variations (continued)
 Another problem with using the milestone approach is that early in the innovation
cycle (the R part of R&D), milestones are often hard to define and even harder to
forecast. With the heavy level of uncertainty in a long-term research project, the
definition of a milestone may be vague, or even made in terms such as define or
develop concept. In these circumstances, project managers feel especially
uneasy in setting milestone dates for which they become accountable. An
approach here is to realize that (1) exact dates are less important, since product
or market need windows probably have not been established (and in some
cases, market needs or even product existence!), and (2) many of these projects
which are early in the innovation cycle will certainly fail, since in the research
phase, many more ideas are explored than have a positive outcome. In this
context, using milestones as a performance metric may not be useful. If
used, the metric should probably be relaxed in some way. For example, instead
of setting a goal of completing 100% of milestones, perhaps a goal of 50, 60, or
80% may be set. Another possibility is to declare any milestone made within 90
or 120 days of the forecast date to be on time. The specific approach taken will
depend on the industry, the research area, and the stage in the innovation cycle.
68


Goal Clarity (2)
 Definition
 This metric uses an interval rating scale assessing the extent to
which project performance objectives are clearly defined and
understood by all participants on the project team.

 While this metric may provide a semi-quantitative assessment there
is a degree of subjectivity for an individual to assess the extent to
which he/she really understands objectives and roles on the team.

 How to use the metric
 A member of the team would provide to all members of the team a
survey form to assess understanding of the project objectives and
commitments. This can be done using some interval rating scheme.
It would be wise to conduct this survey several times through the life
of the project to really assess the level of understanding.

69


Goal Clarity (2)
 A four stage assessment can be constructed with the following as the
highest stage for individual team members:

a) clear understanding of the expected product of team effort
b) personal belief that a team is the right way to develop/achieve the expected
product
c) understanding whether the team is an implementation, recommendation and/or
informational team
d) understanding the team's operational ground rules and end-result boundaries
e) belief that the team has all the appropriate
knowledge,functions, diversity, levels and locations, so that the expected
product of the team effort can be achieved using the minimum number of
people?
f) high personal commitment to achieving the objectives
g) clear understanding of personal role
h) acceptance of the recognized team leader
i) capability to draw on additional resources to keep the core team to a minimum
70


Comparative Manufacturing Cost (1.8)

 Definition
 Benchmarked manufacturing cost data vs. competition for
same type of unit cost.

 It is extremely useful and in some companies paramount to
know how R&D is helping to provide an advantaged cost
position to the operations. This metric reflects the
quantification of that goal. Unfortunately, cost accounting and
even further comparative cost accounting can be extremely
difficult.

71


Comparative Manufacturing Cost (1.8)

 While most firms have very accurate manufacturing cost data
for themselves, the generation of accurate manufacturing cost
data for competition is considerably more difficult.
Therefore, when using this measurement, there should be an
estimate made of the variance of the competitive estimates.

 This metric is intended to be based on unit process
comparisons. There can be many options created that are
aggregates of production processes, but these simplifications
can be misleading. Therefore caution is urged.

72


Gross Margin (1.8)

 Definition
 Gross Profit as a percentage of sales, where gross profit
equals net sales minus cost of goods sold (product costs plus
direct manufacturing costs).

 To some extent, gross profit margin reflects value of the firm's
technology assets and the value created by R&D.
However, raw material, production, and distribution costs also
directly affect the firm's gross profit margin. Each firm should
attempt to understand the correlation of Gross Profit Margin to
R&D.

73


Gross Margin (1.8)

 Value assessment should be based on change in gross profit
margin from period to period. (Periods should be appropriate
to an industry and may be in excess of one year.) Changes in
Gross Profit Margin in relationship to changes in values of
other metrics (e.g., financial return , technology transferred to
manufacturing, sales protected by proprietary position) should
be followed in an attempt to uncouple the contribution of R&D
from other factors. This is a retrospective metric that can be
used as a benchmark with the competition, if gross margin
data from competitive firms are available.

74


Use of Cross-Functional Teams (1.8)
 Number of Cross-functional Teams
 Current management philosophy suggests that the use of cross-functional
teams will improve the effectiveness and efficiency of the R&D Process and
will help R&D to be integrated with the business. The number of such teams
can be counted if they are established on a formal basis.

 Evaluation of the use of Cross-functional Teams
 A firm may rate its practice of using cross-functional teams by using a
subjective scale ranging from 1 to 4. Firms operating at Level 1 are
characterized by the existence of strong organizational boundaries, lack of
cross-functional involvement in R&D projects, and no cross-functional team
structure. Firms operating at Level 4 are characterized by a well developed
and supported team structure which effectively places all R&D work in
cross-functional teams responsible for the entire project rather than
functional silos responsible for parts. This metric is one of many relating to
how well the firm conducts the Practice of the R&D Process. It also relates
to how well R&D is integrated with the business.

75


Use of Cross-Functional Teams (1.8)
 This metric will be important if the use of cross-functional teams contributes to
the effectiveness of the R&D function in the firm, given its peculiar situation. This
is usually thought to be the case, but there may be cases where other factors
are more dominant.

 Evaluation of the use of teams can be used to diagnose problems with the
organization or the R&D process and to plan for improvement.

 How to use the Metric
 A simple count of teams is rarely as valuable as an assessment of the how the
firm uses such teams. This can be accomplished using the rating scale
suggested above. Input for the evaluation should be gathered from a broad
cross-section of the firm. The evaluation is important for planning improvements.

 Trends across time are probably more valuable than benchmarking.

76


Rating of Technology Features
and Benefits (1.8)
 Metrics for Product Features & Benefits:

 Metric 1: Competitive Technical Performance of Product (Project
Metric)
 Comparison of technical performance of a product in those dimensions
where the customer is likely to perceive a benefit. This may be used in a
prospective sense to appraise the value of a feature or in a retrospective
sense to register the value of a benefit which the market has recognized.

 Examples include:
 The use level required to achieve a needed result in the customer
application.
 The yield strength of a high-performance alloy
 The measured softness provided by a textile softener.
 The measured UV resistance of an external architectural coating.
77


and Benefits (1.8)

 Metric 2: Customer rating of Products (Business Segment or
Firm)
 Customer rating using a scale of 1-5 of the technology benefits that is
perceived in a firm‟s products. This can be compared to the rating for
the best competitor, usually in the form of a ratio. This is an
aggregate subjective measure for a business segment or for the firm.

 Metric 3: Economic Value of Products (Project, Business
Segment, or Firm)
 This metric is the price differential per unit obtained by virtue of the
technology feature minus the cost of providing the feature. The
differential can be multiplied by the volume to assess the total benefit
to the firm.
78


and Benefits (1.8)

 Metric 4: Market Share Evaluation (Business Segment or
Firm)
 If differential pricing does not occur, the advantages of superior
product technology can appear as differential market share. In this
case, the relative market share (the firm‟s share divided by the
largest share) can be used as a surrogate for the value of technology
embodied in the products.

79


and Benefits (1.8)
 Metrics for Process Features & Benefits:

 Metric 5: Competitive Technical Performance of Process
(Project Metric)
 Comparison of technical performance of a product in those
dimensions which are important to manufacturing cost or product
performance. This may be used in a prospective sense to appraise
the expected value of a new or improved process or in a
retrospective sense to register the demonstrated value.

 Examples include:
 Manpower requirements
 Efficiencies of raw material conversion
 By-product or co-product costs or values
 Consistency, controllability, and other such quality parameters.
80


and Benefits (1.8)
 Metrics for Process Features & Benefits:

 Metric 6: Economic Value of Processes (Project, Business
Segment, or Firm)
 The differential in profitability (versus the target or competitor)
attributable to new or improved process technology.

 Metric 7: Profitability Evaluation of Processes (Business
Segment or Firm)
 In the same way that Market Share is a surrogate measure of
product performance in those business areas where the basis of
competition is product performance, overall profitability in a business
segment of the firm is a measure of process performance, in those
business areas where the basis of competition is cost and/or quality.

81


and Benefits (1.8)
 Product Metrics 1 through 4 attempt to assign value to technology used in
products. However, differential value or market share are normally the result
of many different factors. These metrics can give some indications if factors
are carefully sifted, but may be misleading if the analysis is superficial.
Objective measurement of product performance and customer ratings relative
to competitors are the most accurate measures of product technology. But
note that comparison of features that the market has not recognized as
benefits may be self-serving and deceptive. Competitive rankings of
measured product performance and or customer ratings may be averaged
over market segments or over the firm to obtain average values.

 Process Metrics 5 through 7 attempt to assign value to new or improved
process technology resulting from R&D. However, economic value and
profitability are normally the result of many different factors. Objective
measurement of process performance is the most accurate measure of
process technology.

82


and Benefits (1.8)
 How to Use the Metric:
 Metric 1
 Define the key parameters which measure features the customer is likely
to perceive as benefits. Measure product performance as accurately as
methods allow. Compare to the same measurements of competitive
products. Rank performance versus best competitors.

 Metric 2
 Ask customers to rate the technology attributes of a product line
relative to solving their problems. This requires a carefully constructed
survey instrument.

 Metric 3
 This metric is valid for products which are differentiated by
performance. It is not applicable to commodity products.
83


and Benefits (1.8)
 How to Use the Metric:
 Metric 4 (See Definition)

 Metric 5
 Define the key parameters which measure or impact cost or quality. Measure
process performance as accurately as methods allow. Compare to the same
measurements of competitive processes. Rank performance versus best
competitors.


 NOTE: These metrics may be used retrospectively to measure the output of R&D over the past period
and prospectively to set targets for future accomplishments.
84


Response Time to Competitors
Moves (1.8)
 Definition
 This metric measures the ability of the firm to respond to new technical innovations
introduced by competitors. Depending upon corporate strategy, it could be the time
required to match or exceed the competitive offering.

 This metric is an indicator of technical leadership in a given field. The technical leader
will not spend a significant amount of time matching competitive innovations, while a
follower will be more reactive than proactive. The utilization of resources for this
function as a percentage of total resources should be tracked over time. An increase in
this percentage would indicate that new technology programs are not as effective as
desired and technical position relative to the competition is eroding. This metric also
measures the flexibility and creativity of the firm to change priorities to meet competitive
challenges. A strong market intelligence function is required to identify competitive
entries at an early stage in the introduction and to assess the technical merits of the
offering so that appropriate responses can be made. The technical merits of
competitive offerings must be critically assessed to differentiate from market
repositioning of existing technology.
85


Moves (1.8)
 The time between the introduction of a competitive offering and the internal
development of a comparable or superior offering can be measured and
compared to product development times for similar products. A rating system of 1
to 4 can be used.

1. Organization slow to recognize significant competitive offering in the marketplace; slow
to launch program to respond; unable to get their offering into the marketplace in
acceptable time to be combatant. Slow to recognize impact of technical innovations.
2. Organization recognizes need for competitive offering; has trouble in launching program
to develop counter-offering and gets offering to market barely in time to have impact.
3. Organization responds to competitive offering and develops counter offering to maintain
relative position.
4. Organization anticipates potential for competitive offering and has counter offering into
the marketplace with superior product allowing a gain in market/competitive position.
Very fast to recognize impact of any hints of technical innovations.

86


Moves (1.8)
 The percentage of resources used in matching competitive moves could be
measured over time. An increase in spending in this function should raise
questions regarding core research and development efforts for organizations who
strive for technical leadership.

 The importance of this metric will depend upon the technology strategy of the
firm. Firms pursuing technical leadership will be very interested in minimizing
response time. The importance may vary across different strategic segments in
the same firm.

87


Comparative Technology
Investment (1.5)
 Definition
 This measures the current annual expenditure for R&D staff and capital compared to
the best competitor and/or the industry average.

 This metric measures the rate of current activity in developing the technology of interest
with the intent of predicting whether the firm is expected to gain or lose ground in the
technology. It should be kept separately for the KEY and PACING technologies most
critical to the strategy. Retrospectively it measures the efficiency of the investment in
meeting new product and technology development goals.

 This metric should be as quantitative as possible, but in some industries it may be
necessary to make estimates as to the size of the development effort of the best
competitor and the industry average. Analyses must be based on a comparison of
similar functions. For example, some organizations include sales support as part of
their report for technology expenditures. The components of the expenditures under
study must be understood in making the comparison.
88


Comparative Technology
Investment (1.5)
 Information on a firm s overall technology expenditures are available in the firm s
annual reports or industry publications. These can be used for comparison to
internal overall investment. Information by industry is available in industry
publications and from organizations such as IRI (IRI/CIMS survey).

 Rationing the firm‟s current investment in technology versus the best competitor
and industry averages provides an insight into the efficiency of the technology
investment. Performance exceeding expectations in value creation goals at
competitive investment rates indicates an efficient organization while sub-standard
performance raises concern about the quality of the investment. Smaller firms may
require an investment higher than industry norms to maintain a competitive
position to offset critical mass issues.

89


Employee Morale (1.5)

Definition
 This metric takes quantitative ratings of key aspects of employee
satisfaction and morale as shown by direct employee survey. It is
recognized that employees may feel good and have high morale, yet
produce nothing of value for the firm -- the real question is are they
motivated and committed to create and innovate profitably?

Advantages and Limitations
 Extensive surveys are time consuming and expensive to conduct and
must be conducted with sufficient frequency to establish base lines and
understand real trends. One must understand also that technical
populations tend to have certain biases in such surveys. Many employees
feel "surveyed-out".
90



How to Use this Metric
 The typical survey uses five point scales for agreement (strongly
agree, agree, neither agree or disagree, disagree, strongly disagree) for
importance (extremely important, very important, somewhat important, of
little importance, not at all important) and for performance (very
good, good, fair, poor, very poor) in answering sets of questions related to
work environment, feelings about the company, ratings of the
company, ratings of the organization/work location/work group, feelings
about the individual's job, and general satisfaction. Opportunities are given
for comments. Such extensive surveys are most often conducted by third
parties to maintain confidentiality.

91



 How to Use this Metric (continued)
 Four tested questions of job satisfaction are:

1. If a good friend was interested in a job like yours for your firm, what
would you tell the friend?
2. All in all, how satisfied are you with your present job?
3. Knowing what you know now, if you had to decide all over
again, would you take your current job?
4. 3.4 How satisfied are you with the overall employee- employer
relations at your firm?

92



 One firm which uses e-mail extensively, every 9-14 months
conducts a broad "pulse" survey and asks employees to
provide two ratings (using 0-10 point scales with descriptors)
one rating the employees work climate/environment and the
other rating personal feelings about the work itself. Employees
spend less than 4 minutes to reply by e-mail, or to be
anonymous by fax or to a voice mailbox. Employees often
write additional comments that give information sought in the
more extensive surveys. Confidentiality is assured and rapid
feedback (within 2 weeks) of survey results to participants
maintains a high level of interest and participation.

93


Metrics for the Stage Gate Model
 Definition of the Innovation Process
 The term "innovation process" refers to the overall process in a company for
conceiving and developing ideas and concepts into profitable products and
processes for the company business(es). It is one of the 5 to 7 overall high-
level business processes within the corporation. In order to
maintain, diagnose, and improve the process, both effectiveness and
efficiency metrics are required to support decision-making.

 Note that in the context discussed herein, the assumption is made that the
innovation model will be technology-based. That is, innovation is founded in
the efforts of the R&D organization, and that the overall function of the
process is the embodiment of technology in the products of the business of
the corporation. Progress through the full innovation process will involve the
efforts of all corporate resources, including
marketing, engineering, manufacturing, and service and support functions.

94


 Relevance of Innovation Process Model to Metric Choice
 Often stakeholders in the innovation process -- R&D management, financial
management, and corporate officers -- need to utilize metrics which supply specific
decision support information about corporate capabilities in innovation. These
metrics are overall, or outcome metrics, which provide a grade or rating of the
overall result of the innovation capability of the corporation or business in question.
Such metrics might relate to value creation by the process, effectiveness of the
overall process in terms of new product revenue or profit versus total R&D
investment, or the ratio of new product revenue or profit to total company or
business revenue or profit.

 However, these metrics are often long-term and retrospective in nature, and while
they serve to evaluate the overall process as a whole (and generally, but not
always, on a lagging basis), they may offer little, if any, insight into the current or
future state of the corporate innovation process or its many and diverse activities.

95


 Relevance of Innovation Process Model to Metric Choice (continued)
 To be useful for the on-going management and evolution of corporate innovation, a
diverse set of metrics must be available which offer insight into current condition
and future vision state of the process and many of its constituent parts. For many
such metrics to be useful, they must be not simply outcome-oriented, but process-
oriented, and specifically diagnostic of discrete sub-processes within the overall
innovation process.

 Each in-process metric must be related to a process model which ties it to the
portion of the process of interest, from early idea or technology exploration through
proof of concept to technology development to productization. The following
process description is a relatively simple process model and defines the various
sub-processes within that model in order to provide anchors for useful metrics
within the innovation process.

96


 Overall Description of the Stage Gate Model for the Innovation Process
Used in this Metrics Guide
 There are a number of possible representations of the corporate innovation process. One
simple yet fairly effective representation is the so-called "stage gate model".

 The model in Stage Gate divides the overall innovation process into four stages, from
concept exploration or "ideation" to the process of productization or commercialization. This
division into four sections is arbitrary. An argument can be made for a three-stage model, or a
more complex model of multiple sections and sub-sections. The four-stage model is chosen
primarily for convenience, and to match a number of corporate models and those defined by
scholars in the area of business processes. The shape of the model recognizes several
realities of modern business.

 For example, the process funnel narrows as the process proceeds from exploratory research
or concept exploration through proof-of-concept to technology development and
commercialization, illustrating pictorially the selective filtration that occurs in the process.
There are typically many more ideas and concepts that are explored than are developed into
significant technology capabilities in a business, and fewer still that emerge finally into
finished products. In each stage of the process, the candidates dwindle, until only the most
promising are brought to full production.
97


 Overall Description of the Stage Gate Model for the Innovation Process
Used in this Metrics Guide (continued)
 Similarly, the four gates, labeled A, B, C, and D, stand for the tests or decision-making
activities that are exercised between the stages in the innovation process. Economic reality
imposes a limit on the total level of technology and product development that each company
can support. A second reality is that as each decision gate is passed, the resources and
funding required to carry a given project to the next stage increases dramatically; a good rule
of thumb is a 10X expenditure increase at each gate. These corporate "facts of life" impose a
stringent set of conditions on the suitability of R&D programs which approach a gate. Each
gate requires a set of metrics which ensure that only those programs most suitable to meet
corporate business needs pass into the next sub-process.

 The following sections describe the decision gates and stage processes shown in the stage
gate model. Underlined terms link to appropriate metrics for the gate or process under
discussion.

98


 Description of the Major Sub-process Divisions in the Innovation
Process Model
 As noted in the introduction to the process model, the division of the innovation process
model is arbitrary, but in general it seeks to approximate reality. The four divisions cover the
very early technology or idea exploration phase of innovation (when possibilities are
defined), a proof-of- concept phase when the mapping of ideas into the realities of the
business world occurs, the development, and finally commercialization. Each of the separate
phases, or sub-processes, is covered in the following sections, together with the entry gate
which defines conditions for admission to that sub-process. Refer to the stage gate model in
the references that follow.

99



Gate A
Gate B
Gate C
Stage I Gate D
Stage II

Stage III Stage IV

100



101


Stage-Gate Model Description
 Process Entry Gate (A) and Exploratory Concept Sub-process (I)
 Gate A is the entry not only to Process I but to the entire innovation process. The purpose of
Process I is to explore new ideas and concepts and set in motion as many promising "seed"
projects as possible. The cost of research and investigation is small here. In companies
where exploratory concepts consist of exploring product ideas and concepts, a single worker
may have a project or even several in work simultaneously. Where true basic research is
involved, it is most often at the University level, with industry participating through
grants, contracts, or research agreements. In either case, there are typically many avenues
being explored and no valid idea or concept is neglected.

 The main consideration at Gate A is whether the idea or concept is strategically appropriate
(at this stage, the alignment to corporate business goals may be ephemeral in some
cases), and whether the expertise available to address the concept or idea is adequate.
Metrics at Gate A should address these issues.

 The purpose of Sub-process I is the validation of concepts or physical principles. Metrics for
this process should simply address the validity of results and whether or not basic principles
are established. Competency metrics may also be valuable to support assessments of
required resources to execute projects.

102


Decision Gate A
 Metrics which can be used to
make good decisions at
this gate
 Strategic alignment  Metrics assessing the
 Financial return efficiency and effectiveness of
 New sales ratio the decision making process
 Cost savings ratio  Delayed Stage kills
 R&D yield
 Cycle time (Decision)
 R&D return
 Decision Gate processes
 Distribution of technology
 Gate effectiveness
investment
 Core technical competency
 Criteria for overall business
success
 Supporting R&D metrics

103


R&D Process Stage I
 Metrics assessing the efficiency of the
Stage I process
 Cycle time
 Market cycle time
 Project management cycle time
 Quality of personnel
 Internal customer rating
 External customer rating
 External recognition
 Published works
 Milestones achieved

104


 Conceptualization Gate (B) and Proof-of-Concept Sub-process (II)
 Gate B primarily tests whether a concept is validated or some physical principle has been
established. Entry to Sub-process II depends mainly on appropriate resources being able to
establish proof-of-principle in a business context, and whether the concept or idea to be
tested has potential application areas within the business goals of the Metrics should test
these concerns. The "filtration" function at this gate is fairly strong, since although the cost of
research in Sub-process II is still not great, there will be many more candidate ideas than
there will be resources to address them.

 In Sub-process II, the emphasis is on proof-of-principle in real business applications.
Business considerations such as market window and competitive reaction begin to be
important, although there will still be concern about options and possible spin-offs of the
technology or concept. Project metrics for tracking milestones and execution time become
more important, although time frames are still lengthy and product requirements may be
vague or broad-brush. Metrics concerning resources and technical capability are also
appropriate.

105


Decision Gate B
 Metrics which can be used to
make a good decision at
this gate
 Strategic alignment
 Financial return  Metrics assessing the efficiency
 New sales ratio and effectiveness of the
 Cost savings ratio decision making process
 R&D yield
 Delayed Stage Kills
 R&D return
 Distribution of technology  Cycle time (Decision)
investment  Decision Gate processes
 Development pipeline milestones  Gate effectiveness
achieved
 Percent achieved
 Performance level at each
 Criteria for overall business success
 Supporting R&D metrics
 Core technical competency
 Probability of success

106


R&D Process Stage II

Stage II process
 Cycle time

107


 Technology Development Gate (C) and Product/Process Development
Sub-process (III)
 The emphasis at Gate C is in suitability for product development. Gate C is also a strong
filter; projects that pass this gate will be the few that are highly promising for
commercialization and meet all the requirements for profitable business products. Candidate
technologies which pass this gate will have forecast long-term corporate benefit, and the
projects entering Sub-process III meet all the strategic requirements of fit, alignment, and
attractiveness for the business. Metrics for Gate C must address these strategic
requirements, as well as the tactical issues of assuring that the candidate projects have
successfully completed the requirements of Sub-process II.

 In Sub-process III, the emphasis shifts to harder-edge issues, such as timing and execution
to assure that market windows are met and product needs are satisfied. There is also
emphasis on maintaining and extending technologies to keep a competitive edge in the
marketplace. Milestones are important due to cycle time issues, and project funding must be
managed more carefully due to budgets which are typically millions of dollars rather than the
100X lower investment that may be typical on a project in Sub-process I. Metrics for Sub-
process III must reflect the concerns of timely, accurate execution and tight budget control.

108


Decision Gate C
 Metrics which can be used
to make a good decision at
this gate
 Strategic alignment  Metrics assessing the
 Financial return efficiency and effectiveness
 New sales ratio of the decision making pr
 Cost savings ratio  Delayed Stage kills
 R&D yield  Cycle time (Decision)
 R&D return
 Customer satisfaction
 Gate effectiveness
 Milestone achieved
 Response time to competitors
moves  Criteria for overall business
 Core technical competency success
 Probability of success  Supporting R&D metrics

109


R&D Process Stage III
Stage III process
 Cycle time
 Customer satisfaction
 External
 Internal
 Product quality and reliability
 Customer evaluation
 Reliability/defects
 Use of cross-functional teams
110


 Product Launch Gate (D) and Commercialization Sub-process (IV)
 Gate D is the last test prior to full product launch in most cases. Where concerns -- and
associated metrics -- at the first three gates will have been primarily strategic (fit with
corporate goals and competencies, strategic timing, alignment with business need), the test
at Gate D is primarily whether execution in Sub-process III was timely and efficient. Concerns
for entering full productization are about whether all major technical hurdles are cleared, and
whether commercialization costs allow for profitable entry into the marketplace. Some
strategic questions must still be addressed, including market need and timing, and metrics
utilized here must address both the strategic and tactical issues.

 Sub-process IV is obviously market-oriented, with careful management of commercialization
and product costs, timing, and execution the key issues. Since budgets of hundreds of
millions of dollars may be at stake, program and resource management are paramount, and
metrics appropriate to the concerns will be chosen.

111


Decision Gate D
 Metrics which can be used
to make a good decision at
this gate  Metrics assessing the
 Strategic alignment efficiency and effectiveness
 Financial return of the decision making pr
 New sales ratio
 Delayed Stage kills
 Cost savings ratio
 Cycle time (Decision)
 R&D yield
 R&D return
 Customer satisfaction  Gate effectiveness
 Milestone achieved
 Response time to competitors  Criteria for overall business
moves success
 Probability of success  Supporting R&D metrics

112


R&D Process Stage IV
 Metrics assessing the efficiency of the Stage IV
process
 Technology transfer to manufacturing
 Cycle time
 Use of cross functional teams

113


 Gate Efficiency
 A key concern at each gate is that decisions be as efficient as possible. That is, that projects
passed through each gate be good candidates to benefit the business, providing the
subsequent sub-processes are navigated successfully. Since the "filtration" function of the
decision process at each gate will be partially imposed by the economic constraints of a
business (only so much total funding is available for the projects in each stage), even some
"fit" projects will simply "miss the cut" at each gate. However, those projects which pass the
fitness criteria, whether actually entering the next stage or not, represent a measure of the
efficiency of the previous gate decision process. A metric (discussed in the section on stage
gate metrics) may be developed in terms of the percent of successful projects in the next
stage for the previous gate.

114


Overall Process Efficiency Metrics

 Metrics assessing the efficiency of the overall process
 Use of project milestones
 Cycle time
 Cost versus budget
 Efficiency of internal technical processes
 Use of cross functional teams

115


Technology Value Pyramid (TVP)

116


 Technology Value Pyramid consists of a hierarchy of categories of
metrics as follow:

1. Value Creation : Metrics which directly demonstrate the value of R&D activities to
the positioning, profitability and growth of the corporation and creation of
shareholder value.
2. Integration of R&D with the Business : Metrics which indicate the degree of
integration, the commitment of the business to the R&D processes and
program, teamwork, and ability to exploit technology across the organization.
3. Portfolio Assessment : Metrics which communicate the total R&D program and
which allow optimization of the total program for the corporation's benefit.
4. Value of Technology Assets : Metrics which indicate the technology strength of
the corporation and relate to the probability of success versus competitors for the
opportunities which have been selected by the firm. An indicator of the potential
for the creation of future value.
5. Practice of R&D: Metrics which relate to the overall efficiency of the R&D
processes.
117


 Supplemental Explanation and Implications for R&D Management
 The TVP encompasses descriptors and a potential 'menu' of metrics of the
fundamental elements of R&D and the relationships with business results in the
short and long term. Not all measurements are right for all companies, or at all
times, for describing or tracking the most important aspects of R&D.

 This is parallel to a comprehensive financial model of a business that is
composed of many more descriptors and 'metrics' than are necessary for
examining the critical factors of the moment that are needed to guide the
decisions of senior managers and investors.

 From an R&D perspective the critical factors of the moment are dependent on the
situation of the company, the perspective required and the basic dynamics of the
model.

118


 Top Down and Output Oriented
 The model provides a top-down perspective that is output oriented. 'Value Creation' indicators are the
prime drivers of overall business returns that are derived from technology-based new
products/processes, predictors of business growth and (implicitly) a critical aspect of the soundness of
strategic business reviews. These indicators are used to answer critical questions, such as:
 Are we spending the right amount on R&D?
 Are we getting good returns on our R&D?
 If the 'Value Creation' indicators are being maintained or going up:
 The corporation has the likely raw material to extend a technology- based or innovation-
based growth program;
 The investors have the possibility of an extended stream of positive returns from the
accumulation of financial pay-offs from technology- based innovations; and
 The R&D units enjoy the likelihood of consistent funding to reinvest in various aspects
of technology application for the near term and base building for the future.
 The key words here are likely or possible. 'Value Creation' is a necessary but not sufficient condition for
growth. It is also only a measure of the moment, whether it is looking to the past or to the future.
And, any downward movements will predict the difficulties the business will have in achieving solid gains
against the competition. These indicators are crucial to assessing the total returns from R&D
investments, whether enough is being spent on R&D, and what is the likely future value of the company
from a technology perspective.

119


 Transformation Strategies Drive ‘Value Creation’
 'Value Creation' is the result of an accumulation of effort by R&D and the business to produce new ideas
and to put the best ones into practice. Due to the lag effects and to all of the factors that are
involved, momentum is built into these factors over time. They will change, but not rapidly. Change is
caused by the drivers of 'Value Creation'. The drivers are strategies that transform the R&D foundations
of competencies, know-how, etc. into specific projects and implementation. These are the strategies that
are represented by the 'Portfolio Assessment' and the 'Integration with Business' indicators.

 When 'Value Creation' is very positive, these strategies are most likely working well. When 'Value
Creation' is going in the wrong direction, look first to these areas as to the cause. In fact, one could look
at 'Value Creation' indicators simply as the double integral of these strategy indicators over time.

 Given this importance to 'Value Creation', it is no surprise that companies have routinely focused in
recent years on methodologies and activities dealing with 'Portfolio Assessment' and 'Integration with
Business' issues. These are correctly perceived as the means to (the ends that) improve the future
stream of results from R&D.

 'Portfolio Assessment' indicators describe the state of the various pipelines that run through the R&D
enterprise, as well as the targets that are being pursued. They provide a view of how the R&D $ are
being spent in terms of the timing, risks and returns that are possible. The 'Portfolio Assessment'
indicators are a prime place to look for answers if there are problems with 'Value Creation', competitive or
market share problems, or if there are problems with internal satisfaction.

120


 Transformation Strategies Drive ‘Value Creation’ (Continued)
 Some of the critical management questions affected by the dynamics of these indicators include
the following:
 Are we allocating the R&D budget optimally (to the elements of the portfolio)?
 Are we maximizing our investment yield from R&D?

 Unlike 'Value Creation' indicators which tend to have modest levels of momentum associated with
them and provide significant underpinnings to the business returns, the 'Portfolio Assessment'
indicators can vary quickly and have little immediate effect on the business. Their major effects
are cumulative. This dynamic often leads to short term, risk averse behaviors that over time
undermine the 'Value Creation' indicators. Maintaining an aggressive monitoring of the 'Portfolio
Assessment' indicators is extremely important to the long term support to 'Value Creation'.

 Similarly, when there are problems within the portfolio that are corrected, it is necessary to give
the solution enough time to work.

 If 'Portfolio Assessment' indicators show strategy by what categories of R&D and targets are
being developed, the 'Integration with Business' indicators show strategy of how it is being
done...and, consequently, with what level of quality and execution.

121


 Transformation Strategies Drive ‘Value Creation’ (Continued)
 The 'Integration with Business' indicators focus on process, culture, teamwork and organization.
They also touch on many of the aspects of cycle time. The issues addressed by these indicators
change slowly in reality and are probably the true pacing factors that are applied by the
organization to the 'Portfolio Assessment' indicators that, in turn, put limits on the 'Value Creation'
that is realizable.

 These indicators are often the subject of efforts aimed at TQM, cycle-time reduction, or re-
engineering. They are difficult to build up to high quality levels because of the various
organizational pressures, agendas, incentives, etc. And, they are easy to degrade. The
organizational stability of these indicators varies from weak to very strong and are the sum of
dozens of behaviors.

 When there are difficulties attributable to barriers, for example, that can be removed, then the
indicators and the results can be changed relatively rapidly. However, when there are difficulties
due to lack of cooperation, lack of contact with the market, lack of good competitive intelligence
or with a lack of risk taking, then new attitudes and new behaviors are required. This takes time to
build into the culture and the indicators and the results will be slow to change.

122


 Business and Technology Leadership
 In all aspects, the dynamics of 'Integration with Business' indicators depend on organizational
matters versus those of the 'Portfolio Assessment" indicators, which depend on investment
decisions. And, a consideration of organizational and investment matters quickly brings the
factors of business and technology leadership into the model. Second and third order dynamics
pushing for immediate change are easy to see as overlays to deeper cultural values.

 It is also logical to see both the importance of these two sets of transformation strategy indicators
to managers who want immediate change and the criticality of both sets to the impacts on
sustained 'Value Creation'.

 Unfortunately, the underlying dynamics do not allow the conversion of management desires for
immediate gratification (at low risk) into sustained profitable growth. There must be allowances
made for the application of enough time to link all of the elements on an ongoing basis. Over
time, all of the indicators will point to a consistent improvement in the transformation strategies
and to the attendant output in 'Value Creation'.

 Conversely, monitoring the transformation strategy indicators of an otherwise healthy technology-
based enterprise, will show the early warning signs of degradation to the degree
possible, presuming the technology foundations are sound.

123


 Foundations
 The foundations for the strategies represented by the 'Portfolio Assessment' and the
'Integration with Business' are built on the 'Asset Value of Technology' and the 'Practice of
R&D Processes'. Some of the critical questions regarding these dynamics are:

 Are we becoming more or less productive with our R&D?

 Are we building a strong enough future base of competencies?

 Are we getting an early warning of any declining capability?

 Foundation indicators have the most momentum of any category. They are very slow to
change and provide the real rate limitations to growth through technology-based innovation.
However, they are also very vulnerable to neglect. They need nurturing, leadership and the
execution of well focused technology strategies to become strong elements of a company's
growth foundation. And, just as when weak transformation strategies lead to degradation of
'Value Creation', they also degrade the foundations.

124


 Foundations (Continued)
 Aside from weak transformation strategies, the other major dynamic that affects foundations are
external technological changes.

 In these cases, technological paradigm shifts occur that undermine the foundations comprised of
traditional technical competencies. These bring in new competitors, topple current competitors
and may, in fact, redefine the structure of an industry. Although these paradigm shifts take time to
develop, R&D and the business are usually both entrenched in the traditional areas, and either
don't see the changes coming or insist on devaluing the importance of them until it is too late. The
foundations which, if strong, took considerable time to build, must nonetheless be constantly
extended and rebuilt to provide an ongoing set of options that are the logical growth paths for the
business's future. Otherwise, over time, they will significantly erode.

 Thus, the dynamics of the foundation indicators are that they are slow to build, a rate limiting
source of quality for the options to be taken up by the transformation strategies, and that they are
relatively easy to degrade in spite of strong momentum. They are also fundamental to a strong
competitive basis for ongoing strategy (based on technical core competencies), to the heart of
R&D creativity, productivity and a significant contributor to cycle time reduction. The drivers of
these areas are the R&D leadership.

125



 Purpose
 The principal use of the model is for communication and
control. This includes the overall R&D program, the research
program, the business technical programs and the
development of external relationships for which new
capabilities are the goal.

 (The model is not intended to aid project evaluation except as
a project would contribute to the improvement of various
categories and their respective indicators.)

126

TVP Model Metrics

TVP Value Creation
 Financial return  Market share
 New sales ratio
 Cost savings ratio  Gross margins
 R&D yield
 R&D return
 Product technology rating
 Customer rating
 Projected value of the R&D  Economic trade-offs
pipeline
 Projected sales
 Sales protected by proprietary
 Projected income
position
 Patents only
 Number of ways technology is  Percent proprietary sales
exploited
 Technology planning
 Intellectual property
management
127

TVP Model Metrics

TVP Portfolio Assessment
 Strategic Alignment  Gross margin

 Projected value of R&D  Comparative technology
pipeline investment
 Projected sales
 Projected income  Information Technology Use in
R&D
 Distribution of technology
investment  Probability of success

 Number of ways technology is  Technology planning
exploited
 Environmental management in
 Market share R&D

128

TVP Model Metrics

TVP Integration with Business
 Strategic alignment  Product quality and reliability
 Use of Project milestones  Reliability / defects

 Customer satisfaction  Use of cross-functional teams
 Internal
 Number of ways technology is
exploited  Decision Gate processes

 Projects with business /  Technology planning
marketing approval

129

TVP Model Metrics

TVP Value of Technical Assets
 Strategic alignment  Response time to competitors
 Cycle time moves
 Market cycle time  Comparative technology
 Project management cycle time investment
 Customer satisfaction  Customer rating of technical
 External capability
 Internal  Number and quality of patents
 Quality of personnel  Percent useful
 Internal customer rating  Value ratio
 External customer rating  Percent retention
 External recognition  Patents per R&D Dollar
 Market share  Sales protected by proprietary
 Comparative manufacturing cost position
 Patents only
 Gross margin
 Percent proprietary sales
 Rating of technology features
and benefits  Core technical competency
 Intellectual property management

130

TVP Model Metrics

TVP Practice of R&D

 Gate effectiveness  Environmental management In R&D

 Defects reported  Idea generation and creativity

 Delayed Stage kills  People development

 Cost vs. budget  Intellectual property management

 Decision Gate processes  R&D climate

 Probability of success  Information Technology use in R&D

131

COMPARE EFFICIENCY & EFFECTIVENESS

Platform 3:
(Stages VIII, Part 4)

Integrating Technology Innovation with Business Function – Part I: Laying the Foundation


How do we Measure
Performance?
 Benchmarking
 Conduct performance and process benchmarking studies to
focus new concept R&D efforts on the most critical / valuable /
useful system components.

133


Benchmarking (Definitions)
 Benchmarking (Generic)
 Comparisons with other organizations to identify and then implement
strategies for improvement in one‟s own organization.
 Identification of superior performance and/or practices and
implementation of those in one‟s own organization.
 Organizations undertake benchmarking exercises to:
 Establish how well they need to perform to achieve „best practice‟
 Go beyond the experience of staff within their own enterprise in
pursuit of ambitious and innovative improvement plans.
 The main categories are performance benchmarking and
process benchmarking, and there are significant differences
between the two.

134

ISSUE PERFORMANCE PROCESS
BENCHMARKING BENCHMARKING
What is being Only compares performance. Compares process (how
compared things are done) and
PLATFORM 3: VIII (3) performance (how well the
process is done).
Degree of focus Generally compares a number Always compares one process
Performance of performance indicators.
These could measure the
at a time.

performance of just one
vs. Process process, but often cover a wider
spectrum of the organisation‟s
Benchmarking: Benchmarking
processes.
The most success approach is It is not advisable to choose
partners to organise a quorum of benchmarking partners before
Overview of benchmarking partners as soon undertaking a thorough
as the broad parameters of the analysis of your own practices
the Key project have been decided. (process, procedures) -
particularly if best practice is
Differences the goal.
Form of comparison Generally by circulating a report Best done through physical
containing all partners' data. visits to partners' places of
This data has been gathered business, although other
from each partner against a forms of comparison can be
predetermined set of appropriate.
performance indicators.
Confidentiality of The identity of the partners as a Because the identity of
partners‟ identity and group is generally known, but partners is known,
of the information each partner‟s data in the final benchmarking partners are
being shared report can be masked by a code either not competitors or the
letter for each partner. information being shared is
Therefore, benchmarking not confidential to the other
partners may be competitors but partner.
information shared remains
confidential.
Outcome from Generally to decide on one or Should be to implement
analysis of findings more process benchmarking improvements to the process
projects or „in-house‟ process that was benchmarked.
improvement projects. 135


Performance Benchmarking
 Different organizations compare performance against a number of agreed
performance measures. Examples of performance measures are:
 student enrolment efficiency - total cost per student enrolled
 student enrolment effectiveness - enrolment errors/1000 enrolments
 student enrolment customer satisfaction - average student elapsed time per enrolment
 student contact hours per staff member
 training hours per staff member
 employee satisfaction indexes
 customer satisfaction indexes
 Key advantages
 useful way of identifying where one stands in relation to others (i.e. identifying the gap)
 can (but may not) show where you most need to improve
 Disadvantages
 does not usually identify the processes or practices that have led to leading or best practice
performance
 provides no guidance on how to improve
 interpretation of results may be difficult (e.g. how much improvement is really possible)
 Conclusion
 It is important to use process benchmarking in conjunction with performance benchmarking
not only to find out how to improve but also how much one needs to improve
136


Process Benchmarking
 Organizations compare procedures or processes as well as performance. The main aim is
to identify how other organizations achieve superior efficiency, quality, innovation or
customer responsiveness in a process. Examples of processes are:
 enrolment
 student records
 handling prospective student enquiries
 fees collection
 project management/timetabling
 course promotion/marketing
 student feedback
 course evaluation
 resources
 sponsorship
 staff selection
 staff development and training
 Advantages
 Can tell how to improve as well as how much improvement is feasible.
 Differences between organizations compared- benchmarking partners are easier to interpret
 Disadvantage
 Might choose inappropriate partners if the organization has not clearly defined what it wants
to benchmark.
137


Strategic Benchmarking
 This examines the fundamentals of:
 How a business is defined
 The customer groups and needs a business serves
 The resources and skills used to satisfy customers.

 This type of benchmarking is most relevant when an organization is
reviewing its mission and strategy. For example, how do niche or
specialist organizations compete against larger organizations (e.g.
Bendigo Bank versus National Australia Bank).

 Issues to be addressed include:
 Customer groups and needs served (broad versus narrow)
 Resources and skills used (employees versus outsourced/partnerships and
alliances)
 Location (single versus multiple campuses)
 How products or services are provided.
138


Informal Benchmarking
 This refers to the well-established practice of informally comparing
one’s performance, process or practice with others.

 Attending conferences or workplace curriculum development
workshops are part of day-to-day responsibilities of staff in the
vocational education and training sector. These activities provide an
opportunity to compare one’s own organization with others.

 While useful, these comparisons should not be confused with proper
process or performance benchmarking. Only the latter have the scope
and rigour to deliver effective strategies for improvement.

139


When should we benchmark?
 Many organizations decide to benchmark when an:
 Improvement in performance is required.
 These organizations hope to achieve considerable
improvement as a result of benchmarking.
 Sometimes organizations undertake their first
benchmarking project to learn the technique for use in
the future.
 In general, benchmarking should be employed when it
is important to gain inspiration for specific
improvements from the practices of other
organizations. That is, when the ideas that are
generated from within are unlikely to result in sufficient
change.

140


Why Should We Benchmark?
 Process Benchmarking:
 Generally results in improvements that are greater than would
be gained from an improvement project conducted solely
within the organization
 Is an effective catalyst for people who need to change. If they
are included in the benchmarking project (preferably on site
visits) they see for themselves what changes are possible.
 Because performance benchmarking can help determine
where the greatest improvements can be made.

141


Who Should We Benchmark?
 Ideally you should benchmark an organization that has achieved a ‘best
practice’ or leading performance in a particular area. While the gap between
your current performance and the best practice organization may be large, you
will learn a lot from them and possibly ‘fast track’ your rate of improvement.
 However, you will learn much from benchmarking ‘above average’ performers.
These may be found in areas such as:
 Other departments, divisions or groups within your own organization
 Competitors (generally only possible through an intermediary)
 Other organizations in the same industry sector, e.g. those offering similar products and
services, but not competing in the same markets
 Any private or public sector organization (sometimes called „generic‟ or „best practice‟
benchmarking). The ultimate goal is to identify the „best-in-class‟ performers who have
processes that are analogous rather than identical to you own.
 Wherever possible you should aim to:
 Benchmark against the „best practice‟ performers (doing so will accelerate your rate of
improvement)
 Benchmark outside your industry (the source of most major improvements)
 Use the experience of organizations in the vocational education and training sector.

142


What Should We Benchmark?

 The short answer is something:

 Particularly important to the success of our organization
 Particularly important to your clients / customers
 That clearly needs to be improved.

143


How does benchmarking fit with
other initiatives and continuous
improvement programs?
 Benchmarking can be integrated into:
 Continuous improvement programs (comparisons with other organizations are
important sources of ideas for improvement)
 Changing workplace culture (people observe first-hand the practices of other
organizations. This means they are more likely to commit to changes implemented
within their own organization as a result of the benchmarking exercise.)
 Strategic and business planning (benchmarking can help identify achievable goals
to be included in the plan and a good strategic plan can identify areas to be
benchmarked)
 Annual budgeting process (benchmarking projects can identify the cost
implications of improvements required)
 Quality assurance systems (process benchmarking can identify improved
processes to incorporate into standard operating procedures)
 „Business Excellence‟ Criteria (Quality Awards assessors expect benchmarking to
play an integral role in an organization‟s approach to process improvement).
144


Process Benchmarking
 Stage 1: Decide What to  Stage 4: Analyze Findings
Benchmark  Analyze Gaps
 Make Initial Choice  Identify Best Practices
 Refine Choice
 Stage 5: Improve Practice
 Stage 2: Plan Project
 Plan Implementation
 Identify Scope
 Establish Team  Manage Implementation
 Manage Project
 Plan Consequences
 Stage 3: Conduct project
 Train Team
 Analyze Practices
 Choose Partners
 Design Questionnaires

145


Performance Benchmarking
 Stage 1: Decide What to  Stage 3: Conduct Project
Benchmark  Design Questionnaire
 Stage 2: Plan Project  Gather Data
 Choose Partners  Analyze Data
 Plan Comparisons  Produce Report
 Manage Project  Stage 4: Analyze results
 Stage 5: Determine Further
Action

146

PLATFORM 3: STAGE IX
INFUSE INNOVATION NORMS & VALUES

PLATFORM 3: IX

Infuse Innovation Norms & Values

 3-Step Process
1. There needs to be an expression of
feelings, emotions, goals, motivators, & the like from each individual
member of the group.
2. Individuals need to interpret & personally internalize them.
3. Through compromise & consensus building, each group member
ultimately reaches agreement on the norms & values of the group.
 Guiding Principles
1. Try to be fair.
2. Use your freedom to grow.
3. Make your own commitments & keep them.
4. Consult with other colleagues prior to any action that may adversely
affect the reputation or financial stability of the company.
148

PLATFORM 3: IX

Change Management

1. Create a Sense of Urgency
2. Produce a Guiding Coalition
3. Develop Vision & Strategy
4. Communicate Change Vision
5. Empower Broad-based Action
6. Generate Short-term Wins
7. Consolidate Gains, Produce More Change
8. Anchor New Approaches in Environment

149

PLATFORM 3: IX (Parts 1-2)

Change Management

1. Create a Sense of Urgency
 Market & competitive realities
 Potential crises & opportunities
 Influence games impacting environment
 Persuasive influence
2. Produce a Guiding Coalition
 Assemble & organize stakeholders for max. influence
 Develop relationships with critical decision-makers
 Make the group work like a cohesive team

150


Change Management

3. Develop Vision & Strategy
 Create a vision to help direct change effort
 Determine leverage points for exerting influence
 Develop strategies to achieve vision
 Build coalitions around unifying points of agreement
Build momentum

4. Communicate Change Vision
 Use every vehicle to CONSTANTLY communicate vision &
strategies
 Frame arguments to convey essential messages with clarity
& persuasion
 Guiding coalition role-model actions / behaviour desired

151


Change Management
5. Empower Broad-based Action
 Get rid of obstacles
 Develop a roadmap
 Change systems / structures undermining change vision
 Benchmark
 Develop an industrial / market cluster
 Apply effective decision-making tools
 Encourage risk taking, innovation & creativity
6. Generate Short-term Wins
 Visible improvements in performance / outcomes
 Arrange optimum strategic partnering
 Execute wins efficiently & effectively
 Visibly recognize & reward people responsible
152


Change Management
7. Consolidate Gains, Produce More Change
 Capitalize on influence, credibility & track record to CHANGE all
system, structures, policies
 Hire, promote, develop people to drive change
 Reinvigorate with new projects, themes, change agents
 Seek continuous improvement
 Develop & apply suitable metrics
8. Anchor New Approaches in Environment
 Optimize all processes, systems, performance
etc., behaviour, leadership, management
 Articulate, model & apply (via NLP) best practice for “success”
 Ensure continuity: leadership development & succession

153

IRL TECHNOLOGY INNOVATION
STRATEGY FRAMEWORK

END

Innovation Benefits Realization for Industrial Research (Part-8)

More Related Content

What's hot (10)

Viewers also liked (20)

Similar to Innovation Benefits Realization for Industrial Research (Part-8) (20)

More from Iain Sanders (18)

Recently uploaded (20)

Innovation Benefits Realization for Industrial Research (Part-8)