From Principles to Strategies for Systems Engineering

Systems Engineering
From Principles to Strategies
How to apply Principles, Practices, and
Processes of Systems Engineering to
solve complex technical, operational,
and organizational problems
1

Course Contents
§ Lesson 0 – Course Introduction
§ Lesson 1 – Overview of SE Discipline
§ Lesson 2 – Principles of Systems Engineering
§ Lesson 3 – Practices of Systems Engineering
§ Lesson 4 – Processes of Systems Engineering
§ Lesson 5 – Applying SE to Notional Project
Hands On Project Used To
Illustrate these lessons
throughout the course
2

Systems Engineering in One Picture
3
† National Airspace Systems Engineering Manual, Federal Aviation Administration, ATO Operations Planning, Oct 11, 2006

Exit Criteria for this Course
§ Provide the skills and knowledge needed to apply
Systems Engineering in any program domain
§ Develop understanding of the Principles,
Practices, and Processes of Systems Engineering
needed to increase the probability of program
success (PoPS)†
– Capabilities Based Planning
– Measures of Effectiveness and Performance of the
program outcomes
– Provide the mechanisms to convert user needs into
user satisfaction
4
† Probability of Program Success (PoPS) is a framework implemented across DOD services. “Implementation Guidance and Methodology for Naval
Probability of Program Success (PoPS), Office of Assistant Secretary Research, Development, and Acquisition, 1000 Navy Pentagon, Washington DC, Oct
06, 2008

Start With The End In Mind
§ Systems Engineering provides the guiding
principles for the analytic foundation of any
development program through
– Technical Frameworks
– Requirements Management
– Integration Management
– Risk Identification and Mitigation
– Evaluation, Verification, and Validation of all work
products
5

We’ve Met the Enemy and He is Us†
§ Unrealistic performance expectations
§ Unrealistic baseline estimates for cost and
schedule
§ Immature technologies or excessive
manufacturing or integration risk
§ Unanticipated design, engineering,
manufacturing, or technology integration
§ Issues arising during program
performance
§ Changes in procurement quantities
§ Inadequate program funding or funding
instability
§ Poor performance by customer or
contractor personnel responsible for
program management
6
† WSARA (2009) lists eight root causes in Decisions Made During Program Execution as a Root Cause of Nunn-McCurdy
Breaches , The Evidence from Root Cause Analyses done by RAND and IDA for PARCA May 16-17 David L. McNicol

Lesson 0
Systems Engineering is a disciplined
approach to solving complex
technical, organizational, and
operational problems, using Systems
Thinking to consider the whole rather
than just a collection of the parts.
7

Lesson 0
§ Why are we here?
§ What is Systems Engineering?
§ Systems Engineering Framework
8
Lesson 0

Why Are We Here
These 3 Days?
§ All the worlds system, we need to learn how
to manage the processes of developing, then
assembling. all these moving parts.
§ Systems Engineering is the means to
delivering the final system made up of these
parts.
§ This course will provide you with the
Principles, Practices, and Processes needed to
increase the probability of program success
using Systems Engineering.
9
Lesson 0

All the worlds’ a
system.
All the parts are
separate.
All the parts are
connected.
10
Lesson 0

Systems Thinking
§ Before moving the Systems Engineering let’s talk
about Systems Thinking
§ Russell Ackhoff† tells us systems are defined by
– Parts
– Relationships
– Purpose
§ Systems Thinking looks at
– Relationships – rather than unrelated objects
– Connectedness – rather than structure
– The whole – rather than just its parts
– Patterns – rather than contents
11
† Systems Thinking for Curious Managers, R. Ackoff, with H. Addison and A. Carey, Triarchy Press 2010
Lesson 0

Systems Thinking
§ Thinking systematically … requires several
shifts in perception, which lead in turn to
different ways to teach, and different ways to
organize society†
§ Our perception needs to move away from …
– Take the thing or event to be understood apart
– Explain the behavior or properties of the parts
taken separately
– Aggregate the explanations of the parts into an
understanding of the whole, the thing to be
explained.
12
† Redesigning Society, Ackhoff and Rovin, Stanford Business School Books, 2003
Lesson 0

Moving to Synthetic Thinking†
§ Managers should never accept the output of a
technologically-based system unless they
understand exactly what the system does and
why.
§ Systems must be understood in the context of
what they can do and the world in which they will
do it.
§ It is not enough to see the system as a sum of the
operations of the component parts.
§ It must be seen as a functioning whole.
§ This is the System Thinking Point of View.
13
† A Primer for Model-Based Systems Engineering, 2nd Edition, David Long and Zane Scott, Vitech, 2011
Lesson 0

Systems Engineering is…†
“an interdisciplinary approach to translating
users’ needs into the definition of a system, its
architecture and design through an iterative
process that results in an effective operational
system. Systems engineering applies over the
entire life cycle, from concept development to
final disposal.”
14
† Pre-Milestone A and Early-Phase Systems Engineering: A Retrospective Review and Benefits for Future Air Force
Acquisition (2008)
Lesson 0

We Will To Learn To Identify All The Parts And
Assemble Them Into A Functioning System
And Learn How All The Elements Of Systems
Engineering Will Aide Us In That Process
15
Lesson 0

World of Engineered Systems†
16
Systems
Engineering
Systems
Architecture
Requirements
Definitions
Stakeholder
Analysis
Trade Space
Exploration
Design Definition
and Optimization
Human Factors
Systems
Integration
Interface
Management
Verification and
Validation
Commissioning
and Operations
Lifecycle
Management
† Derived from MIT Open Courseware, readings for Systems Engineering
Lesson 0

It All Started
Here
The term Systems Engineering dates to Bell Telephone
Laboratories in the early 1940s. The first attempt to teach
Systems Engineering as we know it today came in 1950 at
MIT by Mr. Gilman, Director of Systems Engineering at Bell.
Systems engineering was defined as work functions with
five phases:
1. System studies or program planning
2. Exploratory planning, including problem definition,
selecting objectives, systems synthesis, systems analysis,
selecting best system, and communicating the results.
3. Development planning, repeating phase 2 in more
detail.
4. Studies during development, including development of
parts of the system and integration and testing of these
parts;
5. Current engineering, taking place while the system is
operational and being refined.
RAND Corporation created systems analysis, an important
part of Systems Engineering.
The Department of Defense entered the world of Systems
Engineering in the late 1940s with the development of
missiles and missile-defense systems.
Paul Fitts† codified Systems Engineering by addressing the
allocation of the systems functions to the physical
elements of the system in the late 1940s and early 1950s.
† Paul Fitts (ed) (1951) Human engineering for an
effective air navigation and traffic control system.
National Research Council, Washington, DC 17
Lesson 0

And Now, Systems
Are Everywhere
18
Lesson 0

Complicated, Complex, and
Complexity
§ Complicated – consisting of many interacting
parts or elements
§ Complex – characterized as many parts that
interact with in multiple ways with possibly
unknown outcomes
§ Complexity – the state of being intricate or
complicated
§ Irreducible Complexity - characteristic of
certain complex systems in which all
components needed to function.
19
Lesson 0

Systems Engineering is about
Managing Complexity†
20
Product
Number of
Parts
Number of
Levels
Screwdriver 3 1
Roller Blades 30 2
Inkjet Printer 300 3
Copy Machine 2,000 4
Automobile 10,000 5
Airliner 100,000 6
† Modified from, Ulrich, K.T., Eppinger S.D. , Product Design and Development,
Second Edition, McGraw Hill, 2000, Exhibit 1-3
Lesson 0

The Structure of This Course
§ Lessons, 1 through 4,
– Overview,
– Principles,
– Practices, and
– Processes of Systems Engineering.
§ Learning assessments of these concepts are at
the end of each Lesson.
§ A notional project is used for classroom
experience to apply the Principles, Practices,
and Processes.
22
Lesson 0

We Will Focus On …
§ INCOSE and 15288 Processes
§ The system life cycle management
§ Project Management Processes
§ Using our notional project we will:
– Develop Measures of Effectiveness (MOE),
Performance (MOP), and Technical Performance
(TPM)
– Trace the Principles, Practices, and Processes to
our notional program
23
Lesson 0

Overview Of Our Course
Lesson 2 Principles
Lesson 3 Practices
Lesson 4 Processes
SE
Lifecycle(s)
Vee
Paradigm
SEMP
UAV
Project
ConOps
SOW
WBS
MOE
MOP
TPM
24
Lesson 5
Notional Project
Lesson 0

When We Say Systems
Engineering What Do We Mean?†
§ System
– A combination of interacting elements organized to achieve one
or more stated purposes.
§ Systems of Interest
– The system whose life cycle is under consideration in the
context of INCOSE and 15288 Guidance.
§ System Element
– A member of a set of elements that constitutes a system.
– A system element is a discrete part of a system that can be
implemented to fulfill specified requirements.
§ Enabling System
– A system that complements a system-of-interest during its life
cycle stages but does not necessarily contributed directly to is
function during operation.
– When a system-of-interest enters the production stage, an
enabling production system is required.
25
† Extracted from ISO/IEC 15288
Lesson 0

Through Classroom Examples, We’ll
Learn Systems Engineering for …
§ Capabilities Elicitation
§ Requirements Development
§ Product Development
– Hardware
– Software
– Operations, maintenance, and support
§ Integration And Test
§ Operational Verification and Validation
§ Deployment, Support, and Maintenance
26
Lesson 0

With This Knowledge
We Will Confirm the Needed Capabilities can be
Delivered as Planned†
§ System provides balanced and optimized products that meet the
customers needed capabilities.
§ Effective requirements analysis is applied in the delivery of these
capabilities .
§ Consistent and rigorous application of Systems Engineering management
standards is applied throughout the program’s lifecycle.
§ Effective planning to test these capabilities is accomplished.
§ Effective major technical program reviews to evaluate the emerging
capabilities are conducted.
§ Continuous risk assessments and management to assure capabilities are
delivered as planned is implemented.
§ Reliable cost estimates and policies for the delivery of the capabilities are
developed.
§ Disciplined application of configuration management are demonstrated.
§ System boundaries are well-defined.
† Global Hawk Systems Engineering Case Study, Air Force Center for Systems Engineering, Wright Patterson AFB, 2010. 27
Lesson 0

CAPABILITIES BASED PLANNING†
Before proceeding to the next steps in Systems Engineering, let’s pause
and discuss an issue in all modern program developments.
Many programs provide a multitude of technical and operational features
and functions. We’ve all experienced this. Software tools, automobiles
with more features than we can remember, complex systems like aircraft
with features so complex the pilots have trouble remembering how to
operate then (one cause of the Asiana Air crash in SFO is attributed to
the multiple features in decent control system that created confusion).
One improved approach to engineering a system is to determine what
capability is needed to accomplish the mission or provide a solution to
the business problem.
29
† Guide to Capabilities Based Planning, The Technical Cooperation Program, Joint Systems and Analysis Group, Technical Panel 3.
Lesson 0

Capability Versus Requirement
§ Capabilities describe three characteristics needed
before proceeding to requirements elicitation
– Possible scope – delimits the boundaries of the
problem and the possible solutions
– Possible forms – specific characteristic of a process
not related to its application
– Possible solutions – candidate solutions that have
already been applied to solve similar problems that
should be investigated to determine if they should be
part of the this solution
Lesson 0
Provide safe transport to the public is a capability. No single failure in he
breaking system shall endanger life is a requirement. 30

Capabilities-Based Planning†
§ Capabilities-Based Planning is planning, under
uncertainty, to provide capabilities suitable for
a wide range of business challenges and
circumstances, while working within an
economic framework
§ Capabilities-Based Planning emphasizes
flexibility, adaptiveness and robust capabilities,
implying a modular building-block approach to
Enterprise Services
§ When transformation takes place it is because
new modules have come into use
31
Lesson 0
† Analysis to Support Capabilities-Based Planning, Tom Allen, Institute for Defense Analyses, Capabilities-Based Planning
Workshop, 19-21 October 2004

A Simple Example
Our system of provisioning a new
employee
32
Human Resources
New Employee Ready to Work
Insurance
Orientation
Laptop Account Setup
Charge account setup
Information Technology
Finance
Buying authority available
Supply Chain Management
We Need the Capability To
Provide Buying Authority within 2 working days of
new employee hire
Lesson 0

Capabilities State the Intent of the
Commander
33
Action Outcome
Implement Strategy
Ensure Capabilities
Prioritize Problems And Solutions
Identify Redundancies
Deliver Solutions
† Photo Eustachy
Kossakowski
Lesson 0
In preparing for battle I have always found that
plans are useless, but planning is indispensible

The Role SE in Identify Needed
Capabilities and Managing Their Delivery
What Should We Do?
Where Are We Now?
Identify
Capability
Mismatches
Operational
Concepts
Capability
Goals
Scenarios
Priorities
Customer
Needs
Investment
Balance
Solution
Deployment
Options
Capability
Assessment
Mission
Priorities
Resource
Constraints
Capability
Partitions
Current and
Planned
Capabilities
Affordable
Capabilities
Plan
Abstracted from:
“Capabilities‒Based Planning – How It Is Intended
To Work And Challenges To Its Successful
Implementation,” Col. Stephen K. Walker, United
States Army, U. S. Army War College, March 2005
Future
Environment
Lesson 0
34

A Reminder Before We Start†
§ Systems Engineering Is …
– A logical sequence of activities and decisions that
transforms an operational need into a description of
system performance parameters and a preferred system
configuration. (MIL-STD-499A, Engineering Management)
– An interdisciplinary approach that encompasses the entire
technical effort, and evolves into and verifies an integrated
and life cycle balanced set of system people, products, and
process solutions that satisfies a customer need. (EIA
Standard IS-632, Systems Engineering)
– An interdisciplinary, collaborative approach that derives,
evolves, and verifies a life cycle balanced system solution
that satisfies customer expectations and meets public
acceptability (IEEE P1220 Standard for Application and
Management of Systems Engineering Process)
35
† These definitions have been replaced by INCOSE and 15288, but serve as reminders of the depth and breadth of Systems
Engineering as the foundation of program success.
Lesson 0

One More Reminder
§ Systems Engineering is about the system
aspects of program outcomes.
§ Systems Engineering does not build the
products of the program outcomes, it enables
the right products to be built right, in the right
way, at the right time, to deliver the right
value to the customer.
36
Lesson 0

Oops, One Final Reminder
A system is a collection of elements, that together,
produce result not obtained by the elements alone.
These elements can include people, hardware,
software facilities, policies, and documents – all
interacting to produce on outcome.
37
System Engineering is a product oriented discipline
whose responsibility is to create and execute the
interdisciplinary processes that ensure the
stakeholder needs are satisfied.
Lesson 0

Learning Objectives for the
Systems Engineering Course
§ Understand the most important Systems
Engineering standards and best practices and
how these guide the principles, practices, and
processes of Systems Engineering
§ Understand the key steps in system engineering
process starting with stakeholder analysis and
ending by transitioning the system to operations
§ Understand the role of the human aspects of
Systems Engineering
§ Apply these understandings to a notional project
to Connect the Dots for actual use after the
course
38
Lesson 0

Lesson 0
39
Science determines what IS
Component engineering determines what CAN BE
Systems Engineering determines what SHOULD BE†
† Special Feature, INCOSE Insight, Vol 5, Issue 1, April 2002

Lesson 1
Introduction to the Discipline of
Systems Engineering
Using ISO/IEC 15288 and INCOSE
Systems Engineering Handbook 3.2
40

How To Think Like
A Systems Engineer
§ Systems Thinking says its name
– A framework for solving problems where the
component part of an entity is best understood in
the context of its relationship with other
components of the entity.
§ These are fancy words for
– Understanding any Part, Problem, or Solution
through its relationship with the Whole
41
Lesson 1

As a Discipline, Systems Engineering …†
§ Matches the product to the marketplace
§ Defines the components so the designers can be
design and built them
§ Determines most of the design choices affecting
system cost and performance
§ Ensures that the components will integrate
successfully and perform together as required
§ Provides specifications free of errors, since errors
are very expensive to correct in the latter stages
of design and production.
42
Lesson 1
† Engineering Complex Systems with Models and Objects, David W. Oliver, Timothy P. Kelliher and James G. Keegan, Jr., McGraw-Hill

First Principle of
Systems Engineering
“All Systems Engineering models and processes
are organized around the concept of a life cycle.
The detailed views, implementations, and
terminology that articulate the life cycle differ
across organizations and customers, they share
fundamental elements guided by Systems
Engineering standards.” †
43
Lesson 1
† From MITRE SE Life-Cycle Building Blocks

Some System
Engineering Standards
§ ANSI/GEIA EIA-632, Processes for Engineering a System, 01
Sept 2003
§ ISO/IEC 15288 – System Lifecycle Processes
§ EIA/IS 731.1, Systems Engineering Capability Model,
Electronic Industries Alliance (Interim Standard), 01 Aug
2002
§ IEEE 1220-2005, IEEE Standard for Application and
Management of the Systems Engineering Process, Institute
of Electrical and Electronics Engineers, 09 Sept 2005
§ ISO/IEC 15504: 2004 - Information Technology - Process
Assessment
§ INCOSE Systems Engineering Handbook, Systems
Engineering Handbook A Guide For System Life Cycle
Processes And Activities Incose-TP-2003-002-03.2 January
2010
44
Lesson 1

INCOSE SE Handbook†
§ Generic life cycle
§ Technical Processes
§ Project Processes
§ Agreement process
§ Organizational processes
§ Tailoring processes
§ Specialty Engineering
45
Lesson 1
† INCOSE Systems Engineering Handbook can be found at www.incose.org

ISO/IEC 15288 Standard
§ 25 processes, 123
outcomes, derived from
403 activities and 6
examples covering life cycle
of human made systems
§ System is composed of
elements, where they are
implemented and
integrated into the system
§ Elements can themselves
be considered a system.
46
Lesson 1

ISO/IEC 15288 Framework
Agreement Organization Project Technical
Concept
Develop
Produce
Use Retire
47
Support
Lesson 1

ISO 15288 Systems Engineering
Processes
Enterprise Process
§ Enterprise Process
Management
§ Investment
Management
Processes
§ Systems Lifecycle
Processes
§ Resource
Management
Processes
Agreement Process
§ Acquisition
§ Supply
Project Processes
§ Planning
§ Assessment
§ Control
§ Decision
Management
§ Risk Management
§ Configuration
Management
§ Quality
Management
Technical Processes
§ Stakeholder needs
definition
§ Requirements
analysis
§ Architectural design
§ Implementation
§ Verification
§ Integration
§ Transition
§ Validation
§ Operation
§ Maintenance
§ Disposal
48
Lesson 1

Defense SE and Commercial SE
49
Concept
Stage
Development Stage
System
Definition
Preliminary
Design
Detailed
Design
FAIT
Production
Stage
Utilization
Support
Stage
Customer Support
Retire
ISO 15288
IEEE 1220
Lesson 1

Systems Engineering is Applicable
in a Wide Variety of Domains
§ Aerospace
§ Telecommunications
§ Transportation Systems
§ Military Systems
§ Ship Building
§ Finance and Administrative Systems
§ Information Technology Systems
50
Source: ISO/IEC JTCI/SC7/WG7 Source: ISO/IEC JTCI/SC7/WG7 presentation on ISO/IEC 15288.
Lesson 1

Our Lingua Franca for Engineering
Systems using INCOSE and 15288
§ Every system has a lifecycle, viewed as stages
§ Stages are separated by decision gates
§ Stages may overlap or be concurrent
§ Each stage is accomplished by executing
processes within the stage
§ Any process may be applied to any stage
51
Lesson 1

What is Systems Engineering?
§ Systems Engineering says its name. It …
… is an interdisciplinary field of engineering focused on the design
and management of complex project throughput their life cycle.
It addresses the disciplines of reliability, logistics, coordination of
teams, requirements management, evaluation metrics,
optimization methods, risk management – the System.
… integrates all the disciplines and specialty groups into a team
effort forming a structured development process that proceeds
from concept to production to operation. Systems Engineering
considers both the business and the technical needs of all
customers with the goal of providing a quality product that meets
the user needs.
52
Lesson 1

The Primary Structure of Systems
Engineering
Two Distinct Systems Engineering Activities†
From Partitioning è To Integrating
User Needs è User Satisfaction
User Requirements è User Acceptance Test
System Requirements è System Acceptance Test
System Design è Integration Test
System Development è Subsystem Test
53
Lesson 1
† This is the foundation of the Systems Engineering Vee

Typical Life Cycle Management
54
Cycle Activities
Exploratory Identify enabling technologies
Concept Analyze needs, identify concepts, and develop solutions
Development Engineer system to deliver needed capabilities
Production Build the system to deliver needed capabilities
Utilization Operate the system for the benefit of the user
Support Maintain and support the system
Retire Retire, archive, or dispose of the system
Lesson 1
1
2
3
4
5
6
7

Exploratory
§ Create reference architecture
§ Identify enabling technologies
§ Generate early cost and schedule projections
§ Clearly identify needed customer capabilities
§ Assess technology readiness
55
1
Lesson 1 Life Cycle Management

Concept
§ Refine and broaden any studies
§ Focus on requirements driven approaches rather than
product driven
§ Identify, clarify, and document stakeholder
requirements
§ Build in-depth studies to evaluate multiple candidates
§ Provide sustained justification for system concept
§ Prototypes used to verify feasibility of concepts
§ Expand risk and opportunity evaluation for
affordability, environmental impacts, failure modes,
hazard analysis, technical obsolescence, and system
disposal
56
2

Development
§ Detailed planning, development, and IV&V
§ Incremental and iterative processes apply here
§ Full freedom to apply development model
– Waterfall
– Plan driven
– Agile development
57
3

Production
§ Produce system-of-interest, manufactured
product, or delivered service
§ Modify product to resolve production issues,
reduce costs, enhance product capabilities
§ Systems engineering assessment of any
changes during production.
58
4

Utilization
§ Operate system-of-interest
§ Plan product modification through out
operation of system
§ Upgrades and enhancements to capabilities
59
5

Support
§ Provide services that enable continued
operation.
§ Propose modifications to resolve:
– Supportability problems
– Reduce operational costs
– Extend life of the system
§ Assess changes to avoid loss of system
capabilities while under operation.
60
6

Retirement
§ System of interest and related services
removed from operation
§ Ensure retirement requirements satisfied
§ Retirement planning done during Concept
process
61
7

Consensus of INCOSE Fellows for Core
Framework for Systems Engineering†
State the Problem
Investigate Alternatives
Model the System
Integrate Developed Components
Launch Resulting System
Access Performance
Re-Evaluate
1
2
3
4
5
† A Consensus of INCOSE Fellows on What is Systems Engineering? The Systems Engineering Process from A. T. Bahill and B. Gissing, Re-evaluating Systems
Engineering concepts using systems thinking, IEEE Transaction on Systems, Man and Cybernetics, Part C: Applications and Reviews, 28 (4), 516-527, 1998.
This Is An Iterative And Parallel Process
6
7
62
Lesson 1

State the Problem†
§ Describe the top level functions in terms of
capabilities the system must provide for
success
§ State the problem to be fixed (deficiency that
will be ameliorated)
§ State mandatory needed for acceptability
§ State the Measures of Effectiveness and
Measures of Performance needed for success
1
63
Lesson 1 Core Framework
† Derived from Bahill, A. T. and Dean, F. F., Discovering system requirements, Chapter 4 in Handbook of
Systems Engineering and Management, A. P. Sage and W. B. Rouse

Investigate Alternatives
§ Create an evaluate alternatives based on
technical and operational performance, cost,
and schedule
§ Use multi-criteria decision making processes
to assess alternatives
§ Establish model elements, their coupling and
cohesion, to be tested next
2
64

Model the System
§ Models can be used to test alternatives for most
system designs.
§ Models represent the essential characteristics of
the system under development
§ The preferred model can be expanded and used
throughout the program lifecycle
§ Many models available
– Physical analogs – plywood and lead weights for
spacecraft
– Mathematical models – 3D CATIA† dynamic models
– Block diagrams, state machines functional flow
models, computer simulations – can model dynamic
behaviors
3
65

Integrate Developed Components
§ The system, the business processes
implemented by the system, and the people
participating in that business process must all
be integrated.
§ Define the subsystems on natural boundaries
– This is a hard problem, but start with coupling and
cohesion considerations
– Use an Interface Control Document paradigm to
manage connections to the subsystems
§ Integration between co-evolving subsystems is
high risk and must be explicitly managed
4
66

Launch Resulting System
§ Deploy the system, run it, and produce outputs.
– More complex than it sounds of course
– So need a Plan for the Plan to deploy, run and produce
with minimum risk and maximum value
§ SE products include:
– Mission capabilities description
– Technical and operational requirements
– Allocation of functions to physical components
– Measures of Effectiveness, Performance, and
Technical attributes used to assess capabilities will be
provided
5
67

Assess System Performance
§ Each level of the system, from abstract to
tangible, needs a measures of its outcome
6
System Measure
Capability Measures of Effectiveness (MOE)
Requirement Measures of Performance (MOP)
Product Technical Performance Measures (TPM)
Mission Attribute Key Performance Parameters (KPP)
68

Measures of
Effectiveness (MOE)
69
Measures of Effectiveness …
§ Are stated in units meaningful to the buyer,
§ Focus on capabilities independent of any
technical implementation,
§ Are connected to the mission success.
The operational measures of success that are closely related to the
achievements of the mission or operational objectives evaluated in
the operational environment, under a specific set of conditions.
Measures of Effectiveness Belong to the End User
Lesson 1
6
Core Framework

Measures of
Performance (MOP)
70
Measures of Performance are …
§ Attributes that assure the system has the
capability to perform,
§ Assessment of the system to assure it meets
design requirements to satisfy the MoE.
Measures that characterize physical or functional attributes
relating to the system operation, measured or estimated
under specific conditions.
Measures of Performance belong to the Program – Developed by
the Systems Engineer, Measured By CAMs, and Analyzed by PP&C
Lesson 1
6
Core Framework

Key Performance
Parameters (KPP)
71
Key Performance Parameters …
§ Have a threshold or objective value,
§ Characterize the major drivers of performance,
§ Are considered Critical to Customer (CTC).
Represent the capabilities and characteristics so
significant that failure to meet them can be cause for
reevaluation, reassessing, or termination of the program
The acquirer defines the KPPs during the operational
concept development – KPPs say what DONE looks like
Lesson 1
6
Core Framework

Technical Performance
Measures (TPM)
72
Technical Performance Measures …
§ Assess design progress,
§ Define compliance to performance requirements,
§ Identify technical risk,
§ Are limited to critical thresholds,
§ Include projected performance.
Attributes that determine how well a system or system
element is satisfying or expected to satisfy a technical
requirement or goal
Lesson 1
6
Core Framework

Dependencies Between These
Measures
73
MoE
KPP
MoP TPM
Mission
Capabilities
Acquirer Defines the Needs and Capabilities
in terms of Operational Scenarios
Supplier Defines Physical Solutions that
meet the needs of the Stakeholders
Operational
measures of success
related to the
achievement of the
mission or
operational
objective being
evaluated.
Measures that
characterize physical
or functional
attributes relating
to the system
operation.
Measures used to
assess design
progress,
compliance to
performance
requirements, and
technical risks.
6

Re-evaluate Resulting Outputs
§ The most important function of the systems
Engineering framework is the re-evaluation of
each phases outcomes at each phase.
§ This is continuous feedback
§ Optimism is the disease of all technical
development, feedback is the cure†
7
74
† Kent Beck, software engineer and creator of eXtreme Programming and Test Driver Development
Customer
Need
State the
Problem
Investigate
Alternatives
Model the
System
Integrate
Launch the
System
Assess
Performance
Outputs
Re-Eval Re-Eval Re-Eval Re-Eval Re-Eval Re-Eval

What Roles Do Systems Engineers
Play In This Framework?†
Enterprise Perspective
Engineering Life Cycle Paradigm
Engineering Planning and Management
Technical Specialties
Collaboration
75
Lesson 1
1
2
3
4
5
† Adapted from MITRE Systems Engineering (SE) Competency Model, September 1, 2007, Version 1.13E Sework Program

System Engineering Roles†
76
Lesson 1 Roles
† MITRE Corporation Systems Engineering Competency Model, http://www.mitre.org/publications/technical-papers/systems-engineering-competency-model

Enterprise Perspective†
§ Comprehensive viewpoint
– A broad understanding of the system and the
context in which it “Lives”
– Discover new approaches and ideas to modeling
complex systems
– Provides long term strategies to achieve mission
objectives and exploit opportunities
77
Lesson 1
1
Roles

Enterprise Perspective†
§ Innovative approaches
– Partial solution provided to ambiguous problems
– Frame the “essence” of the problem
– Provides scalable and adaptable solutions to
complex system solutions
§ Enable stakeholder relations
– Leverage engineering resources
– Communicate independent, objective, direct
information
– Facilitate decision-making
78
Lesson 1
1
Roles

Systems Engineering Life Cycle†
§ Concept definition
– Establish agreement on mission need and Concept
or Operations
– Develop high-level conceptual design
§ Requirements Engineering
– Integrate mission and operational needs into
system requirements
– Facilitate agreement on changes to requirements
79
Lesson 1
2
Roles

Systems Life Cycle Engineering†
§ Architecture
– Identify alternative architecture solutions
§ System design and development
– Establish agreement on design review and
milestone decision approaches
§ Systems integration
– Provide integration strategies to meet mission
need, integration and interoperability challenges
80
Lesson 1
2
Roles

Systems Engineering Life Cycle†
§ Test and Evaluation
– Guide test and evaluation strategies of an
effective and interoperable system
– Influence stakeholders on mitigation strategy and
system acceptance
§ System Implementation, OM&T
– Develop agreement on transitional approach and
system deployment
– Influence approaches to system modification and
technology insertion
81
Lesson 1
2
Roles

Planning and Management†
§ Transformational Planning
– Develop strategy for transforming stakeholder
organization, structure, processes, system
interactions with other organizations
– Collaborate and build consensus with stakeholders
for transforming organization
82
Lesson 1
3
Roles

§ Government Acquisition Support
– Collaborate with stakeholders to establish
acquisition program and program office
– Work with stakeholder to select the Systems
Engineering life cycle approach
– Recommends best value offeror to stakeholder
§ Contractor Evaluation
– Influence stakeholder decision during contractor
evaluations and milestone reviews
– Recommend changes based on contractor
performance
83
Lesson 1
3
Roles

§ Risk Management
– Influence risk management approach
– Influence risk mitigation strategies and program
direction
§ Configuration Management
– Influence configuration management approach
§ Integrated Logistics Support
– Recommend and implement integrated life cycle
logistics support strategy
– Guide approval and implementation of ILS alternatives
84
Lesson 1
3
Roles

§ Quality assurance and Measurement
– Guide establishment and direction of quality
assurance program in the systems acquisition
– Influence resolution of corrective actions to
ensure adherence to processes
§ Continuous Process Improvement
– Influence approach for implementing and
improving Systems Engineering processes
– Influence stakeholder organizations to finalize,
implement, and continuously improve Systems
Engineering processes
85
Lesson 1
3
Roles

Technical Engineering
Specialties†
§ Cost / Benefit Analysis
– Provide direction to analyst for scope, key
parameters, products, and tradeoffs of the
analysis
– Provide strategic, programmatic and enterprise-
wide perspective of cost/benefit analysis
86
Lesson 1
4
Roles

Specialties†
§ Human Centered Engineering
– Recommend design and trade-off decisions during
the early acquisition phases, based on a human-
centered engineering approach.
– Collaborates with the HCE and the
sponsor/customer to resolve HCE issues.
87
Lesson 1
4
Roles

Specialties†
§ Modeling And Simulation
– Recommend M&S scope, approaches, and
changes to operational capabilities
– Facilitates cooperative modeling arrangements.
88
Lesson 1
4
Roles

Specialties†
§ Security Engineering
– Identify security engineering approaches and
constraints
– Plan for certification and accreditation,
– Determine security related trade-offs.
89
Lesson 1
4
Roles

Specialties†
§ Reliability, Maintainability, And Availability
– Collaborates with RMA specialist and the
sponsor/customer to identify approaches,
interpret model results
– Determine design changes and prioritized
corrective actions.
90
Lesson 1
4
Roles

Specialties†
§ Safety Engineering
– Recommends and gains consensus on safety
engineering approaches, design trade-offs, and
solutions
91
Lesson 1
4
Roles

Specialties†
§ Network And Communications Engineering
– Interacts with the specialist to facilitate task
completion and makes recommendations to the
sponsor/customer on network and
communication issues.
92
Lesson 1
4
Roles

Specialties†
§ Software and Information Engineering
– Facilitates interaction among the
sponsor/customer, end-users, and software
engineers to determine system expectations,
problems, and potential solutions
– Gains consensus with the sponsor/customer on an
information-centric view of the enterprise
– Communicates risks and mitigation strategies to
the sponsor/customer based on software testing
and/or software reliability model results.
93
Lesson 1
4
Roles

Specialties†
§ Collaborating with technical Specialties
– Obtains support and resources from the
sponsor/customer for studies and engineering
efforts requiring specialized expertise.
– Recommends appropriate specialists,
communicates pertinent information to the
sponsor/customer, and helps to build consensus
among key stakeholders.
94
4
Lesson 1 Roles

Collaboration†
§ Build trust
§ Build successful team
§ Communicating the Impact
§ Being persuasive and influencing outcomes
§ Facilitating, managing, and championing
change
§ Setting quality standards
§ Focused on results
95
5
Lesson 1 Roles

Enough Of Theory Let’s …
96
… and actually start DOING
Systems Engineering
Lesson 1

Preparing the SEP and SEMP
97
Lesson 1

SEP and SEMP
98
System Engineering Plan
System Engineering
Management Plan
What
Who
When
Why
What
Purpose
When
How
The SEM and SEMP are designed to work together. The SEM answers SE questions
related to what, how, and when, while the SEMP answers SE questions related to
what, who, when, and why (i.e., why a particular organization or program is
implementing or not implementing a particular SE element versus the SEM discussion
regarding an SE element’s purpose).
The what, or products and activities of SE, directly connects the SEM and SEMP, and
the when provides a secondary correlation.
Lesson 1

SEP and SEMP
Needed for Success
System Engineering Plan
§ What needs to happen from
the customer perspective
§ Approach to key areas
– Requirements
– Technical Staffing and
Organizational Planning
– Technical Baseline
Management
– Technical Review Planning
– Integration with Program
Management
§ Provides contractor guidance
for Systems Engineering
applied to program
System Engineering
Management Plan
§ Response to SEP and contract
§ Defines contractor technical
planning processes
§ Articulates details of
– Processes
– Tools
– Organization
§ Describes activities involved in
transforming requirements
into deliverables
§ Includes subcontract
management
99
Lesson 1

What is the Systems Engineering Plan?†
§ Articulates and communicates technical planning and
management approach to program team, stakeholders,
and contractor teams (including bidders if provided
with Request for Proposal (RFP)).
§ Captures integration of both government and
contractor Systems Engineering (SE) activities, roles,
and responsibilities over the acquisition and
sustainment life cycle.
§ Provides expected management interactions and
impacts of their respective processes not only by
addressing program-tailored processes, but also the
"who, when, and to what result(s)”
100
† How to Prepare a Systems Engineering Plan (SEP) that Works, Sharon Vannucci and Lisa Reuss, Systems and Software
Engineering Office of the Deputy Under Secretary of Defense for Acquisition and Technology, ODUSD(A&T) 2009 INCOSE SE
Summit March 4, 2009
Lesson 1 SEP

Traits of the SEP†
§ Defines government (customer) technical planning
expectations
– What needs to happen from customer perspective
§ Describes overall approach in key areas
– Requirements
– Technical Staffing and Organizational Planning
– Technical Baseline Management
– Technical Review Planning
– Integration with Program Management
§ Provides contractor guidance for Systems Engineering as
applied to the acquisition program at hand
§ Identifies to program management and contract personnel
the essential Systems Engineering activities and products
required
101
Lesson 1
† Defense Acquisition Guide, 16 September 2012, Section 4, Systems Engineering
SEP

Preparing the SEP†
Program Technical Requirements
Engineering Resources and Management
Technical Activities and Products
102
1
2
3
† Systems Engineering Plan (SEP) Outline, 20 April 2011 Version 1.0, 04/20/2011, OPR: ODASD (Systems Engineering)
Lesson 1 SEP

Program Technical
Requirements
§ Architectures and Interface Control
– Program’s DODAF architecture development efforts.
– A system physical architecture diagram (delineating
physical interfaces), if available.
– A system functional architecture diagram (delineating
functional interfaces), if available.
– How software architecture priorities will be developed and
documented.
– How architecture products are related to requirements
definition.
– How engineering and architecture activities are linked.
§ Technical Certifications
– Current technical and compliance certifications
103
1
Lesson 1 SEP

Engineering Resources and
Management
§ Technical Schedule and Schedule Risk
Assessment
§ Engineering Resources and Cost/Schedule
Reporting
§ Engineering and Integration Risk Management
§ Technical Organization
§ Relationships with External Technical
Organizations
§ Technical Performance Measures and Metrics
104
2
Lesson 1 SEP

Technical Activities
and Products
§ Results of Previous Phase SE Activities
§ Planned SE Activities for the Next Phase
§ Requirements Development and Change
Process
§ Technical Reviews
§ Configuration and Change Management
Process
§ Design Considerations
§ Engineering Tools
105
3
Lesson 1 SEP

Traits of the SEMP†
§ Responsive to the contract and the SEP
§ Defines contractor (supplier) technical planning
– How it will be accomplished from the contractor perspective
§ Contractor further develops planning outlined in the SEP
§ Project (Supplier) team articulates details of their
– Processes
– Tools
– Organization
– etc.
§ Describes activities involved in the transformation from
requirements to solution
§ Includes integration of subcontractor planning
106
† Systems Engineering Plan and Systems Engineering Management Plan Alignment, NDIA
11th Annual Systems Engineering Conference October 21, 2008
Lesson 1 SEMP

What is the Systems Engineering
Management Plan (SEMP)?
The System Engineering Management Plan
(SEMP) establishes the overall plan for the
system engineering management of the project.
The SEMP identifies and describes the project
organization, roles and responsibilities, overall
tasks, and engineering management planning
required to control the design, development,
fabrication, and tests associated with the
project.
107
Lesson 1 SEMP

Preparing the Systems Engineering
Management Plan (SEMP)†
Technical Program Planning and Control
System Development Process
Engineering specialty Integration
108
Lesson 1
1
2
3
SEMP
† INCOSE Systems Engineering Handbook v. 3.2, INCOSE-TP-2003-002-03.2, January 2010, §5.1.2.2

Technical Program Planning and
Control
§ Systems Engineering Organization
§ Systems Engineering IPT interfaces
§ Technical Risk Analysis
§ Engineering methods
§ Program and Technical Reviews
§ Interface control
109
Lesson 1 SEMP
1

Systems Development
Processes
§ Integrated Product Review Process
§ Systems Engineering Processes
§ System Design Processes
§ Change Control Processes
110
Lesson 1 SEMP
2

Specialty Engineering
Integration†
§ Test & Evaluation
§ Software Engineering
§ Integrated Logistics Support
§ Design Engineering
§ Manufacturing and
Producibility
§ Quality Assurance
§ Reliability and
Maintainability
§ Spectrum Management
§ Concept Development
§ Architecture Engineering
§ System Safety Engineering
§ Acquisition Systems
Protection & International
Program Security
§ Survivability
§ Human Systems Integration
§ Mass Properties
§ EMI/EMC
§ Parts, Materials & Processes
§ Information Assurance
§ Netcentric Engineering
§ Environmental Engineering
§ Prognostics, Diagnostics
Health Management
111
Lesson 1 SEMP
3
† SMC Specialty Systems Engineering, Specialty Engineering Disciplines, Framework and Descriptions, Space and Missile
Systems Center, Volume 2, 1st
Edition, 3 October 2011,

SEMP and Technical Plans
112
Program
Management Plan
(inc IMP/IMS)
Systems
Engineering
Management Plan
Software Development Plan Hardware Development Plan
Configuration Management
Plan
Quality Assurance Plan
IV&V Plan
Lesson 1 SEMP

Other Plans
Plans directly flowing from the SEMP
§ Risk and Opportunity Management Plan
§ Configuration Management Plan
Other plans related to the SEMP
§ Material Management Plan
§ Subcontractor Management Plan
§ Property Management Plan
§ Mission Assurance Implementation Plan
§ Measurements and Analysis Plan
§ Software Development Plan
§ Facilities Management Plan
§ System Security Management Plan
113
Lesson 1

All This Guidance Needs to
Connect with Business Benefits for
it to actually be useful
114
Lesson 1

Benefits of Applying Systems
Engineering†
§ SE ensures the effective development and delivery of
capability through the implementation of a balanced
approach with respect to cost, schedule, performance, and
risk using integrated, disciplined, and consistent activities
and processes regardless of the acquisition life cycle.
§ SE enables the development of engineered resilient
systems that are trusted, assured, and easily modified
(agile).
§ SE planning identifies the most effective and efficient path
to deliver a capability, from identifying user needs and
concepts through delivery and sustainment.
§ SE event-driven technical reviews and audits assess
program maturity and determine the status of the technical
risks associated with cost, schedule, and performance
goals.
115
† Defense Acquisition Guide 16 September 2013
Lesson 1

Benefits of Applying a SE Framework†
§ Focuses systems management across the life
cycle by providing:
– Insight into what should be assessed
– Holistic view of engineering a system (software,
hardware, humans, and processes)
– A process framework that:
• Is easy to tailor to meet project and organization needs
• Reduce development risk
– A basis for
• Stage-gate life cycle models including agile development
• Communication between all stakeholders
• Coordination of work processes
• Integration of management agreements with technical
management processes
116
† Adapted from ISO/IEC JTCI/SC7/WG7
Lesson 1

Critical Success Factors
Before Proceeding to Lesson 2
§ Principles are Immutable
§ Practices are many times domain specific
§ Processes tailored to business need
§ Systems Engineering Management Plan
(SEMP) states how Practices and Processes are
implemented on specific programs
118
Lesson 1

Another Critical
Success Factor†
§ Systems Engineering Management Plan (SEMP) is
the supplier plan to conduct, manage, and
control of the integrated engineering effort.
– The SEMP States how the program will reach DONE
§ Systems Engineering Plan (SEP) is the acquirer’s
technical planning document required for
milestone approval on the program.
– The SEP States what DONE looks like
§ Keeping these aligned is a Critical Success Factor
for all programs applying Systems Engineering.
† Systems Engineering Plan and Systems Engineering Management Plan Alignment NDIA 11th Annual Systems Engineering Conference October
21, 2008, Chet Bracuto DoD OUSD A&T (SSE) Bob Scheurer P.E., P.M.P. Boeing Integrated Defense Systems 119
Lesson 1

Lesson 2
The Immutable Principles of System
Engineering
These principles are applicable to all
domains, all technologies, all
businesses, services or outcomes and
all missions or products
120

Principles of Systems Engineering†
Concept Development
Requirements Engineering
System Architecture
System Design and Development
System Integration
Test and Evaluation
Transition to Operations and Maintenance
121
Lesson 2
1
2
3
4
6
5
† MITRE Systems Engineering Guide, http://www.mitre.org/publications/systems-engineering-guide/systems-engineering-guide
7

Concept Development
§ Operation Needs Assessment
§ Concept of Operations
§ Operational Requirements
§ High-Level Conceptual Definition
122
Lesson 2
1
Concept Development

Operational Needs
Assessment
§ Determine the specific requirements of the needs
assessment process that apply.
§ Identify specific stakeholders in needs assessment
process.
§ Identify and obtain support from operational or
capability domain experts.
§ Develop a needs assessment plan and schedule,
including scenario-driven experiments, gap analysis,
tradeoff studies.
§ Identify analytical tools necessary to define and
quantify needs.
§ Implement and conduct needs assessment.
123
1
Concept Development
Lesson 2

Concept of Operations
§ The objective of the ConOps is to
– Provide end-to-end traceability between operational
needs and captured source requirements.
– Establish a high-level basis for requirements that
supports the system over its life cycle.
– Establish a high-level basis for test planning and
system-level test requirements.
– Support the generation of operational analysis models
(use cases) to test the interfaces.
– Provide the basis for computation of system capacity.
– Validate and discover implicit requirements.
124
1
Concept Development
Lesson 2

Concept of
Operations (ConOps)
§ Contents of the ConOps includes
– The existing system (manual or automated) the
user wants to replace.
– Justification for a new or modified system
(including restrictions on that system).
– A description of the proposed system.
– Scenarios highlighting use of the system in the
user's environment including internal and external
factors.
125
1
Concept Development
Lesson 2

Operational
Requirements
§ The operational requirements must answer:
– Who needs this requirement?
– Who will be operating the system?
– What functions/capabilities must the system
perform?
– Where will the system be used?
– When will the system be required to perform its
intended function and for how long?
– How will the system accomplish its objective?
– How will the requirements be verified?
126
1
Concept Development
Lesson 2

Operational
Requirements
§ Developing the Operational Requirements
– Identify stakeholders
– Elicit requirements for what the system must
accomplish and how well.
– Define constraints imposed by agreements or
interfaces
– Establish critical and desired user performance
– Establish measures of effectiveness and suitability
127
1
Concept Development
Lesson 2

High-Level
Conceptual Design
§ Main objectives of the system
§ Who will use the resulting system
§ What are the properties of the resulting
system
§ What capabilities does the system provide
§ What other systems or processes does this
system interface with
128
1
Concept Development
Lesson 2

§ Analyze and define requirements
§ Elicit, collect, and develop requirements
§ Address uncertainty in requirements
129
2
Lesson 2

A Requirement is …”A statement identifying a capability, a
physical characteristic, or a quality factor that bounds a
product or process need for which a solution will be
pursued.”
– IEEE Standard 1220–2007–05–15
The hardest single part of building a system is deciding what
to build …
... No other part of the work so cripples the resulting system
if done wrong. No other part is more difficult to rectify later.
– Fred Brooks “No Silver Bullet,” 1987
What Is a Requirement?
130
Performance-Based Project Management(sm), Copyright © Glen B. Alleman, 2012, 2013
Lesson 2
2

We’ve All Been Here Before
131
I want it
over my
ears
Got it
2
Lesson 2 Requirements Engineering

Steps in the Requirements
Development Process
§ Perform Fact finding
– Identify and capture needed capabilities
§ Gather and Classify Requirements
– Capture requirements needed to implement
capabilities
§ Prioritize Requirements
– Allocate requirements to appropriate components in
the system hierarchy
§ Integrate and Validate Requirements
– Establish methods for verification of each requirement
– Ensure product complies with requirements to deliver
capabilities
132
Lesson 2
2

Performance Fact Finding
§ Produce an overall statement of the problem
in the operational context.
§ Develop the overall operational and technical
objectives of the target system.
§ Defined the boundaries and interfaces of the
target system.
133
Lesson 2
2

Gather and Classify
Requirements
§ Gather required system capabilities,
functional, nonfunctional and environmental
requirements, and design constraints.
§ Build the Top Down capabilities and functional
decomposition of the requirements in a
Requirements Management System.
134
Lesson 2
2

Evaluate and
Rationalize Requirements
§ Answer the question “why do I need this?” in
terms of operational capabilities.
§ Build a cost / benefit model using probabilistic
assessment of all variables, their
dependencies, and impacts.
§ For all requirements, perform a risk
assessment to cost and schedule.
135
Lesson 2
2

Prioritize Requirements
§ Determine criticality for the functions of the
system.
§ Determine trade off relationships for all
requirements to be used when option
decisions must be made.
§ For all technical items, prioritize their cost and
dependency.
136
Lesson 2
2

Integrate and
Validate Requirements
§ Address the completeness of requirements by
removing all “TBD” items.
§ Validate that the requirements are traceable
to system capabilities, goals, and mission.
§ Resolve any requirements inconsistencies and
conflicts
137
Lesson 2
2

Systems Architecture
§ Architecture Development
§ Architecture Frameworks
– U.S. Department of Defense Architecture
Framework (DoDAF)
– The Open Group Architecture Framework (TOGAF)
– Object-oriented with Unified Modeling Language
– Spewak architecture process and Zachman
Framework
138
Lesson 2
3

Architecture Development
§ Define the architecture purpose, value, and decisions it will
support.
§ Create, refine, and update the architecture in an iterative way
throughout the acquisition life cycle.
§ Validate the architecture will meet expectations when
implemented.
§ Define roles for team members to guide and coordinate their
efforts.
§ Create estimates and schedules based on the architectural
blueprint.
§ Use the architecture to gain insight into project performance.
§ Establish a lightweight, scalable, tailorable, repeatable process
framework
139
Systems Architecture
3
Lesson 2

System Design
and Development
§ Assessing Design’s Ability to Delivery the
Needed Capabilities defined in the technical
and operational requirements.
§ System-Level Technical Requirements
§ Top-Level System Design
140
Lesson 2
4

Systems Integration
§ Integration Testing
§ Develop and Evaluate Integration and
Interoperability
§ Identify and Assess Integration and
Interoperability
§ Interface Management
141
Lesson 2
5

Test and Evaluation
§ Test and Evaluation Plans and Procedures
§ Certification Patterns
§ Test and Evaluation
§ Implementation, O&M, and Transition
§ Verification and Validation
142
Lesson 2
6

Transition to Operations and
Maintenance†
§ Evaluate the end product, personnel, and
enabling product readiness for product transition
§ Prepare the end product for transition
§ Transition the end product to the customer with
required documentation based on the type of
transition required
§ Prepare sites, as required, where the end product
will be stored, assembled, integrated, installed,
used, and/or maintained
§ Capture product implementation work products
143
Lesson 2
7
† NASA Systems Engineering Handbook, SP-2007-6105

Lesson 3
System Engineering Practices
Practices put the Principles to work.
Without Practices, Principles are just
nice book learning waiting for a
problem to solve
144

Systems Engineering Practices†
Product Implementation
Product Integration
Product Verification
Product Validation
Cross Cutting Technical Management
General Management
145
Lesson 3
1
2
3
4
5
6
†NASA Systems Engineering Handbook, SP-2007-6105

Systems Engineering
Practices Live in the Vee
146
We’ll develop the Processes of these in Lesson 4 after we
development the Practices
Lesson 3

Systems Engineering Management
Plan (SEMP) Is Home For Practices
The System Engineering Management Plan
(SEMP) describes the efforts for planning,
controlling and conducting a fully integrated
engineering effort.
The SEMP is used to understand and evaluate
the engineering work efforts as part of the
program monitoring process.
147
Lesson 3

Two Primary Practices
Product Realization
§ Implementation
§ Integration
§ Verification
§ Validation
§ Transition
Technical Management
§ Technical Planning
§ Requirements
Management
§ Risk Management
§ Configuration
Management
§ Data Management
§ Technical Assessment
§ Decision Analysis
148
Lesson 3

PRODUCT REALIZATION PROCESS
The product realization processes are applied to each
operational/mission product in the system structure starting from the
lowest level product and working up to higher level integrated products.
149
Lesson 3

Product Implementation†
§ Prepare to conduct implementation
§ Purchase, Make, Reuse
– Purchase from commercial vendor
– Make or develop product
– Reuse existing product
§ Capture work products
– Drawings
– Designs
– Model descriptions
150
Lesson 3
1

Product Integration†
§ Prepare production integration strategy
– Detail planning
– Integration sequences and procedures
– Product configuration
§ Prepare lower level products
§ Prepare integration environments
§ Assemble and integrate products into end
product
§ Conduct functional testing
§ Prepare product support documentation
§ Capture work product
151
Lesson 3
2

The Right Product That Does
The Right Things
Verification
The assurance that a product,
service, or system meets the
needs of the customer and
other identified stakeholders.
Validation
The evaluation if the product,
service, or system complies
with a regulation,
requirement, specification, or
imposed condition. It is often
an internal process.
152
Lesson 3
3 4

Product Verification†
§ Verification planning
§ Verification preparation
§ Conduct verification
§ Analyze verification results
§ Capture verification work products
153
Lesson 3
3

Types of Verification†
§ Analysis
– Modeling or analytical techniques to predict suitability
of a system to meet stakeholder expectations
§ Demonstration
– Showing the use of product achieves specified
requirements
§ Inspection
– Visual examination of end product
§ Testing
– Use of the end product to obtain detailed date
needed to verify performance
154
Lesson 3
3

Product Validation†
§ Validation planning
§ Perform Validation
§ Analyze outcomes of validation
§ Report validation outcomes
§ Capture work products
155
Lesson 3
4

TECHNICAL MANAGEMENT
Are used to manage the technical development of the system
increments, including the supporting or enabling systems. Consist of:
Technical Planning, Requirements Management, Interface Management,
Risk Management, Configuration Management, Technical Data
Management, Technical Assessment and Decision Analysis.
156

Cross Cutting
Technical Management†
§ Technical Planning
§ Requirements Management
§ Technical Risk Management
§ Technical Data Management
§ Technical Assessment
§ Decision Analysis
157
Lesson 3
5
† NPR 7123.1B NASA Systems Engineering Processes and Requirements

General Management
§ Planning
§ Cost And Schedule
§ Subcontractor Management
§ Integration Engineering
§ Communications
§ Risk And Opportunity
§ Decision Processes
§ Program Reviews
158
Lesson 3
6

General Management
Planning
§ Program planning
– Establish PMB
– Identify on-ramps/off-ramps
– Develop PM plans
§ Execution management
– Maintain baselines
§ Execution monitoring
– Technical performance measures
– Earned value management
– Risk and opportunity management
§ Execution control
– Analysis and assessment
– CCB/ERB
159
Lesson 3
6

General Management
Cost and Schedule
160
Program Manager
§ Establish initial
program
baseline
§ Develop cost
control process
(EVM)
Execution Management
§ Execute processes
§ Maintain baselines
§ Use CAIV in Trade
Studies and Decision
making
Monitor
§ Take
performance
measures
§ Conduct cost
review
Control
§ Analyze and Assess
§ Corrective actions
§ Architecture and PM decisions
based on study results
§ Anticipate future trends
Status
Lessons
Learned
Processes
Baselines
Actions
Lesson 3
6

General Management
Risk and Opportunity
161
Risk and
Opportunity
Management
Program
Manager
Chief Engineer
Customer
Operations
Manager
Subcontract
Manager
Risk and
Opportunity
Manager
Lesson 3
6

Roles and Responsibilities in
Risk and Opportunity Management
§ Risk and Opportunity Manager
– Process owner
– Assess mitigation processes
– Risk forums and meetings
§ Program Manager
– Chairs ROMB
– Provides resources
– Validates scope impacts
– Ensures compliance across program,
§ Chief engineer
– Supports analysis
– Supports mitigation and realization plans
– Reviews TPM plans
162
Lesson 3
6

Program Risk Analysis†
§ Program risks are elicited from:
– Requirements
– Technology
– Logistics
– Management
– Schedule
163
† Systems Engineering Management Plan (SEMP) for the Unmanned Control & Tracking Systems (UCATS), George Mason University, Department
of Systems Engineering and Operations Research, SYST 798 Applied Project Course, Fall 2009
6
Lesson 3

Lesson 4
System Engineering Processes
Processes are how the Principles and
Practices are applied to a specific
problem. In this course, we’ll apply
them to our UAV Program.
164

Systems Engineering Processes
Agreement Processes
Enterprise Processes
Project Management Processes
Technical Management Processes
165
Lesson 4
2
3
4
1

AGREEMENT PROCESSES
Specifies the requirements for the establishment of agreements between
organizations
166
1
Lesson 4

Agreement Processes
§ Acquisition processes
§ Supply processes
167
Lesson 4

Acquisition Processes
§ Establish a plan for how the acquisition will be
conducted.
§ Prepare a request for the supply of a product or
service.
§ Communicate the request for the supply of a product
or service to identified suppliers.
§ Select a supplier.
§ Negotiate an agreement with the supplier.
§ Assess the execution of the agreement.
§ Confirm that the delivered product or service complies
with the agreement.
§ Make payment or provide other agreed consideration
to the supplier for the product or service rendered.
168
Lesson 4

Supply Processes
§ Determine the existence and identity of an acquirer who has, or
who represents a party or parties having a need for a product or
service.
§ Evaluate a request for the supply of a product or service to
determine feasibility and how to respond.
§ Prepare a response that satisfies the solicitation.
§ Negotiate an agreement with the acquirer.
§ Execute the agreement according to the Supplier’s established
project plans and in accordance with the agreement.
§ Assess the execution of the agreement.
§ Deliver the product or service in accordance with the agreement
criteria.
§ Accept and acknowledge payment or other agreed consideration.
§ Transfer the responsibility for the product or service to the acquirer,
or other party, as directed by the agreement.
169
Lesson 4

ENTERPRISE PROCESSES
Enterprise processes manage the organization’s capability to acquire or
supply products or services throughout the lifecycle. Enterprise processes
provide resources and infrastructure to support projects and ensure
satisfaction of organizational objectives and established agreements.
170
2
Lesson 4

Enterprise Processes
§ Environment management
§ Investment management
§ Life cycle management
§ Resource management
§ Quality management
171
Lesson 4

PROJECT MANAGEMENT
PROCESSES
Managing the project and assessing the technical and operational
performance starts with a Systems Engineering process. Using Measures
of Effectiveness, Measures of Performance, and Technical Performance
Measures – developed in Lesson 4 - the project status presents Physical
Percent Complete in units of measure meaningful to the decision maker.
This is the process needed to increase the probability of project success.
172
3
Lesson 4

§ Planning
§ Assessment
§ Project Controls
§ Decision-making
§ Risk Management
§ Information Management
173
Lesson 4

Performance Targets Are Required Before
Performance Measures Can Be Useful
174
Lesson 4

Project Management
Processes
§ Identifying what “done” looks like
§ Planning and schedule the work to reach
“done”
§ Identifying impediments to reaching “done”
§ Measuring progress toward “done”
§ Taking corrective action to arrive on-time, on-
budget, and deliver needed capabilities to the
customer
175
Lesson 4
4

Event Based Planning
§ SRR (Systems Requirements Review)
§ SFR (System Functional Review)
§ PDR (Preliminary Design Review)
§ CDR (Critical Design Review)
§ ASFUT/GSFUT (Air System/Ground System
Functional Unit Test)
§ TRR (Test Readiness Review)
§ SVR (System Validation Review)
§ PRR (Production Readiness Review)
176
Lesson 4

INCOSE VEE and Our IMP/IMS
177
Combine DT&E/O Demonstration`
System to Specified User Needs and
Environmental Constraints
Interpret User Needs, Refine System
Performance Specifications, and
SRR
Develop System Functional Specifications
and System Verification Plan
SFR
Evolve Functional Performance
Specifications into CI Functional (Design To)
Specification and CI Verification Plans
PDR
System DT&E, Verify System
Functionality & Constraints Compliance
to Specifications
TRR
Integrated DT&E, Verify Performance
Compliance to Specifications CI
Verification DT&E
Specifications into Product (Build To)
Documentation and Verification Plans
CDR Fabricate, Assemble, Unit Test to
Build To Documentation
Individual CI Verification DT&E
ASFUT GSFUT
System Decomposition System Realization
System Development and Demonstration
SVR PRR
Lesson 4
4

Different Levels of Detail in the UAV
Model and Associated Activities†
178
† System Testing in the Avionics Domain, von Aliki Ott, zur Erlangung des Grades einer Doktorin der Ingenieurwissenschaften, Vorgelegt
im Fachbereich 3 (Mathematik & Informatik) der Universit¨at Bremen im Oktober 2007
Lesson 4
4

Connecting The Dots for the Project
Management Processes
§ Integrated Master Plan and Integrated Master
Schedule vertically and horizontally traceable.
§ The left side of the Vee and the right side are
connected by the IMP program events.
179
4
Lesson 4

Quick View to IMP/IMS
§ Vertical traceability defines the increasing maturity of
key deliverables
§ Horizontal traceability defines the work activities
needed to produce this increasing maturity
§ Both are needed, but the vertical traceability is the
starting point
§ Program Events, Significant Accomplishments, and
Accomplishment Criteria must be defined before the
horizontal work activities can be identified
§ For all IMP elements, Key Risks must be identified and
assigned from Day One, even without mitigations
180
4
Lesson 4

A Critical Understanding of the IMP
The IMP defines the connections between the Product maturity – Vertical – and the
implementation of this Product maturity through the Functional activities – the Horizontal 181
4
Lesson 4

The 1st Principle
IMP Building is a Full Contact Sport
182
4
Lesson 4

TECHNICAL MANAGEMENT
PROCESSES
Systems Engineering provides the Programmatic framework for the
success of the project. Technical disciplines produce the products and
services, within that framework that enable the needed capabilities to be
delivered on time, on budget, that meet the requirements of the
customer
183
Lesson 4
4

Technical Management
Processes
Work Breakdown Structure (WBS)
Integrated Master Plan (IMP)
Integrated Master Schedule (IMS)
Cost Basis of Estimate (BoE)
Risk Management Plan (RMP)
184
Lesson 4
1
2
3
4
5

WORK BREAKDOWN STRUCTURE
The WBS is a product-oriented family tree composed of hardware,
software, services, data, and facilities. The family tree results from
Systems Engineering efforts during the acquisition of a defense materiel
item.
A WBS displays and defines the product, or products, to be developed
and/or produced. It relates the elements of work to be accomplished to
each other and to the end product. In other words, the WBS is an
organized method to breakdown a product into sub-products at lower
levels of detail.†
185
† MIL-STD-881C, Work Breakdown Structure for Defense Materiel Items, 3 October 2011, AMSC 9231
Lesson 4
1

§ Defines the total System products and services that
produce those products
§ Provides the framework for planning, prioritizing,
managing, and tracking all work done on the program
– Products
– Supporting services
– Facilities
§ Provides the framework for
– Cost Structure for estimating and cost reporting
– Resource allocation
– Status Reporting
– Performance Measurement
– Identify and managing program risk
186
What is the Work Breakdown
Structure?
Lesson 4
1

Physical Architecture
and the WBS
187
Lesson 4
1

Systems Architecture Drives
the Work Breakdown Structure
188
Lesson 4
1

§ End Products used to implement the mission
– Based on the System (Physical) Architecture
§ Enabling Products and Services
– Products and services required to develop, produce,
and support the end items
– Based on Life Cycle
§ The first three (end product) WBS levels:
– Level 1: Overall System
– Level 2: Major Element (or Segment)
– Level 3: Subordinate Components (or Prime Items)
§ Levels 4+ continue the decomposition to the
Configuration Item (CI) level
189
Structure of our UAV WBS
Lesson 4
1

Three WBS’s for Our UAV
Program WBS
§ High-Level (First 3
levels)
§ Provides Program
Structure
§ Generally
developed/controlled
by Customer
(Government)
Contract WBS
§ Detailed (Levels 4+)
§ Provides framework
for Contract Work
Packages and Costing
§ Generally developed
by Contractor
§ Generally follows
Program WBS
190
Subcontract WBS
§ Detailed (Level 4+)
§ Provides framework
for Subcontract Work
Packages and Costing
§ Generally developed
by Subcontractor
§ Generally follows
Contract WBS
Lesson 4
1

§ Systems Engineering generally provides structure for:
– Levels 1-3 of all WBS based on MIL-HDBK-881C
– Levels 4+ of the End Product and Systems Engineering portions of the
WBS
§ Program Control and suborganization participation
– Develop Project Management and SE portions of WBS
– Develop other parts of WBS as appropriate SE support
– Oversee entire WBS development since it serves as framework for
project control and management (costing, resourcing, reporting, risk,
etc.)
§ Design and Development IPT participation (development of End Product
portion of WBS)
§ OT&E participation (development of T&E portion of WBS)
§ Other suborganization participation (to ensure all work activities are
properly represented)
How is the WBS Built?
If we do this By the book, Systems Engineering generally leads WBS development,
since the WBS should be based on the system (physical) architecture
But in reality, Program Controls often develops the WBS with SE support 191
Lesson 4
1

Using a Generic UAV WBS to
Capture Technical Architecture
192
§ Technical performance for
each subsystem with
impacts on cost, schedule,
and technical performance
are defied in the WBS
Dictionary.
§ Each risk held in the Risk
Register, marked with WBS.
§ WBS number used to trace
the risk to the IMS, PE, SA,
AC, CA, WP.
§ Risk retirement plans can
then be shown by PE or
Milestone in a risk waterfall
Lesson 4
1

Level 2 Unmanned
Air Vehicle (UAV)
193
Lesson 4
1

Level 3 UAV Avionics
194
Lesson 4
1

Level 4 UAV Avionics – GN&C
195
Lesson 4
1

What’s Our Goal?
196
Count Cows without having to go
count cows
Lesson 4

§ The WBS is an essential tool for the organization and coordination of
Systems Engineering processes, and is a product of the Systems
Engineering process
§ The importance of the WBS extends to business professionals and
contracting officers.
§ The needs of all stakeholders must be considered in its development
§ The Program Office develops the Program WBS (and a high level
contract WBS for each contract). Each contractor develops the lower
levels of the Contract WBS for their contract
§ The System (Physical) Architecture provides the basic structure for
the “Product” part of the WBS
§ SOW tasks should flow from the WBS
§ The WBS provides a structure for organizing IPTs and tracking
metrics
197
Summary of the WBS
Lesson 4
1

INTEGRATED MASTER PLAN
The IMP is an event-based plan consisting of a hierarchy of program
events, with each event being supported by specific accomplishments,
and each accomplishment associated with specific criteria to be satisfied
for its completion.
198
2
Lesson 4

199
The IMP tells us where is the program going?
The Plan describes where we are going, the various paths we can take to reach our
destination, and the progress or performance assessment points along the way to
assure we are on the right path.
These assessment points measures the “maturity” of the product or service against
the planned maturity. This is the only real measure of progress – not the passage of
time or consumption of money.
The Integrated Master Plan (IMP) Is A Strategy For
The Successful Completion Of The Project
Lesson 4

Integrated Master Plan
§ Vertical traceability defines the increasing maturity of
key deliverables
§ Horizontal traceability defines the work activities
needed to produce this increasing maturity
§ Both are needed, but the vertical traceability is the
starting point
§ Program Events, Significant Accomplishments, and
Accomplishment Criteria must be defined before the
horizontal work activities can be identified
§ For all IMP elements, Key Risks must be identified and
assigned from Day One, even without mitigations
200
Lesson 4
2

Risk Management
Building the IMP Starts at the RFP
201
SOO
ConOps
SOW
Technical and Operational
Requirements
CWBS &
CWBS Dictionary
(IMP)
Integrated Master Schedule
(IMS)
Earned Value Management
System
Objective Status and Essential Views to support the proactive management
processes needed to keep the program GREEN
Performance Measurement Baseline
Measures of
Effectiveness
Measures of
Performance
Measures of
Progress
Key Performance Parameters
Program Specific
Key Performance Parameters
Technical Performance
Measures
WBS
CWBS
Lesson 4
2

The IMP / IMS Structure
202
IMS
IMP
Describes how program
capabilities will be
delivered and
how these
capabilities will
be recognized
as ready for
delivery
Supplemental Schedules
Work Packages and Tasks
Criteria
Accomplishment
Events
or
Milestones
Lesson 4
2

The IMP/IMS provides Horizontal
and Vertical Traceability of
progress to plan
§ Vertical traceability AC è SA è PE
§ Horizontal traceability WP è WP è AC
203
Program Events
Define the maturity
of a Capability at a point in
time.
Significant Accomplishments
Represent requirements
that enable Capabilities.
Accomplishment Criteria
Exit Criteria for the Work
Packages that fulfill Requirements.
Work
Package
Work
Package
Work
Package
Work
Package
Work
Package
Work
Package
Work
package
Lesson 4
2

The IMP’s role
during Execution
204
Program Execution
PMB for IBR
Proposal Submittal
DRFP & RFP
Performance Measurement Baseline
Tasks (T)
BOE
% Complete
Statement of Work
Program Deliverables
IMP
Accomplishments (A)
Criteria (C)
EVMS
Events (E)
Budget Spreads by CA & WP
CAIV
Capabilities Based Requirements
X BCWS =
Probabilistic Risk Analysis
=
Time keeping and ODC =
Technical Performance Measure
BCWP
ACWP
Cost & Schedule Risk Model
BCWS
Decreasing technical and programmatic risk using Risk Management Methods
IMS
Physical % Complete
Continuity and consistency from DRFP through Program Execution
WBS
Lesson 4
2

INTEGRATED MASTER SCHEDULE
The IMS is an integrated, networked schedule containing all the detailed
work packages and planning packages necessary to support the
Integrated Master Plan†
205
3
Lesson 4
† Integrated Master Plan and Integrated Master Schedule Preparation and Use Guide, V0.9, October 21, 2005

§ The WBS is Paramount
§ The IMP defines the increasing maturity of the
program’s deliverables (end item)
§ The IMS sequences the Work Packages
containing the work activities to produce the
End Item Deliverables
§ The IMS is built from the IMP, with WBS
coding to assure coverage of all deliverables
206
Quick View of Building the IMS
Lesson 4

The Integrated Master Schedule
§ The horizontal sequence of work activities
that produce increasing maturity of the
product or services delivered by the program
207
Lesson 4

GAO Schedule Assessment Guide†
tells us to …
Capture All Activities
208
1
2
3
4
5
6
7
8
9
10
Sequence These Activities
Assign Resources To These Activities
Establish Duration For These Activities
Verify Schedule Is Traceable Horizontally And Vertically
Confirm Valid Critical Path – schedule matches program
Ensure Reasonable Total Float
Conduct Schedule Risk Analysis
Update Schedule With Actual Progress
Maintain Baseline with Repeatable Process
Lesson 4
† GAO Schedule Assessment Guide GAO-12-120G

Scheduling and Planning Excellence
Guide (PASEG) tells us to† …
Capture all discrete, authorized work effort in the IMS
1
2
3
4
5
6
7
8
Integrate IMS logic Vertically 1st than Horizontally
Assure IMS is complete, traceable, documented
Assure accurate progress through the status date
Show meaningful critical paths – schedule and program
Use IMS for timely and effective decision making
Align IMS with actual and projected resource demands
Baseline and Maintain IMS using repeatable processes
† The PASEG or GAO Guide is not IMP centric, so this advice is edited to focus on the IMP first, then to build
the credible IMS. Without the IMP, the sequence of work in the IMS fails to show the increasing maturity
of the deliverables through their Measures of Performance, Technical Performance Measures, and Risk
Retirement assessment. Start with the IMP, and only then development the IMS. 209
Lesson 4

As Systems Engineers Why Do We Care
About the IMS?
The planned activities for the project that assure
the Measures of Effectiveness (MOE)s, Measures
of Performance (MOP), Technical Performance
Measures (TPM) all in support of delivered
capabilities have to show up on time, on
schedule, and actually work for the customer to
be satisfied.
The IMS tells us in what order to perform the
work to make this happen.
210
Lesson 4

This is Where the Action Takes Place
211
Lesson 4

BASIS OF ESTIMATE
The Basis of Estimate (BoE) is a description of how cost estimate was
obtained for each WBS element for which a cost is estimated
212
4
Lesson 4

The Cost Estimating Problem
214
I need an estimate for all the software for the GN&C boxes
for the proposal, and I need it in 2 weeks
Here’s a 80% confidence with a
95% Bayesian interval
Hey, none of these intervals turned out
to contain the true cost of that software
Of course not! This was
De Finetti style†, fully
coherent representation
of your beliefs. They’re
not confidence intervals.
† In probability theory, de Finetti's theorem explains why
exchangeable observations are conditionally independent given
some latent variable to which an epistemic probability distribution
would then be assigned. It is named in honor of Bruno de Finetti.
The variables in the cost
estimate are not independent,
identically, distributed (i.i.d.)
Lesson 4

The following excerpts from an Air Force study done 50
years ago titled: "Predictability of the Costs, Time, and
Success of Development" RAND 1959† are revealing:
§ "To a large extent, the differences which arise do so
over the question of the "extent" of the uncertainty in
development - over questions like, "Are estimates of
cost of production likely to be off by 25 percent or 300
percent?" . . . In this paper, we present the results of
some recent research into the extent and nature of the
uncertainty in new developments.
1) Early estimates of important parameters are usually quite inaccurate …
such estimates are strongly "biased" toward over-optimism …
2) The accuracy of estimates is a function of the stage of development, i.e.
estimates improve as development of the item progresses …
215
Before we start …
Lesson 4

§ Do we know what kind of cost
estimate we’re looking for?
216
A bigger problem in cost estimating
Accurate And
Precise?
Accurate But Not
Precise?
Precise But Not
Accurate
Neither Accurate
Nor Precise
Estimating accuracy is an indication of the degree to which the
final cost outcome may vary from the single point estimate –
Larry Dysert, AACE Technical Board Chairman
Lesson 4

RISK MANAGEMENT PLAN
“It is moronic to predict without first establishing an error rate for the
prediction and keeping track of one’s past record of accuracy”
— Nassim Nicholas Taleb, Fooled By Randomness
217
5
Lesson 4

INCOSE Guide Says†
§ The purpose of the Risk Management Process
is to identify, analyze, treat and monitor the
risks continuously.
§ The Risk Management Process is a continuous
process for systematically addressing risk
throughout the life cycle of a system product
or service. It can be applied to risks related to
the acquisition, development, maintenance or
operation of a system.
218
Lesson 4
5
† INCOSE Systems Engineering Handbook, 3.2, §5.4 Risk Management Processes

§ The effectiveness of risk management
depends on the people who set it up and
coordinate the risk management process
§ On many program risk management consists
only of having a policy and oversight
§ If we treat red flags as false alarms rather than
early warnings of danger this incubates the
threats to program success
§ Group think of dominate leaders often inhibits
good thinking about risks
219
Core Elements of Program Risk
Management†
† Towards a Contingency Theory of Enterprise Risk Management, Anette Mikes Robert Kaplan,
Working Paper 13-063 October 17, 2013
Lesson 4
5

A Risk Management Plan
§ Risk Management
Processes
§ Risk Reporting and
Metrics
§ Risk Transfer
§ Risk Management
Training
§ Risk Management
Process and Data Reviews
§ Risk Management
Assurance Process Map
220
Lesson 4
5

221
Risk Management is How Adults Manage Projects – Tim Lister, IBM
Aleatory
Epistemic

§ Uncertainty creates the opportunity for risk
§ Reducing uncertainty may reduce risk
§ Two types of uncertainty†
– One that can be reduced
– One that cannot
§ A risk informed PMB starts with the WBS
§ 8 steps are needed to build a risk informed PMB
222
Quick View of How to Manage in the
Presence of Uncertainty and Risk
Risk informed program performance management is the goal
† Distinguishing Two Dimensions of Uncertainty, Craig Fox and Gülden Ülkumen, in Perspectives of Thinking, Judging, and Decision Making
Lesson 4
5

Assemble a credible WBS and the Integrated Master
Plan / Integrated Master Schedule (IMP/IMS)
– WBS Dictionary says what will be built
– IMP/IMS Narrative says how, where, and what
processes are used to built it
Assess the aleatory uncertainties in the WBS and IMS
Adjust activity durations and sequence to create the
needed margin to handle the aleatory uncertainty
Assign schedule and cost margin to protect end item
deliverables
223
How to Build a Risk Adjusted IMS
in 8 Steps
0
1
2
3
Lesson 4

Identify Event Based uncertainties from WBS
Dictionary and IMP Narratives
Assign these uncertainties to the Risk Register
Determine risk retirement plans and place them in
the IMS
Determine cost and schedule impacts of unmitigated
risks and place them in Management Reserve
Assemble mitigated aleatory and epistemic
uncertainties with the unmitigated epistemic risk into
the Total Allocated Budget
224
Building a Risk Adjusted IMS in 8
Steps (Concluded)
4
5
6
7
8
Lesson 4

§ Lack of precision about the underlying uncertainty
§ Lack of accuracy about the possible values in the
uncertainty probability distributions
§ Undiscovered Biases used in defining the range of
possible outcomes of project processes
§ Natural variability from uncontrolled processes
§ Undefined probability distributions for project
processes and technology
§ Unknowability of the range of the probability
distributions
§ Absence of information about the probability
distributions
225
Sources of Uncertainty
Lesson 4
5

226
Uncertainties are
things we can not
be certain about.
Uncertainty is
created by our
incomplete
knowledge; not
by our ignorance
Lesson 4

§ When we say uncertainty, we speak about a
future state of an external system that is not
fixed or determined
§ Uncertainty is related to three aspects of our
program management domain:
– The external world – the activities of the program
– Our knowledge of this world – the planned and
actual behaviors of the program
– Our perception of this world – the data and
information we receive about these behaviors
227
Some Words about
Uncertainty
Lesson 4
5

§ Risk has two dimensions
– The degree of possibility that an event will take place
or occur sometime in the future
– The consequences of that event, once it has occurred
§ The degree of possibility is qualified as the
Probability of Occurrence (event based) or a
Probability Distribution Function (a distribution of
variability's of a random number)
§ The consequences are usually taken to be
undesirable and qualified as the magnitude of
harm and the remaining probability of a
recurrence of the same risk. 228
Some Words About the Risk That
Results from Uncertainty
Lesson 4
5

§ Naturally occurring uncertainty
and its resulting risk, impacts
the probability of a successful
outcome.
What is the probability of
making a desired completion
date or cost target?
229
All Program Activities have
Naturally Occurring Uncertainty
§ The irreducible statistical behavior of these activities,
their arrangement in a network of activities, and
correlation between their behaviors creates risk.
§ Adding margin protects the outcome from the impact of
this naturally occurring uncertainty
Lesson 4
5

§ Uncertainty is present when probabilities
cannot be quantified in a rigorous or valid
manner, but can described as intervals within
a probability distribution function (PDF)
§ Risk is present when the uncertainty of the
outcome can be quantified in terms of
probabilities or a range of possible values
§ This distinction is important for modeling the
future performance of cost, schedule, and
techncial outcomes of a program
230
Relationship Between
Uncertainty and Risk†
Lesson 4
5
† Paul Garvey, Probability Methods for Cost Uncertainty Analysis – A systems
Engineering Perspective, Marcel Deker, CRC Press, 2000

Areas for Risk Management In
Systems Engineering†
231
Area for Risk
Engagement
Rationale Steps
System
interface
change
management
Each time a change
to an interface is
proposed, the
change should be
evaluated to see if
new uncertainties
and risks are being
introduced.
• Work with systems engineering and the change
management function to ensure that assessments
are made of the risks associated with each proposed
interface change at a level appropriate to the scope
of the change.
• Ensure that doing the assessments and making use
of the results in interface change decisions are
written into the change management process and
any other relevant documents.
• Work with the systems engineering and change
management functions to ensure that these steps
are written into their processes and any other
relevant documents.
• Provide training and support on an ongoing basis as
required and agreed upon.
† Why an INCOSE Systems Approach is Needed, Dick Kitterman, Northrop Grumman Sixteenth Annual International
Symposium of the International Council On Systems Engineering (INCOSE) 8 - 14 July 2006
Lesson 4
5

232
Area for Risk
Engagement
Rationale Steps
System test
definition and
planning
It is necessary to
understand where
there are risks of
being able to test a
particular
requirement, or
related group of
requirements.
• Work with systems engineering to ensure that risks
are identified, analyzed and handled as an inherent
part of each stage of test definition and planning.
• Work with systems engineering to ensure that these
steps are written into their processes and any other
relevant documents.
Lesson 4
5

233
Area for Risk
Engagement
Rationale Steps
System
operational
use planning
Operational
scenarios that are
not clearly defined
may pose risks to
successful
execution, so all
such scenarios
need to be
examined for risk.
• Work with systems engineering and appropriate user
and support stakeholders to ensure that risks are
identified, analyzed and handled, e.g., scenario
modeling and analysis, as an inherent part of each
stage of operational use planning.
• Work with systems engineering and the other
stakeholders to ensure that these steps are written
into their processes and any other relevant
documents.
required and agreed upon
Lesson 4
5

234
Area for Risk
Engagement
Rationale Steps
System support
planning
It is necessary to
determine which
areas of life cycle
support have
uncertainties of
successful
execution, hence
pose risks to
ongoing use of the
system under
consideration
identified, analyzed and handled as an inherent part
of support planning.
documents.
Lesson 4
5

235
Area for Risk
Engagement
Rationale Steps
System
retirement /
replacement
planning
It is necessary to
determine the risks
associated with
planning to either
retire or replace an
in-use system,
including
development,
transition and
disposition.
identified, analyzed and handled as an inherent part
of planning for system retirement, with or without
replacement.
documents.
Lesson 4
5

Beware of Black Swans
236
Lesson 4
5

Lesson 5
Building Our UAV
Putting the Principles, Practices, and
Processes to work on a Notional
Program – a small unmanned aircraft
237

The Notional Project
§ We work for an
Unmanned Aerial Vehicle
(UAV) firm entering the
ranching and farming
market place
§ We know how to build
complex equipment,
including flying machines,
electronics, and training
systems
238
Lesson 5

This Lesson Will Be Hands On
§ Using the ConOps, SOW,
we’ll develop MOEs and
MOPs
§ We’ll connect those to
TPMs
§ Then we’ll build an
IMP/IMS
§ And then look for risks
§ Using the Vee we’ll put
together a SEMP for the
project
239
Lesson 5

Market Opportunity
§ Normalized Difference Vegetation Index
(NDVI), infrared, and thermal imagery allows:
– Searching out a lost calf separated from the herd
after a snowstorm.
– Checking water troughs and fence lines
– Assess crop health
– Variable rate fertilizer applications
– Show which hybrids are performing against others
240
Lesson 5

Delivering Our UAV by Applying What
We Learned About Systems
Engineering Principle, Practices and
Processes
Time to go Flying
The Systems Engineering Vee
Technical Management Processes
Technical Development Processes
241
Lesson 5
1
2
3
4

We Always Start with the Vee
242
Lesson 5

Our Project Starts With Documents
§ ConOps
§ SOW
§ MOE
§ MOP
§ TPM
§ WBS
§ Risk / Opportunities
§ Deliverable Outcomes
244
Our UAV has many parts all interacting with each other and external systems.
We’ll start with the Concept of Operations (ConOps). The Statement of Work
is built for the over all system. Then we’ll develop the Effectiveness and
Performance capabilities for the mission.
From there the Technical Performance Measures and the WBS elements that
implement them.
Lesson 5

A Thought on the Complexity Of
Systems Before We Start
“Complex systems usually come to grief, when they do, not
because they fail to accomplish their nominal purpose. Complex
systems typically fail because of the unintended consequences of
their design …
“I like to think of system engineering as being fundamentally
concerned with minimizing, in a complex artifact, unintended
interactions between elements desired to be separate.
Essentially, this addresses Perrow’s concerns about tightly
coupled systems. System engineering seeks to assure that
elements of a complex artifact are coupled only as intended.”
– Michael Griffin, NASA Administrator, Boeing Lecture, Purdue
University, March 28, 2007
245
Lesson 5

Business Needs Analysis
Out of the many applications for UAV technology,
cattle tracking is one where technology can be
applied off the shelf.
Ranchers in the western united states have a
difficult time tracking down their cattle and rely
heavily on personal recognizance of their farm
ground to do cattle counts.
With the availability of low cost unmanned aerial
vehicle systems, the rancher can now know where
their livestock located may be, using a UAV system
to replace the role of the manual livestock surveyor.
246
Lesson 5

Concept of Operations
§ Our UAV is a suite of sensor based aerial and
stationary ground, platforms that provide real-
time Reconnaissance, Surveillance, and Asset
information to a managers and staff of a
commercial cattle ranch.
§ It supports this staff as they plan, coordinate, and
execute operations through increased situational
awareness (SA) by integrating intelligence,
surveillance, and reconnaissance (ISR) into a
single platform about their assets – cows on the
open range.
247
Lesson 5

CONOPS (Continued)
§ The user sends a UAV to an area of
interest (within a 5 mile radius)
from a pickup truck mounted
launch system.
§ The user pre-programs the gas-
powered autonomous UAV for the
designated 77 square mile area.
§ The UAV must be able to launch
and be recovered on a rough
surface less than 200 meters in
length, be able to reach the
designated area within 10 minutes
hour, loiter as high as 1,000 feet
above ground level.
§ The UAV and payload sensors are to
provide 4 hour persistent coverage
of 77 square mile area four times in
a 24 hour loiter time and provide
imagery between three and ten
seconds of data capture.
§ UAV will contain EO/IR to provide
the user with initial asset situation.
§ Based on this preliminary
information, the user may direct the
UAVs to fly at lower levels to capture
Full Motion Videos (FMVs) of the
herd.
248
As documented in the Business Plan
Lesson 5

§ Because the system must be packaged and
transported with limited cargo capacity of
ranch vehicles, the UAV suite must conform to
limits on power, weight and size.
§ We will be using weight as the TPM, as
derived from the Business Case and an Energy
Key Performance Parameters (KPP), to
demonstrate better cost and schedule
performance
249
CONOPS (Concluded)
Lesson 5

MOEs and MOPs Specify Project
Objectives
§ MOE.1 Data for Decision
Making
§ MOE.2 Multi-Spectral
Collection
§ MOE.3 Radius of Action
§ MOE.4 Transport Radius
§ MOE.5 Standard Data
Format
§ MOE.6 Pre-Programmed
Flight Instruction
§ MOE.7 Launch and
Recovery
§ MOP.1 Collection Area
§ MOP.2 Surveillance
Coverage
§ MOP.3 Collection Coverage
§ MOP.4 Data Sources
§ MOP.5 Data Timeliness
§ MOP.6 Data Format
§ MOP.7 UAV Availability
§ MOP.8 Launch Recovery
Area
§ MOP.9 Pre-Programmed
Flight Instruction Accuracy
§ MOP.10 Transport Volume
§ MOP.11 Transport Weight
250
Lesson 5

Measures of Effectiveness
§ Range of flight
§ Resolution of images
§ Ability to return home safely when
commanded to do so, or with loss of
communication or any other disruptive event
§ Reliability and maintainability in a ranching
environment
251
Lesson 5

Measures of Performance
§ Speed
§ Loiter time
§ Survivability
§ Weight
§ Fuel consumption
§ Accuracy of location
§ Sensor performance
252
Lesson 5

Technical Performance Measures
§ Weight limits
§ Full Motion Video resolution and frame rate
§ Electroptical / Infrared sensitivity
§ Fuel consumption
253
Lesson 5

Connecting the MOEs, MOPs, and TPMs
to the Integrated Master Plan
254
SOW
ConOps
WBS
CWBS
Technical
Performance
Measures
Messages of
Performance
Measures of
Effectiveness
JROC KPPs
Integrated
Master Plan
Integrated
Master
Schedule
Performance
Measurement
Baseline
Techncial and Operational
Requirements
Lesson 5

Approach and Structure of our
SE Processes Applied to the UAV
255
Agreement
Processes
Technical
Processes
Support Processes
Technical Management of Inputs
and Outputs
Lesson 5

How Systems Engineering Contributes
To These Programmatic Elements
Risk
SOW
Cost
WBS
IMP/IMS
TPM
PMB
❺ Deliverables
defined in the SOW,
traced to the WBS,
with narratives and
Measures of
Performance (MoP)
❹ Budget at the Work
Package level, rolled to
the Control Accounts
showing cost spreads for
all work in the IMS
❻ Measures of
Performance (MoP) for
each critical deliverable in
the WBS and identified in
each Work Package in the
IMS, used to assess
maturity in the IMP
❶ The Products and
Processes in a “well
structured” decomposition,
traceable to the deliverables
❷ IMS contains all
the Work Packages,
BCWS, Risk mitigation
plans, with traces to
the IMP measuring
increasing maturity
through Measures of
Effectiveness (MoE)
and KPPs (JROC and
Program)
❸ Technical and Programmatic
Risks Connected through the
WBS, Risk Register, IMP and IMS
The PMB is the Document
of Record for the Program
Performance is Measured
through the PMB 256
Lesson 5

The System Engineering Process Model
Apply to Our Program†
257
† Roving UAV IED Interdiction System, MSSE Cohort 311-093A, Naval Post Graduate School, March 2001
Lesson 5

Topology of SE Based Organization
258
ConOps, SOO, SRD, Goals
and Specifications
WBS
SOW
ORG Specs
IMP
IMS
EVMS Provides basis for cost and schedule performance management
Defines single schedule to design, build, and verify the product
MOE
Cost
Risks
KPP/TPM
EV
MOP
Key development, integration, and operational methods to assure
product’s performance measures and parameters are
measureable and all work to produce the deliverables defined in
the Integrated Master Schedule.
For each deliverable, risks are defined and mitigated.
Measures of Effectiveness used to assure mission needs are
fulfilled within the cost parameters to allow tradeoffs to be made.
Defines the single authorities program plan for technical and
programmatic activities needed to provide mission capabilities
Defines the design that meets specification requirements
Trades
Analysis
Models
Defines product based organization that parallel the WBS
Defines the work activities needed to meet the project objectives
Defines the product structure and support processes
Defines product function, performance and verification
requirements, program and product objectives
Lesson 5

Let’s Apply the “Vee” To Our UAV
§ Start with an Integrated Master Plan
§ Show the increasing maturity of the products
using the Vee
§ Conduct Program Reviews to confirm we have
the right artifacts – from the SE view – for the
program
259
Lesson 5

INCOSE Vee and the IMP/IMS
260
Combine DT&E/O Demonstration`
System to Specified User Needs and
Interpret User Needs, Refine System
Performance Specifications, and
SRR
Develop System Functional Specifications
and System Verification Plan
SFR
Specifications into CI Functional (Design To)
Specification and CI Verification Plans
PDR
System DT&E, Verify System
Functionality & Constraints Compliance
to Specifications
TRR
Integrated DT&E, Verify Performance
Compliance to Specifications CI
Verification DT&E
Specifications into Product (Build To)
Documentation and Verification Plans
CDR Fabricate, Assemble, Unit Test to
Build To Documentation
Individual CI Verification DT&E
ASFUT GSFUT
System Decomposition System Realization
System Development and Demonstration
SVR PRR
Lesson 5

261
Lesson 5

Integrated Master Schedule
262
Lesson 5

Risk Management Plan†
§ Risks that may exist on our UAV project
– Safety Risks – will our UAV endanger people,
property, or livestock?
– Navigation Risks – will our UAV get lost and not be
able to find its way home – safely?
– Collision Risks – will our UAV collide with fixed
objects, moving objects including other UAVs or
ground vehicles?
– Mission Risks – what are the risks to nit fulfilling
the mission of counting cows in the pasture?
263
Lesson 5
† Analysis of Unmanned Aerial Vehicles Concept of Operations in ITS Applications, Department of
Aerospace Engineering and Mechanics, University of Minnesota

Resources
Knowledge is of two kinds.
We know a subject
ourselves, or we know
where we can find
information upon it.
- Samuel Johnson
264

References and Bibliography
§ Guide to the Systems Engineering Body of Knowledge (SEBOK), version 1.2, Pyster, A. and D.
Olwell (eds). 2013.
§ Global Hawk Systems Engineering Case Study, Bill Kinzig, Air Force Center for Systems
Engineering, Write-Patterson AFB, Dayton, Ohio.
§ MITRE Systems Engineering (SE) Competency Model, September 1, 2007, Version 1.13E.
§ Systems Engineering Guide for System of Systems, Version 1.0, August 2008, Director, Systems
and Software Engineering, Deputy Under Secretary of Defense (Acquisition and Technology)
Office of the Under Secretary of Defense (Acquisition, Technology and Logistics)
§ NASA/SP-2007-6105, Rev1, NASA Systems Engineering Handbook.
§ SMC Systems Engineering, Specialty Engineering Disciplines, Space & Missile Systems Center,
U.S. Air Force Volume 2, 1st Edition, 3 October 2011
§ Systems Requirements Analysis, Jeffrey O. Grady, Academic Press, 2006.
§ The Requirements Engineering Handbook, Ralph R. Young, Archtech House, 2004
§ The Art of Systems Architecting, Maier and Rechtin, 2000, CRC Press
§ “Re-evaluating Systems Engineering Concepts Using Systems Thinking,” A. Terry Bahill, Fellow,
IEEE, and Bruce Gissing, IEEE Transactions On Systems, Man, And Cybernetics—part C:
Applications And Reviews, VOL. 28, NO. 4, November 1998
265

§ Complexity and Postmodernism, Paul Cilliers, Routledge, 1998.
§ Systems Theory as the Foundation for Understanding Systems, Kevin MacG. Adams,* Patrick T.
Hester, Joseph M. Bradley, Thomas J. Meyers, and Charles B. Keating, National Centers for
System of Systems Engineering, Old Dominion University, Norfolk, VA 23508-2563, Systems
Engineering, Vol. 17, No. 1.
§ Engineering Complex Systems with Models and Objects, David W. Oliver, Timothy P. Kelliher,
James G. Keegan, Jr., McGraw-Hill
§ Systems Engineering for Capabilities, Dr. Judith S. Dahmann and George Rebovich Jr. and Jo
Ann Lane, CROSSTALK The Journal of Defense Software Engineering, November 2008.
§ VS10, Multiprogram/Project Common-Use Document, Project Management and Systems
Engineering Handbook, MSFC-HBBK-3173, May 30, 2001.
§ Technical Measurement: A Collaborative Project of PSM, INCOSE, and Industry, INCOSE-TP-
2003-020-01.
§ Coming to Grips with Measures of Effectiveness, Noel Sproles, University of Australia, 1998.
§ Analysis of Alternatives, (AoA) Handbook, A Practical Guide to Analysis of Alternatives, July
2008, Office of Aerospace Studies, Air Force Materiel Command (AFMC) OAS/A9.
266

§ Introduction to Systems Cost Uncertainty Analysis: An Engineering Systems Perspectives,
National Institute of Aerospace (NIA) Systems Analysis & Concepts Directorate, NASA Langley
Research Center, 2 May 2006.
§ Establishing System Measures of Effectiveness, John M. Green, Senior Principal Systems
Engineer, Raytheon Naval & Maritime Integrated Systems.
§ The Seven Samurai of Systems Engineering: Dealing with the Complexity of 7 Interrelated
Systems, James N Martin The Aerospace Corporation,
http://www.incose.org/wma/library/docs/seven_samurai-martin-paper-v040316a.pdf
§ NASA Systems Engineering Processes and Requirements, NPR 7123.1B
§ System Management: Planning, Enterprise Identity, and Deployment, Second Edition, Jeffrey
O. Grady, CRC Press, 2011.
§ Systems Synthesis: Product and Process Design, Jeffry O. Grady, CRC Press,
§ Systems Verification, Jeffry O. Grady, CRC Press, 2007.
§ Using the Incremental Commitment Model to Integrate System Acquisition, Systems
Engineering, and Software Engineering, Barry Boehm and Jo Ann Lane, University of Southern
California, Center for Systems and Software Engineering
267

§ Systems Engineering Principles And Practice, Second Edition, Alexander Kossiakoff, William N.
Sweet, Samuel J. Seymour, Steven M. Biemer, John Wiley & Sons, 2011
268

From Principles to Strategies for Systems Engineering

More Related Content

What's hot

Similar to From Principles to Strategies for Systems Engineering

More from Glen Alleman

Recently uploaded

From Principles to Strategies for Systems Engineering