SQL Server Integration Services and Analysis Services

SQL Server Integration
Services and Analysis Services
Mohan

Consultant
Certification
Mohan Arumugam
Technologies Specialist
E-mail : moohanan@gmail.com
Phone : +91 99406 53876
Profile
Blogger
Trainer
Who Am I ?

Agenda • Overview of Integration Services
• Principles of Good Package Design
• Component Drilldown
• Performance Tuning

A DW architecture
Datawarehouse
(SQL Server, Oracle,
DB2, Teradata)
SQL/Oracle SAP/Dynamics Legacy Text XML
Integration Services
Reports Dashboards Scorecards Excel BI tools
Analysis Services

Session Objectives
 Assumptions
 Experience with SSIS and SSAS
 Goals
 Discuss design, performance, and scalability for building ETL packages and cubes (UDMs)
 Best practices
 Common mistakes

SQL Server 2005 BPA availability!
 BPA = Best Practice Analyzer
 Utility that scans your SQL Server metadata and recommends best practices
 Best practices from dev team and Customer Support Services
 What’s new:
 Support for SQL Server 2005
 Support for Analysis Services and Integration Services
 Scan scheduling
 Auto update framework
 CTP available now, RTM April
 http://www.microsoft.com/downloads/details.aspx?FamilyId=DA0531E4-E94C-4991-82FA-F0E3FB
D05E63&displaylang=en

Agenda
 Integration Services
 Quick overview of IS
 Principles of Good Package Design
 Component Drilldown
 Performance Tuning
 Analysis Services
 UDM overview
 UDM design best practices
 Performance tips

What is SQL Server Integration
Services?
 Introduced in SQL Server 2005
 The successor to Data Transformation Services
 The platform for a new generation of high-performance data integration technologies

Call center data: semi structured
Legacy data: binary files
Application database
ETL
Warehouse
Reports
Mobile
data
Data mining
Alerts & escalation
•Integration and warehousing require separate, staged operations.
•Preparation of data requires different, often incompatible, tools.
•Reporting and escalation is a slow process, delaying smart responses.
•Heavy data volumes make this scenario increasingly unworkable.
Hand
coding
Staging
Text mining
ETL Staging
Cleansing
&
ETL
Staging
ETL
ETL Objective: Before SSIS

Call center:
semi-structured data
Legacy data: binary files
Application database
•Integration and warehousing are a seamless, manageable operation.
•Source, prepare, and load data in a single, auditable process.
•Reporting and escalation can be parallelized with the warehouse load.
•Scales to handle heavy and complex data requirements.
SQL Server Integration Services
Text mining
components
Custom
source
Standard
sources
Data-cleansing
components
Merges
Data mining
components
Warehouse
Reports
Mobile
data
Alerts & escalation
Changing the Game with SSIS

SSIS Architecture
 Control Flow (Runtime)
 A parallel workflow engine
 Executes containers and tasks
 Data Flow (“Pipeline”)
 A special runtime task
 A high-performance data pipeline
 Applies graphs of components to data movement
 Component can be sources, transformations or destinations
 Highly parallel operations possible

Agenda
 Overview of Integration Services
 Principles of Good Package Design
 Component Drilldown
 Performance Tuning

Principles of Good Package Design -
General
 Follow Microsoft Development Guidelines
 Iterative design, development & testing
 Understand the Business
 Understanding the people & processes are critical for success
 Kimball’s “Data Warehouse ETL Toolkit” book is an excellent reference
 Get the big picture
 Resource contention, processing windows, …
 SSIS does not forgive bad database design
 Old principles still apply – e.g. load with/without indexes?
 Platform considerations
 Will this run on IA64 / X64?
 No BIDS on IA64 – how will I debug?
 Is OLE-DB driver XXX available on IA64?
 Memory and resource usage on different platforms

Architecture
 Process Modularity
 Break complex ETL into logically distinct packages (vs. monolithic design)
 Improves development & debug experience
 Package Modularity
 Separate sub-processes within package into separate Containers
 More elegant, easier to develop
 Simple to disable whole Containers when debugging
 Component Modularity
 Use Script Task/Transform for one-off problems
 Build custom components for maximum re-use

Infrastructure
 Use Package Configurations
 Build it in from the start
 Will make things easier later on
 Simplify deployment Dev  QA  Production
 Use Package Logging
 Performance & debugging
 Build in Security from the start
 Credentials and other sensitive info
 Package & Process IP
 Configurations & Parameters

Development
 SSIS is visual programming!
 Use source code control system
 Undo is not as simple in a GUI environment!
 Improved experience for multi-developer environment
 Comment your packages and scripts
 In 2 weeks even you may forget a subtlety of your design
 Someone else has to maintain your code
 Use error-handling
 Use the correct precedence constraints on tasks
 Use the error outputs on transforms – store them in a table for processing later, or use downstream if
the error can be handled in the package
 Try…Catch in your scripts

Component Drilldown - Tasks &
Transforms
 Avoid over-design
 Too many moving parts is inelegant and likely slow
 But don’t be afraid to experiment – there are many ways to solve a problem
 Maximize Parallelism
 Allocate enough threads
 EngineThreads property on DataFlow Task
 “Rule of thumb” - # of datasources + # of async components
 Minimize blocking
 Synchronous vs. Asynchronous components
 Memcopy is expensive – reduce the number of asynchronous components in a flow if
possible – example coming up later
 Minimize ancillary data
 For example, minimize data retrieved by LookupTx

Debugging & Performance Tuning -
General
 Leverage the logging and auditing features
 MsgBox is your friend
 Pipeline debuggers are your friend
 Use the throughput component from Project REAL
 Experiment with different techniques
 Use source code control system
 Focus on the bottlenecks – methodology discussed later
 Test on different platforms
 32bit, IA64, x64
 Local Storage, SAN
 Memory considerations
 Network & topology considerations

Volume
 Remove redundant columns
 Use SELECT statements as opposed to tables
 SELECT * is your enemy
 Also remove redundant columns after every async component!
 Filter rows
 WHERE clause is your friend
 Conditional Split in SSIS
 Concatenate or re-route unneeded columns
 Parallel loading
 Source system split source data into multiple chunks
 Flat Files – multiple files
 Relational – via key fields and indexes
 Multiple Destination components all loading same table

Application
 Is BCP good enough?
 Overhead of starting up an SSIS package may offset any performance gain over BCP for small data sets
 Is the greater manageability and control of SSIS needed?
 Which pattern?
 Many Lookup patterns possible – which one is most suitable?
 See Project Real for examples of patterns:
http://www.microsoft.com/sql/solutions/bi/projectreal.mspx
 Which component?
 Bulk Import Task vs. Data Flow
 Bulk Import might give better performance if there are no transformations or filtering required, and the destination is SQL Server.
 Lookup vs. MergeJoin (LeftJoin) vs. set based statements in SQL
 MergeJoin might be required if you’re not able to populate the lookup cache.
 Set based SQL statements might provide a way to persist lookup cache misses and apply a set based operation for higher performance.
 Script vs. custom component
 Script might be good enough for small transforms that’re typically not reused

Case Study - Patterns
105 seconds 83 seconds
Use Error Output for
handling Lookup miss
Ignore lookup errors and check for null
looked up values in Derived Column

Debugging & Performance Tuning – A
methodology
 Optimize and Stabilize the basics
 Minimize staging (else use RawFiles if possible)
 Make sure you have enough Memory
 Windows, Disk, Network, …
 SQL FileGroups, Indexing, Partitioning
 Get Baseline
 Replace destinations with RowCount
 Source->RowCount throughput
 Source->Destination throughput
 Incrementally add/change components to see effect
 This could include the DB layer
 Use source code control!
 Optimize slow components for resources available

Case Study - Parallelism
 Focus on critical path
 Utilize available resources
Memory Constrained Reader and CPU Constrained
Let it rip! Optimize the slowest

Summary
 Follow best practice development methods
 Understand how SSIS architecture influences performance
 Buffers, component types
 Design Patterns
 Learn the new features
 But do not forget the existing principles
 Use the native functionality
 But do not be afraid to extend
 Measure performance
 Focus on the bottlenecks
 Maximize parallelism and memory use where appropriate
 Be aware of different platforms capabilities (64bit RAM)
 Testing is key

Agenda
Server architecture and UDM basics
Optimizing the cube design
Partitioning and Aggregations
Processing
Queries and calculations
Conclusion

Analysis
Server
OLEDB
ADOMD
.NET
AMO IIS
TCP
HTTP
XMLA
ADOMD
Client
Apps
BIDS
SSMS
Profiler
Excel
Client Server Architecture

Dimension
An entity on which analysis is to be performed
(e.g. Customers)
Consists of:
Attributes that describe the entity
Hierarchies that organize dimension members in meaningful ways
Customer
ID
First
Name
Last
Name
State City Marital
Status
Gender … Age
123 John Doe WA Seattle Married Male … 42
456 Lance Smith WA Redmond Unmarried Male … 34
789 Jill Thompson OR Portland Married Female … 21

Attribute
 Containers of dimension members.
 Completely define the dimensional space.
 Enable slicing and grouping the dimensional
space in interesting ways.
 Customers in state WA and age > 50
 Customers who are married and male
 Typically have one-many relationships
 City  State, State  Country, etc.
 All attributes implicitly related to the key

Hierarchy
Ordered collection of attributes into levels
Navigation path through dimensional space
User defined hierarchies – typically multiple
levels
Attribute hierarchies – implicitly created for
each attribute – single level
Customers by Geography
Country
State
City
Customer
Customers by Demographics
Marital
Gender
Customer

Customer
City
State
Country
Gender Marital
Country
State
City
Customer
Gender
Customer
Marital
Gender
Customer
Customer
City
State
Country
Gender
Marital
Attributes Hierarchies
Age
Dimension Model

Cube
Collection of dimensions and measures
Measure  numeric data associated with a set
of dimensions (e.g. Qty Sold, Sales Amount,
Cost)
Multi-dimensional space
Defined by dimensions and measures
E.g. (Customers, Products, Time, Measures)
Intersection of dimension members and measures is
a cell (USA, Bikes, 2004, Sales Amount) = $1,523,374.83

A Cube
Product
Peas Corn Bread Milk Beer
M
a
r
k
e
t
Bos
NYC
Chi
Sea
Jan
Mar
Feb
Time
Units of beer sold in
Boston in January

Measure Group
Group of measures with same dimensionality
Analogous to fact table
Cube can contain more than one measure
group
E.g. Sales, Inventory, Finance
Multi-dimensional space
Subset of dimensions and measures in the cube
AS2000 comparison
Virtual Cube  Cube
Cube  Measure Group

Sales Inventory Finance
Customers X
Products X X
Time X X X
Promotions X
Warehouse X
Department X
Account X
Scenario X
Measure Group
Measure Group

Agenda
 Server architecture and UDM Basics
 Optimizing the cube design
 Partitioning and Aggregations
 Processing
 Queries and calculations
 Conclusion

Top 3 Tenets of Good Cube Design
Attribute relationships

Attribute Relationships
One-to-many relationships between attributes
Server simply “works better” if you define them where applicable
Examples:
City  State, State  Country
Day  Month, Month  Quarter, Quarter  Year
Product Subcategory  Product Category
Rigid v/s flexible relationships (default is flexible)
Customer  City, Customer  PhoneNo are flexible
Customer  BirthDate, City  State are rigid
All attributes implicitly related to key attribute

Customer
City
State Country
Gender Marital Age
Attribute Relationships (continued)

Customer
City
State
Country
Gender Marital Age
Attribute Relationships (continued)

Where are they used?
MDX Semantics
Tells the formula engine how to roll up measure values
If the grain of the measure group is different from the key attribute (e.g. Sales
by Month)
Attribute relationships from grain to other attributes required (e.g. Month  Quarter,
Quarter  Year)
Otherwise no data (NULL) returned for Quarter and Year
MDX Semantics explained in detail at:
http://www.sqlserveranalysisservices.com/OLAPPapers/AttributeRelationships.htm

Storage
Reduces redundant relationships between dimension members – normalizes
dimension storage
Enables clustering of records within partition segments (e.g. store facts for a
month together)
Processing
Reduces memory consumption in dimension processing – less hash tables to
fit in memory
Allows large dimensions to push 32-bit barrier
Speeds up dimension and partition processing overall

Query performance
Dimension storage access is faster
Produces more optimal execution plans
Aggregation design
Enables aggregation design algorithm to produce effective set of
aggregations
Dimension security
DeniedSet = {State.WA} should deny cities and customers in WA – requires
attribute relationships
Member properties
Attribute relationships identify member properties on levels

How to set them up?
Creating an attribute relationship is easy, but
…
Pay careful attention to the key columns!
Make sure every attribute has unique key columns (add composite keys as
needed)
There must be a 1:M relation between the key
columns of the two attributes
Invalid key columns cause a member to have
multiple parents
Dimension processing picks one parent arbitrarily and succeeds
Hierarchy looks wrong!

How to set them up?
Don’t forget to remove redundant
relationships!
All attributes start with relationship to key
Customer  City  State  Country
Customer  State (redundant)
Customer  Country (redundant)

Example
Time dimension
Day, Week, Month, Quarter, Year
Year: 2003 to 2010
Quarter: 1 to 4
Month: 1 to 12
Week: 1 to 52
Day: 20030101 to 20101231
Day
Week
Month
Quarter
Year

Example
Time dimension
Year: 2003 to 2010
Quarter: 1 to 4 - Key columns (Year, Quarter)
Month: 1 to 12
Week: 1 to 52
Day: 20030101 to 20101231
Day
Week
Month
Quarter
Year

Time dimension
Year: 2003 to 2010
Month: 1 to 12 - Key columns (Year, Month)
Week: 1 to 52
Day: 20030101 to 20101231
Example
Day
Week
Month
Quarter
Year

Time dimension
Year: 2003 to 2010
Month: 1 to 12 - Key columns (Year, Month)
Week: 1 to 52 - Key columns (Year, Week)
Day: 20030101 to 20101231
Example
Day
Week
Month
Quarter
Year

Defining Attribute Relationships

User Defined Hierarchies
Pre-defined navigation paths thru dimensional
space defined by attributes
Attribute hierarchies enable ad hoc navigation
Why create user defined hierarchies then?
Guide end users to interesting navigation paths
Existing client tools are not “attribute aware”
Performance
Optimize navigation path at processing time
Materialization of hierarchy tree on disk
Aggregation designer favors user defined hierarchies

Natural Hierarchies
1:M relation (via attribute relationships)
between every pair of adjacent levels
Examples:
Country-State-City-Customer (natural)
Country-City (natural)
State-Customer (natural)
Age-Gender-Customer (unnatural)
Year-Quarter-Month (depends on key columns)
How many quarters and months?
4 & 12 across all years (unnatural)
4 & 12 for each year (natural)

Natural Hierarchies
Best Practice for Hierarchy Design
Performance implications
Only natural hierarchies are materialized on disk during processing
Unnatural hierarchies are built on the fly during queries (and cached in
memory)
Server internally decomposes unnatural hierarchies into natural components
Essentially operates like ad hoc navigation path (but somewhat better)
Create natural hierarchies where possible
Using attribute relationships
Not always appropriate (e.g. Age-Gender)

Best Practices for Cube Design
Dimensions
Consolidate multiple hierarchies into single dimension (unless they are
related via fact table)
Avoid ROLAP storage mode if performance is key
Use role playing dimensions (e.g. OrderDate, BillDate, ShipDate) - avoids
multiple physical copies
Use parent-child dimensions prudently
No intermediate level aggregation support
Use many-to-many dimensions prudently
Slower than regular dimensions, but faster than calculations
Intermediate measure group must be “small” relative to primary measure group

Attributes
Define all possible attribute relationships!
Mark attribute relationships as rigid where appropriate
Use integer (or numeric) key columns
Set AttributeHierarchyEnabled to false for attributes not used for navigation
(e.g. Phone#, Address)
Set AttributeHierarchyOptimizedState to NotOptimized for infrequently used
attributes
Set AttributeHierarchyOrdered to false if the order of members returned by
queries is not important
Hierarchies
Use natural hierarchies where possible

Measures
Use smallest numeric data type possible
Use semi-additive aggregate functions instead of MDX calculations to achieve
same behavior
Put distinct count measures into separate measure group (BIDS does this
automatically)
Avoid string source column for distinct count measures

Partitioning
Mechanism to break up large cube into
manageable chunks
Partitions can be added, processed,
deleted independently
Update to last month’s data does not affect prior months’ partitions
Sliding window scenario easy to implement
E.g. 24 month window  add June 2006 partition and delete June 2004
Partitions can have different storage settings
Partitions require Enterprise Edition!

Benefits of Partitioning
Partitions can be processed and queried
in parallel
Better utilization of server resources
Reduced data warehouse load times
Queries are isolated to relevant partitions  less data to scan
SELECT … FROM … WHERE [Time].[Year].[2006]
Queries only 2006 partitions
Bottom line  partitions enable:
Manageability
Performance
Scalability

Best Practices for Partitioning
No more than 20M rows per partition
Specify partition slice
Optional for MOLAP – server auto-detects the slice and validates against user
specified slice (if any)
Must be specified for ROLAP
Manage storage settings by usage patterns
Frequently queried  MOLAP with lots of aggs
Periodically queried  MOLAP with less or no aggs
Historical  ROLAP with no aggs
Alternate disk drive - use multiple controllers to avoid I/O
contention

Best Practices for Aggregations
Define all possible attribute relationships
Set accurate attribute member counts and fact
table counts
Set AggregationUsage to guide agg designer
Set rarely queried attributes to None
Set commonly queried attributes to Unrestricted
Do not build too many aggregations
In the 100s, not 1000s!
Do not build aggregations larger than 30% of
fact table size (agg design algorithm doesn’t)

Best Practices for Aggregations
Aggregation design cycle
Use Storage Design Wizard (~20% perf gain) to design initial set of
aggregations
Enable query log and run pilot workload (beta test with limited set of users)
Use Usage Based Optimization (UBO) Wizard to refine aggregations
Use larger perf gain (70-80%)
Reprocess partitions for new aggregations to
take effect
Periodically use UBO to refine aggregations

Improving Processing
 SQL Server Performance Tuning
 Improve the queries that are used for extracting data from SQL Server
 Check for proper plans and indexing
 Conduct regular SQL performance tuning process
 AS Processing Improvements
 Use SP2 !!
 Processing 20 partitions: SP1 1:56, SP2: 1:06
 Don’t let UI default for parallel processing
 Go into advanced processing tab and change it
 Monitor the values:
 Maximum number of datasource connections
 MaxParallel – How many partitions processed in parallel, don’t let the server decide on its own.
 Use INT for keys, if possible.
Parallel processing requires Enterprise Edition!

Improving Processing
 For best performance use ASCMD.EXE and
XMLA
 Use <Parallel> </Parallel> to group processing tasks together until Server is
using maximum resources
 Proper use of <Transaction> </Transaction>
 ProcessFact and ProcessIndex separately
instead of ProcessFull (for large partitions)
 Consumes less memory.
 ProcessClearIndexes deletes existing indexes
and ProcessIndexes generates or reprocesses

Best Practices for Processing
Partition processing
Monitor aggregation processing spilling to disk (perfmon counters for temp
file usage)
Add memory, turn on /3GB, move to x64/ia64
Fully process partitions periodically
Achieves better compression over repeated incremental processing
Data sources
Avoid using .NET data sources – OLEDB is faster for processing

Agenda
 Server architecture
 UDM Basics
 Optimizing the cube design
 Partitioning and Aggregations
 Processing
 Queries and calculations
 Conclusion

Non_Empty_Behavior
Most client tools (Excel, Proclarity) display non empty results –
eliminate members with no data
With no calculations, non empty is fast – just checks fact data
With calculations, non empty can be slow – requires evaluating
formula for each cell
Non_Empty_Behavior allows non empty on calculations to just
check fact data
Note: query processing hint – use with care!
Create Member [Measures].[Internet Gross Profit] As
[Internet Sales Amount] - [Internet Total Cost],
Format_String = "Currency",
Non_Empty_Behavior = [Internet Sales Amount];

Auto-Exists
Attributes/hierarchies within a dimension are always existed
together
City.Seattle * State.Members returns {(Seattle, WA)}
(Seattle, OR), (Seattle, CA) do not “exist”
Exploit the power of auto-exists
Use Exists/CrossJoin instead of .Properties – faster
Requires attribute hierarchy enabled on member property
Filter(Customer.Members,
Customer.CurrentMember.Properties(“Gender”) = “Male”)
Exists(Customer.Members, Gender.[Male])

Conditional Statement: IIF
Use scopes instead of conditions such as Iif/Case
Scopes are evaluated once statically
Conditions are evaluated dynamically for each cell
Always try to coerce IIF for one branch to be null
Create Member Measures.Sales As
Iif(Currency.CurrentMember Is Currency.USD,
Measures.SalesUSD, Measures.SalesUSD * Measures.XRate);
Create Member Measures.Sales As Null;
Scope(Measures.Sales, Currency.Members);
This = Measures.SalesUSD * Measures.XRate;
Scope(Currency.USA);
This = Measures.SalesUSD;
End Scope;
End Scope;

Best Practices for MDX
Use calc members instead of calc cells where possible
Use .MemberValue for calcs on numeric attributes
Filter(Customer.members, Salary.MemberValue > 100000)
Avoid redundant use of .CurrentMember and .Value
(Time.CurrentMember.PrevMember, Measures.CurrentMember ).Value can be replaced with
Time.PrevMember
Avoid LinkMember, StrToSet, StrToMember, StrToValue
Replace simple calcs with computed columns in DSV
Calculation done at processing time is always better
Many more at:
Analysis Services Performance Whitepaper:
http://download.microsoft.com/download/8/5/e/85eea4fa-b3bb-4426-97d0-7f7151b2011c/SSAS2005P
erfGuide.doc
http://sqljunkies.com/weblog/mosha
http://sqlserveranalysisservices.com

Conclusion
AS2005 is major re-architecture from AS2000
Design for perf & scalability from the start
Many principles carry through from AS2000
Dimensional design, Partitioning, Aggregations
Many new principles in AS2005
Attribute relationships, natural hierarchies
New design alternatives – role playing, many-to-many, reference dimensions, semi-additive
measures
Flexible processing options
MDX scripts, scopes
Use Analysis Services with SQL Server Enterprise Edition to get max
performance and scale

Resources
 SSIS
 SQL Server Integration Services site – links to blogs, training, partners, etc.:
http://msdn.microsoft.com/SQL/sqlwarehouse/SSIS/default.aspx
 SSIS MSDN Forum: http://forums.microsoft.com/MSDN/ShowForum.aspx?ForumID=80&SiteID=1
 SSIS MVP community site:
http://www.sqlis.com
 SSAS
 BLOGS: http://blogs.msdn.com/sqlcat
 PROJECT REAL-Business Intelligence in Practice
 Analysis Services Performance Guide
 TechNet: Analysis Services for IT Professionals
 Microsoft BI
 SQL Server Business Intelligence public site: http://www.microsoft.com/sql/evaluation/bi/default.asp
 http://www.microsoft.com/bi

Mohan Arumugam
Technologies Specialist
E-mail : moohanan@gmail.com
Phone : +91 99406 53876
Profile
Thank You

SQL Server Integration Services and Analysis Services

More Related Content

Similar to SQL Server Integration Services and Analysis Services (20)

More from Mohan Arumugam (7)

Recently uploaded (20)

SQL Server Integration Services and Analysis Services

Editor's Notes