Dynamics 365 saturday 2018 - data migration story

The Dynamics 365 Data Migration Story
Andre Margono (MVP)
Solutions Architect | Barhead Solutions

Disclaimer
 This story is based on personal experience and I’m not a data expert 

Background Story #1
 I was assigned to a project as a CRM Consultant/Developer.
 During the course of the project, there was a need for an immediate
replacement for a data migration resource.
 I was “accidentally” selected as the data migration resource…..

Background Story #2
 Years later, I had the privilege to lead a project as the technical delivery
lead/architect.
 Need to define data migration approach within the high-level design
document.
 I’m fortunate enough to have a lead consultant that could work together on
the complex data migration piece, even though he has no Dynamics
background. And we exchanged our knowledge and approach.

Common Data Migration Issues in D365 Projects
 Data is one of the most important aspects in the
adoption and success of a project, yet often
neglected.
 Data migration is often underestimated in CRM
implementation projects.
 Lack of knowledge could leads to disaster.
Dynamics 365 Specialists are not necessarily a data
migration experts. But inevitably, sometimes we are
tasked to handle data migration.
 Dynamics 365 projects often delivered in Agile
methodology. Thus, the data schema is not “stable”.

Start with a proper planning
 Understand the scope of the data migration: Identify
Sources, How far away, the volume, how recent and
"active"?
 Understand the data by profiling them: Size, Format,
Quality? Does it requires clean up and dedupe?
 Any archival strategy/consideration?
 Start thinking of decommission/transition process.
 Establish the ETL approach.
 Discuss on how to handle changes in the schema.

Start with a proper planning - Continued
 Start to engage the "owner" of the data (a.k.a "the
business people")
 Start with the estimation and make sure to cover the
time for the load time, not just the "build" effort
 Plan on how many "Dress-rehearsals" needed
 Plan the cutover strategy
 Make sure to have the rollback strategy
 Make a call: Within the main project? Or as a
separate project running in parallel?

Example of Simple ETL Approach
Staging DB
 Contact
 Account
 Lead
 Oppty
 etc..
Source
Systems
Data
Cleanse
 Contact
 Account
 Lead
 Oppty
 etc..
Extract
 Contact
 Account
 Lead
 Oppty
 etc..
Transform
 Contact
 Account
 Lead
 Oppty
 etc..
Target
Extract Transform Load
Validation

The plan is ready, what’s next?

Build
 Make sure to get the data mapping right
 Do it incrementally and get early verification from the business.
 Make sure load/exception reports are included within the build cycle
 Start timing the load to get the estimated time needed to load for the rehearsals
 Keep environments “in-Sync”
 Note down any process/automation that need to be toggled off
 Classic Dynamics 365 environments :Plugins, Workflow, Business Rules.
 The "now" Dynamics 365: Flow, Logic Apps, Azure Functions/Webhook, any other
3rd Party Integration Platform
 Note down the sequence of load when necessary
 Start writing down the "Migration Checklist"

Before The Day…
 Make sure the Migration Checklist is up to date
 Perform some "dress-rehearsals" - capture
anomalies, fix problems, note the timing, run
migration report post rehearsal, update the
migration checklist as needed
 Finalising the cutover strategy

Execution (a.k.a The Day”)
 Run through the migration checklist
 Make sure the environments dependencies are
fulfilled (e.g: latest solutions have been imported)
 All reference data has been imported
 All Dynamics 365 specific configurations are in place
 Turned off all unnecessary
plugins/workflows/external integrations
 Make sure clear communication within the project
team on each migration milestones

Now, I’ve got the concept?
What toolsets that I can use?

Let’s start with the simplest
one…

Dynamics 365 Native Data Import
Typically useful for simple migration in a straight forward implementation
 Pros:
 Simple to use, by using the Excel/csv mapping
 Doesn’t need to develop complex migration package/script (can be done by non-techy)
 Cons:
 Dependant to the Async Service of Dynamics 365.
 Could not handle complex mapping
 Extraction and Transformation handled externally
 Only suitable if Dynamics 365 is the single target system (no SharePoint or other
components).
 Could not be automated

Quick Tips: Handling cyclic
relationship with the native Data
Import

Quick Tips: Handling cyclic relationship with
the native Data Import
 Sample data set

 Put it in a folder and zip it up

Cobalt Migration Dynamics
 https://www.cobalt.net/migration-dynamics/
 Pros:
 Best usage for lift-and-shift migration, where the on-prem to online data model are
identical
 Typically handle the common Dynamics to Dynamics migration issue, such as: disabling
WF, plugins, etc
 Mapping and basic transformation rules via configuration (no-code)
 Designed with performance in mind
 Cons:
 Inflexible in terms of data model. E.g: Could not have custom mapping/transformation
 Only suitable if Dynamics 365 is the single target system (no SharePoint or other
components).

Now with the more complex
toolset…

SSIS + KingswaySoft
 http://www.kingswaysoft.com/products
 Pros:
 Flexibility to transform the data and perform clean up by leveraging SQL Server capability
 Could load the data to multiple systems (SharePoint, ERP, etc) when the implementation is
leveraging the other Microsoft or 3rd party products
 Powerful toolkit that can deliver migration with low-code/scripting
 Better control in load performance (parallel, using batch a.k.a ExecuteMultiple)
 Cons:
 Requires more time to develop the migration package/script
 Might need Infrastructure (SQL server/VM to host the staging DB)

Off-Topic:
KingswaySoft now is supporting running SSIS on
the cloud via Azure Data Factory v2 Integration
Runtime – a.k.a: ADF v2 IR
http://www.kingswaysoft.com/blog/2018/04/30/Running-SSIS-Integration-Toolkit-
on-the-Cloud

Tips & Tricks
 Fine tuning of load performance with Batching (ExecuteMultiple) and Parallel
Loading using Balanced Data Distributor (BDD)
 Dynamics 365 API Limit: 60,000 API requests, per organization instance, within
five minute sliding interval (https://docs.microsoft.com/en-
us/dynamics365/customer-engagement/developer/api-limits)
 ExecuteMultiple limit: https://msdn.microsoft.com/en-au/library/jj863631.aspx
 Maximum batch size – there is a limit to how many requests can be added to a request
collection. If that limit is exceeded, a fault is thrown before the first request is ever
executed. Recommended batch size: 200
 Throttling of concurrent calls – for Microsoft Dynamics 365 (online) there is a limit of 2
concurrent ExecuteMultipleRequest executions per organization. If that limit is exceeded, a
“Server Busy” fault is thrown before the first request is ever executed.


Tips & Tricks
 If the order of the records is important, do not run parallel loading
 Always run a few dress rehearsals
 Always keep the timing of each milestone
 Stage the data extract in a staging environment/DB.
 Always have the migration report (Exceptions, Errors, Successful Load,
Warnings, Timing)
 Do not to run a direct source to target data load and have the transformation
running on the fly

Conclusion
• Plan ahead!
• Build carefully
• Keep the checklist up to date
• Rehearse! Rehearse! Rehearse!
• Just do it 

Dynamics 365 saturday 2018 - data migration story

More Related Content

What's hot (20)

Similar to Dynamics 365 saturday 2018 - data migration story (20)

Recently uploaded (20)

Dynamics 365 saturday 2018 - data migration story