Scripted Data Loads with Salesforce DataLoader and ANT

Scripted Data Loads with
Salesforce Data Loader and ANT
 Scott Geosits
 Salesforce.com Architect, Varsity Brands Inc.
 sjgeosits@herﬀjones.com
 @ScottGeosits

Scott Geosits
Salesforce.com Architect
Varsity Brands, Inc.

  Why Scripted Data Loads?
  Tools – Salesforce Data Loader and ANT
  Setting Things Up
  Creating And Running Scripts
  Use Cases
  Summary
Agenda

Why Scripted Data Loads?
Full Copy Sandbox
Sandbox
Sandbox
Sandbox
Sandbox

Full Copy Sandbox
Sandbox
Sandbox
Sandbox
Sandbox
SANDBOX REFRESH

Full Copy Sandbox
Sandbox
Sandbox
Sandbox
Sandbox
DATA IN SYNC

Full Copy Sandbox
Sandbox
Sandbox
Sandbox
Sandbox
SANDBOX REFRESH
data jenga

 There HAS to be a better way!

 Predictable and Repeatable data sets
 Easy re-load of data after sandbox refreshes
•  You SHOULD be refreshing from production OFTEN – after every prod release at a minimum
•  git versioning + scripted data loads means you are productive much sooner after a refresh
 CI for Selenium / Cucumber style testing in addition to Apex Unit Tests
 Much better handling of object relationships
•  All but most simple data models have relationships
•  Data loader client requires you to manipulate CSV ﬁles to support relationships
•  External ID with UPSERT is how we load data with relationships
Beneﬁts of Scripted Data

 Salesforce-provided utility for exporting and importing data from a Salesforce org or sandbox
 CSV ﬁle format
 Available from within Salesforce
 Save mapping ﬁles for repeated operations
 Load up to 5,000,000 records
 Run from desktop or command line
Salesforce Data Loader

 Open source scripting tool
 Part of Apache project – https://ant.apache.org
 Define build steps via set of XML target definitions (build file)
 Supports external build tasks, dependency chains
 Like make but a hundred thousand times better
 We will use this to run data loader by making calls to the data loader’s JAR file
Apache ANT

Scripted Data Loads with Salesforce DataLoader and ANT

 Everything we talk about here is in a github repo for you to experiment with
 Use consistent structure across projects
github

github
ANT build.xml file

github
Executable utilities

github
SDL mapping files
Pre-built CSV data files

github
Supporting Java JAR files

github
Credentials for SFDC orgs

github
Sample import / export
scripts

Salesforce Object Considerations

 Salesforce record IDs do not match between production and sandboxes
•  Well, not exactly: full copy sandbox IDs match, but we are not typically data loading those
 External ID is needed to preserve object relationships
 Our convention will be to create a ﬁeld called Load_ID__c on all of our objects.
•  Text ﬁeld with length = 18
•  External ID = true
Setting Up Objects

 Why do we need to do this? Let’s look at a simple example
 We have an org that has two accounts
•  ABC Computers – Salesforce ID = 001F000000yHO3ZIAW
•  XYZ Consulting – Salesforce ID = 001F000000yHO3ZIAX
 Each of these accounts has two contacts:
•  John Jones (ABC Computers) – Salesforce ID = 003a000000xJP2QIAW
•  Jane Peters (ABC Computers) – Salesforce ID = 003a000000xJP3QIAW
•  Jim Smith (XYZ Consulting) – Salesforce ID = 003a000000xJP4QIAW
•  Maria Brown (XYZ Consulting) – Salesforce ID = 003a000000xJP5QIAW
Load ID
Setting Up Objects

 If we do a field-by-field export of both objects, we end up with the following two files:
Load ID
Setting Up Objects

 Now if we import these into a sandbox, they will import with diﬀerent Salesforce IDs:
•  ABC Computers – Salesforce ID = 001g000000yCW4xIAN
•  XYZ Consulting – Salesforce ID = 001g000000yCW4yIAN
 Looking back at our original records, we see we have a problem as the IDs do not match so we
won’t be able to match the contact records up correctly:
 Using desktop Data Loader, here’s where you would go in by hand and switch all of the Account
values in the Contact CSV ﬁle to match the new IDs.
Load ID
Setting Up Objects

 If we use external IDs, however, we can upsert on those external IDs
 Similar to loading a contact list by email address.
 Email address is not a primary key in Salesforce but set up as external ID and upsert can match on
it.
 We will steal borrow this pattern and use it to our advantage.
Load ID
Setting Up Objects

 When we export, we put the Salesforce ID into a ﬁeld called Load_ID__c in the CSV and end up
with the following two ﬁles:
Load ID
Setting Up Objects

 Now when we import, we tell the data loader to upsert that record into the database using the
Load_ID__c external ID, and we get:
•  ABC Computers – Salesforce ID = 001g000000yCW4xIAN, Load_ID__c = 001F000000yHO3ZIAW
•  XYZ Consulting – Salesforce ID = 001g000000yCW4yIAN, Load_ID__c = 001F000000yHO3ZIAX
 Comparing against our originals, the Salesforce IDs are diﬀerent but we’ve preserved the ID from
the source org in our Load ID ﬁeld:
Load ID
Setting Up Objects

 We also tell it when establishing the relationships to do so using the Load ID external ID and NOT
the record primary key Salesforce ID.
 We then get the record inserted with the correct relationship established to the new account
record:
•  Contact John Jones
•  Salesforce ID = 003J000000xZD7GIAW
•  Load ID = 003a000000xJP2QIAW
•  Account = 001g000000yCW4xIAN
 Note that once the upsert occurs the record is in the target system with the NEW account
Salesforce ID, not the actual Load ID from the old system.
Load ID
Setting Up Objects

 Define mapping of fieldnames from CSV to Salesforce object and vice versa
 Text file containing source field name, equals sign, destination field name
•  Must use Salesforce API Field Name, not Label
•  Keep things simple! Use Salesforce API fieldnames for CSV column names.
 Need to create one mapping file for export and another one for import
•  Naming conventions used by ANT build script:
•  Export file: {Salesforce object API Name}ExportMap.sdl - ex. Candidate__cExportMap.sdl
•  Import file: {Salesforce object API Name}ImportMap.sdl - ex. Candidate__cImportMap.sdl
 Export mappings have Salesforce field name on left, CSV column name on right
Export Files
SDL Mapping Files

 Id=Load_ID__c
 Candidate__c=Candidate__c
 Position__c=Position__c
 Stage__c=Stage__c
 Status__c=Status__c
Export Example
SDL Mapping Files

 Special notation for relationship ﬁelds
 Need to specify that when a record is imported, it links the object to the related object (which
must already be in the database) via the related object’s Load_ID__c ﬁeld.
 We do this with the __r: notation:
•  Candidate__c=Candidate__r:Load_ID__c
Import Files
SDL Mapping Files

 Special notation for relationship fields
 Need to specify that when a record is imported, it links the object to the related object (which
must already be in the database) via the related object’s Load_ID__c field.
 We do this with the __r: notation:
•  Candidate__c=Candidate__r:Load_ID__c
 This tells data loader to establish the relationship to the Candidate__c object by matching the
value in the CSV file (Candidate__c) to the Candidate__c record with a matching Load_ID__c
value.
Import Files
SDL Mapping Files

 Load_ID__c=Load_ID__c
 Candidate__c=Candidate__r:Load_ID__c
 Position__c=Position__r:Load_ID__c
 Stage__c=Stage__c
 Status__c=Status__c
Import Example
SDL Mapping Files

 Store Salesforce authentication credentials
•  Username
•  Encrypted password
•  Org server and name
•  Salesforce server
 Each environment gets one
 Do NOT version these in git
Setting Up ANT Property Files

 Must encrypt password using Data Loader encryption utility
•  Encryption utility found under bin/ in Data Loader install directory
•  Also available under bin/ in my github repository
•  Encryption key needed – seed value for hash generator
•  There is one called dataloaderencryptionkey.txt under datascripts/ directory – feel free to modify
•  If you rename, you must rename in template ﬁles under datascripts/ as well
 Encrypt password + security token
Creating Encrypted Passwords

 Copy result to your properties ﬁle
 One time operation, until your password or security token is modiﬁed
Creating Encrypted Passwords
./encrypt.sh –e {password + securitytoken} ../datascripts/dataloader_encryption_key.txt

 orgname=fullcopy
 sf.username=myuser@myorg.com
 sf.passwordencrypted=640b322f24f318d8c2a2fdeb0cdbdd10e3727e16a5a7d580b92379c0461f80dcb3d31964
 apiVersion=34.0
 sf.server=https://test.salesforce.com
 sf.maxPoll=20
 sf.orgserver=cs7
Sample ANT Property File

ANT build ﬁle and conﬁguration

 All build targets are defined in build.xml file in repository root.
 We will just touch on capabilities of ANT
•  For complete documentation, visit http://ant.apache.org
 Specify which property file to get credentials from with a -propertyfile {path to property file}
option
 Macros and Targets are primary components of the build.xml file.
ANT Build File

 Pre-defined steps to invoke Data Loader ProcessRunner Java program.
•  Create process template copy
•  Perform token substitution based on property file values and command line arguments
•  Invoke Java program with process file
 The following macros are already defined for you in the build.xml file:
•  sfExport, sfUpsert, sfDelete
Macros
ANT Build File

 A target defines one or more commands to execute sequentially.
 Target definitions can also specify dependent targets that must run first.
 The following targets are already defined in the github build.xml file:
•  initDataDirectory
•  exportData
•  updateDataFiles
•  upsertData
•  deleteData
Targets
ANT Build File

 Creates org-specific temp subdirectory under orgs and data directory beneath that.
 Parameters:
 -propertyfile {path to property file}
initDataDirectory Target
ANT Build File
ant initDataDirectory –propertyfile properties/sandbox.properties

 Exports data in CSV format from specified org to org specific data subdirectory
 Output file location: orgs/{orgname}/data/Obj__cExport.csv
 Parameters:
 -Dobject={object name}
 -Dsoql={SOQL to filter data}
exportData Target
ANT Build File
ant exportData –Dobject=Obj__c –Dsoql=“SELECT Id FROM Obj__c” –propertyfile properties/
sandbox.properties

 Copies all export files from org data subdirectory to datascripts/datafiles staging directory.
 Also renames Export.csv portion of the file name to Import.csv.
 Parameters:
updateDataFiles Target
ANT Build File
ant updateDataFiles –propertyfile properties/sandbox.properties

 Upserts data from CSV format file in datascripts/datafiles directory to org specified in property
file.
 Input file location: datascripts/datafiles/Obj__cImport.csv
 Parameters:
upsertData Target
ANT Build File
ant upsertData –Dobject=Obj__c –propertyfile properties/sandbox.properties

 Deletes data for the requested object in the org specified in the property file.
 Parameters:
 -Dsoql={SOQL query to define filter for data to delete}
deleteData Target
ANT Build File
ant deleteData –Dobject=Obj__c –Dsoql=“SELECT Id FROM Obj__c” –propertyfile properties/
sandbox.properties

Demo
Export Object Records to CSV

Demo
Import Object Records from CSV

 Call initDataDirectory first to create / initialize directories
 Call exportData for each object you want to export – order does not matter
When creating SOQL, remember to select ALL of the fields specified in export mapping file
Export Scripting Tips
Scripting Imports and Exports

Call upsertData for each object you want to import
Order matters with imports
•  Upsert objects with no lookups or M-D relationships ﬁrst
•  Then import in dependency order
Disable your triggers prior to running import to avoid governor limit exceptions
 Make sure to run updateDataFiles target prior to running import script.
•  You could work this into either the export or import scripts depending on your typical workﬂow
Import Scripting Tips
Scripting Imports and Exports

Demo
Export and Import Full Data Model

 Pre-loading sandboxes
•  Developers need data
•  QA needs data too
•  Sometimes very diﬀerent data needs
 Continuous Integration
•  Jenkins / Bamboo / Travis etc.
•  Pre-populate sandboxes
 Production issue troubleshooting and resolution
•  Do not make code ﬁxes directly in production!
Use Cases for Scripted Data Loads

 Varsity Colors application is a product management and e-commerce application built on
Salesforce and Heroku.
 Main Problem:
•  How to get data for items that are having issues in production into a sandbox for isolated troubleshooting
and resolution?
Case Study: Varsity Colors

 Highly configured products.
•  Apparel with color, size options
•  Designs are customized for each school by layer colors, school names, mascots, fonts, etc.
 400+ base products configured for 180 different color schemes and 800 different schools.
 Non-trivial data model for product catalog (> 40 objects) to store configuration options.
Data Model

 Took about eight hours to build all the SDL mapping ﬁles out
•  Some of the objects have over 40 custom ﬁelds
•  Product object hierarchy is approximately 9 levels deep through a combination of master-detail and
lookup relationships.
 Took some additional time (via trial and error) to determine correct load order for imports.
Solution

 On full catalog import, needed to disable a few triggers as there are a few objects with almost 1M
records and we hit governor limits on triggers.
•  Re-run trigger logic with Apex batch scripts after.
 Developer orgs are too small for full production catalog.
•  Use SOQL to deﬁne subset for development work.
Issues

 Full data export of the product catalog from production can be done in approximately 30 minutes
with a single command from a terminal window.
 SOQL can be used to selectively choose smaller data sets to export for development or
debugging.
 Full import into target org can be done in approximately the same time with also a single
command.
 Extended this solution to export account and team catalog data relevant to a subset of schools to
avoid having to manually set up account storefronts.
Results

 Don’t set up data by hand – your time is too valuable
 Invest the time to set data scripts up for all of your applications and target stakeholders – it will
pay huge dividends pretty quickly
 Work with stakeholders (developers, QA teams) to deﬁne data needs
 Add Load_ID to all objects you create to future-proof your orgs for scripted data
 Have your developers and admins build SDL export and import ﬁles whenever they create new
objects
Summary And Tips

 Use my github repository as a starting point, then adjust and innovate to meet your needs
 Version your SDL mapping ﬁles, but not your SFDC property ﬁles
 Be aware of governor limits on imports and developer sandbox data size limitations and plan
accordingly
 Use SOQL judiciously to reduce your data sets
Summary And Tips

 Github repository
•  git clone https://github.com/scottgeosits/sfdc-dataload-tutorial
 Scripting Data Loads for Salesforce, parts 1 & 2
•  http://bit.ly/1GGHfq8
•  http://bit.ly/1HSnr4u
 Data Loader Guide
•  http://sforce.co/1xUWr0I
 Data Loader github: https://github.com/forcedotcom/dataloader
For More Information

Scripted Data Loads with Salesforce DataLoader and ANT

More Related Content

What's hot (17)

Viewers also liked (20)

Similar to Scripted Data Loads with Salesforce DataLoader and ANT (20)

More from Salesforce Developers (20)

Recently uploaded (20)

Scripted Data Loads with Salesforce DataLoader and ANT