A 5-step methodology for complex E&P data management
Download as PPTX, PDF•2 likes•3,063 views
The document outlines a 5-step methodology to enhance data management standards in the oil and gas industry's exploration and production (E&P) sector, aimed at addressing the increasing complexity of data integration. The steps include separating data models, detaching model mechanics from storage, specifying relationships, implementing error detection processes, and executing a runtime link for integration. Utilizing tools like the transformation manager, the approach seeks to streamline data handling while maintaining high quality and reducing project risks.
A 5-step methodology for complex E&P data management
- 1. A 5-step methodology for complex E&P data management Raising data management standards www.etlsolutions.com
- 2. The increasing complexity of E&P data New devices are being used Timescales are collapsing. in every phase of Exploration Once, drilling and logging & Production (E&P) in the Oil data were distinct activities & Gas industry, gathering separated by days, but now more data with which better they happen decisions can be made. simultaneously. Metadata (in the Dublin core These changes are being and ISO 19115 sense) are factored into the becoming ever more development of industry important in providing standards (such as context. This has a direct PPDM), driving their impact on proprietary evolution to ensure database design and continued use. functionality. The price of progress is growing data complexity.
- 3. A 5-step methodology for managing this data To make a robust and repeatable approach work, we use Transformation Manager, our data The Transformation Manager integration toolset. software is coupled with the approach we have adopted over many years in the Oil & Gas industry The result is a five-stage methodology.
- 4. Step 1 Separate source and target data models and the logic which lies between them. • This means that we can isolate the pure model structure and clearly see the elements, attributes and relationships in each model. • We can also see detail such as database primary keys and comments. • As exposing relationships is the key in handling PPDM and other highly normalized models, this is a critical step.
- 5. Step 2 Separate the model from the mechanics of data storage. • The mechanics define physical characteristics such as ‘this is an Oracle database’ or ‘this flat file uses a particular delimiter or character set’. It is the model that tells us things like ‘a well can have many bores’, ‘a wellbore many logs’, and that ‘log trace mnemonics’ are catalogue controlled. • At a stroke, this separation abolishes a whole category of complexity. • For both source and target we need a formal data model, because this enables us to read or write to database, XML, flat file, or any other data format.
- 6. Step 3 Specify relationships between source and target. • In all data integration projects, determining the rules for the data transfer is a fundamental requirement usually defined by analysts working in this field, often using spreadsheets. • But based on these or other forms of specification, we can create the integration components in Transformation Manager using its descriptive mapping language. This enables us to create a precisely defined description of the link between the two data models. • From this we can generate a runtime system which will execute the formal definitions. Even if we chose not to create an executable link, the formal definition of the mappings is still useful, because it shows where the complexity in the PPDM integration is and the formal syntax can be shared with others to verify our interpretation of their rules.
- 7. Step 4 Follow an error detection procedure. • To ensure that only good data is stored, Transformation Manager has a robust process of error detection that operates like a series of filters. For each phase, we detect errors relevant to that phase and we don't send bad data to the next phase, where detection becomes even more complex. • We detect mechanical and logical errors separately. If the source is a flat file, a mechanical error could be malformed lines; logical errors could include dangling foreign key references or missing data values. • Next, we can detect errors at the mapping level, inconsistencies that are a consequence of the map itself. Here, for example, we could detect that we are trying to load production data for a source well which does not exist in the target. • Finally there are errors where the data is inconsistent with the target logical model. Here, simple tests (a string value is too long, a number is negative) can often be automatically constructed from the model. More complex tests (well bores cannot curve so sharply, these production figures are for an abandoned well) are built using the semantics of the model. • A staging store is very useful in providing an isolated area where we can disinfect the data before letting it out onto a master system. Staging stores were an integral part of the best practice data loaders we helped build for a major E&P company, and it is now common practice that these are stored until issues are resolved.
- 8. Step 5 Execute a runtime link to generate the code required to generate the integration. • This will generate integration components, in the form of Java code, which can reside anywhere in the architecture. • This could be on the source, target or any other system to manage the integration between PPDM and non-PPDM data sources.
- 9. Our offerings: E&P data management Transformation Support, Transformation Manager data training and Manager loader mentoring software developer kits services Data loader Data and migration connector packaged development services
- 10. Why Transformation Manager? For the user: Everything under one roof Greater control and transparency Identify and test against errors iteratively Greater understanding of the transformation requirement Automatically document Re-use and change management Uses domain specific terminology in the mapping
- 11. Why Transformation Manager? For the business: Reduces cost and effort Reduces risk in the project Delivers higher quality and reduces error Increases control and transparency in the development Single product Reduces time to market
- 12. Contact information Karl Glenn [email protected] +44 (0) 1912 894040 Raising data management standards www.etlsolutions.com