FIWARE Training: Introduction to Smart Data Models
0 likes225 views
The document introduces the Smart Data Models program which provides standardized data models for various domains. It explains that the program aims to enable agile standardization through contributions from the community. It outlines the governance structure and current status of the program, including the available domains, data models, contributors and tools. Participants are then guided through an exercise to turn a data source into a Smart Data Model by generating a JSON schema, example payload and submitting it as a pull request to the incubated repository on GitHub.
FIWARE Training: Introduction to Smart Data Models
- 1. Introduction to Smart Data Models Agile standardization with the Smart Data Models Program 27-6-21 Alberto Abella Data Modelling expert FIWARE Foundation @smartdatamodels @fiware @aabella [email protected]
- 2. ■ About: ■ During this part of the session you will get introduced to the Smart Data Models Program ■ This session will: ■ Explain basics about the Smart Data Models program and how it is governed ■ Introduce you to data models for specific verticals that are already available ■ Explain how you may contribute extensions to existing data models under Smart Data Models ■ Explain how you may contribute new data models under Smart Data Models ■ Goals: ■ After this session you will be able to implement a service using a Smart Data Model ■ You will be able to explain how to become an active contributor to the Smart Data Model program Contents and goals
- 3. ■ Question: ■ What industrial sector are you interested in so we can visit this while the online demonstration (use the chat) ■ This presentation is available at https://bit.ly/i4trust2 (Recommended to get the links) ■ Write it down in this link http://bit.ly/github_users Contents and goals
- 4. ■ What is the Smart Data Models Program. ■ Structure and governance ■ Current Status ■ Becoming a contributor of Smart Data Models Program ■ Practical exercises ■ Services for creating a data model ■ Creating an official data model Agenda
- 5. Smart Data Models: Introduction and Governance
- 6. Steering Board : ■ Four members (June-2022) ■ IUDX: Indian Urban Data Exchange ■ Public entity supporting data interchange for Smart cities in India ■ TMForum: ■ Worldwide Telecommunications association ■ FIWARE Foundation ■ Curator of the open source FIWARE platform and its ecosystem ■ Open and Agile Smart Cities ■ Smart cities association for the cooperation and mutual learning Introduction and Governance
- 7. Introduction and Governance ▪ A community site with detailed data models available for open use for multiple sectors ▪ Together with other relevant organizations in the curation of the different domains and subjects ▪ Providing coherence and consistency between data models across different domains ▪ To create a method for AGILE standardization and evolution these data models ▪ To provide extended usefulness to FIWARE platform users in terms of: − Extended interoperability − Reduced time dedicated to data model codying − Accumulated experience tested in real case scenarios − Mapped to be integrated with other platforms ▪ Using open licensing to allow extensive use and adoption ▪ Allowing customization and evolution of data models ▪ Based on real case scenarios ▪ Based on git platform and github as development frontend ▪ Connecting with linked data ▪ Based on widely adopted standards (including ontologies and international schemas (i.e. schema.org)
- 8. Standardization bodies Current markets Emerging markets Agile standardization weeks days years years asap Research weeks months 8 years start end Standardization in a digital market
- 9. By Pieter Brueghel the Elder - bAGKOdJfvfAhYQ at Google Cultural Institute, zoom level maximum, Public Domain, https://commons.wikimedia.org/w/index.php?curid=22178101 Differential advantages 1. Agile standardization. Standardization time takes Weeks vs months/years 2. Easy contribution. One single file source of truth. 3. Based on actual experience. All data models are based on real case scenarios. 4. Multilanguage (6). Automation translation of specs. Introduction and Governance 5. 1 change for all domains. Changing a data model impact on all the domains related to it.
- 10. ▪ Don’t reinvent the wheel. Take advantage of the experience of other FIWARE users ▪ Many attributes (>18.000 already available in the database). Don’t create definitions yourself unless you really want to do it. ▪ Specification automatic generated in 6 languages (EN,FR, GE, SP, IT, JA). No work for the creators ▪ Examples of the data models available ▪ Following open and adopted standards ▪ Everything is open-licensed (Free reuse, free modification and free sharing). Customize to your needs. ▪ It does nos impose restriction on using additional attributes (e.g. internal ones) ▪ It does nos impose restrictions on how many attributes has to implement in your use case Why should you use Smart Data Models Trusted Spark DT Basic Spark DT
- 11. Smart Data Models: Structure and contents. Verticals
- 12. ■ What is a Smart Data Model: ■ It is the definition of a set of attributes, their definitions and their data types that can be combined into a single entity. ■ It is the combination of 3 elements ■ A technical description of the data types and relationships of the attributes of an element that has to be modeled ■ A documented description of these attributes aligned with the technical description ■ Some examples of the use of the data model ■ Based on real case scenario (not theory or academic research) ■ With an open license allowing its use, share and modification ■ What is NOT a Smart Data Model ■ It is not an ontology describing the elements of an area of knowledge ■ A new standard ■ An academic exercise Smart Data Models: Structure and contents. Verticals
- 13. Structure and contents. Verticals data-models Umbrella repo Smart Water Subject 1 (sewage) Smart Cities Smart Building Smart Environment Smart Destinations Smart Energy Smart Manufacturing Smart Agrifood Smart Robotics Subject 2 (parking) Subject 3 (weather) Subject 4 (...) DOMAINS REPOSITORIES Readme pointing to the list of subjects General info or shared resources DATA-MODELS - Guides for coding new data models - Template for new data models and examples - Directory for scripting tools to check data models - Official list of domains and data models SUBJECTS’ REPOSITORIES Readme pointing to the list of data models for the objects Contributors.md subject-schema.json DATA MODELS README.md /doc/spec.md /examples schema.json Adopters LIFECYCLE MANAGEMENT REPOSITORIES Incubated Harmonization
- 14. Structure and contents. Verticals GITHUB http://github.com/smart-data-models - Oriented to developers - All resources available - Contribution by PR - Issues on data models - Demo visit - Oriented to end users - News on updates (subscription) - Check attributes and enumerations - Check descriptions - Demo visit SITE (wp) http://smartdatamodels.org
- 15. Current status (domains, subjects & data models) 15 Smart Energy 424 1 Smart Sensoring* 138 2 Smart Cities 85 3 Cross Sector 77 4 Smart Water 35 5 Smart Agrifood 30 6 Smart Environment 23 7 Smart Aeronautics 13 8 Smart Robotics 12 9 Smart Destination 11 10 Smart Manufacturing 3 11 Smart Health 2 12 Updated 27-6-22 * Many sensors are specific from other domains but not counted there Smart cities 27, Health 19, Environment 12, Energy 5, Water 4, Agrifood 1, Robotics 1
- 16. Current status: New subjects new data models 16 CPSV-AP 1 AutonomousMobileRobot 2 IT 3 STAT-DCAT-AP 4 OCF 5 807 Official Smart Data Models 89 on the queue to be accepted 28 DM Harmonization 75 DM Incubated Digital Innovation Hub 6 Updated 27-6-22 56 subjects (Groups of data models) Incubated RAMI 4.0 Transit Management Verifiable Credentials GS1 mapping OSLO (transport) Vineyards (Agri) CCVC Vessel
- 17. Contributors and dissemination ▪ 116 active contributors ▪ 226 contribution in data models ▪ 22 services to contributors in data models ▪ Contributors belong to 75 different organizations ▪ Terms available for search 18.271 ▪ Documented adopters 130 ▪ Every term in data models has an associated page https://smartdatamodels.org/term ▪ Google finds 1570 pages in smartdatamodels.org Updated 20-6-22
- 18. 18 Format: json schema Include descriptions validates examples in keyvalues format 1.-Schema 2.-Examples File name schema.json at the root of the data model folder Format: json, jsonld, csv, etc Always id and type File names exampleXXXXX.NNN at examples directory in data model folder Origin: from actual use cases or derived from them Always in actual systems ● keyvalues only export format ● normalized usual format Single-source-of-truth for the attributes’ definitions Never stored in a context broker Defines attributes’ names and their data types Conceptual elements 1/ 2 Manual contribution From actual use case
- 19. 19 Format: markdown 6 languages 3.-Specification File name specXX.md at the docs data model folder Generated automatically. Never edited Never contributed Explanatory of the data model Conceptual elements 2/ 2 Format: repository in github Contains several folders for data models Root of the repository 4.-Subject include file CONTRIBUTORS (list of people) README: List of data models included Created by the organization Belongs to one or several domains
- 20. 20 Format: jsonld Long URI for all subject’s attributes 5.-Context File name context.jsonld at subject root Generated automatically. Never edited Never contributed For linked data solutions Conceptual elements 2/ 2 Format: yaml List of entities using the data model 6.-Adopters Voluntary. Incentives At the root of every data model Contributed manual
- 21. 21 Format: yaml At the root of the subject 7.-Contributors File name CONTRIBUTORS.yaml Manually contributed. Incremental Voluntary to appear Conceptual elements 2/ 2 Format: yaml Text for customzie specifications and README 8.-notes.yaml Voluntary content File name notes.yaml at the root of subject or root of data models Contributed manual
- 22. Structure and contents. Verticals ■ Contents: ■ schema.json single source of truth of the model. Validates only the key-values payloads ■ examples : Directory for examples ■ ADOPTERS.yaml use cases of the data model ■ doc: directory for specifications ■ LICENSE.md license of the data model. I.e. CC-BY ■ README.md pointer to the main elements of the data model, including examples, specifications and other services ■ model.yaml technical description of the attributes for embedding into the specifications ■ notes.yaml additional file for customization contents for the specifications ■ swagger.yaml yaml file required for the interactive specification and future test of services ■ Mandatory ■ Optional ■ Automatic
- 23. Tools for support (option at the main menu) Tools List data models: Find a data model from data model name or subject name Generator of data model from json example: Drafts a 1.- Schema from a 3.-Example in json Create data model from csv payload: Drafts a 1.- Schema from a 3.-Example in json Data Model Editor: Drafts a 1.- Schema manually (text editor) Data model documentation checker: Checks that a 1.- Schema includes all the descriptions Generator of NGSI-LD normalized: Creates a normalized 3.-Example from an existing 1.- Schema Generator of NGSI-LD keyvalues: Creates a keyvalues 3.-Example from an existing 1.- Schema GMapper @context to external ontologies: Generates a 5.-Context mapped to external ontologies Subjects’ @context merger: Generates a 5.-Context from several subjects mapped with SDM URI
- 24. Tools for support (option at the main menu) Support Slack Channel Subscription to updates Submit an issue Documentation Contribution manual Code recommendations Learning zone About FAQ Governance Sstatistics Community Data models in progress Events
- 25. Smart Data Models: Become a contributor.
- 26. 26 1 Do you have a use case? Exception management No Yes Is any data model related? Yes Access contribution manual and draft yor data model (tool) in your repository Ask permission to contribute in the incubated repository Option 1 Option 2 (recommended) Pull request on the incubated repository after completing the contribution checklist Work on the incubated repository and ask for support through issues or mail Pass the manual review? No Fix the comments Publication process 2 LEGEND 1: Start new model 2: End new model 3: Start update existing model 3 Create your proposal of update Pull request on the official repository of the data model 3 No Yes Pass the manual review? Fix the comments No Yes Smart Data Models: Become a contributor.
- 27. 27 Smart Data Models: Contribution Manual ▪ Available at https://bit.ly/contribution_manual
- 28. Smart Data Models: Exercises Turn a data source definition into a data model
- 29. ■ Expected knowledge from the participants ■ Knowledge of NGSI-v2 and NGSI-LD and their differences ■ Some work with a Context Broker (any of them) ■ JSON and JSON Schema ■ Git and GitHub concepts ■ A code editor like PyCharm Exercise 29
- 30. ■ Generate the key-values of payload ■ Use the script or manually with your code editor Before we start. Checking services @smartdatamodels 30
- 31. ■ It is required your github users to be written here (http://bit.ly/github_users) to grant you access to the incubated repository. ■ If you do not have a github user, then go here: http://bit.ly/register_github ■ Use the repository incubated. https://github.com/smart-data-models/incubated/tree/master ■ The final exercise is to submit a complete data model. ■ Complete the creation of an official data model through all its steps. ■ It will be done with official sources, so the result of the exercise, if completed, will become an official data model of the program. ■ Comments to the contribution manual will be incorporated ■ If not completed during session time It can be completed afterwards Exercise 31
- 33. Data sources These data sources have dozens of properties. We’ll only take a few for the exercise. ■ https://www.schema.org/MedicalCondition ■ https://www.schema.org/MedicalGuideline ■ https://www.schema.org/Drug ■ https://www.schema.org/MedicalScholarlyArticle ■ https://www.schema.org/LocalBusiness ■ https://www.schema.org/Restaurant ■ Other available. https://www.schema.org/docs/health-lifesci.home.html Take a minimum of 5 properties, one an object / array. Exercise 33
- 34. Data sources Other data sources could be valid as well as: 1. It has documented attributes 2. It has clear data types for the attributes 3. It is on use in some real case scenario Exercise 34
- 35. ■ Steps: 1. Access to the data source 2. Open the editor 3. Paste the data definitions in the editor according to the instructions. 4. Generate the json schema 5. Validate the json schema 6. Generate the example of payload 7. Validate the example against the schema 8. Submit the new data model Exercise 35
- 36. ■ Review the contribution manual 1. Contribution manual is linked in the main page or in https://bit.ly/contribution_manual Exercise 36
- 37. ■ Access to the data source. Examples available. a. https://www.schema.org/MedicalCondition b. https://www.schema.org/MedicalGuideline c. https://www.schema.org/Drug d. https://www.schema.org/MedicalScholarlyArticle e. https://www.schema.org/LocalBusiness f. https://www.schema.org/Organization g. https://www.schema.org/Restaurant h. https://www.schema.org/docs/health-lifesci.home.html Exercise 37
- 38. ■ Open the online editor Exercise 38
- 39. ■ Paste the data definitions in the notepad according to the contribution manual. ■ We’ll see this in these 3 videos (9 minutes). Learning zone at smartdatamodels.org Exercise 39
- 40. ■ Generate the json schema ■ Fix possible errors ■ Provide more detail / limits ■ Include Units ■ Include Enumerations ■ Include Privacy ■ Include limitations to values ■ Required Properties Exercise 40
- 41. ■ Validate the json schema ■ Validation of the json schema https://www.jsonschemavalidator.net/ ■ Validation of the documentation https://smartdatamodels.org/index.php/da ta-models-contribution-api/ Exercise 41
- 42. ■ Generate the normalized example of payload ■ I.e. NGSI-LD is available ■ Needs edit for meaningful data Exercise 42
- 43. ■ Validate the payload (keyvalues). https://www.jsonschemavalidator.net/ ■ Exercise 43
- 44. ■ Fork the incubated repository ■ PR your changes to the data model They will appear in the front page in 5 minutes maximum (right column, bottom) Exercise 44
- 45. Smart Data Models: Summary
- 46. ■ Goal ■ Allow real interoperability between NGSI data sources ■ Contribution: ■ Always possible as long as it has a use case, and comply with contribution workflow ■ Use of the data models ■ Search tools for finding the right data model ■ Best not to reinvent the wheel ■ Differential advantages of agile standardization ■ Quick answer ■ Do not invent ■ Easy contribution ■ Single source of truth ■ Better simple and useful than technically ‘correct’ and powerful Summary