Data Governance with JSON Schema
Download as PPTX, PDF0 likes597 views
This document discusses using JSON schema and document validation in MongoDB to validate documents. It provides examples of using MongoDB's schema validation features to validate fields, data types, and values in documents. The document shows how to validate embedded documents in arrays and express complex validation rules to check that calculated fields like totals are correct. It emphasizes that validation allows enforcing business rules to catch errors and improve data quality and regulation compliance.
![#MDBlocal
Document Validation (v3.2)
Use query filters to define schema constraints. E.g.,
db.createCollection("orders", {
validator: {
item: {$type: "string"},
price: {$type: "decimal"},
quantity: {$type: "int"},
color: {$in: ["red", "green", "blue"]}
}
});
(ex1 - ex6)](https://image.slidesharecdn.com/mdb-180222232145/85/Data-Governance-with-JSON-Schema-9-320.jpg)
![#MDBlocal
Good for Starter Examples
{
_id: 34556,
item: "Flannel T-shirt",
price: NumberDecimal("27.99"),
quantity: NumberLong(1),
color: "green"
}
{
_id: 34556,
total: NumberDecimal("141.00"),
VAT: NumberDecimal("0.20"),
totalWithVAT: NumberDecimal("169.20"),
lineitems: [
{ sku: "MDBTS001",
name: "Flannel T-shirt",
quantity: 10,
unit_price: NumberDecimal("9.00") },
{ sku: "MDBTS002",
quantity: 5,
unit_price: NumberDecimal("10.00")}]
}
More Realistic](https://image.slidesharecdn.com/mdb-180222232145/85/Data-Governance-with-JSON-Schema-13-320.jpg)
![#MDBlocal
Good for Starter Examples
{
_id: 34556,
item: "Flannel T-shirt",
price: NumberDecimal("27.99"),
quantity: NumberLong(1),
color: "green"
}
{
_id: 34556,
total: NumberDecimal("141.00"),
VAT: NumberDecimal("0.20"),
totalWithVAT: NumberDecimal("169.20"),
lineitems: [
{ sku: "MDBTS001",
name: "Flannel T-shirt",
quantity: 10,
unit_price: NumberDecimal("9.00") },
{ sku: "MDBTS002",
quantity: 5,
unit_price: NumberDecimal("10.00")}]
}
More Realistic](https://image.slidesharecdn.com/mdb-180222232145/85/Data-Governance-with-JSON-Schema-14-320.jpg)
![#MDBlocal
$expr
Check that totalWithVAT == total * (1+VAT):
$expr: {
$eq: [
"$totalWithVAT",
{$multiply: [
"$total",
{$sum: [1, "$VAT"]}
]}
]
}
Let's add this as a validation rule alongside our JSON Schema validation ()](https://image.slidesharecdn.com/mdb-180222232145/85/Data-Governance-with-JSON-Schema-19-320.jpg)
![#MDBlocal
Ensure total equals sum of line items
$expr: {
$eq: [
"$total",
{$sum: {
$map: {
"input": "$lineitems",
"as": "item",
"in": {
"$multiply": [
"$$item.quantity",
"$$item.unit_price"
] } } }} ] }](https://image.slidesharecdn.com/mdb-180222232145/85/Data-Governance-with-JSON-Schema-21-320.jpg)
Data Governance with JSON Schema
- 1. #MDBlocal Just in Time Validation with JSON Schema
- 2. #MDBlocal Shannon Bradshaw VP of Developer Education, Snowboarding Enthusiast, Father MongoDB @shannonbradshaw
- 7. #MDBlocal
- 8. #MDBlocal { _id: 34556, item: "Flannel T-shirt", price: NumberDecimal("27.99"), quantity: NumberInt(1) color: "green" } Simple Schema to Start
- 9. #MDBlocal Document Validation (v3.2) Use query filters to define schema constraints. E.g., db.createCollection("orders", { validator: { item: {$type: "string"}, price: {$type: "decimal"}, quantity: {$type: "int"}, color: {$in: ["red", "green", "blue"]} } }); (ex1 - ex6)
- 10. #MDBlocal JSON Schema (v3.6) ● JSON Schema Draft 04 ● Is becoming an industry standard ● Widely used; lots of community support (ex5)
- 12. #MDBlocal (ex6 - ex8b) Let's make our validator a little more strict
- 13. #MDBlocal Good for Starter Examples { _id: 34556, item: "Flannel T-shirt", price: NumberDecimal("27.99"), quantity: NumberLong(1), color: "green" } { _id: 34556, total: NumberDecimal("141.00"), VAT: NumberDecimal("0.20"), totalWithVAT: NumberDecimal("169.20"), lineitems: [ { sku: "MDBTS001", name: "Flannel T-shirt", quantity: 10, unit_price: NumberDecimal("9.00") }, { sku: "MDBTS002", quantity: 5, unit_price: NumberDecimal("10.00")}] } More Realistic
- 14. #MDBlocal Good for Starter Examples { _id: 34556, item: "Flannel T-shirt", price: NumberDecimal("27.99"), quantity: NumberLong(1), color: "green" } { _id: 34556, total: NumberDecimal("141.00"), VAT: NumberDecimal("0.20"), totalWithVAT: NumberDecimal("169.20"), lineitems: [ { sku: "MDBTS001", name: "Flannel T-shirt", quantity: 10, unit_price: NumberDecimal("9.00") }, { sku: "MDBTS002", quantity: 5, unit_price: NumberDecimal("10.00")}] } More Realistic
- 15. #MDBlocal Our New Orders Schema ● Orders must contain a lineitems array of 1 to 10 documents ● Documents embedded in lineitems have fields for: ○ sku (required) ○ name (optional) ○ unit_price (required) ○ quantity (required) ● Value of quantity must be an integer ● (Must also contain fields: total, VAT, totalWithVAT, but we'll get to that.)
- 16. #MDBlocal Let's validate the line item documents (embedded in an array) (ex9 - 10b)
- 17. #MDBlocal Now Let's Deal with Totals and VAT The most recent order we looked at actually contains a few errors: ● The totalWithVAT value should be 141 * 1.20 = 169.2 ● The total value should be the sum of each line item sub-total ● I.e., (10 * 9) + (10 * 5) = 140 Can we enforce that totalWithVAT and total are correct using validation?
- 18. #MDBlocal Yes we can! ● MongoDB 3.6 -- more expressive query syntax ● Can express complex business rules in validations ● E.g, compute and compare multiple values at runtime
- 19. #MDBlocal $expr Check that totalWithVAT == total * (1+VAT): $expr: { $eq: [ "$totalWithVAT", {$multiply: [ "$total", {$sum: [1, "$VAT"]} ]} ] } Let's add this as a validation rule alongside our JSON Schema validation ()
- 20. #MDBlocal Now let's make sure the total is the sum of the line item subtotals
- 21. #MDBlocal Ensure total equals sum of line items $expr: { $eq: [ "$total", {$sum: { $map: { "input": "$lineitems", "as": "item", "in": { "$multiply": [ "$$item.quantity", "$$item.unit_price" ] } } }} ] }
- 22. #MDBlocal All together now (ex11 - 14c)
- 26. #MDBlocal Special thanks to Raphael Londner
- 27. #MDBlocal Learn More ● What's New Guide ● MongoDB Documentation ○ Schema Validation ○ Expressive Query Syntax ● Free MongoDB 3.6 Online Course from MongoDB University