Link Discovery Tutorial Part III: Benchmarking for Instance Matching Systems
1 like639 views
The document is a tutorial on benchmarking instance matching systems, discussing the importance, methodologies, and characteristics of effective benchmarks. It emphasizes the need for standardized evaluation processes to measure the performance of various tools and promote improvement in instance matching technologies. Additionally, it highlights the existing benchmarks, their attributes, and calls for advancements to address complex variations and scalability challenges in the field.
![Semantic Web Instance Generation
(SWING) [FMN+11]
Semi automatic generator of Instance Matching Benchmarks
Contributed in the generation of IIMB Benchmarks of OAEI in 2010, 2011
and 2012 Instance Matching Tracks
Freely available at (https://code.google.com/p/swing-generator/)
All kind of variations supported into the benchmarks except multilinguality
Automatically produced gold standard
Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 13 / 36](https://image.slidesharecdn.com/ldtutorialbenchmarking-161108153343/85/Link-Discovery-Tutorial-Part-III-Benchmarking-for-Instance-Matching-Systems-13-320.jpg)
![Lance [SDF+15b]
Flexible, generic and domain-independent benchmark generator which takes
into consideration RDFS and OWL constructs in order to evaluate instance
matching systems.
Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 14 / 36](https://image.slidesharecdn.com/ldtutorialbenchmarking-161108153343/85/Link-Discovery-Tutorial-Part-III-Benchmarking-for-Instance-Matching-Systems-14-320.jpg)
![Lance [SDF+15b]
Lance provides support for:
Semantics-aware transformations
Complex class definitions (union, intersection)
Complex property definitions (functional properties, inverse functional
properties)
Disjointness (properties)
Standard value and structure based transformations
Weighted gold standard based on tensor factorization
Varying degrees of difficulty and fine-grained evaluation metrics
Available at http://github.com/jsaveta/Lance
Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 15 / 36](https://image.slidesharecdn.com/ldtutorialbenchmarking-161108153343/85/Link-Discovery-Tutorial-Part-III-Benchmarking-for-Instance-Matching-Systems-15-320.jpg)
Link Discovery Tutorial Part III: Benchmarking for Instance Matching Systems
- 1. Link Discovery Tutorial Benchmarking for Instance Matching Systems Axel-Cyrille Ngonga Ngomo(1) , Irini Fundulaki(2) , Mohamed Ahmed Sherif(1) (1) Institute for Applied Informatics, Germany (2) FORTH, Greece October 18th, 2016 Kobe, Japan Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 1 / 36
- 2. The Questions(s) Instance matching research has led to the development of various systems. What are the problems that I wish to solve? What are the relevant key performance indicators? What is the behavior of the existing engines w.r.t. the key performance indicators? Which are the tool(s) that I should use for my data and for my use case? Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 2 / 36
- 3. Importance of Benchmarking Benchmarks exist To allow adequate measurements of systems To provide evaluation of engines for real (or close to real) use cases Provide help Designers and Developers to assess the performance of their tools Users to compare the different available tools and evaluate suitability for their needs Researchers to compare their work to others Leads to improvements: Vendors can improve their technology Researchers can address new challenges Current benchmark design can be improved to cover new necessities and application domains Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 3 / 36
- 4. The Answer: Benchmark your engines! Instance Matching/Linking Benchmark comprises of Datasets: The raw material of the benchmarks. These are the source and the target dataset that will be matched together to find the links between resources Test Cases: Address heterogeneities (structural, value, semantic) of the datasets to be matched Gold Standard (Ground Truth / Reference Alignment): The "correct answer sheet" used to judge the completeness and soundness of the instance matching algorithms Metrics: The performance metric(s) that determine the systems behaviour and performance Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 4 / 36
- 5. Benchmark Datasets: Characteristics Nature Real Datasets: Widely used datasets from a domain of interest + Realistic conditions for heterogeneity problems + Realistic distributions - Error prone, hard to create Reference Alignment Synthetic Datasets: Produced with a data generator (that hopefully produces data with interesting characteristics + Fully controlled test conditions + Accurate, Easy to create Reference Alignments - Unrealistic distributions - Systematic heterogeneity problems Schema Datasets to be matched have the same or different schemas Domain Datasets come from the same or different domains Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 5 / 36
- 6. Benchmark Test Cases: Variations Value Name style abbreviations, Typographical errors, change format (date/gender/number), synonym change, language change (multilinguality) Structural Change property depth, Delete/add property, split property values, transformation of object/data to data/object type property Semantics class deletion/modification, invert property assertions, change class/property hierarchy, assert class disjointness Combinations of Variations Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 6 / 36
- 7. Benchmark: Gold Standard The "correct answer sheet" used to judge the completeness and soundness of the instance matching algorithms Characteristics Existence of errors / missing alignments Representation: owl:sameAs and skos:exactMatch Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 7 / 36
- 8. Benchmark: Metrics Precision P = tp (tp+fn) Recall R = tp (tp+fp) F-measure F = 2 × P × R (P+R) Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 8 / 36
- 9. Instance Matching Benchmarks: Desirable Attributes Systematic Procedure matching tasks should be reproducible and the exe- cution must be comparable Availability benchmark should be available Quality precise evaluation rules and high quality ontologies must be provided Equity evaluation process should not privilege any system Dissemination benchmark should be used to evaluate instance matching systems Volume dataset size Gold Standard gold standard should exist and be as accurate as pos- sible Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 9 / 36
- 10. What about Benchmarks? Instance matching techniques have, until recently, been benchmarked in an ad-hoc way. There is no standard way of benchmarking the performance of the systems, when it comes to Linked Data. Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 10 / 36
- 11. Ontology Alignment Evaluation Initiative IM benchmarks have been mainly driven forward by the Ontology Alignment Evaluation Initiative (OAEI) organizes annual campaign for ontology matching since 2005 hosts independent benchmarks In 2009, OAEI introduced the Instance Matching (IM) Track focuses on the evaluation of different instance matching techniques and tools for Linked Data Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 11 / 36
- 12. Instance Matching Benchmarks Bechmark Generators Synthetic Benchmarks Real Benchmarks Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 12 / 36
- 13. Semantic Web Instance Generation (SWING) [FMN+11] Semi automatic generator of Instance Matching Benchmarks Contributed in the generation of IIMB Benchmarks of OAEI in 2010, 2011 and 2012 Instance Matching Tracks Freely available at (https://code.google.com/p/swing-generator/) All kind of variations supported into the benchmarks except multilinguality Automatically produced gold standard Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 13 / 36
- 14. Lance [SDF+15b] Flexible, generic and domain-independent benchmark generator which takes into consideration RDFS and OWL constructs in order to evaluate instance matching systems. Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 14 / 36
- 15. Lance [SDF+15b] Lance provides support for: Semantics-aware transformations Complex class definitions (union, intersection) Complex property definitions (functional properties, inverse functional properties) Disjointness (properties) Standard value and structure based transformations Weighted gold standard based on tensor factorization Varying degrees of difficulty and fine-grained evaluation metrics Available at http://github.com/jsaveta/Lance Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 15 / 36
- 16. Lance Architecture Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 16 / 36
- 17. Synthetic Benchmarks Ontology Alignment Evaluation Benchmarks Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 17 / 36
- 18. Synthetic Instance Matching Benchmarks: Overview (1) IIMB 2009 IIMB 2010 PR 2010 IIMB 2011 Sandbox 2012 IIMB 2012 RDFT 2013 ID- REC 2014 Author Task 2015 Systematic Procedure √ √ √ √ √ √ √ √ √ Availability √ √ √ √ √ - - √ √ Quality √ √ √ √ √ √ √ √ √ Equity √ √ √ √ √ √ √ √ √ Dissemination 6 3 6 1 3 4 4 5 5 Volume 0.2K 1.4K 0.86K 4K 0.375K 1.5K 0.43K 2.650K 10K Gold Standard √ √ √ √ √ √ √ √ √ Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 18 / 36
- 19. Synthetic Instance Matching Benchmarks: Overview (2) IIMB 2009 IIMB 2010 PR 2010 IIMB 2011 Sandbox 2012 IIMB 2012 RDFT 2013 ID- REC 2014 Author Task 2015 Value Variations √ √ √ √ √ √ √ √ √ Structural Variations √ √ √ √ - - - + + Logical Variations √ √ - √ - √ - - - Multilinguality - - - - - - √ √ √ IIMB 2009 IIMB 2010 PR 2010 IIMB 2011 Sandbox 2012 IIMB 2012 RDFT 2013 ID- REC 2014 Author Task 2015 Blind Evaluations - - - - - - √ √ √ 1-n Mappings - - √ - - - √ √ - Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 19 / 36
- 20. Synthetic Instance Matching Benchmarks: Overview (3) IIMB 2009 IIMB 2010 PR 2010 IIMB 2011 Sandbox 2012 IIMB 2012 RDFT 2013 ID- REC 2014 Author Task 2015 Lance 2015 Systematic Procedure √ √ √ √ √ √ √ √ √ √ Availability √ √ √ √ √ - - √ √ √ Quality √ √ √ √ √ √ √ √ √ √ Equity √ √ √ √ √ √ √ √ √ √ Dissemination 6 3 6 1 3 4 4 5 5 2 Volume 0.2K 1.4K 0.86K 4K 0.375K 1.5K 0.43K 2.650K 10K > 1M Gold Standard √ √ √ √ √ √ √ √ √ √ Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 20 / 36
- 21. Synthetic Instance Matching Benchmarks: Overview (4) IIMB 2009 IIMB 2010 PR 2010 IIMB 2011 Sandbox 2012 IIMB 2012 RDFT 2013 ID- REC 2014 Author Task 2015 Lance 2015 Value Variations √ √ √ √ √ √ √ √ √ √ Structural Variations √ √ √ √ - - - + + + Logical Variations √ √ - √ - √ - - - + Multilinguality - - - - - - √ √ √ √ Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 21 / 36
- 22. Synthetic Instance Matching Benchmarks: Overview (5) IIMB 2009 IIMB 2010 PR 2010 IIMB 2011 Sandbox 2012 IIMB 2012 RDFT 2013 ID- REC 2014 Author Task 2015 Lance 2015 Blind Evaluations - - - - - - √ √ √ √ 1-n Mappings - - √ - - - √ √ - - Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 22 / 36
- 23. Real Benchmarks Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 23 / 36
- 24. Real Instance Matching Benchmarks: Overview (1) ARS DI 2010 DI 2011 Systematic Procedure √ √ √ Availability √ √ - Quality √ √ √ Equity √ √ √ Dissemination 5 2 3 Volume 100K 6K NA Gold Standard √ √ + Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 24 / 36
- 25. Real Instance Matching Benchmarks: Overview (2) ARS DI 2010 DI 2011 Value Variations √ √ √ Structural Variations √ √ - Logical Variations - - - Multilinguality - - - Blind Evaluations - - - Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 25 / 36
- 26. Wrapping Up Multilinguality Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 26 / 36
- 27. Wrapping Up Value Variations Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 27 / 36
- 28. Wrapping Up Structural Variations Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 28 / 36
- 29. Wrapping Up Logical Variations Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 29 / 36
- 30. Wrapping Up Combinations of Variations Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 30 / 36
- 31. Wrapping Up Scalability Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 31 / 36
- 32. Open Issues Only one benchmark that tackles both, combination of variations and scalability issues Not enough IM benchmark using the full expressiveness of RDF/OWL language Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 32 / 36
- 33. Systems Systems can handle the value variations, the structural variation, and the simple logical variations separately. More work needed for complex variations (combination of value, structural, and logical) More work needed for structural variations Enhancement of systems to cope with the clustering of the mappings (1-n mappings) Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 33 / 36
- 34. Conclusions Many instance matching benchmarks have been proposed Each of them answering to some of the needs of instance matching systems. It is essential to start creating benchmarks that will “show the way to the future” Extend the limits of existing systems. Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 34 / 36
- 35. Acknowledgment This work was supported by grants from the EU H2020 Framework Programme provided for the project HOBBIT (GA no. 688227). Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 35 / 36
- 36. References I Ngonga Ngomo et al. (AKSW & FORTH) LD Tutorial:Benchmarking October 17, 2016 36 / 36