More ways of symbol grounding for knowledge graphs?
Download as PPTX, PDF•1 like•4,345 views
This document discusses various ways to ground the symbols used in knowledge graphs. It describes the traditional "symbol grounding problem" where symbols are defined based only on other symbols. It then outlines several approaches to grounding symbols in non-symbolic ways, such as by linking them to perceptual modalities like images, audio, and simulation. It also discusses grounding symbols via embeddings, relationships to physical entities, and operational semantics. The document argues that richer grounding could help integrate these notions and enhance interoperability, exchange, identity, and reasoning over knowledge graphs.
More ways of symbol grounding for knowledge graphs?
- 1. More ways of symbol grounding for knowledge graphs? Paul Groth (@pgroth) Elsevier Labs pgroth.com Dagstuhl Seminar 18371 Knowledge Graphs: New Directions for Knowledge Representation on the Semantic Web
- 2. "How can you ever get off the symbol/symbol merry-go-round? How is symbol meaning to be grounded in something other than just more meaningless symbols? This is the symbol grounding problem.” (Harnard, 1990) Harnad, S. (1990) The Symbol Grounding Problem. Physica D 42: 335-346. http://cogprints.org/3106/ What does http://dbpedia.org/resource/Netherlands mean?
- 3. Symbol Grounding & the Semantic Web Key notion: Social commitment (Cregan, 2007) • designation - what is being referred to • entailment - what are the (logical)consequences of something Good enough? Cregan A.M. (2007) Symbol Grounding for the Semantic Web. In: Franconi E., Kifer M., May W. (eds) The Semantic Web: Research and Applications. ESWC 2007. Lecture Notes in Computer Science, vol 4519. Springer, Berlin, Heidelberg
- 4. Designation & Dereferenceablity Looking definitions up – Natural Language and Programmatic
- 6. schema:dateModified a rdf:Property ; rdfs:label "dateModified" ; schema:domainIncludes schema:CreativeWork, schema:DataFeedItem ; schema:rangeIncludes schema:Date, schema:DateTime ; rdfs:comment "The date on which the CreativeWork was most recently modified or when the item's entry was modified within a DataFeed." . schema:datePublished a rdf:Property ; rdfs:label "datePublished" ; schema:domainIncludes schema:CreativeWork ; schema:rangeIncludes schema:Date ; rdfs:comment "Date of first broadcast/publication." . schema:disambiguatingDescription a rdf:Property ; rdfs:label "disambiguatingDescription" ; schema:domainIncludes schema:Thing ; schema:rangeIncludes schema:Text ; rdfs:comment "A sub property of description. A short description of the item used to disambiguate from other, similar items. Information from other properties (in particular, name) may be necessary for the description to be useful for disambiguation." ; rdfs:subPropertyOf schema:description . https://www.w3.org/TR/rdf11-mt/ Entailment – logics
- 7. Are relations good enough to describe entities? A knowledge graph is "graph structured knowledge bases (KBs) which store factual information in form of relationships between entities" (Nickel et al. 2015). Nickel, M., Murphy, K., Tresp, V., & Gabrilovich, E. (2015). A Review of Relational Machine Learning for Knowledge Graphs, 1–18.
- 8. Other ways of grounding symbols
- 9. Sub-symbolic representations (aka embeddings) Yang, Fan, Zhilin Yang, and William W. Cohen. "Differentiable learning of logical rules for knowledge base reasoning." Advances in Neural Information Processing Systems. 2017. Rocktäschel, T., & Riedel, S. (2017). End-to-end differentiable proving. In Advances in Neural Information Processing Systems (pp. 3791-3803).
- 10. Grounding in physical reality http://cynthia.matuszek.org/Icra2014GestureLanguage.html https://www.csee.umbc.edu/~cmat/ “Grounded Language Acquisition: Learning models of language using data from the noisy, probabilistic physical world in which robots and humans both reside. This makes language learning easier (how do you learn the meaning of "green" without a camera?) and makes robots more able to understand instructions and descriptions.” Wiriyathammabhum, P., Summers-Stay, D., Fermüller, C., & Aloimonos, Y. (2017). Computer vision and natural language processing: recent approaches in multimedia and robotics. ACM Computing Surveys (CSUR), 49(4), 71.
- 11. Grounding in Perception – Audio / Images Kiela, Douwe, and Stephen Clark. "Learning neural audio embeddings for grounding semantics in auditory perception." Journal of Artificial Intelligence Research 60 (2017): 1003-1030. Kiela, Douwe. Deep embodiment: grounding semantics in perceptual modalities. No. UCAM-CL-TR-899. University of Cambridge, Computer Laboratory, 2017. Kiela, D., Conneau, A., Jabri, A., & Nickel, M. (2017). Learning visually grounded sentence representations. arXiv preprint arXiv:1707.06320.
- 12. Image and Video Grounding Datasets visualgenome.org Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li Jia- Li, David Ayman Shamma, Michael Bernstein, Li Fei-Fei Gella, Spandana, and Frank Keller. "An Analysis of Action Recognition Datasets for Language and Vision Tasks." Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). Vol. 2. 2017 Xu, J., Mei, T., Yao, T., & Rui, Y. (2016). Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 5288-5296). Miech, A., Laptev, I., & Sivic, J. (2018). Learning a Text-Video Embedding from Incomplete and Heterogeneous Data. CoRR, abs/1804.02516. https://www.di.ens.fr/willow/research/mee/
- 13. Grounding in Simulation https://ai2thor.allenai.org
- 14. Operational Semantics (or actually just Javascript) https://mixedreality.mozilla.org
- 15. Thoughts • Potential richer ways to ground the symbols within a knowledge graph. • How do we integrate with these notions? • Things that can be brought to this work • Interoperability • Exchange • Identity • Reasoning • Things not mentioned but in the same boat: • Abstract Meaning Representation • Universal Dependencies