Applying Noisy Knowledge Graphs to Real Problems
Download as PPTX, PDF1 like262 views
The document discusses the application of noisy knowledge graphs to address complex real-world challenges, such as human trafficking and other illicit domains, highlighting the issues of messy data and the active role of users as producers. It emphasizes the importance of strategic design decisions in representing raw data through knowledge graphs and improving query reformulation techniques for better user intent satisfaction. Additionally, it notes the impact of related tools and academic contributions to these areas over the past few years.
Applying Noisy Knowledge Graphs to Real Problems
- 1. Applying Noisy Knowledge Graphs to Real Problems Mayank Kejriwal USC Information Sciences Institute May 2019
- 5. Web has lowered the barrier to entry! 5
- 6. 6
- 7. 7 Pump and dump schemes proliferate online
- 8. 8
- 9. Quechua Fula Odiya Maithili Bhojhpuri Uighyur Mayan languages Aboriginal languages Tasmanian languages Fang Umbundu Setswana Afro-Asiatic Khoisan Fon Yoruba Peulh Adangame Erzya Bashkir Khakas Udmurt Ingush Tagalog Hilgaynon Bikol Waray Native American dialects
- 12. 12
- 13. 13
- 14. What do these problems have in common (besides being really hard)?
- 15. 1. Very messy, raw data, with both redundancy and irrelevance 2. Users are also producers i.e. we cannot just ‘build’ the system and hand it off 3. Domains are largely non-analytic (e.g., we don’t have a model/equations for human trafficking)
- 16. 1. Very messy, raw data, with both redundancy and irrelevance 2. Users are also producers i.e. we cannot just ‘build’ the system and hand it off 3. Domains are largely non-analytic (e.g., we don’t have a model/equations for human trafficking)
- 17. 1. Very messy, raw data, with both redundancy and irrelevance 2. Users are also producers i.e. we cannot just ‘build’ the system and hand it off 3. Domains are largely non-analytic (e.g., we don’t have a model/equations for human trafficking)
- 19. 19 Space of design decisions Raw data Search+GUI ? ?? Representation + Infrastructure ? ? ?
- 20. Space of design decisions Search+GUI Producer Consumer
- 21. 21
- 22. 22
- 23. 23 Space of design decisions Raw data Search+GUI Knowledge Graph ?? Representation + Infrastructure ? ? ?
- 25. 25 Space of design decisions: example from human trafficking Raw data Search+GUI Knowledge Graph (KG) Domain discovery Define KG schema Representation + Infrastructure Flexible inputs Query reformulation KG Construction
- 27. The Knowledge Graph is noisy…how do we cope?
- 28. Answer: Strategize around each triangle Search+GUI ConsumerProducer
- 29. Example from DIG: consumer triangle Search+GUI Consumer
- 31. 31 Anti-fragile query reformulation to satisfy user intent SELECT ?ad ?ethnicity WHERE { ?ad a :Ad ; :hair_color 'Auburn' ; :review_site_id 'cg9469f' ; :price_per_hour '500' ; :name ’Claire Gold’ ; :ethnicity ?ethnicity . } query 1 query 2 query 3 query 4 query n Query Reformulation Keyword expansion • Context broadening • Constraint relaxation Precision Recall Elastic Search 100M entities Ranked Candidates
- 33. 33 Infrastructure: Leverage existing ecosystems (there are many!)
- 35. 35 Prosecutions
- 37. Showcasing THOR
- 38. 38 Other domains Narcotics Illegal weapons sales Fraudulent shipments Securities fraud Causal exploration Geopolitical forecasting Cyberattack prediction
- 41. BACKUP
- 42. THOR: Text-enabled Humanitarian Operations in Real-time
- 45. 45 Controlled (i.e. academic measurements) 0 10 20 30 40 50 60 70 80 90 100 0 - 0.1 < 0.2 < 0.3 < 0.4 < 0.5 < 0.6 < 0.7 < 0.8 < 0.9 <= 1.0 Average Precision of Retrieved Pages DARPA MEMEX Eval (90K pages) Point Fact Cluster ID Aggregate Facet %Questions Average Precision
- 46. 46 In-use impact (sex trafficking) 100 million+ escort ads 3 years data coverage 2 billion triples 100 law enforcement offices 3 convictions
- 47. 47 NY County District Attorney (HTRU) MEMEX tools getting rolled out
- 48. 48 Memex tools getting rolled out
- 49. 49 Academic Output ~15 publications over the course of the program • 7 more currently under review • 2 best paper awards • Upcoming special issue call on knowledge construction and management • 2 upcoming books, incl. graduate-level textbook on knowledge graphs (MIT Press, 2018) Multiple tutorials/demonstrations at top-tier academic conferences • Tutorials on knowledge graph construction and data mining over Web corpora/unusual domains in KDD17, ISWC17, AAAI18, WWW18 • At ISWC17, only full-day tutorial accepted; had near-capacity attendance • Demos at ISWC17, AAAI18 (nominated for Best Demo) • Case study at CHI18 Selected papers • Knowledge Graphs for Social Good: An Entity-centric Search Engine for the Human Trafficking Domain (IEEE Transactions on Big Data, 2017) • Information Extraction in Illicit Domains (WWW, 2017) • Unsupervised Entity Resolution on Multi-type Graphs (ISWC 2016)
- 50. 50 Broker Rich club effectStar cluster Web formation Social Science Studies
- 51. 51 • Subjective issue • Architecture-level evaluation • Ablation analysis “Ideal” Evaluation
- 53. 53 Space of design decisions Raw data Search+GUI ? ?? ? ?
- 55. 55 Structured query execution on noisy data SELECT ?ad ?ethnicity WHERE { ?ad a :Ad ; :hair_color 'Auburn' ; :review_site_id 'cg9469f' ; :price_per_hour '500' ; :name ’Claire Gold’ ; :ethnicity ?ethnicity . } query 1 query 2 query 3 query 4 query n Query Reformulation Keyword expansion • Context broadening • Constraint relaxation Precision Recall Elastic Search 100M entities Ranked Candidates
- 56. 56 DIG capabilities Aggregations Facets Dossier Generation Networks Provenance Structured Queries Interface Customization • Capabilities that generic search engines like Google do not currently support • Domain-specific –Allows a user to specify her schema –No prior constraints • Insight –Supports aggregations, network analysis, faceted search, dossiers... • Graph –Uses a knowledge graph representation + efficient NoSQL query reformulation
- 57. Users want Situational Awareness i.e. equipped with actionable insights
- 58. • Advanced name matching algorithm based on machine learning, phonetic similarity and illicit webpage-specific word embeddings 58 How can we tell when two actors are really one and the same? Abbie Candy Kim Lea Nicki Abby Kandy Kimmy Leah Nikki
- 59. • Evaluated on five investigative domains beyond human trafficking, each with its own domain-specific needs –Narcotics –Counterfeit Electronics Manufacturing –Securities Fraud –Mail Shipment Fraud –Illegal Weapons Sales • User engagement was high –Investigators were able to customize their domain in just one day, with less than an hour of training –Have expressed interest in continuing to refine and use the search engine internally 59 Other use-cases
- 60. Relevance score Matching search criteria highlighted Image extraction+face and pose analytics using deep learning Original URL
Editor's Notes
- #3: Beautify this slide further
- #4: people
- #5: people
- #10: Current state
- #11: Current state
- #12: Current state
- #13: THOR
- #19: people
- #20: Modify slide
- #21: Modify slide
- #24: Modify slide
- #26: Modify slide
- #27: people
- #29: Modify slide
- #30: Modify slide
- #35: people
- #41: people
- #43: Current state
- #44: Current state
- #45: people
- #53: people
- #54: Modify slide