AIISC Projects via Posters at 2022 Retreat.pptx

# Title of the poster
1 Generalized Automated Planning
2 Extrapolation Error in Offline RL
3 Machine Learning for Predicting the Likelihood of Autism Spectrum Disorder from Infant ECG Recordings
4 Characterization of Neutrophils in Lupus
5 Rating of AI Systems Through a Causal Lens
6 Inductive Logic Programming for Explainable Artificial Intelligence
7 Evaluating Student-generated Analogies
8 Knowledge Graphs Construction & Alignment At Scale
9 Knowledge-Infused Self Attention Transformer
10 Group Recommendation and a Case Study in Team Formation with ULTRA
11 Sensor Data Based Insight Extraction
12 Building a Digital Twin for Information Environment
13 Can I Eat this Food or Not? Knowledge-Infused Explainable Food Recommendation System
14 Neuro-symbolic AI for Scene Understanding in Autonomous Driving
15 Aging Brain Cohort - AI: Cognitive Decline Detection
16 Connect the Goal and the State Using Language For an RL agent
17 Convergence Guarantees of the Softmax Policy Gradient in Markov Potential Games
18 Predicting Picture Naming task from the Brain Damages Measured with Structural MRI
19 IMAGINATOR - Pre-trained Image+Text Joint Embeddings
20 Explainable AI to Learn Puzzles for Collaborative Education
21 Towards Rare Event Prediction in Manufacturing Domain
22 CausalKG: Causal Knowledge Graph for explainability using interventional and counterfactual reasoning
23 CPR: Causal Process Knowledge-Infused Reasoning
24 Link to flyer on DECEPTION

Research Poster:
Generalized Automated Planning
Vishal Pallagani*, Bharath Chandra*, Biplav Srivastava*, Francesca Rossi^, Keerthiram Murugesan^, Lior
Horesh^, Francesco Fabiano$, Andrea Loreggia&
*University of South Carolina, ^IBM Research, $University of Udine, &University of Brescia
What is Automated Planning?
• Automated planning is a fundamental area of AI, concerned
with computing behaviors which when executed in an initial
state realize the goals and objectives of the agent. Refer to
Figure 1 to get an idea of how NASA is making use of AI
Planning techniques.
• Nevertheless, industrial-level scalability remains a
fundamental challenge to the broad applicability of AI
automated planning techniques. This is especially notable
when the space of objects is (possibly) infinite or when there
is inherent uncertainty about the initial plan parameters.
What is the aim of Generalized Planning?
Generalized planning studies the representation, computation,
and evaluation of solutions that are valid for multiple
planning instances. With recent traction in the research
community toward generalized planning, the following core
challenges have been identified:
• achieving scalability through plans that include cyclic flow of
control and solve large classes of problems (e.g., solving
nxn rubik's cube)
• acquisition (through learning or search) of domain control
knowledge for reducing the cost of planning
• synthesis of program-like structures from partial programs or
goal-specifications.
Proposed Approach
In order to realize the various identified challenges of
approaches, we are making use of Natural Language
Processing (NLP) techniques combined with Knowledge
Representation Ontologies.
Figure 1. ISAAC: Integrated System for Autonomous and
Adaptive Caretaking Credits: NASA
Motivating Research Questions
• How can we effectively find, represent and utilize high-
level knowledge about planning domains?
• How can we create a single planner that can
assimilate and solve multiple planning domains of
varying types?
• What separates planning problems from program
synthesis?
• How can abstraction techniques for understanding,
analyzing and reasoning about programs be utilized for
generalized planning?
• How can we restore the explainability aspect of
current Automated Planning techniques while
achieving generalization?
Planning + NLP
We explore the use of LLMs for automated planning. We
introduce Plansformer; an LLM fine-tuned on planning
problems and capable of generating plans with favorable
behavior in terms of correctness and length with minimal
knowledge-engineering efforts. Plansformer is capable of
adapting to solving different planning domains with varying
complexities, owing to the transfer learning abilities of LLMs.
Plansformer achieves ~97% valid plans, out of which ~95% are
optimal for Towers of Hanoi. Refer to Figure 2,3.
Planning + Ontology*
We aimed to consolidate all the available information relevant
to the Planning community in the form a Planning Knowledge
Graph (PlanKG). This effort is in line to effectively find, and
utilize knowledge about planning domains, and further infuse it
in Plansformer to perform Knowledge-infused Planning.
Refer to Figure 3.
*presenting the work led by Bharath Chandra
Planning @ AIISC
People: Forest Agostinelli, Qi Zhang
Discussions: Planning and RL Tech Discussions (Spring 22),
Language Models and relevance to Planning (Fall 21)
Figure 3. Plansformer
Architecture (left), PlanKG (right)
Figure 2. Comparison of various language models with
Plansformer in their ability to generate a plan for a
blocksworld problem instance

RESEARCH POSTER PRESENTATION DESIGN © 2019
www.PosterPresentations.com
• There are differences in HRV between infants classified
as EL and LL. This information implies that ECG could
be a biomarker for ASD.
•We performed individual group comparisons for each
HRV measures as shown in Table 1.
•Fig 3 Shows all machine learning models we evaluated
with the metrics. Random Forrest and Decision Tree
performed best among all the metrics.
• Our results compares favourably with the results from [2],
which did not report any other metrics then accuracy. They
used an ensemble classifier with 75% accuracy for valence
recognition in ASD vs. typically developed using HRV.
RESULTS
Table 1: Group wise mean difference of HRV features
Create a pipeline that focuses on cardiac activity (ECG)
during object/parent interaction experiments and uses time-
series analysis and machine learning to
a. pre-process,
b. analyze, and
c. predict the likelihood of ASD at young (3-6 mo) age.
Impact: Reduce waiting times for access to therapy and
treatment and to reach underserved populations better.
ABSTRACT
METHODS
• Our approach combines multiple HRV measures and
machine learning to identify a composite HRV metric
with identification of participants at very young age
with high likelihood or low likelihood with ASD
• We discovered HRV measures that are useful in
characterizing ANS patterns associated with ASD
• EL infants demonstrated increased mean, median,
pNN20, HTI, CVNN, sd1sd2, CVI and max inter beat
interval compared to LL infants.
• These findings suggest that EL infants demonstrate
lower-paced breathing and a lower heart rate.
• Results with such a high accuracy in classifying LL and
EL demonstrate promise for using wearable, wireless,
and user-friendly ECG devices in health settings.
• This preliminary study shows the potential of ECG as
biomarkers in ASD.
DISCUSSION CONCLUSIONS
• This study concludes that these HRV measurements as
potential features for specific changes to ANS function
associated with ASD.
• We plan to further study if HRV can help to classify
severity of symptoms in EL group.
REFRENCES
1. Y.-C. Cheng,et al. Heart rate variability in
individuals with autism spectrum disorders: a meta-
analysis. Neurosci. Biobehav Rev, 118, 08 2020.
2. M. Frasch, et al Can a composite heart rate
variability biomarker shed new insights about autism
spectrum disorder in school-age children? J Autism
Dev Disord, 51, 05 2020.
3. Makowski,et al (2021). NeuroKit2: A Python
toolbox for neurophysiological signal processing.
Behavior Research Methods, 53(4)
4. M. Maenner, et al. Prevalence and characteristics of
autism spectrum disorder among children aged 8 years
- autism and developmental disabilities monitoring
network, 11 sites, united states, 2018. MMWR
Surveillance Summaries, 70(11):1–16, Dec. 2021.
5. S. Laborde, et al. Thayer. Heart rate variability and
cardiac vagal tone in psychophysiological research –
recommendations for experiment planning, data
analysis, and data reporting. Frontiers in
Psychology, 8:213, 2017
Data Collection
Fig 1: Data collection pipeline
Data Pre-processing:
NN intervals were inspected for possible artifacts (e.g., missed or
spurious heartbeats).
• Segments which having length of >200 clean beats are considered.
• We interpolated missing beats and further rejected the segments
where interpolation % is >25% from the length of actual beats.
• We used NN interval time series from N = 41 participants to
compute HRV measures and build classification models.
• We extracted HRV features using Neurokit23.
Heart Rate Variability:
We selected a set of linear and nonlinear HRV1 measures for machine
learning models after doing an ablation study.
Classification using Machine Learning Models:
• Our approach focuses on raw ECG to predict the familial outcome
for ASD as EL vs. LL We calculated the mean across each extracted
feature for allowing subject-wise classification.
• We used the stratified 5-fold cross validation technique to ensure
that training and testing have both classes and it is adequately
evaluated.
Machine Learning for Predicting the Likelihood of Autism Spectrum Disorder from Infant ECG Recordings
• ASD is difficult to diagnose until 3 years of life.
• Prevalence is increasing, 1 out of 444 children.
• Behavioral assessments and other tests are very time
consuming.
• No standardized assessment tools available to identify
individual strengths, weaknesses, and needs.
• Understanding behavioral and neurological
developments at early age is becoming crucial.
INTRODUCTION
LIMITATION
ACRONYMS
ASD: Autism Spectrum Disorder
CVI: Cardiovagal Index
CVNN: Coefficient of variation of normal beat intervals
CSI: Cardiac Sympathetic Index
DT: Decision tree
ECG : Electrocardiograms
ET : Extra Tree
EL : Elevated Likelihood
fMRI: Functional Magnetic Resonance Imaging
GB: Gradient Boosting
HRV : Heart Rate Variability K
HTI: Triangular index of HRV
NN: K Nearest Neighbour
LL: Low Likelihood
LogReg: Logistic Regression
NN: Normal Beat Intervals
pNN: Proportion of successive normal beat intervals
RF: Random Forest
SD: Standard Deviation
SGD: Stochastic Gradient Descent
XGB: Extra Gradient Boosting
Labels LL EL
MeanNN 425± 31 429±23
MedianNN 422±29 425±23
pnn20 8.35±7 10.84±6
sd1sd2 0.30±0.15 0.31±0.14
CvNN 0.06±0.03 0.08±0.05
HTI 6.93±2 8.02±1
CSI 4.04±1 3.66±1
CVI 3.72±0.4 3.95±0.4
MaxNN 607±201 659±230
MinNN 339±57 323±93
Fig 3: Classification performance of Machine Learning models on HRV
Deepa Tilwani1 , Christian O’Reilly1, Jessica Bradshaw2, Amit Sheth1
1) AI Institue UofSC, 2) IMB UofSC

Vandana Srivastava1, Dr. Gustaf Wigerblad2,Dr Mariana Kaplan2
1. PhD student, AI Institute, University of South Carolina 2. National Institute of Arthritis and Musculoskeletal & Skin Diseases
Introduction
• Systemic lupus erythematosus (SLE), is an
autoimmune disease
• Neutrophils are a type of white blood cell, the most
abundant (50–75%) leukocyte in human blood, first
responders in the innate immune system
Clusters in Healthy vs Lupus Patients Based on Similarity of Gene
Expression
Fig 1 &2: Neutrophil Markers in Healthy (“No”) vs Lupus Patients (‘Yes”) Fig 5: Gene Ontology for Top 100 Diff Marker Genes
Characterization of Neutrophils in Lupus
Motivation
• Studies have established the roles of the innate immune
system in the initiation and propagation of
autoimmunity and the development of organ damage in
SLE [1]
• Neutrophils are an important and most found leukocyte
in the innate immune system
Data and methods
• Purified neutrophil single-cell RNA sequence (one cell of
mRNA) of 7 healthy controls and 7 lupus patients (3
females, 4 males)
• Healthy dataset: 20,004 genes, 90,956 cells
• Lupus dataset: 26,257 genes, 117,136 cells
• Analysis software SEURAT in R for single-cell sequences
• Clustering (KNN)
• Differential expression (Wilcoxon rank sum test)
• Dimension reduction (PCA, UMAP)
Significant change in the expression of neutrophil markers in lupus patients. Gene CSF3HR, S100A8,
FCGR3B, and CXCR2 are less expressed in lupus
Advisor: Dr. Christian O’ Reilly, AI Institute, University of South Carolina
Fig 3 & 4: Gene Expression Change Across Different Clusters in Lupus Patients
Results
• 15 clusters, 1174 differential expression marker genes
• Expression shift in neutrophil markers genes (Fig 1&2)
• GO terms indicate toxicity in cells, degranulation (losing
important constituents- genes LYZ,EEF1A1 in cluster
2/8), and neutrophil activation (Fig 5)
• Increased expression of IFN (interferon)-related genes
(IFIT1, IFITM3, ISG15 in cluster 4/7) (Fig 4)
Left: The size of the dot encodes the percentage of cells within a class, while the color encodes the average expression level
across all cells within a class (blue is high). In cluster 2, the DE genes LYZ, VCAN, EEF1A1, RPL23A are heavily expressed;
in cluster 3/4, EPSTI1 and ZCCHC2;in cluster 6, 8-10, RPS18/19 and RPL10; in cluster 8-10, EEF1A1, RPL23A, in cluster
10,CD74, and HLA-DRA are upregulated
Right: Single-cell heatmap of the gene expression within clusters
Conclusion
• Shift in the gene expression in lupus
• Increased expression of IFN (interferon)-related genes,
confirming that lupus is an IFN-derived disease
• General degranulation and activation of neutrophils
Acknowledgements / References
• Dr. Mariana Kaplan’s lab, NIAMS
• https://maayanlab.cloud/Enrichr/ -- for GO terms
• [1] Bite of the wolf: innate immune responses propagate
autoimmunity in lupus, Sarthak Gupta, Mariana J. Kaplan, J Clin
Invest. 2021;131(3):e144918. https://doi.org/10.1172/JCI144918
• [2] Wigerblad et al, Single-cell analysis reveals the range of
transcriptional states of circulating human neutrophils, bioRxiv
preprint doi: https://doi.org/10.1101/2022.02.22.481522
Significance / Future Work
• Understanding the potential role of neutrophil
dysregulation in autoimmune diseases can lead to
developing targeted therapeutic solutions [1]
• Machine learning models can be developed for the
classification of cells as lupus or healthy

Kausik Lakkaraju
Mentors: Dr. Biplav Srivastava, Dr. Marco Valtorta
Undesirable
Attributes
Desirable
Attributes
AI System
Outcome
• Causal models allow us to define the cause-effect
relationships between each of the attributes in a system.
• Each node represents an attribute, and the arrowhead
direction shows the causal direction from cause to effect.
• In the above diagram, the red color arrows represent
undesirable paths and green arrow represents desirable path.
• The ‘?’ indicates that we test the validity of these causal links
using appropriate statistical tests like t-test along with some
causality-based techniques like backdoor adjustment.
• Based on the validity, we assign a rating to the AI system.
Causal analysis allows us to answer the question of ‘why’ and
rating tells us ‘how’ biased a system is.
? ?
?
• In [1], the authors rated automated machine language
translators for gender bias.
• In [2], the authors proposed a personalized rating
methodology for chatbots.
• However, in these works, purely statistical methods were
used to calculate the rating.
• I recently presented a student paper on ‘Rating of AI
Systems through a Causal Lens’ [3] at the AIES 2022
conference which was held in Oxford, UK.
• Most of the existing Machine Learning models are black-
box and they only learn the correlations between different
attributes but not the causal relations.
• Many of the existing statistical fairness definitions used to
evaluate different AI systems for bias are proved to be
insufficient a lot of times in the past.
START
Gender
and
Race?
Confounder
?
Create
Group-4
Compute the
Weighted Rejection
Score (WRS)
Confounder
?
Create
Group-2
Create
Group-1
Create
Group-3
Compute the
Deconfounding
Impact Estimation
(DIE) score
Compute the partial
order
Compute rating based
on rating level, L.
STOP
• We built a multi-modal, explainable chatbot
called ALLURE [4] that teaches students how
to solve a Rubik’s cube.
• One of our future works is to make the
conversation of this chatbot free from any
abusive language or hate speech.
• We are working on rating object recognition
systems using ALLURE as the setting.
Background
Prior Work: Rewind
The Quest for ‘Why’ The Curious Case of Sentiment Analysis Systems (SASs)
The Tale of German Credit Data through a Causal Lens Future Work: Forward
• How could we estimate the bias present in AI systems?
• Wouldn’t it be helpful if one could give a rating that can
have a real-world impact like legal accountability?
• This is the research problem I am working on as a part of my
Ph.D. dissertation.
Journey to the Center of the Problem
Slack channel for AI Ethics Discussion
On every Friday at 9AM EST, we meet virtually to discuss various
research papers in the ‘AI Ethics’ domain.
Topics include but not limited to: Bias in AI systems, Model
uncertainty, Testing / RCTs, Adversarial attacks, Data privacy.
Contact: kausik@email.sc.edu
Fig. 1: Generalized Causal Diagram
Fig. 2: Proposed Causal Diagram for SASs
Fig. 1: Proposed Causal Diagram for German Credit Dataset
References:
1. Biplav Srivastava and Francesca Rossi. 2020. Rating AI Systems for Bias to Promote
Trustable Applications. In IBM Journal of Research and Development.
2. Biplav Srivastava, Francesca Rossi, Sheema Usmani, and Mariana Bernagozzi. 2020.
Personalized Chatbot Trustworthiness Ratings. In IEEE Transactions on Technology
and Society.
3.Kausik Lakkaraju. 2022. Why is my System Biased?: Rating of AI Systems through a Causal Lens. In Proceedings of the 2022
AAAI/ACM Conference on AI, Ethics, and Society (AIES '22). Association for Computing Machinery, New York, NY, USA, 902.
https://doi.org/10.1145/3514094.3539556
4.Lakkaraju, K., Hassan, T., Khandelwal, V., Singh, P., Bradley, C., Shah, R., Agostinelli, F., Srivastava, B., & Wu, D. (2022). ALLURE: A
Multi-Modal Guided Environment for Helping Children Learn to Solve a Rubik’s Cube with Automatic Solving and Interactive
Explanations. Proceedings of the AAAI Conference on Artificial Intelligence, 36(11), 13185-13187.
https://doi.org/10.1609/aaai.v36i11.21722
5.Judea Pearl and Dana Mackenzie. 2018. The Book of Why: The New Science of Cause
and Effect (1st ed.). Basic Books, Inc., USA.
6.https://en.wikipedia.org/wiki/Four_causes
Research Log: Journey so far
Number of papers: 1 ([3]); Number of manuscripts: 2;
Number of patents: 1
Rating of AI Systems through a Causal Lens

Inductive Logic Programming for
Explainable Artificial Intelligence
Vedant Khandelwal, Forest Agostinelli, Amit Sheth
University of South Carolina, Columbia
Inductive Logic Programming: An Introduction
• Induction forms general rules (hypothesis) from specific observations (training examples).
• Machine Learning automates Induction. It induces a hypothesis (model) that generalizes the
training examples (observations), based on given background knowledge (input features).
• Logic Programming corresponds to expressing any statement through logical facts and rules.
• Inductive Logic Programming is a form of ML that tries to induce hypotheses that generalize
specific training examples, where data is represented as a logic program.
Future Work: End-to-End Explainable System (Neuro-Symbolic Approach)
Extracting Explainable Solutions for Sliding Tile Puzzle
• Sliding Tile Puzzle – A challenge in which the player is allowed to
slide tile pieces in the blank space to attain a particular goal
configuration.
• The current example mentions about 8-puzzle, where the goal state
configuration is “1,2,3,4,5,6,7,8,0”. Goal State – 8-Puzzle
• Deep Learning Algorithms have solved various complex games such
as sliding tile puzzles and Rubik’s cubes but are not explainable.
• ILP can be used with these algorithms to obtain an explainable
solution to these problems.
desc(S) 🡨 goal(S)
desc(S) 🡨 inplace_until(S,A), inplace_clause(S,B), adjacent(S,C,D,E),
tile4(A), tile7(B), tile6(C), blank(D), left(E).
steps(S,G) 🡨 move(S,A,S1), move(S1,B,S2), move(S2,C,G), right(A),
up(B), left(C).
Real World Use Case: Mental Health
Input Data – She is currently completing a
taper of Fluoxetine and Ziprasidone for
depressive Symptoms. But the drugs are not
working. They are causing trouble with sleeping
and irregular heartbeats.
Relational Fact Extractor
External Knowledge Sources
Pos/Neg Examples Background File
Pos(symptom(Flouxetine,
Ziprasidone, irregular_heartbeat)).
Pos(symptom(Flouxetine,
Ziprasidone, trouble_sleeping)).
• symptom(A,B,C) 🡨 counteract(A,B), side_effect(A,C), drug(A), name(A,Flouxetine), drug(B),
name(B, Ziprasidone), condition(C), name(C, irregular_heartbeat).
• symptom(A,B,C) 🡨 counteract(A,B), map_sim(C,D), side_effect(A,D), drug(A),
name(A,Flouxetine), drug(B), name(B,Ziprasidone), phrase(C,trouble_sleeping), condition(D),
name(C,restlessness).
Background Knowledge (BK)– These are a set of input
logical statements that are True. All the other features not
present in this are considered False by default.
Training Examples – These are the set
of observations over which a hypothesis
generalizes.
Drawbacks of Logic Programming–
• It does not work with ambiguous inputs.
• Difficult to scale.
“Inductive Inference is the only process known to us by which
essentially new knowledge comes into the world”
– Ronald Fisher, British Polymath
Input Features: (BK)
parent(a,b). parent(a,c).
parent(d,b). male(a).
female(c).
female(d).
Observations: (Pos/Neg)
father(a,b).
father(a,c).
father(d,b).
Observations: (Pos/Neg)
mother(d,b).
mother(a,b).
mother(a,c).
father(X,Y) 🡨 parent(X,Y), male(X). mother(X,Y) 🡨 parent(X,Y), female(X).
Ambiguous Input Sample:
Prob(2):68%, prob(3):32%,
Prob(6):29%.
Learning Explanatory Rules from Noisy Data
Neuro Symbolic
Data Efficient Not Always Yes
Interpretable No Yes
Robust
Mislabeled Data
Yes Not Very
Robust
Ambiguous Data
Yes No
Generalizability Not Always Yes
Hypotheses
Hypothesis – It is the induced logical rule which can be further used on unseen examples.

Biomedical Vocabularies Alignment at Scale in the UMLS
Metathesaurus [9-10]
The Unified Medical Language System (UMLS) Metathesaurus is a
biomedical terminology integration system developed by the US
National Library of Medicine to integrate biomedical terms by
organizing clusters of synonymous terms into concepts. With 214
source vocabularies, the current construction and maintenance
process of the UMLS Metathesaurus heavily relies on lexical
similarity algorithms and arduous human curation to identify
candidates for synonymy. We developed a novel supervised
learning approach based on lexical information from the source
vocabularies for improving the task of suggesting synonymous
pairs that can scale to the size and diversity of the UMLS source
vocabularies and demonstrated the strong performance in terms of
recall (91- 92%), precision (88-99%), and F1 score (89-95%) [9].
We also investigated the role of multiple types of contextual
information available to the UMLS editors, namely source
synonymy (SS), source semantic group (SG), and source
hierarchical relations (HR), represented in a graphical form and
vectorized via popular graph embedding algorithms for the
alignment task. The results showed a significant performance
improvement, +5.0% in precision (93.75%), +0.69% in recall
(93.23%), +2.88% in F1 (93.49%) over [9], demonstrating the
importance of contextualization (Figure 5) [10].
Graph Embeddings and Reconstruction
Graph embeddings aim at learning graphical representations in a
low dimensional vector space. While approaches in [10] have
shown promising results for the downstream alignment task, [11]
has found existing (state of the art) graph embedding approaches
can only preserve up to 62% (graph and dataset dependent) of the
original graph topological structure. There is a need to go beyond
the applicability of these approaches and investigate the underlying
graph reconstruction process (for explainability), especially when
aligning and integrating graphs originating from different
distributions. As an ongoing work, we are proposing a Metagraph
Learning framework to (a) learn graphs from different projection
spaces, then (b) encode the different graph representations, and (c)
evaluate the (reconstruction) information entropy as well as (d) the
applicability in downstream alignment, question answering, and
text summarization tasks.
Knowledge Graphs Construction & Alignment At Scale
Hong Yung (Joey) Yip1, Amitava Das1, and Amit Sheth1
1. Artificial Intelligence Institute, University of South Carolina
PROPOSED SYSTEM & ARCHITECTURE
Knowledge Graph (KG) is an encapsulation of structured knowledge in a graphical representation and is used
for a variety of information processing and management tasks such as
● Improving automation
● Data & knowledge integration from diverse sources
● Empowering new generation of Artificial Intelligence (AI) applications with domain knowledge*
*Motivating example (Figure 1): Is 150 a normal blood pressure? To answer this question is not a
straightforward task and require additional knowledge (other than what is provided in text) and context before
deducing an answer.
KGs are increasingly used in a wide array of dynamic, high-volume, and real-time domains such as healthcare,
enterprise, and applications ranging from search engines, question answering, summarization, text
simplification to chatbots and recommender systems. These applications often draw knowledge from
interdisciplinary domains and operate under environments with siloed, incomplete, and heterogeneous
knowledge. To enable intelligent systems in these domains, there is a need for a Knowledge Model and
platform that supports multi-faceted KG aspects relating to contextualization, temporal, personalization, and
large-scale knowledge constructions, integrations as well as expansions.
INTRODUCTION
LITERATURE REVIEW
There exists tools and platforms that support construction of KGs for various domains such as [1] which
explored an automated process using logistic regression, naive Bayes classifier and a Bayesian network to
learn high quality knowledge bases directly from electronic medical records; [2] which generated an AI-KG
using a combination of deep learning and semantic technologies; [3] which proposed T2KG, an automatic KG
creation framework for natural language text using a hybrid combination of a rule-based approach and a
similarity-based approach; [4] which constructed a disease-centric knowledge graph from multiple large
knowledge sources; [5] which constructed a PubMed knowledge graph (PKG) by extracting and integrating
bio-entities from 29 million PubMed abstracts; [6] which studied and evaluated a semi-automatic method
consisting of a multi-step pipeline that generates knowledge graphs (KGs) from biomedical texts in the
scientific literature; and [7] which proposed a programmable and integrated big knowledge graph platform
supporting the construction, management, and operation of large to massive scale knowledge graphs.
However, these existing tools and platforms lack [8]:
● An open architecture that supports ingestion and learning of data from heterogeneous and distributed
sources
● Dynamicity: Dynamic schema generations to reflect real world information updates and changes as they
occur
● Provenance: Ability to link disparate information by traversing links across the network while preserving
the originating knowledge source
● Temporal: Ability to reflect knowledge that are time-bounded in nature
● Ability to deduce linkages among entities
The AIISC KG effort involves the development of a comprehensive tool and platform (Figure 2, 3, and 4) for
KG construction and alignment with the objective to (a) empower data with knowledge based on a
combination of bottom-up data-driven statistical-learning based approaches, and top-down declarative
approaches and (b) to improve and address the limitations of existing KG platforms.
Figure 2: Stages of KG construction
Figure 1: Semantic data annotation
Figure 3: AIISC KG platform architecture
Figure 4: A walkthrough of KG construction from text
*Enrichment: Increase quality and applicability of the graph by entity disambiguation, by aligning
and mapping the schema and data to existing ontologies and graph, and other measures that increase
data quality and can make integration decisions explicit, reusable, and reversible.
[1] Rotmensch, M., Halpern, Y., Tlimat, A., Horng, S., & Sontag, D. (2017). Learning a health knowledge graph from electronic medical records. Scientific reports, 7(1), 1-11.
[2] Dessì, D., Osborne, F., Reforgiato Recupero, D., Buscaldi, D., Motta, E., & Sack, H. (2020, November). Ai-kg: an automatically generated knowledge graph of artificial intelligence. In International Semantic Web Conference
(pp. 127-143). Springer, Cham.
[3] Kertkeidkachorn, N., & Ichise, R. (2018). An automatic knowledge graph creation framework from natural language text. IEICE TRANSACTIONS on Information and Systems, 101(1), 90-98.
[4] Huang, Z., Yang, J., Harmelen, F. V., & Hu, Q. (2017, October). Constructing knowledge graphs of depression. In International conference on health information science (pp. 149-161). Springer, Cham.
[5] Xu, J., Kim, S., Song, M., Jeong, M., Kim, D., Kang, J., ... & Ding, Y. (2020). Building a PubMed knowledge graph. Scientific data, 7(1), 1-15.
[6] Rossanez, A., Dos Reis, J. C., Torres, R. D. S., & de Ribaupierre, H. (2020). KGen: a knowledge graph generator from biomedical scientific literature. BMC medical informatics and decision making, 20(4), 1-24.
[7] Ruqian, L. U., Chaoqun, F. E. I., Chuanqing, W. A. N. G., Shunfeng, G. A. O., Han, Q. I. U., Zhang, S., & Cungen, C. A. O. (2020). HAPE: A programmable big knowledge graph platform. Information Sciences, 509, 87-103.
[8] https://beta.nsf.gov/funding/opportunities/encouraging-research-open-knowledge-networks
[9[ Nguyen, V., Yip, H. Y., & Bodenreider, O. (2021, April). Biomedical vocabulary alignment at scale in the UMLS metathesaurus. In Proceedings of the Web Conference 2021 (pp. 2672-2683).
[10] Nguyen, V., Yip, H. Y., Bajaj, G., Wijesiriwardene, T., Javangula, V., Parthasarathy, S., ... & Bodenreider, O. (2022, April). Context-Enriched Learning Models for Aligning Biomedical Vocabularies at Scale in the UMLS
Metathesaurus. In Proceedings of the ACM Web Conference 2022 (pp. 1037-1046).
[11] Liu, X., Zhuang, C., Murata, T., Kim, K. S., & Kertkeidkachorn, N. (2019). How much topological structure is preserved by graph embeddings?. Computer Science and Information Systems, 16(2), 597-614.
REFERENCES
RESEARCH
Figure 5: Concepts alignment and disambiguation [9-10]

Knowledge Infused Self Attention Transformer
Yuxin Zi
Artificial Intelligence Institute South Carolina
Highlights
● Encourages graph topology
● Trade-off between data and graph context
● Easily explained to the end-user
Ongoing Research Based on KSAT
■ Find mathematically motivated architectural
components for KSAT (heuristic-based at the
moment);
■ Exploring the use of knowledge beyond the bias term
(e.g. regularization term, or a combination of both) to
control the learnable function classes;
■ Using KSAT for Knowledge Intensive Language
Tasks (KILT): e.g. question-answering; Quora
question pair similarity.
Applications
■ Language modeling (multiple task-specific
contexts:linguistic/common-sense/broad-
base/domain-specific…);
■ Mental health applications: (multiple domain-specific
contexts: suicidality, depression, anxiety…)
■ Cooking/nutrition; autonomous driving

Group Recommendation and a Case Study in Team Formation with ULTRA
Biplav Srivastava, Sai Teja Paladi, Siva Likitha Valluru, Aniket Gupta2
(Former members: Tarmo Koppel1, Rohit Sharma, Owen Bond, Ronak Shah, Austin Hetherington)
University of South Carolina, 1Tallinn University of Technology, 2Intern
Introduction – Group Recommendation
Background:
• This is an instance of the general problem of building
teams where demand opportunities come only periodically
and potential members that best fit the criteria and
requirements may vary over time.
Sample use cases:
• Crowdfunding: recommending projects and ventures for
investors by accounting for each project's current progress
and status, future plans, preferences of individual
members, and collective preferences of the group.
• Balanced meal plans for diabetics: recommend meal items
(appetizer, main dish, dessert) based on, e.g., dieticians’
and doctors’ recommendations, calorie count, added
sugars/fat grams count, portion sizes, and types of dishes
(e.g., vegetables, dairy, poultry, and other meat).
Team Recommendation
Background:
• Existing literature has encouraged the use of greedy methods, genetic and heuristic algorithms, topic diversification, and cost constraint bi-
objective optimizations. The constraints we consider in our work are team building cost, overall project budget, and time allocated to finish the
project.
• Funding agencies issue Requests For Proposals (RFPs) for themes they are looking to fund ideas for. Researchers in turn respond to RFPs with
innovative ideas and list the types of activities and research that will be conducted during the assigned period of work. For maximized efficiency,
they also look to team with their colleagues to respond to such calls.
• An optimized team building method is therefore necessary to make the best use of funding resources and ensure high chance of success. The
image below shows the architecture for ULTRA, where users can be either admins or candidate team members.
Sample use cases:
• Forming negotiation teams in hostage crises
http://casy.cse.sc.edu/mapper/
Status of Project
• This project is currently in its prototype stages. Regarding the
tools being used to build this system, we have 3 papers and 1
patent in review.
• We built two tools so far that assist with this problem: (1) KITE
(bottom right), an unsupervised system that draws domain-
specific insights (e.g., person, politics) from unstructured data,
and (2) a text-to-classification mapper, a tool that displays
classification codes (for articles, dissertations, books, etc.)
from ACM and JEL guides.
• Sending relief teams during natural disasters
• Forming and selecting sports teams based on
criteria such as individual skill level, brand value,
success rate in past performances, physical fitness,
psychological factors, and overall team budget.
• Building surgical teams with factors, such as best
skill levels, experience, and past operation success
rates, to maximize patient’s chance of survival and
recovery.
• Having optimized supply chain management team,
using individual agent performance and aggregate
performance of other agents.
Funders: University of South Carolina and SCRA,
Papers: 2, Patent: 1

Sensor Data Based Insight Extraction
Bharath Muppasani 1, Kausik Lakkaraju 1, C.J. Anand 2, Chinmayi Appajigowda 2,
Biplav Srivastava 1, Lokesh Johri 2
AI Institute, University of South Carolina 1, Tantiv4 2
Overview
• Data Sources: Smart Routers, MiDAS IoT Sensor
• MiDAS Chatbot working with networking and power data
• Power Insights:
○ Power: Anomaly Detection and Forecasting
○ Power: State Identification
MiDAS Chatbot*
Publications
• “Data-Based Insights for the Masses: Scaling Natural
Language Querying to Middleware Data” - presented at
DASFAA-2022 in April, 2022.
• “A Dataset and Baseline Approach for Identifying Usage
States from Non-Intrusive Power Sensing With MiDAS IoT-
based Sensors” - submitted at IAAI-2023. Received
conditional acceptance.
• Made the power datasets for 8 different locations available
in public domain.
Power: State Identification Problem
• The objective of state identification (SIP) is to identify power
usage patterns of any system, like buildings or factories, of
interest. We consider it from the perspective of power
systems and as a data analysis problem.
Power: Data Collection
*work led by Kausik Lakkaraju
• The data used for this research is being collected by the
MiDAS IoT sensor installed by Tantiv4. The sensor is capable
of collecting electrical power data and harmonics data (32
harmonic levels) for an installed location with a frequency of
3.33Hz and 2Hz respectively.
• ARIMA based Anomaly Detection:
• LSTM based Power Forecasting:
Power: Anomaly Detection and Forecasting
• Customer support is vital for any business to retain its
customers, but it can be costly. So, companies often look for
decision-support technology, like chatbots, to improve
customer satisfaction.
Funding: Tantiv4; Papers: 2, Patent: 1

Building a Digital Twin for Information Environment
Sai Teja Paladi, Bharath Muppasani, Eric Cappuccio, Erik Austin, Murray, Peyton Tucker
Vignesh Narayanan, Michael Huhns, Biplav Srivastava
Research Objective
• Build digital twins(ICS) of specific populations to simulate, test
and monitor information campaigns in an information
environment
NetLogo and Ontology Integration
• NetLogo (multi-agent programmable modeling environment)
• NetLogo UI when we setup the environment
• NetLogo UI When we start the simulation - during a physical
event
● Scaling - Efforts to model and simulate one million agents
● Modeling topic dependencies -
○ Multiple related topics can influence how opinions of agents
evolve
○ Current models do not take this interaction into account
● Simulated agent population cannot accurately mimic an actual
human population
○ Our goal is to not replicate exact human behaviour but get to
a reasonable approximation of it - useful for understanding
how influence campaigns affect populations, how interest
and other groups emerge - working with domain experts to
improve and scale our agent models
Visualization
Agent Types and Setup
● Passive consumers - Basic agents
● Active consumers - Basic agents, Live players and
Spokesperson agents
● Content creators - Information dissemination agents,
Physical event agents
● Content manipulators - Flow manipulator agents
System Architecture
• Agents, topics, groups and IPs required for simulation are
generated using - Input csv files
• Creating the environment and starting the information spread -
NetLogo Logic
• State of the simulation environment is tracked at each tick and
saved in – Output log files
NetLogo GUI
Ongoing Work
Tools
● Agent-based modelling - Simulation model of Human –
Information Interactions
● ICS Ontology

CNN Food Recognition
model
Volume Estimator
(Computer Vision
techniques)
Food name
Food Volume
NUTRITION KG
Calorie Count
ANALYZER
REASONER
Personalized Health KG
- Glucose level
- Gut microbiome*
- Comorbidities*
DOMAIN KGs
Disease specific (Diabetes)
Vegetables, whole grains ----> Healthy
CHO
Sugar, white rice ----> Unhealthy CHO
ConceptNet
Apple, banana ----> Fruit
Pork, bacon ----> meat
Effects of cooking
Meat+Grilling -----> Carcinogenic agents
Fat based cooking -----> Trans-Fat
Can I eat this? YES
Healthy CHO: Carrot, Kale, Broccoli
Unhealthy CHO: None
Cooking method: Boiling
Your DV of CHO: within limit
Allergy: None
KNOWLEDGE GRAPHS
(explainability and personalization)
Rule based + ML/DL
TAGGER
(Cooking Methods)
Ingredients
Recipe
Frying, boiling, stir-
fry
2.3M recipes
900K food items
40+ major diets/allergens
150+ nutrients
Cooking
method KG
[1] Min, Weiqing, et al. "A survey on food computing." ACM Computing
Surveys (CSUR) 52.5 (2019): 1-36.
[2] Salvador, Amaia, et al. "Learning cross-modal embeddings for
cooking recipes and food images." Proceedings of the IEEE conference
on computer vision and pattern recognition. 2017.
Can I eat this? Yes
Healthy CHO: carrot, kale, broccoli, bean
Unhealthy CHO: None
Cooking method: Boiling
Your DV of CHO: Within limit
Allergy: None
Can I eat this? No
Healthy CHO: Potato
Unhealthy CHO: None
Cooking method: Frying
Allergy: None
Slow-Cooker Chicken & Bean Stew Potato Fries
Can I eat this? No
Healthy CHO: None
Unhealthy CHO: Added sugar
Cooking method: Baking
Allergy: Dairy
Cheesecake
It contains unhealthy CHO while other
factors are acceptable.
Cooking method is frying which introduces
unhealthy fat.
All good!
We would like extend our system to all chronic
conditions and also plan to incorporate estimation of
food volume through user input or from food images.
Further, we would like to conduct a clinical trial to
evaluate the suitability our system to patients.
Frying
Air
Frying
Oil
Frying
Stir
Frying
Deep
Frying
In the recent past, people have become more cautious
about their food choices due to its impact on their health
and chronic diseases. Due to this, there has been an
increase in various dietary assessment systems that
estimates the calorie intake of food items [1]. Of which, a
few of the systems also recommend meals to nudge
users towards healthy eating habits [1]. However, the
models do not provide the explanation for the food
choices made by the recommendation system. User’s
trust plays a critical role in determining the
appropriateness and acceptability of food
recommendation systems.
MOTIVATION
We fine-tune Stanford NER parser to extract cooking
actions from the recipe instructions. We utilize implicit
entity matching approaches to match the extracted
cooking actions to entities in our Cooking KG. After
analysing the effects of cooking methods and the type of
ingredients used utilizing various knowledge sources, we
evaluate whether a given recipe is suitable for a given
diet.
METHODS
We use Recipe1M [2] dataset for evaluation. In addition
to that, we use or curate knowledge graphs from various
sources such as ConceptNet, Edamam, WikiData,
MayoClinic and User Health Records.
DATASET FUTURE WORK REFERENCES
ACKNOWLEDGEMENTS: Edamam for Nutrition KG
In this work, we propose a food recommendation system
that employs an analyser to recommend whether the
food is advisable to the user and a reasonser that
provide an explanation for the decisions made by the
analyser. This work also incorporates a personalized
health knowledge graph to recommend meals based on
user’s health condition and food preferences.
PROPOSED SYSTEM
ARCHITECTURE
EXPECTED RESPONSES
CAN I EAT THIS FOOD OR NOT?
KNOWLEDGE INFUSED EXPLAINABLE FOOD RECOMMENDATION SYSTEM
Revathy Venkataramanan and Amit Sheth
revathy@email.sc.edu, amit@sc.edu
Artificial Intelligence Institute, University of South Carolina

Aging Brain Cohort - AI:
Cognitive Decline Detection
Usha Lokala1, Sarah E. Newman-Norlund2 , Sara Sayers2, Julius Fridriksson2, Amit Sheth1
1Artificial Intelligence Institute, 2Department of Communication Sciences, Arnold School of Public Health
ABC – AI : An Interdisciplinary Project
• Bridge between the Artificial Intelligence
Institute at the University of South Carolina
and the Aging Brain Cohort (ABC) at the
University of South Carolina.
• The study conducted at AIISC on the data
from the ABC study will help researchers
understand the relationship between various
biologic measures and age-related cognitive
decline.
• This work aims to analyze cognitive decline
or impairment characteristics among healthy
older adults.
• Analyze the language features recorded in
discourse transcripts
• Identify factors in the speech to text analysis
that contributed to their severity of cognitive
impairment.
• Determine the variables that contribute to
Aphasia recovery to understand how brain
works and support people with Aphasia and
stroke survivors.
• Visit us at: AIISC: https://aiisc.ai/
• ABC AI: https://wiki.aiisc.ai/index.php/ABC-AI
-AI

Predicting Picture Naming task from the brain damages measured with
Structural MRI
Raxit Goswami, Deepa Tilwani, Dr. Vignesh Narayanan, Dr. Christian O'Reilly, Dr. Rutvik Desai, Dr. Amit Sheth
Problem
Based on lesion damages in specific brain regions, we are attempting
to automate a multivariate technique that can predict the picture
naming task in patients with aphasia post-stroke.
Approach Results
• Brain Regions are connected with each other and form a
graph structure. We are planning to represent all the lesion
scores in graph format and apply Graph neural network to
predict PNT score
• Also, actual data (lesion score for different regions) are in 3D
images. So, we are planning to apply image processing
methods e.g., 3D CNN to predict PNT score.
Data
Sample Lesion Data : (P - MXXXXX)
WQ and PNT Score Data :
190 Patients for region wise damages and PNT Scores
About : Autoencoders are a special type of neural network
architecture in which the output is the same as the input.
Autoencoders are trained in an unsupervised manner in
order to learn the extremely low-level representations of the
input data. These low-level features are then deformed
back to project the actual data.
Use : Latent view represents the lowest level space in
which the inputs are reduced, and information is preserved.
These can be used for Dimensionality Reduction and
remove noise from the data.
Architecture :
1.Autoencoder
Autoencoder
References
Future Work
• Yourganov G, Smith KG, Fridriksson J, Rorden C. Predicting aphasia type from
brain damage measured with structural MRI. Cortex. 2015 Dec;73:203-15. doi:
10.1016/j.cortex.2015.09.005. Epub 2015 Sep 25. PMID: 26465238; PMCID:
PMC4689665.
• Kristinsson S, Zhang W, Rorden C, Newman-Norlund R, Basilakos A, Bonilha L,
Yourganov G, Xiao F, Hillis A, Fridriksson J. Machine learning-based multimodal
prediction of language outcomes in chronic aphasia. Hum Brain Mapp. 2021
Apr 15;42(6):1682-1698. doi: 10.1002/hbm.25321. Epub 2020 Dec 30. PMID:
33377592; PMCID: PMC7978124.
• Yourganov G, Fridriksson J, Rorden C, Gleichgerrcht E, Bonilha L. Multivariate
Connectome-Based Symptom Mapping in Post-Stroke Patients: Networks
Supporting Language and Speech. J Neurosci. 2016 Jun 22;36(25):6668-79. doi:
10.1523/JNEUROSCI.4396-15.2016. PMID: 27335399; PMCID: PMC4916245.
• Hanik, M., Demirtaş, M.A., Gharsallaoui, M.A. et al. Predicting cognitive scores
with graph neural networks through sample selection learning. Brain Imaging
and Behavior 16, 1123–1138 (2022). https://doi.org/10.1007/s11682-021-
00585-7
• BANSAL, SHIVAM. “How Autoencoders Work: Intro and UseCases.” Kaggle,
2018. https://www.kaggle.com/code/shivamb/how-autoencoders-work-intro-
and-usecases.
• Stark, Brielle & Rorden, Chris & Yourganov, Grigori & Hanayik, Taylor. (2017).
User Manual and Tutorial for NiiStat, a tool for the neuroimaging of behavior.
• Liu, S., Masurkar, A.V., Rusinek, H. et al. Generalizable deep learning model for
early Alzheimer’s disease detection from structural MRIs. Sci Rep 12, 17106
(2022). https://doi.org/10.1038/s41598-022-20674-x
# Regions Lesion Score
1 Superior frontal gyrus (posterior segment) 0.9329
2 Middle frontal gyrus (posterior segment) 0.3434
3 Inferior frontal gyrus pars opercularis 0.1839
4 Lateral fronto-orbital gyrus 0
5 Entorhinal Area 0.3497
… … …
… … …
… … …
86 Lenticular fasciculus 0
87 OlfactoryRadiation 0
88 Lateral ventricle_frontal 0
89 Posterior insula 0.0032
90 Posterior inferior temporal gyrus 0.0613
Patient PNT Score AQ Score
MXXXXX 77.5 47.3
Fig 2a. Autoencoder Architecture
Fig 2b. Autoencoder with various ML Regression methods
Fig 1. Brain Left, Right and Posterior Views

IMAGINATOR - Pre-trained Image+Text Joint Embeddings
Megha Chakraborty
University of South Carolina, Columbia
Contextuality in Joint Embeddings
Word embeddings are learned representations such that words with similar
meaning are represented similarly. Deriving such a representation over
images is a challenging task.
Contemporary Joint Embeddings Techniques
●CLIP (Radford, 2021) - Text transformer and visual transformer with
contrastive training.
●Stanford’s Joint Embedding (Kolluru, 2019) - VGG-19 and GLoVe along
with a triplet loss function.
Most related works, perform embedding generation on sentence level.
Our focus is on word level grounding of images to learn finer features.
Creating the Joint Embeddings
Intrinsic Evaluation Extrinsic Evaluation Conclusion
● The current model is limited to
only 21k object classes, we
intend to expand on this.
● Further, to draw meaningful
cross-modal connection
between sections of text and
parts of images, we are working
on implementing positional
encoding schemes in Visual
Transformer
The motivation is to create JEs that can represent such
real world analogies, and such distributional
semantics can aid in solving several multimodal tasks.
Using Detic (Zhou, 2022) dataset that has 21k object classes
detected for our training, we capture three aspects of the input
data while generating joint embeddings:
● object-object collocation - voo
● word-object collocation - vwo
● word-object correlation - vwor
The vectors obtained are merged by taking a weighted
average having weights 0.1, 0.1, 0.8 respectively for voo, vwo,
vwor to get the final joint embedding representation.
Text-based
similarity
Image-based similarity
Results (average pairwise euclidean
distance) for intrinsic evaluation of our
JEs based on object similarity.
Results (average euclidean distance) for intrinsic valuation of our
JEs based on word analogy.
Image Captioning using BLEU@4 Image2Tweet task on CIDEr metric

Explainable AI to Learn Puzzles for Collaborative Education
Forest Agostinelli1, Rojina Panta1, Vedant Khandelwal1, Biplav Srivastava1, Kausik Lakkaraju1, Dezhi Wu2
1. AI Institute, University of South Carolina, Columbia, South Carolina, USA
2. Department of Integrated Information Technology, University of South Carolina, Columbia, South Carolina, USA
Motivation Why use puzzles ?
Shortcomings of Current AI ?
❖ Provide simplified explanation to solution provided by
Artificial Intelligence(AI) system.
❖ To enable collaboration of Artificial Intelligence (AI) with
human to solve problems in robotics, mathematics,
program synthesis.
❖ Thus, to create a personalized education system using
AI.
❖ Have properties that an Explainable AI (XAI) can discover
like symmetry and recursion.
❖ Are challenging and interesting to humans.
❖ Existing algorithms like DeepCubeA that use Deep
reinforcement with search can solve these problems but
are not explainable.
❖ Logic programs are explainable to humans, but with lack
of domain-specific heuristic function, logic cannot be
discovered on its own.
Approach: DeepXube Results
Future Work
❖ DeepXube, combines DeepCubeA with Inductive Logic
Program to solve the planning problem in an explainable
manner.
❖ It can currently support three puzzles:- Rubik’s cube, tower
of Hanoi and 15 Puzzle.
❖ Create more robust explanations using graphs to extract
hierarchies.
❖ Have user studies of to analyze impact of research.
❖ Incorporating language models to generate natural language to
simplify collaboration between AI and human.
❖ Expand scope to problems like chemical synthesis, theorem proving,
robotics and program synthesis, etc.
The result of program for Rubik Cube to have one piece in
place can be shown as:
Collaboration and Personalization?
❖ Enable collaboration using multimodal interface :
❖ 3D visualization to visualize Rubik’s cube.
❖ Chatbot system to provide hints to solve Rubik’s cube.
❖ Personalization:
❖ Use visual interface and natural language:
❖ Get student feedback on the algorithm.
❖ Adapting algorithm based on the feedback.
solve(S,S):- goal(S).
solve(V0,A_out):-
edge_cbl(V1),has_stk_col(V1,V5),face(V6),onface(V0,V1,V5,V6),face_col(V6,V5),
has_stk_col(V1,V2),dif_col(V2,V5),dif_col(V5,V2),direction(V4),face(V3),dif_face(V3,V6),dif_fac
e(V6,V3),stk_f_dir(V0,V1,V2,V3,V4),face_col(V3,V2), move(V0,V6,V4,A_out).
solve(V0,A_out):-
edge_cbl(V1),has_stk_col(V1,V5),face(V6),onface(V0,V1,V5,V6),has_stk_col(V1,V2),
dif_col(V2,V5),dif_col(V5,V2),direction(V4),face(V3),dif_face(V3,V6),dif_face(V6,V3),stk_f_dir(V
0,V1,V2,V3,V4),face_col(V3,V2),move(V0,V6,V4,A_out).
solve(V0,A_out):-
edge_cbl(V1),has_stk_col(V1,V2),face(V3),onface(V0,V1,V2,V3),clockwise(Dir1),
move(V0,V3,Dir1,A_out).
The rough English translation: If one sticker is on face F and the other sticker is some
direction D away from the face that it matches, then move face F in direction D.
Otherwise, there is some sticker on face F, move face F clockwise.
The result of NLG for predicates can be shown as:
edge_cbl('E',) --> There is an edge cubelet E.
has_stk_col('E', 'G') --> Edge cubelet E has a sticker color
G.
has_stk_col('E', 'B') --> Edge cubelet E has a sticker color B.
onface('A', 'E', 'B', 'C') --> Sticker E of the edge cubelet B is
aligned with the face C.
stk_f_dir('A', 'E', 'G', 'D', 'F') --> Sticker E of edge cubelet G
is in direction D of the face F.
face('D',) --> There is a face D.
face('C',) --> There is a face C.
face_col('D', 'G') --> Face D has color G.
face_col('C', 'B') --> Face C has color B.
dif_col('G', 'B') --> G and B are different colors.
dif_face('D', 'C') --> D and C are different faces.
direction('F',) --> There is a direction F.
DeepCubeA: Agostinelli, F., McAleer, S., Shmakov, A. et al. Solving the Rubik’s cube with deep
reinforcement learning and search. Nat Mach Intell 1, 356–363 (2019). https://doi.org/10.1038/s42256-

Towards Rare Event Prediction in Manufacturing Domain
Problem Statement
Methodology
• Rare Event Classification in Multivariate Time Series (Ranjan, 2018)
[Focus : Rare Events]
◦ A real-world dataset from a pulp-and-paper manufacturing industry
• Other related datasets
• CONTEXT: An Industry 4.0 Dataset of Contextual Faults in a Smart Factory
(Kaupp, 2021)
Objective: Given a dataset from manufacturing domain, develop a neuro-symbolic AI approach leveraging the domain semantics for predicting events.
Future Steps
Intuition to Neuro-symbolic AI
◦ Incorporating relevant background knowledge & domain
semantics in predicting rare events.
◦ Explicit information related to sensors, cameras, weather,
belts, shuttles, or any physical parts etc.
◦ Data collection from physical
prototypes
◦ Identification & finalizing of relevant
events
◦ Integration with neXt Future Factory
Data Infrastructure
Datasets for Event Prediction Methods for Event Prediction
◦ What are Rare Events ?
◦ The occurrences that take place with a significantly lower frequency
than more common events. E.g., paper breaks, component failures
• These events are costly for industries, resumption time is higher.
• Identifying them lead to reducing defects, lowering equipment downtime,
and optimizing energy consumption.
• Ensure optimization, quality & safety standards in manufacturing processes.
Challenges to Focus
Visualization of features in time domain that lead to a break Visualization of features in frequency domain that lead to a break
Autoencoders & Deep
LSTM-based Stacked
Autoencoder for
Multivariate Time Series
Traditional Statistical
Approaches
CNN-based Unsupervised Approach
Vector Autoregression (VAR),
Vector Autoregression Moving-
Average (VARMA),
MARIMA, XGBoost, AdaBoost,
Random Forest
https://ieeexplore.ieee.org/document/8581424
https://www.nature.com/articles/s41598-019-55320-6
Can we improve the event
understanding in Smart
Manufacturing systems by
leveraging multimodal data
along with relevant
background knowledge and
manufacturing context ?
Chathurangi Shyalika1 , Ruwan Wickramarachchi1 & Amit Sheth1
1. Artificial Intelligence Institute, University of South Carolina
Distribution of the response variable
Dataset
statistics
Top 10 important
variables
contribute to
response variable

CausalKG: Causal Knowledge Graph for explainability using interventional and
counterfactual reasoning
Utkarshani Jaimini, Dr. Amit Sheth
Artificial Intelligence Institute, University of South Carolina
Figure 1: A smart robot contemplating the causal ramification of its action. The noise by the vacuum
cleaner caused the human to wake up. The smart robot interpretation should not be “Do not vacuum
ever” but “Do not vacuum when the human is sleeping in the room”. (Source: Book of Why, Drawing by
Maayan Harel)
This research is support in part by National Science Foundation (NSF) Award #2133842 “EAGER:
Advancing Neuro-symbolic AI with Deep Knowledge- infused Learning,” and Award #2119654, “RII
Track 2 FEC: Enabling Factory to Factory (F2F) Networking for Future Manufacturing.” Any opinions,
findings, and conclusions or recommendations expressed in this material are those of the author(s)
and do not necessarily reflect the views of the NSF. * Contact: ujaimini@email.sc.edu
Artificial Intelligence, Causality, and Counterfactuals
Acknowledgement
CausalKG: Causal Hyper-relational Knowledge graph
• Causality is a complex phenomenon.
• Unlike humans, artificial intelligence does not have inherent understanding of causality,
implications of their actions, and the ability to retrospect their past actions.
Figure 2. Explainability inspired from the Ladder of causation (Judea Pearl) and Ladder of thinking
Figure 3. (i) Causal Bayesian Network (CBN); (ii) Representation of information from CBN into a
knowledge graph (KG) (a) A triple-based representation of causality in a KG resulting in missing
information (such as mediator) needed for explainability, intervention and counterfactual queries. (b), (c)
hyper-relational graph-based representation which captures richer information about the causal concept
such as mediator and associated causal effects.
(i) (ii)
Figure 4. A snapshot of CausalKG: A hyper-relational KG of causality for asthma condition with pollen and
AQI trigger as Treatment variable shown in Figure 3, symptom and medication as Mediator and Outcome
variables, respectively. From the given scenario, we can conclude 1) AQI is a likely cause for a patient’s
medication intake (due to higher total causal effect). 2) What if the patient does not experience an
asthma symptom and given there is a pollen (AQI) trigger in the outdoor environment, the intake of
medication is 6.64% (5.06%) more likely. 3) What if the patient experiences an asthma symptom and
pollen (AQI) trigger is not present in the outdoor environment, the intake of medication is 3.04% (8.68%)
more likely for an asthma symptom being experienced.

CPR: Causal Process Knowledge Infused Reasoning
Kaushik Roy and Jinendra Malekar
Artificial Intelligence Institute South Carolina
Future Work:
1. Efficient Re-discovery of Causal Models
2. Efficient learning methods to modify
parameters
3. Enhancing explanation visuals for the
end-user
Applications in
Other Domains
● Cooking/Nutrition
Management
● Autonomous
Driving
● Game Playing
CPR-Reddit Web Plug-in
https://mentalhealthcpr.herokuapp.com

AIISC Projects via Posters at 2022 Retreat.pptx

AIISC Projects via Posters at 2022 Retreat.pptx

More Related Content

Similar to AIISC Projects via Posters at 2022 Retreat.pptx

Recently uploaded

AIISC Projects via Posters at 2022 Retreat.pptx