Accountable and Robust Automatic Fact Checking

Accountable and Robust
Automatic Fact
Checking
Isabelle Augenstein
augenstein@di.ku.dk
@IAugenstein
http://isabelleaugenstein.github.io/
ECIR
11 April 2022

False Information Online
https://ec.europa.eu/info/live-work-travel-eu/coronavirus-response/fighting-disinformation/tackling-coronavirus-disinformation_en

Types of False Information
http://www.contentrow.com/tools/link-bait-title-generator

https://rokokoposten.dk/2021/02/23/mars-indfoerer-indrejseforbud-efter-fund-af-britisk-virusvariant/

• Disinformation:
• Intentionally false, spread deliberately
• Misinformation:
• Unintentionally false information
• Clickbait:
• Exaggerating information and under-delivering it
• Satire:
• Intentionally false for humorous purposes
• Biased Reporting:
• Reporting only some of the facts to serve an agenda

Full Fact Checking Pipeline
Claim Check-
Worthiness Detection
Evidence Document
Retrieval
Stance Detection /
Textual Entailment
Veracity Prediction
“Immigrants are a drain
on the economy”
not check-worthy
check-worthy
on the economy”
“Immigrants are a drain on the
economy”, “EU immigrants have a
positive impact on public finances”
positive
negative
neutral
true
false
not enough info
on the economy”

Explainability – what is it and why do we need it?
Architect Trainer End User
Terminology borrowed from Strobelt et al. (2018), “LSTMVis: A tool for visual analysis of hidden state dynamics in recurrent
neural networks. IEEE transactions on visualization and computer graphics.”
true

Claim: “In the COVID-19 crisis, ‘only 20%
of African Americans had jobs where they
could work from home.’”
Evidence: “20% of black workers said they
could work from home in their primary
job, compared to 30% of white workers.”
Claim: “Children don’t seem to be getting
this virus.”
Evidence: “There have been no reported
incidents of infection in children.”
Claim: “Taylor Swift had a fatal car
accident.”
Reason: overfitting to spurious patterns
(celebrity death hoaxes are common)
Claim: “Michael Jackson is still alive,
appears in daughter’s selfie.”
Reason: overfitting to spurious patterns
(celebrity death hoaxes are common)
Right prediction Wrong prediction
Right reasons
Wrong reasons

Architect Trainer End User
?
true

Overview of Today’s Talk
• Introduction
• Fact checking – detecting false information online
• Explainability – making the right prediction for the right reasons
• Diagnostic Properties
• Evaluating explanations using different diagnostic properties
• Diagnostics-guided explanation generation
• Generating Free-Text Explanations for Fact Checking
• Supervised generation of explanations
• Unsupervised post-editing of explanations
• Detecting Vulnerabilities of Fact Checking Models
• Prediction without sufficient evidence
• Wrap-up

A Diagnostic Study of Explainability
Techniques for Text Classification
Pepa Atanasova, Jakob Grue Simonsen,
Christina Lioma, Isabelle Augenstein
EMNLP 2020
19

A Typical Fact Checking Pipeline
Claim Check-
Worthiness Detection
Evidence Document
Retrieval
Stance Detection /
Textual Entailment
Veracity Prediction
on the economy”
not check-worthy
check-worthy
on the economy”
“Immigrants are a drain on the
economy”, “EU immigrants have a
positive impact on public finances”
positive
negative
neutral
true
false
not enough info
on the economy”

Explanation via Rationale Selection -- Sentences
Claim: The Faroe Islands are no longer part of a kingdom.
Evidence document: The Faroe Islands, also called the Faeroes, is an
archipelago between the Norwegian Sea and the North Atlantic, about halfway
between Norway and Iceland, 200 mi north-northwest of Scotland.
The Islands are an autonomous country within the Kingdom of Denmark.
Its area is about 1,400 km2 with a population of 50,030 in April 2017.
(…)
Label: refutes
James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Arpit Mittal (2018). FEVER: a
large-scale dataset for Fact Extraction and VERification. In Proc. of NAACL.
https://arxiv.org/abs/1803.05355

Explanation via Rationale Selection -- Words
Hypothesis: An adult dressed in black holds a stick.
Premise: An adult is walking away empty-handedly.
Label: contradiction
Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, Phil Blunsom (2018). e-SNLI: Natural Language
Inference with Natural Language Explanations. In Proc. of NeurIPS.
https://proceedings.neurips.cc/paper/2018/hash/4c7a167bb329bd92580a99ce422d6fa6-Abstract.html

Post-Hoc Explainability Methods via Rationale Selection
Example from Twitter Sentiment Extraction (TSE) dataset
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein (2020). A Diagnostic Study of Explainability
Techniques for Text Classification. In EMNLP. https://www.aclweb.org/anthology/2020.emnlp-main.263/

Gradient-Based Approaches
• Compute gradient of input w.r.t. output
• Different aggregation methods used to produce one score per
input token (mean average, L2 norm aggregation)
• Saliency, InputX-Gradient, Guided Backpropagation
Perturbation-Based Approaches
• Replace tokens in input with other tokens to compute their relative
contributions
• Occlusion, Shapley Value Sampling
Simplification-Based Approaches
• Train local linear models to approximate local decision boundaries
• LIME
Post-Hoc Explainability Methods via Rationale Selection

• How can explainability methods be evaluated?
• Proposal: set of diagnostic properties
• What are characteristics of different explainability methods?
• How do explanations for models with different architectures differ?
• How do automatically and manually generated explanations differ?
Post-Hoc Explainability via Rationale Selection:
Research Questions
Atanasova, et al., 2020 ”A Diagnostic Study of Explainability Techniques for Text Classification”

• Agreement with Human Rationales (HA)
• Degree of overlap between human and automatic saliency scores
• Faithfulness (F)
• Mask most salient tokens, measure drop in performance
• Rationale Consistency (RC)
• Difference between explanations for models trained with different
random seeds, with model with random weights
• Dataset Consistency (DC)
• Difference between explanations for similar instances
• Confidence Indication (CI)
• Predictive power of produced explanations for model’s confidence
Post-Hoc Explainability Methods via Rationale Selection:
Diagnostic Properties

Selected Results
Spider chart for Transformer model on e-SNLI
HA: Agreement with
human rationales
F: Faithfulness
RC: Rationale Consistency
DC: Dataset Consistency
CI: Confidence indication
T: Computing Time

Aggregated Results
0,4
0,45
0,5
0,55
0,6
0,65
0,7
R
a
n
d
o
m
S
h
a
p
S
a
m
p
l
L
I
M
E
O
c
c
l
u
s
i
o
n
S
a
l
i
e
n
c
y
_
u
S
a
l
i
e
n
c
y
_
l
2
I
n
p
u
t
X
G
r
a
d
_
u
I
n
p
u
t
X
G
r
a
d
_
l
2
G
u
i
d
e
d
B
P
_
u
G
u
i
d
e
d
B
P
_
l
2
Mean of diagnostic property measures for e-SNLI
Transformer CNN LSTM

Summary
• Different explainability techniques
• Produce different explanations
• More than one explanation can be correct
• Different aspects to what makes a good explanation
• Diagnostic properties allow one to assess different aspects of
explainability techniques

Diagnostics-Guided Explanation
Generation
AAAI 2022
32

Diagnostic-Guided Explanations
34
• Explanations currently suffer from:
• ! lack of faithfulness to the underlying model
• " lack of consistency for similar data points
• # lack of confidence in explanation predictions
• We present the first method to learn those diagnostic
properties in an unsupervised way
• We directly optimise for them to improve the quality
of generated explanations

Diagnostic-Guided Explanations
Affiliation
name@example.com
g model’s ratio-
in its reasoning
ypically trained
ons. When such
re often selected
e a downstream
ptimising an ex-
ithfulness is one
hich prior work
y of an explana-
agnostic proper-
w similar expla-
nfidence Indica-
eflects the confi-
w to directly op-
raining a model
ch markedly im-
uman rationales,
complex reason-
Question: What is Bernardo's last name?
Answer Option: Smith
S1: ...
S15: It amuses me.
S16: Show me around, and then I shall decide.
S17: Of course, Señor Flynn.
S18: And stop calling me señor.
S19: Not even Los Mundos is so polite.
S20: Call me Bernardo.
Model
2
[MASK](4)
[MASK](4)
Prediction is preserved
when removing
(masking) non-selected
explanation sentences.
Faithful!
Explanation is
similar for
similar(masked)
input.
Consistent!
Can predict
confidence from
explanation.
Indicates
Confidence!
4
Target
Prediction
1
Explanation
Prediction
1
[M
A
S
K]
(2)
False
p=0.9
S1 S15 S16 S17 S18 S19 S20
0.1 0.3 0.2 0.7 0.5 0.4 0.9
3
4
4
2
[MASK](4)
4
[MASK](4)
2
Figure 1: Example instance from MultiRC with predicted
target and explanation (Step 1), where sentences with con-
35
!
"
#

Selected Results (MultiRC)
36
65,5
51,4
7,1
71,1
78,3
16,2
71,7
79,4
19,3
71,0
78,6
16,1
70,6
78,1
16,5
70,5
79,6
18,8
0,0
10,0
20,0
30,0
40,0
50,0
60,0
70,0
80,0
F1-Classification F1-Evidence Acc-Joint
SOTA Supervised
Sup+Data Consistency Sup+Faithfulness
Sup+Confidence Indication Sup+All

Effects of Diagnostic Property Objectives
37
! Data Consistency
• Improves explanations by removing sentences unrelated to the
target prediction.
Question: What colors are definitely used in the picture Lucy drew?
Answer: Yellow and purple Label: True
Pred: True, p=.99
E: She makes sure to draw her
mom named Martha wearing a
purple dress, because that is her
favorite. She draws many yellow
feathers for her pet bird named
Andy.
Pred: True, p=.98
E: She makes sure to draw her
mom named Martha wearing a
purple dress, because that is her
favorite. She draws many yellow
feathers for her pet bird named
Andy. She draws a picture of her
family.
Supervised Supervised + DC
Remove unrelated
Keep related to
target prediction

38
! Faithfulness
• Improves the explanations to reflect the rationale used to
predict the target for instances classified correctly by the
supervised model.
Claim: Zoey Deutch did not portray Rosemarie Hathaway in
Vampire Academy. Label: REFUTE
Pred: Refute, p=.99
E: She is known for portraying
Rosemarie ``Rose'' Hathaway in
Vampire Academy(2014), Beverly
in the Richard Link later film
Everybody Wants Some!!
Pred: Refute, p=.99
E: Zoey Francis Thompson Deutch
(born November 10, 1994) is an
American actress.
Supervised Supervised + F
Does not lead to
correct prediction

39
! Confidence Indication
• Re-calibrates the prediction probabilities of generated
explanations and predicted target tasks.
• Does not change many target predictions.
Label: Positive
E (both Sup-Sup-CI): For me, they calibrated my creativity as a
child; they are masterful, original works of art that mix moving
stories with what were astonishing special effects at the time
(and they still hold up pretty well).
Predicted Sup: negative, p=.99
Predicted Sup+CI: positive, p=.99
Not aligned with the
high confidence of
the explanation for
the positive class.

Generating Fact Checking
Explanations
ACL 2020
40

Extracted Wikipedia Sentences as Explanations
• FEVER dataset (Thorne et al., 2018 )
• Claims are re-written sentences from
Wikipedia.
• Explanation consists of evidence
sentences extracted from Wikipedia
pages.
• Biggest dataset for fact checking
• Brings small or no improvements for
veracity prediction with real-world
claims (Wadden et al., 2020, Ostrowski
et al., 2020)
42
Thorne et al., 2018 “FEVER: a large-scale dataset for Fact Extraction and VERification”
Thorne et al., 2021 “Evidence-based Verification for Real World Information Needs”
Ostrowski et al., 2021 “Multi-Hop Fact Checking of Political Claims”
Wadden et al., 2020 “Fact or Fiction: Verifying Scientific Claims”
claim was conducted in a separate annotation pro-
cess by annotators who were aware of the page but
not the sentence from which original claim was
extracted and thus in 31.75% of the claims more
than one sentence was considered appropriate ev-
idence. Claims require composition of evidence
from multiple sentences in 16.82% of cases. Fur-
thermore, in 12.15% of the claims, this evidence
was taken from multiple pages.
To ensure annotation consistency, we developed
suitable guidelines and user interfaces, resulting
in inter-annotator agreement of 0.6841 in Fleiss 
(Fleiss, 1971) in claim verification classification,
and 95.42% precision and 72.36% recall in evi-
dence retrieval.
To characterize the challenges posed by FEVER
we develop a pipeline approach which, given a
claim, first identifies relevant documents, then se-
lects sentences forming the evidence from the doc-
uments and finally classifies the claim w.r.t. ev-
idence. The best performing version achieves
31.87% accuracy in verification when requiring
correct evidence to be retrieved for claims SUP-
PORTED or REFUTED, and 50.91% if the correct-
ness of the evidence is ignored, both indicating the
Claim: The Rodney King riots took place in
the most populous county in the USA.
[wiki/Los Angeles Riots]
The 1992 Los Angeles riots,
also known as the Rodney King riots
were a series of riots, lootings, ar-
sons, and civil disturbances that
occurred in Los Angeles County, Cali-
fornia in April and May 1992.
[wiki/Los Angeles County]
Los Angeles County, officially
the County of Los Angeles,
is the most populous county in the USA.
Verdict: Supported
Figure 1: Manually verified claim requiring evidence
from multiple Wikipedia pages.
remains too challenging for the ML/NLP methods
currently available. Wang (2017) extended this ap-
proach by including all 12.8K claims available by
Politifact via its API, however the justification and
the evidence contained in it was ignored in the ex-

Real World Fact Checking
Statement: “The last quarter, it was just
announced, our gross domestic product was below
zero. Who ever heard of this? Its never below zero.”
Speaker: Donald Trump
Context: presidential announcement speech
Label: Pants on Fire
Wang, William Yang. "“Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection." Proceedings ACL’2017.
0,204 0,208
0,247
0,274
0
0,1
0,2
0,3
Validation Test
Fact checking macro F1
score
Majority Wang et. al

Real-World Fact Checking Explanations
Ruling
Comments
Justification
https://www.politifact.com/factchecks/2020/apr/30/donald-trump/trumps-boasts-testing-spring-cherry-picking-data/

Generating Explanations from Ruling Comments
Claim:
We’ve tested more than every country
combined.
Ruling Comments:
Responding to weeks of criticism over his administration’s
COVID-19 response, President Donald Trump claimed at a
White House briefing that the United States has well
surpassed other countries in testing people for the virus.
"We’ve tested more than every country combined,"
Trump said April 27 […] We emailed the White House for
comment but never heard back, so we turned to the data.
Trump’s claim didn’t stand up to scrutiny.
In raw numbers, the United States has tested more
people than any other individual country — but
nowhere near more than "every country combined" or,
as he said in his tweet, more than "all major countries
combined.”[…] The United States has a far bigger
population than many of the "major countries" Trump often
mentions. So it could have run far more tests but still
have a much larger burden ahead than do nations like
Germany, France or Canada.[…]
Joint Model
Veracity Label
Justification/
Explanation
Atanasova, et al., 2020 ”Generating Fact Checking Explanations."

Related Studies on Generating Explanations
• Camburu et. al; Rajani et. al generate abstractive
explanations
• Short input text and explanations;
• Large amount of annotated data.
• Real world fact checking datasets are of limited size and
the input consists of long documents
• We take advantage of the LIAR-PLUS dataset:
• Use the summary of the ruling comments as a gold explanation;
• Formulate the problem as extractive summarization.
• Camburu, Oana-Maria, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. "e-SNLI: Natural language inference with natural language
explanations." In Advances in Neural Information Processing Systems,. 2018.
• Rajani, Nazneen Fatema, Bryan McCann, Caiming Xiong, and Richard Socher. "Explain Yourself! Leveraging Language Models for
Commonsense Reasoning." In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4932-4942. 2019.

Joint Explanation and Veracity Prediction
Ruder, Sebastian, et al. "Latent multi-task architecture learning." In Proceedings of the AAAI 2019.
Atanasova, et al., 2020 ”Generating Fact Checking Explanations." In Proceedings of ACL 2020.
Cross-stitch layer
Multi-task objective
Which sentences should be selected for the explanation?

Generating Free Text Explanations:
Automatic Evaluation Measures
• Indirectly: Fact Checking Performance
• F1. Predicting fact checking labels: using a joint model, from
generated explanations, gold explanations, etc.
• Directly: Explanation Quality
• ROUGE-N. Overlap of n-grams between the oracle and the generated
summaries.
• ROUGE-L. Longest Common Subsequence (LCS) between the oracle
and the generated summaries.

Generating Free Text Explanations:
Manual Evaluation Measures
• Explanation Quality
• Coverage. The explanation contains important, salient information
and does not miss any points important to the fact check.
• Non-redundancy. The explanation does not contain any information
that is redundant/repeated/not relevant to the claim or the fact
check.
• Non-contradiction. The explanation does not contain any pieces of
information contradictory to the claim or the fact check.
• Overall. Overall explanation quality.
• Explanation Informativeness. Veracity label for a claim
provided based on the automatically generated
explanations only.

Selected Results: Explanation Quality
1,48 1,48 1,45
1,58
1,89
1,75
1,40
2,03
1,68
1,79
1,48
1,90
-
0,50
1,00
1,50
2,00
2,50
Coverage Non-redundancy Non-contradicton Overall
Mean Average Rank (MAR).
Lower MAR is better! (higher rank)
Gold Justification Extractive Extractive MT

Explanation Informativeness
-
0,050
0,100
0,150
0,200
0,250
0,300
0,350
0,400
0,450
0,500
Agree-Correct Agree-Not Correct Agree-Not
Sufficient
Disagree
Manual veracity labelling, given a particular
explanation as percentages of the dis/agreeing
annotator predictions.
Justification Extractive Extractive-MT
Atanasova, et al., 2020 ”Generating Fact Checking Explanations.” In Proceedings of ACL 2020.

Summary
• First study on generating real-world veracity
explanations
• Jointly training veracity prediction and explanation
• improves the performance of the classification system
• improves the coverage and overall performance of the
generated explanations

Generating Fluent Fact Checking
Explanations with Unsupervised Post-
Editing
Shailza Jolly, Pepa Atanasova, Isabelle
Augenstein
Preprint, December 2021
53

Ongoing Work: Unsupervised Post-Editing
Shailza Jolly, Pepa Atanasova, Isabelle Augenstein (2021). Generating Fluent Fact Checking Explanations with
Unsupervised Post-Editing. Preprint. https://arxiv.org/abs/2112.06924

Vulnerabilities of
Fact Checking
Models
55

Fact Checking with Insufficient
Evidence
TACL, Vol 10 (2022), to appear
56

Known Vulnerabilities of Fact Checking Models
• Claim-only bias
• 61.7 accuracy of a BERT given only the claim
• Lexical biases
• E.g. negation and the REFUTES class
• Prior proposed solutions:
• FEVERSymmetric dataset to measure the bias of fact checking
models
• Re-weighted training objective
• Increase the importance of claims with different labels containing those phrases
58
Schuster, et al., 2019 ”Towards Debiasing Fact Verification Models”. In Proceedings of EMNLP 2019.

Known Vulnerabilities of Fact Checking Models
053
054
055
056
057
058
059
060
061
062
063
064
065
066
067
068
069
070
071
nymous TACL submission
ocess
exter-
models
there
indi-
work,
mation
oduc-
three
uct an
BERT RoBERTa AlBERT
NOT ENOUGH
INFO
NOT ENOUGH
INFO
Claim: By April 9 , less than 9000 people who tested positive for COVID-19
in the UK died of the virus .
Evidence: [[2020 coronavirus pandemic in the United Kingdom]] As of 9
April , there have been 65,077 confirmed cases of COVID-19 in the UK ,
and 8.958 people with confirmed infection have died .
Label: SUPPORTS
SUPPORTS SUPPORTS
Human Annotator
Evidence Omission:
Delete 8.958 from Evidence
Figure 1: An example from the VitaminC test set,
59
• Prediction with insufficient evidence
• SufficientFacts dataset – fluency-preserving removal of constituents,
e.g. adjective/adverb/noun/number modifiers, prepositional phrases
• Models still predict the original label in most cases:

Evidence Omission
Sentence (S), Prepositional Phrase (PP), Noun Modifier (NOUNM), Adjective Modifier (ADJM), Adverb
Modifier (ADVM), Number Modifier (NUMM), Date Modifier (DATEM), Subordinate Clause (SBAR)

SufficientFacts Dataset Construction
• Ensemble of 3 trained Transformer-based FC models (BERT,
RoBERTa, ALBERT)
• Predict labels for instances from FC datasets (FEVER, HoVer,
VitaminC) from which information has been removed
• Manually annotate instances for which:
• Models agree that evidence is sufficient
• Agree that evidence is insufficient
• Disagree
• Overall findings from manual annotation
• Model disagreements are mostly for NEI instances
• When models agree that information is insufficient, they are correct in
97.2% of cases
62

Fine-Grained Analysis By Removal Type
# NEI / EI instances
All = all models agree on prediction; higher proportion of correct predictions -> better
hardest
easiest

Fine-Grained Analysis By Removal Type
# NEI / EI instances
All = all models agree on prediction; higher proportion of correct predictions -> better
Difference
NEI & EI

New Task: Evidence Omission Detection
• Given an instance, find a distractor sentence from the document
of the gold evidence that is most semantically similar to claim
(word overlap)
• Append distractor sentence to original evidence texts to serve as
anchors
• Omit information from original evidence texts (using evidence
omission procedure)
65

• For each of the 3 trained Transformer-based FC models:
• Select candidates instances for which the other two supervised
models previously predicted NEI
• Two types of positive instances:
• Sentences from original evidence without distractor sentence
• Sentence from original evidence + distractor sentence with highest
word overlap with the claim, with one consituent omitted
• Negative instances: original evidence with constituents/sentences
removed that all three models predict as NEI
66

67
Contrastive instances: negative ones, where models agree that the
remaining evidence is insufficient; positive ones, where a distractor
sentence with high lexical overlap is added

Fact Checking with Evidence Omission Detection
• Contrastive learning (CL):
• Auxiliary task:
• bring the anchor and the positive instance closer in the representation space
• drive the anchor and the negative instances further apart
• Counterfactual data augmentation (CAD):
• Augment fact checking datasets with instances from Evidence
Omission Detection dataset
• Treat it as the same task
68

Selected Results for Fact Checking on HoVer
69
55
60
65
70
75
80
85
Base CL CAD
Original Test
BERT RoBERTa ALBERT Ensemble
55
60
65
70
75
80
85
Base CL CAD
SufficientFacts
BERT RoBERTa ALBERT Ensemble

Summary of Findings
• Comprehensive study of FC with omitted information
• Fluency-preserving omission methods
• FC models struggle most when adverbial modifiers are
removed; least when date modifiers are removed
• Significant differences between Transformer
architectures
• AlBERT better at correctly predicting NEI
• BERT and RoBERTa better at correctly predicting with EI
70

Summary of Findings
• SufficientFacts is challenging test bed for FC
• Performance ~30 F1 points lower than on standard test set
• Training on additional data for evidence omission
detection task improves performance on original test
set and on SufficientFacts
71

• Why explainability?
• understanding if a model is right for the right reasons
• Generated explanations can help users understand:
• inner workings of a model (model understanding)
• how a model arrived at a prediction (decision understanding)
• Explainability can enable:
• transparency for end users
• human-in-the-loop model development
• human-in-the-loop data selection
Overall Take-Aways

Overall Take-Aways
• Diagnostic Properties
• Enable automatic evaluation of explanations
• Additional training objectives can improve the properties of the
explanations and the general model performance
• Generating Free-Text Explanations
• Complex task; we propose first solutions, framing the task as
summarisation
• How to generate fact checking explanations from evidence
documents directly?
• How to collect a less noisy dataset of real-world fact checks?

Overall Take-Aways
• Detecting Vulnerabilities of Fact Checking Models
• Fact checking models contain many vulnerabilities
• Easily fooled with lexical trigger words
• Easily tricked into making predictions without sufficient evidence
• Training on adversarially generated instances improves robustness
• Training on instances with omitted information improves
robustness
• How to generate explanations that are useful both for debugging
and for the end user?

Thank you!
isabelleaugenstein.github.io
augenstein@di.ku.dk
@IAugenstein
github.com/isabelleaugenstein
11/04/2022 76

Thanks to my PhD students and colleagues!
77
CopeNLU
https://copenlu.github.io/
Pepa Atanasova Shailza Jolly Jakob Grue Simonsen Christina Lioma

Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein. A
Diagnostic Study of Explainability Techniques for Text Classification. In
Proceedings of EMNLP 2020.
Pepa Atanasova, Jakob Grue Simonsen, Christina Lioma, Isabelle Augenstein.
Diagnostics-Guided Explanation Generation. In Proceedings of AAAI 2022.
Generating Fact Checking Explanations. In Proceedings of ACL 2020.
Shailza Jolly, Pepa Atanasova, Isabelle Augenstein.
Generating Fluent Fact Checking Explanations with Unsupervised Post-Editing.
Preprint, abs/2112.06924, December 2021.
Fact Checking with Insufficient Evidence. TACL, Volume 10 (2022), to appear.
Presented Papers

Pepa Atanasova, Dustin Wright, Isabelle Augenstein. Generating Label
Cohesive and Well-Formed Adversarial Claims. In Proceedings of EMNLP 2020.
Dustin Wright, David Wadden, Kyle Lo, Bailey Kuehl, Isabelle Augenstein, Lucy
Lu Wang. Generating Scientific Claims for Automatic Scientific Fact Checking.
In Proceedings of AAAI 2022.
Dustin Wright, Isabelle Augenstein. Semi-Supervised Exaggeration Detection
of Health Science Press Releases. In Proceedings of EMNLP 2021.
Shailza Jolly, Pepa Atanasova, Isabelle Augenstein. Time-Aware Evidence
Ranking for Fact-Checking. Journal of Web Semantics, Special Issue on Content
Credibility, Volume 71, November 2021.
Wojciech Ostrowski, Arnav Arora, Pepa Atanasova, Isabelle Augenstein. Multi-
Hop Fact Checking of Political Claims. In Procedings of IJCAI 2021.
Other Recent Relevant Papers

Accountable and Robust Automatic Fact Checking

More Related Content

What's hot (20)

Similar to Accountable and Robust Automatic Fact Checking (20)

More from Isabelle Augenstein (20)

Recently uploaded (20)

Accountable and Robust Automatic Fact Checking