[Paper Reading] Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Supervised Learning of Universal Sentence Representations
from Natural Language Inference Data
2017/11/7 B4 Hiroki Shimanaka
Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo ̈ıc Barrault, Antoine Bordes
EMNLP 2017

Abstract & Introduction (1)
Many modern NLP systems rely on word embeddings, previously
trained in an un-supervised manner on large corpora, as base features.
Word embeddings
 Word2Vec (Mikolov et al., 2013)
 GloVe (Pennington et al., 2014)
Efforts to obtain embeddings for larger chunks of text, such as
sentences, have however not been so successful.
Unsupervised Sentence embeddings
 SkipThought (Kiros et al., 2015)
 FastSent (Hill et al., 2016)
1

2
In this paper, they show how universal sentence representations
trained using the supervised data of the Stanford Natural Language
Inference datasets.
It can consistently outperform unsu-pervised methods like SkipThought
vec-tors (Kiros et al., 2015) on a wide range of transfer tasks.
Abstract & Introduction (2)

3
Approach
This work combines two research directions, which they describe in
what follows.
First, they ex-plain how the NLI task can be used to train univer-sal sentence
encoding models using the SNLI task.
Second, they describe the architectures that we investigated for the sentence
encoder, which, in their opinion, covers a suitable range of sentence
encoders currently in use.

4
SNLI (Stanford Natural Language Inference) dataset
The SNLI dataset consists of 570k human-generated English sentence
pairs, manually la-beled with one of three categories: entailment,
contradiction and neutral.
https://nlp.stanford.edu/projects/snli/

5
The Natural Language Inference task
Once the sentence vectors are generated, 3
matching methods are applied to extract
relations between 𝑢 and 𝑣.
(i) concatenation of the two representa-tions (u, v)
(ii) element-wise product u ∗ v
(iii) absolute element-wise difference |u − v|
The resulting vector, which captures
information from both the premise and the
hypothesis, is fed into a 3-class classifier
consisting of multiple fully-connected layers
culminating in a softmax layer.

6
Sentence encoder architectures (1)
LSTM (Hochreiter and Schmidhuber, 1997) and GRU (Cho et al., 2014)
A sentence is represented by the last hid-den vector.
BiGRU-last
It con-catenates the last hidden state of a forward GRU, and the last hidden
state of a backward GRU.

7
BiLSTM with mean/max pooling

8
It uses an attention mecha-nism
over the hidden states of a
BiLSTM to gen-erate a
representation 𝑢 of an input
sentence.
 self-attentive sentence encoder (Liu et al., 2016; Lin et al., 2017)

9
Hierarchical ConvNet
It is like AdaSent (Zhao et al., 2015).
The final representation 𝑢 =
[𝑢1, 𝑢2, 𝑢3, 𝑢4] concatenates
representations at different levels of
the input sentence.

10
Training details
For all their models trained on SNLI, they use SGD with a learning rate
of 0.1 and a weight decay of 0.99.
At each epoch, they divide the learning rate by 5 if the dev accuracy
decreases.
they use mini-batches of size 64 and training is stopped when the
learning rate goes under the threshold of 10−5.
For the classifier, they use a multi-layer perceptron with 1 hidden-layer
of 512 hidden units.
They use open-source GloVe vectors trained on Common Crawl 840B
with 300 dimensions as fixed word embeddings.

11
Evaluation of sentence representations (1)
Binary and multi-class classification
sentiment analysis (MR, SST)
question-type (TREC)
product reviews (CR)
sub-jectivity/objectivity (SUBJ)
opinion polarity (MPQA)
Entailment and semantic relatedness
They also evaluate on the SICK dataset for both entailment (SICK-E) and
semantic relatedness (SICK-R).
Semantic Textual Similarity (STS14 (Agirre et al., 2014)).

12
Evaluation of sentence representations (2)
Paraphrase detection
Sentence pairs have been human-annotated according to whether they cap-
ture a paraphrase/semantic equivalence relation-ship.
Caption-Image retrieval
The caption-image retrieval task evaluates joint image and language feature
models (Hodosh et al., 2013; Lin et al., 2014).
The goal is either to rank a large collec-tion of images by their relevance with
respect to a given query caption (Image Retrieval), or ranking captions by
their relevance for a given query image (Caption Retrieval).

16
Conclusion
This paper studies the effects of training sentence embeddings with
supervised data by testing on 12 different transfer tasks.
They showed that mod-els learned on NLI can perform better than
mod-els trained in unsupervised conditions or on other supervised tasks.
They showed that a BiLSTM network with max pooling makes the best
current universal sen-tence encoding methods, outperforming existing
approaches like SkipThought vectors.

17
Reference
Supervised Learning of Universal Sentence Representations from
Natural Language Inference Data, Conneau et al., EMNLP 2017

[Paper Reading] Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

More Related Content

Similar to [Paper Reading] Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (20)

More from Hiroki Shimanaka (7)

Recently uploaded (20)

[Paper Reading] Supervised Learning of Universal Sentence Representations from Natural Language Inference Data

Editor's Notes