ICASSP 2018 Tutorial: Generative Adversarial Network and its Applications to Signal Processing and Natural Language Processing

Generative Adversarial Network
and its Applications to Signal Processing
and Natural Language Processing
Hung-yi Lee and Yu Tsao

Mihaela Rosca, Balaji Lakshminarayanan, David Warde-Farley, Shakir Mohamed, “Variational Approaches for Auto-Encoding
Generative Adversarial Networks”, arXiv, 2017
All Kinds of GAN … https://github.com/hindupuravinash/the-gan-zoo
GAN
ACGAN
BGAN
DCGAN
EBGAN
fGAN
GoGAN
CGAN
……

ICASSP Keyword search on session index page,
so session names are included.
Number of papers whose titles include the keyword
0
5
10
15
20
25
30
35
40
45
2012 2013 2014 2015 2016 2017 2018
generative adversarial reinforcement
It is a wise choice to
attend this tutorial.

Outline
Part I: General Introduction of Generative
Adversarial Network (GAN)
Part II: Applications to Signal Processing
Part III: Applications to Natural Language
Processing

Part I: General Introduction

Outline of Part 1
Generation by GAN
Conditional Generation
Unsupervised Conditional Generation
Relation to Reinforcement Learning

Outline of Part 1
Generation by GAN
• Image Generation as Example
• Theory behind GAN
• Issues and Possible Solutions

Anime Face Generation
Draw
Generator
Examples

Basic Idea of GAN
Generator
It is a neural network
(NN), or a function.
Generator
0.1
−3
⋮
2.4
0.9
imagevector
Generator
3
−3
⋮
2.4
0.9
Generator
0.1
2.1
⋮
5.4
0.9
Generator
0.1
−3
⋮
2.4
3.5
high
dimensional
vector
Powered by: http://mattya.github.io/chainer-DCGAN/
Each dimension of input vector
represents some characteristics.
Longer hair
blue hair Open mouth

Discri-
minator
scalar
image
Basic Idea of GAN It is a neural network
(NN), or a function.
Larger value means real,
smaller value means fake.
Discri-
minator
Discri-
minator
Discri-
minator1.0 1.0
0.1 Discri-
minator
0.1

• Initialize generator and discriminator
• In each training iteration:
DG
sample
generated
objects
G
Algorithm
D
Update
vector
vector
vector
vector
0000
1111
randomly
sampled
Database
Step 1: Fix generator G, and update discriminator D
Discriminator learns to assign high scores to real objects
and low scores to generated objects.
Fix

DG
Algorithm
Step 2: Fix discriminator D, and update generator G
Discri-
minator
NN
Generator
vector
0.13
hidden layer
update fix
Gradient Ascent
large network
Generator learns to “fool” the discriminator

DG
Learning
D
Sample some
real objects:
Generate some
fake objects:
G
Algorithm
D
Update
Learning
G
G D
image
1111
image
image
image
1
update fix
0000vector
vector
vector
vector
vector
vector
vector
vector
fix

100 updates
Source of training data: https://zhuanlan.zhihu.com/p/24767059

1000 updates

2000 updates

5000 updates

10,000 updates

20,000 updates

50,000 updates

The faces
generated by
machine.
The images are generated
by Yen-Hao Chen, Po-Chun
Chien, Jun-Chen Xie, Tsung-
Han Wu.

0.0
0.0
G
0.9
0.9
G
0.1
0.1
G
0.2
0.2
G
0.3
0.3
G
0.4
0.4
G
0.5
0.5
G
0.6
0.6
G
0.7
0.7
G
0.8
0.8
G

(Variational) Auto-encoder
As close as possible
NN
Encoder
NN
Decoder
code
NN
Decoder
code
Randomly generate
a vector as code Image ?
= Generator
= Generator

Auto-encoder v.s. GAN
As close as possibleNN
Decoder
code
Genera-
tor
code
= Generator
Discri-
minator
If discriminator does not simply memorize the images,
Generator learns the patterns of faces.
Just copy an image
1
Auto-encoder
GAN
Fuzzy …

FID[Martin Heusel, et al., NIPS, 2017]: Smaller is better
[Mario Lucic, et al. arXiv, 2017]

Outline of Part 1
Generation

Generator
• A generator G is a network. The network defines a
probability distribution 𝑃𝐺
generator
G𝑧 𝑥 = 𝐺 𝑧
Normal
Distribution
𝑃𝐺(‫)ݔ‬ 𝑃𝑑𝑎𝑡𝑎 𝑥
as close as possible
How to compute the divergence?
𝐺∗ = 𝑎𝑟𝑔 min
𝐺
𝐷𝑖𝑣 𝑃𝐺, 𝑃𝑑𝑎𝑡𝑎
Divergence between distributions 𝑃𝐺 and 𝑃𝑑𝑎𝑡𝑎
𝑥: an image (a high-
dimensional vector)

Discriminator
𝐺∗
= 𝑎𝑟𝑔 min
𝐺
Although we do not know the distributions of 𝑃𝐺 and 𝑃𝑑𝑎𝑡𝑎,
we can sample from them.
sample
G
vector
vector
vector
vector
sample from
normal
Database
Sampling from 𝑷 𝑮
Sampling from 𝑷 𝒅𝒂𝒕𝒂

Discriminator 𝐺∗
= 𝑎𝑟𝑔 min
𝐺
Discriminator
: data sampled from 𝑃𝑑𝑎𝑡𝑎
: data sampled from 𝑃𝐺
train
𝑉 𝐺, 𝐷 = 𝐸 𝑥∼𝑃 𝑑𝑎𝑡𝑎
𝑙𝑜𝑔𝐷 𝑥 + 𝐸 𝑥∼𝑃 𝐺
𝑙𝑜𝑔 1 − 𝐷 𝑥
Example Objective Function for D
(G is fixed)
𝐷∗ = 𝑎𝑟𝑔 max
𝐷
𝑉 𝐷, 𝐺Training:
Using the example objective
function is exactly the same as
training a binary classifier.
[Goodfellow, et al., NIPS, 2014]
The maximum objective value
is related to JS divergence.

Discriminator 𝐺∗
= 𝑎𝑟𝑔 min
𝐺
Discriminator
: data sampled from 𝑃𝑑𝑎𝑡𝑎
: data sampled from 𝑃𝐺
train
hard to discriminatesmall divergence
Discriminator
train
easy to discriminatelarge divergence
𝐷
𝑉 𝐷, 𝐺
Training:
(cannot make objective large)

𝐺∗
= 𝑎𝑟𝑔 min
𝐺
𝐷𝑖𝑣 𝑃𝐺, 𝑃𝑑𝑎𝑡𝑎max
𝐷
𝑉 𝐺, 𝐷
The maximum objective value
is related to JS divergence.
Step 1: Fix generator G, and update discriminator D
Step 2: Fix discriminator D, and update generator G
𝐷
𝑉 𝐷, 𝐺
[Goodfellow, et al., NIPS, 2014]

Using the divergence
you like 
[Sebastian Nowozin, et al., NIPS, 2016]
Can we use other divergence?

Outline of Part 1
Generation
More tips and tricks:
https://github.com/soumith/ganhacks

JS divergence is not suitable
• In most cases, 𝑃𝐺 and 𝑃𝑑𝑎𝑡𝑎 are not overlapped.
• 1. The nature of data
• 2. Sampling
Both 𝑃𝑑𝑎𝑡𝑎 and 𝑃𝐺 are low-dim
manifold in high-dim space.
𝑃𝑑𝑎𝑡𝑎
𝑃𝐺
The overlap can be ignored.
Even though 𝑃𝑑𝑎𝑡𝑎 and 𝑃𝐺
have overlap.
If you do not have enough
sampling ……

𝑃𝑑𝑎𝑡𝑎𝑃𝐺0 𝑃𝑑𝑎𝑡𝑎𝑃𝐺1
𝐽𝑆 𝑃𝐺0
, 𝑃𝑑𝑎𝑡𝑎
= 𝑙𝑜𝑔2
𝑃𝑑𝑎𝑡𝑎𝑃𝐺100
……
𝐽𝑆 𝑃𝐺1
= 𝑙𝑜𝑔2
𝐽𝑆 𝑃𝐺100
= 0
What is the problem of JS divergence?
……
JS divergence is log2 if two distributions do not overlap.
Intuition: If two distributions do not overlap, binary classifier
achieves 100% accuracy
Equally bad
Same objective value is obtained. Same divergence

Wasserstein distance
• Considering one distribution P as a pile of earth,
and another distribution Q as the target
• The average distance the earth mover has to move
the earth.
𝑃 𝑄
d
𝑊 𝑃, 𝑄 = 𝑑

Wasserstein distance
Source of image: https://vincentherrmann.github.io/blog/wasserstein/
𝑃
𝑄
Using the “moving plan” with the smallest average distance to
define the Wasserstein distance.
There are many possible “moving plans”.
Smaller
distance?
Larger
distance?

𝑃𝑑𝑎𝑡𝑎𝑃𝐺0 𝑃𝑑𝑎𝑡𝑎𝑃𝐺1
𝐽𝑆 𝑃𝐺0
= 𝑙𝑜𝑔2
𝑃𝑑𝑎𝑡𝑎𝑃𝐺100
……
𝐽𝑆 𝑃𝐺1
= 𝑙𝑜𝑔2
𝐽𝑆 𝑃𝐺100
= 0
What is the problem of JS divergence?
𝑊 𝑃𝐺0
= 𝑑0
𝑊 𝑃𝐺1
= 𝑑1
𝑊 𝑃𝐺100
= 0
𝑑0 𝑑1
……
……
Better!

WGAN
𝑉 𝐺, 𝐷
= max
𝐷∈1−𝐿𝑖𝑝𝑠𝑐ℎ𝑖𝑡𝑧
𝐸 𝑥~𝑃 𝑑𝑎𝑡𝑎
𝐷 𝑥 − 𝐸 𝑥~𝑃 𝐺
𝐷 𝑥
Evaluate wasserstein distance between 𝑃𝑑𝑎𝑡𝑎 and 𝑃𝐺
[Martin Arjovsky, et al., arXiv, 2017]
How to fulfill this constraint?D has to be smooth enough.
real
−∞
generated
D
∞
Without the constraint, the
training of D will not converge.
Keeping the D smooth forces
D(x) become ∞ and −∞

• Original WGAN → Weight Clipping [Martin Arjovsky, et al.,
arXiv, 2017]
• Improved WGAN → Gradient Penalty [Ishaan Gulrajani,
NIPS, 2017]
• Spectral Normalization → Keep gradient norm
smaller than 1 everywhere [Miyato, et al., ICLR, 2018]
Force the parameters w between c and -c
After parameter update, if w > c, w = c; if w < -c, w = -c
Keep the gradient close to 1
𝑉 𝐺, 𝐷 = max
𝐷∈1−𝐿𝑖𝑝𝑠𝑐ℎ𝑖𝑡𝑧
𝐸 𝑥~𝑃 𝑑𝑎𝑡𝑎
𝐷 𝑥 − 𝐸 𝑥~𝑃 𝐺
𝐷 𝑥
real
samples
Keep the gradient
close to 1
[Kodali, et al., arXiv, 2017]
[Wei, et al., ICLR, 2018]

Energy-based GAN (EBGAN)
• Using an autoencoder as discriminator D
Discriminator
0 for the
best images
Generator is
the same.
-
0.1
EN DE
Autoencoder
X -1 -0.1
[Junbo Zhao, et al., arXiv, 2016]
Using the negative reconstruction error of
auto-encoder to determine the goodness
Benefit: The auto-encoder can be pre-train
by real images without generator.

Mode Collapse
: real data
: generated data
Training with too many iterations ……

Mode Dropping
BEGAN on CelebA
Generator
at iteration t
Generator
at iteration t+1
Generator
at iteration t+2
Generator switches mode during training

Ensemble
Generator
1
Generator
2
……
……
Train a set of generators: 𝐺1, 𝐺2, ⋯ , 𝐺 𝑁
Random pick a generator 𝐺𝑖
Use 𝐺𝑖 to generate the image
To generate an image

Objective Evaluation
Off-the-shelf
Image Classifier
𝑥 𝑃 𝑦|𝑥
Concentrated distribution
means higher visual quality
CNN𝑥1
𝑃 𝑦1|𝑥1
Uniform distribution
means higher variety
CNN𝑥2
𝑃 𝑦2|𝑥2
CNN𝑥3
𝑃 𝑦3|𝑥3
…
𝑃 𝑦 =
1
𝑁
𝑛
𝑃 𝑦 𝑛|𝑥 𝑛
[Tim Salimans, et al., NIPS, 2016]
𝑥: image
𝑦: class (output of CNN)
e.g. Inception net,
VGG, etc.
class 1
class 2
class 3

Outline of Part 1
Generation

• Original Generator
• Conditional Generator
G
condition c
𝑧 𝑥 = 𝐺 𝑧
Normal
Distribution G
𝑃𝐺 𝑥 → 𝑃𝑑𝑎𝑡𝑎 𝑥
𝑧
Normal
Distribution 𝑥 = 𝐺 𝑐, 𝑧
𝑃𝐺 𝑥|𝑐 → 𝑃𝑑𝑎𝑡𝑎 𝑥|𝑐
[Mehdi Mirza, et al., arXiv, 2014]
NN
Generator
“Girl with red hair
and red eyes”
“Girl with yellow
ribbon”
e.g. Text-to-Image

Target of
NN output
Text-to-Image
• Traditional supervised approach
NN Image
Text: “train”
a dog is running
a bird is flying
A blurry image!
c1: a dog is running
as close as
possible

Conditional GAN
D
(original)
scalar𝑥
G
𝑧Normal distribution
x = G(c,z)
c: train
x is real image or not
Image
Real images:
Generated images:
1
0
Generator will learn to
generate realistic images ….
But completely ignore the
input conditions.
[Scott Reed, et al, ICML, 2016]

Conditional GAN
D
(better)
scalar
𝑐
𝑥
True text-image pairs:
G
𝑧Normal distribution
x = G(c,z)
c: train
Image
x is realistic or not +
c and x are matched or not
(train , )
(train , )(cat , )
[Scott Reed, et al, ICML, 2016]
1
00

x is realistic or not +
c and x are matched
or not
Conditional GAN - Discriminator
[Takeru Miyato, et al., ICLR, 2018]
[Han Zhang, et al., arXiv, 2017]
[Augustus Odena et al., ICML, 2017]
condition c
object x
Network
Network
Network
score
Network
Network
(almost every paper)
condition c
object x
c and x are matched
or not
x is realistic or not

Conditional GAN
paired data
blue eyes
red hair
short hair
Collecting anime faces
and the description of its
characteristics
red hair,
green eyes
blue hair,
red eyes
The images are generated
by Yen-Hao Chen, Po-Chun
Chien, Jun-Chen Xie, Tsung-
Han Wu.

Conditional GAN - Image-to-image
G
𝑧
x = G(c,z)
𝑐
[Phillip Isola, et al., CVPR, 2017
Image translation, or pix2pix

as close as
possible
• Traditional supervised approach
NN Image
It is blurry.
Testing:
input L1
e.g. L1

Testing:
input L1 GAN
G
𝑧
Image D scalar
GAN + L1
L1

Conditional GAN
- Video Generation
Generator
Discrimi
nator
Last frame is real
or generated
Discriminator thinks it is real
[Michael Mathieu, et al., arXiv, 2015]

https://github.com/dyelax/Adversarial_Video_Generation

Domain Adversarial Training
• Training and testing data are in different domains
Training
data:
Testing
data:
Generator
Generator
The same
distribution
feature
feature
Take digit
classification as example

blue points
red points
feature extractor (Generator)
Discriminator
(Domain classifier)
image Which domain?
Always output
zero vectors
Domain Classifier Fails

feature extractor (Generator)
Discriminator
(Domain classifier)
image
Label predictor
Which digits?
Not only cheat the domain
classifier, but satisfying label
predictor at the same time
More speech-related applications in Part II.
Successfully applied on image classification
[Ganin et al, ICML, 2015][Ajakan et al. JMLR, 2016 ]
Which domain?

Domain X Domain Y
male female
It is good.
It’s a good day.
I love you.
It is bad.
It’s a bad day.
I don’t love you.
Transform an object from one domain to another
without paired data (e.g. style transfer)
Part II
Part III
GDomain X Domain Y

Unsupervised
• Approach 1: Direct Transformation
• Approach 2: Projection to Common Space
?𝐺 𝑋→𝑌
Domain X Domain Y
For texture or
color change
𝐸𝑁 𝑋 𝐷𝐸 𝑌
Encoder of
domain X
Decoder of
domain Y
Larger change, only keep the semantics
Domain YDomain X Face
Attribute

?
Direct Transformation
𝐺 𝑋→𝑌
Domain X
Domain Y
𝐷 𝑌
Domain Y
Domain X
scalar
Input image
belongs to
domain Y or not
Become similar
to domain Y

𝐺 𝑋→𝑌
Domain X
Domain Y
𝐷 𝑌
Domain Y
Domain X
scalar
Input image
belongs to
domain Y or not
Become similar
to domain Y
Not what we want!
ignore input

𝐺 𝑋→𝑌
Domain X
Domain Y
𝐷 𝑌
Domain X
scalar
Input image
belongs to
domain Y or not
Become similar
to domain Y
Not what we want!
ignore input
[Tomer Galanti, et al. ICLR, 2018]
The issue can be avoided by network design.
Simpler generator makes the input and
output more closely related.

𝐺 𝑋→𝑌
Domain X
Domain Y
𝐷 𝑌
Domain X
scalar
Input image
belongs to
domain Y or not
Become similar
to domain Y
Encoder
Network
Encoder
Network
pre-trained
as close as
possible
Baseline of DTN [Yaniv Taigman, et al., ICLR, 2017]

𝐺 𝑋→𝑌
𝐷 𝑌
Domain Y
scalar
Input image
belongs to
domain Y or not
𝐺Y→X
Lack of information
for reconstruction
[Jun-Yan Zhu, et al., ICCV, 2017]
Cycle consistency

𝐺 𝑋→𝑌 𝐺Y→X
𝐺Y→X 𝐺 𝑋→𝑌
𝐷 𝑌𝐷 𝑋
scalar: belongs to
domain Y or not
scalar: belongs to
domain X or not

Cycle GAN
Dual GAN
Disco GAN
[Jun-Yan Zhu, et al., ICCV, 2017]
[Zili Yi, et al., ICCV, 2017]
[Taeksoo Kim, et
al., ICML, 2017]
For multiple domains,
considering starGAN
[Yunjey Choi, arXiv, 2017]

Issue of Cycle Consistency
• CycleGAN: a Master of Steganography
[Casey Chu, et al., NIPS workshop, 2017]
𝐺Y→X𝐺 𝑋→𝑌
The information is hidden.

Domain X Domain Y
𝐸𝑁 𝑋
𝐸𝑁𝑌 𝐷𝐸 𝑌
𝐷𝐸 𝑋image
image
image
image
Face
Attribute
Projection to Common Space
Target

Domain X Domain Y
𝐸𝑁 𝑋
𝐷𝐸 𝑋image
image
image
image
Minimizing reconstruction error
Training

𝐸𝑁 𝑋
𝐷𝐸 𝑋image
image
image
image
Because we train two auto-encoders separately …
The images with the same attribute may not project
to the same position in the latent space.
𝐷 𝑋
𝐷 𝑌
Discriminator
of X domain
Discriminator
of Y domain
Training

Sharing the parameters of encoders and decoders
Training
𝐸𝑁 𝑋
𝐸𝑁𝑌
𝐷𝐸 𝑋
𝐷𝐸 𝑌
Couple GAN[Ming-Yu Liu, et al., NIPS, 2016]
UNIT[Ming-Yu Liu, et al., NIPS, 2017]

𝐸𝑁 𝑋
𝐷𝐸 𝑋image
image
image
image
The domain discriminator forces the output of 𝐸𝑁𝑋 and
𝐸𝑁𝑌 have the same distribution.
From 𝐸𝑁𝑋 or 𝐸𝑁𝑌
𝐷 𝑋
𝐷 𝑌
Discriminator
of X domain
Discriminator
of Y domain
Training
Domain
Discriminator
𝐸𝑁𝑋 and 𝐸𝑁𝑌 fool the
domain discriminator
[Guillaume Lample, et al., NIPS, 2017]

𝐸𝑁 𝑋
𝐷𝐸 𝑋image
image
image
image
𝐷 𝑋
𝐷 𝑌
Discriminator
of X domain
Discriminator
of Y domain
Training
Cycle Consistency:
Used in ComboGAN [Asha Anoosheh, et al., arXiv, 017]

𝐸𝑁 𝑋
𝐷𝐸 𝑋image
image
image
image
𝐷 𝑋
𝐷 𝑌
Discriminator
of X domain
Discriminator
of Y domain
Training
Semantic Consistency:
Used in DTN [Yaniv Taigman, et al., ICLR, 2017] and
XGAN [Amélie Royer, et al., arXiv, 2017]
To the same
latent space

Basic Components
EnvActor
Reward
Function
Video
Game
Go
Get 20 scores when
killing a monster
The rule
of GO
You cannot control

• Input of neural network: the observation of machine
represented as a vector or a matrix
• Output neural network : each action corresponds to a
neuron in output layer
…
…
NN as actor
pixels
fire
right
left
Score of an
action
0.7
0.2
0.1
Take the action
based on the
probability.
Neural network as Actor

Start with
observation 𝑠1 Observation 𝑠2 Observation 𝑠3
Example: Playing Video Game
Action 𝑎1: “right”
Obtain reward
𝑟1 = 0
Action 𝑎2 : “fire”
(kill an alien)
Obtain reward
𝑟2 = 5

Start with
observation 𝑠1 Observation 𝑠2 Observation 𝑠3
Example: Playing Video Game
After many turns
Action 𝑎 𝑇
Obtain reward 𝑟𝑇
Game Over
(spaceship destroyed)
This is an episode.
We want the total
reward be maximized.
Total reward:
𝑅 =
𝑡=1
𝑇
𝑟𝑡

Actor, Environment, Reward
𝜏 = 𝑠1, 𝑎1, 𝑠2, 𝑎2, ⋯ , 𝑠 𝑇, 𝑎 𝑇
Trajectory
Actor
𝑠1
𝑎1
Env
𝑠2
Env
𝑠1
𝑎1
Actor
𝑠2
𝑎2
Env
𝑠3
𝑎2
……

Reward Function → Discriminator
Reinforcement Learning v.s. GAN
Actor
𝑠1
𝑎1
Env
𝑠2
Env
𝑠1
𝑎1
Actor
𝑠2
𝑎2
Env
𝑠3
𝑎2
……
𝑅 𝜏 =
𝑡=1
𝑇
𝑟𝑡
Reward
𝑟1
Reward
𝑟2
“Black box”
You cannot use
backpropagation.
Actor → Generator Fixed
updatedupdated

Imitation Learning
We have demonstration of the expert.
Actor
𝑠1
𝑎1
Env
𝑠2
Env
𝑠1
𝑎1
Actor
𝑠2
𝑎2
Env
𝑠3
𝑎2
……
reward function is not available
𝜏1, 𝜏2, ⋯ , 𝜏 𝑁
Each 𝜏 is a trajectory
of the expert.
Self driving: record
human drivers
Robot: grab the
arm of robot

Inverse Reinforcement Learning
Reward
Function
Environment
Optimal
Actor
Inverse Reinforcement
Learning
Using the reward function to find the optimal actor.
Modeling reward can be easier. Simple reward
function can lead to complex policy.
Reinforcement
Learning
Expert
demonstration
of the expert
𝜏1, 𝜏2, ⋯ , 𝜏 𝑁

Framework of IRL
Expert 𝜋
Actor 𝜋
Obtain
Reward Function R
𝜏1, 𝜏2, ⋯ , 𝜏 𝑁
𝜏1, 𝜏2, ⋯ , 𝜏 𝑁
Find an actor based
on reward function R
By Reinforcement learning
𝑛=1
𝑁
𝑅 𝜏 𝑛 >
𝑛=1
𝑁
𝑅 𝜏
Reward function
→ Discriminator
Actor
→ Generator
Reward
Function R
The expert is always
the best.

𝜏1, 𝜏2, ⋯ , 𝜏 𝑁
GAN
IRL
G
D
High score for real,
low score for generated
Find a G whose output
obtains large score from D
𝜏1, 𝜏2, ⋯ , 𝜏 𝑁
Expert
Actor
Reward
Function
Larger reward for 𝜏 𝑛,
Lower reward for 𝜏
Find a Actor obtains
large reward

Concluding Remarks
Generation

Reference
• Generation
• Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-
Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio, Generative Adversarial
Networks, NIPS, 2014
• Sebastian Nowozin, Botond Cseke, Ryota Tomioka, “f-GAN: Training
Generative Neural Samplers using Variational Divergence Minimization”,
NIPS, 2016
• Martin Arjovsky, Soumith Chintala, Léon Bottou, Wasserstein GAN, arXiv,
2017
• Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, Aaron
Courville, Improved Training of Wasserstein GANs, NIPS, 2017
• Junbo Zhao, Michael Mathieu, Yann LeCun, Energy-based Generative
Adversarial Network, arXiv, 2016
• Mario Lucic, Karol Kurach, Marcin Michalski, Sylvain Gelly, Olivier Bousquet,
“Are GANs Created Equal? A Large-Scale Study”, arXiv, 2017
• Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec
Radford, Xi Chen Improved Techniques for Training GANs, NIPS, 2016
• Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler,
Sepp Hochreiter, GANs Trained by a Two Time-Scale Update Rule Converge
to a Local Nash Equilibrium, NIPS, 2017

Reference
• Generation
• Naveen Kodali, Jacob Abernethy, James Hays, Zsolt Kira, “On Convergence
and Stability of GANs”, arXiv, 2017
• Xiang Wei, Boqing Gong, Zixia Liu, Wei Lu, Liqiang Wang, Improving the
Improved Training of Wasserstein GANs: A Consistency Term and Its Dual
Effect, ICLR, 2018
• Takeru Miyato, Toshiki Kataoka, Masanori Koyama, Yuichi Yoshida, Spectral
Normalization for Generative Adversarial Networks, ICLR, 2018

Reference
• Generational Generation
• Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt
Schiele, Honglak Lee, Generative Adversarial Text to Image Synthesis, ICML,
2016
• Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, Alexei A. Efros, Image-to-Image
Translation with Conditional Adversarial Networks, CVPR, 2017
• Michael Mathieu, Camille Couprie, Yann LeCun, Deep multi-scale video
prediction beyond mean square error, arXiv, 2015
• Mehdi Mirza, Simon Osindero, Conditional Generative Adversarial Nets,
arXiv, 2014
• Takeru Miyato, Masanori Koyama, cGANs with Projection Discriminator,
ICLR, 2018
• Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei
Huang, Dimitris Metaxas, StackGAN++: Realistic Image Synthesis with
Stacked Generative Adversarial Networks, arXiv, 2017
• Augustus Odena, Christopher Olah, Jonathon Shlens, Conditional Image
Synthesis With Auxiliary Classifier GANs, ICML, 2017

Reference
• Generational Generation
• Yaroslav Ganin, Victor Lempitsky, Unsupervised Domain Adaptation by
Backpropagation, ICML, 2015
• Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario
Marchand, Domain-Adversarial Training of Neural Networks, JMLR, 2016

Reference
• Unsupervised Conditional Generation
• Jun-Yan Zhu, Taesung Park, Phillip Isola, Alexei A. Efros, Unpaired Image-to-
Image Translation using Cycle-Consistent Adversarial Networks, ICCV, 2017
• Zili Yi, Hao Zhang, Ping Tan, Minglun Gong, DualGAN: Unsupervised Dual
Learning for Image-to-Image Translation, ICCV, 2017
• Tomer Galanti, Lior Wolf, Sagie Benaim, The Role of Minimal Complexity
Functions in Unsupervised Learning of Semantic Mappings, ICLR, 2018
• Yaniv Taigman, Adam Polyak, Lior Wolf, Unsupervised Cross-Domain Image
Generation, ICLR, 2017
• Asha Anoosheh, Eirikur Agustsson, Radu Timofte, Luc Van Gool, ComboGAN:
Unrestrained Scalability for Image Domain Translation, arXiv, 2017
• Amélie Royer, Konstantinos Bousmalis, Stephan Gouws, Fred Bertsch, Inbar
Mosseri, Forrester Cole, Kevin Murphy, XGAN: Unsupervised Image-to-
Image Translation for Many-to-Many Mappings, arXiv, 2017

Reference
• Unsupervised Conditional Generation
• Guillaume Lample, Neil Zeghidour, Nicolas Usunier, Antoine Bordes, Ludovic
Denoyer, Marc'Aurelio Ranzato, Fader Networks: Manipulating Images by
Sliding Attributes, NIPS, 2017
• Taeksoo Kim, Moonsu Cha, Hyunsoo Kim, Jung Kwon Lee, Jiwon Kim,
Learning to Discover Cross-Domain Relations with Generative Adversarial
Networks, ICML, 2017
• Ming-Yu Liu, Oncel Tuzel, “Coupled Generative Adversarial Networks”, NIPS,
2016
• Ming-Yu Liu, Thomas Breuel, Jan Kautz, Unsupervised Image-to-Image
Translation Networks, NIPS, 2017
• Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim,
Jaegul Choo, StarGAN: Unified Generative Adversarial Networks for Multi-
Domain Image-to-Image Translation, arXiv, 2017

Part II: Speech Signal
Processing

Outline of Part II
Speech Signal Generation
• Speech enhancement
• Postfilter, speech synthesis, voice conversion
Speech Signal Recognition
• Speech recognition
• Speaker recognition
• Speech emotion recognition
• Lip reading
Conclusion

Speech Signal Generation (Regression Task)
G Output
Objective function
Paired

Speech, Speaker, Emotion Recognition and Lip-reading
(Classification Task)
Output
label
Clean data
E
G
𝒚
𝒙
Emb.
Noisy data
𝒙
𝒛 = 𝑔( 𝒙)
𝑔(∙)
ℎ(∙)
Accented
speech
𝒙
Channel
distortion
𝒙
Acoustic Mismatch

Speech Enhancement
• Typical objective function
• Typical objective function
 Mean square error (MSE) [Xu et al., TASLP 2015], L1 [Pascual et al., Interspeech
2017], likelihood [Chai et al., MLSP 2017], STOI [Fu et al., TASLP 2018].
Enhancing
G Output
Objective function
 GAN is used as a new objective function to estimate the parameters in G.
Model structures of G: DNN [Wang et al. NIPS 2012; Xu et al., SPL 2014], DDAE
[Lu et al., Interspeech 2013], RNN (LSTM) [Chen et al., Interspeech 2015;
Weninger et al., LVA/ICA 2015], CNN [Fu et al., Interspeech 2016].

Speech Enhancement
• Speech enhancement GAN (SEGAN) [Pascual et al., Interspeech 2017]
z

Table 1: Objective evaluation results. Table 2: Subjective evaluation results.
Fig. 1: Preference test results.
Speech Enhancement (SEGAN)
SEGAN yields better speech enhancement results than Noisy and Wiener.
• Experimental results

• Pix2Pix [Michelsanti et al., Interpsech 2017]
D Scalar
Clean
Noisy
(Fake/Real)
Output
Noisy
G
Noisy Output Clean
Speech Enhancement

Fig. 2: Spectrogram comparison of Pix2Pix with baseline methods.
Speech Enhancement (Pix2Pix)
• Spectrogram analysis
Pix2Pix outperforms STAT-MMSE and is competitive to DNN SE.
NG-DNN STAT-MMSE
Noisy Clean NG-Pix2Pix

Table 3: Objective evaluation results.
Speech Enhancement (Pix2Pix)
• Objective evaluation and speaker verification test
Table 4: Speaker verification results.
1. From the objective evaluations, Pix2Pix outperforms Noisy and
MMSE and is competitive to DNN SE.
2. From the speaker verification results, Pix2Pix outperforms the
baseline models when the clean training data is used.

• Frequency-domain SEGAN (FSEGAN) [Donahue et al., ICASSP 2018]
D Scalar
Clean
Noisy
(Fake/Real)
Output
Noisy
G
Noisy Output Clean
Speech Enhancement

Fig. 2: Spectrogram comparison of FSEGAN with L1-trained method.
Speech Enhancement (FSEGAN)
FSEGAN reduces both additive noise and reverberant smearing.

Table 5: WER (%) of SEGAN and FSEGAN. Table 6: WER (%) of FSEGAN with retrain.
Speech Enhancement (FSEGAN)
• ASR results
1. From Table 5, (1) FSEGAN improves recognition results for ASR-Clean.
(2) FSEGAN outperforms SEGAN as front-ends.
2. From Table 6, (1) Hybrid Retraining with FSEGAN outperforms Baseline;
(2) FSEGAN retraining slightly underperforms L1–based retraining.

= 𝐸𝒔 𝑓𝑎𝑘𝑒
log(1 − 𝑫 𝑺 𝒔 𝑓𝑎𝑘𝑒, 𝜃 )
+𝐸 𝒏 𝑓𝑎𝑘𝑒
log(1 − 𝑫 𝑵 𝒏 𝑓𝑎𝑘𝑒, 𝜃 )
• Adversarial training based mask estimation (ATME)
[Higuchi et al., ASRU 2017]
Speech Enhancement
Estimated
speech
𝑫 𝑵𝑫 𝑺
𝑮 𝑴𝒂𝒔𝒌
Clean speech
data
Noise data
Noisy data
Estimated
noise
True noiseTrue speech
Noisy speech
True or FakeTrue or Fake
𝑉 𝑀𝑎𝑠𝑘
oror

Fig. 3: Spectrogram comparison of (a) noisy; (b) MMSE with
supervision; (c) ATMB without supervision.
Speech Enhancement (ATME)
The proposed adversarial training mask estimation can
capture speech/noise signals without supervised data.
Speech mask Noise mask 𝑀𝑓,𝑡
𝑛

 The estimated mask parameters are used to compute spatial covariance
matrix for MVDR beamformer.
 𝑠𝑓,𝑡= 𝐰𝑓
H
𝐲 𝑓,𝑡 , where 𝑠𝑓,𝑡 is the enhanced signal, and 𝐲 𝑓,𝑡 denotes the
observation of M microphones, 𝑓 and 𝑡 are frequency and time indices;
𝐰𝑓 denotes the beamformer coefficient.
 The MVDR solves 𝐰𝑓 by: 𝐰𝑓=
(𝑅 𝑓
𝑠+𝑛
)−1 𝐡 𝑓
𝐡 𝑓
H
(𝑅 𝑓
𝑠+𝑛
)−1 𝐡 𝑓
 To estimate 𝐡 𝑓, the spatial covariance matrix of the target signal, 𝑅𝑓
𝑠
, is
computed by : 𝑅𝑓
𝑠
= 𝑅𝑓
𝑠+𝑛
－𝑅𝑓
𝑛
, where 𝑅𝑓
𝑛
=
𝑀 𝑓,𝑡
𝑛
𝐲 𝑓,𝑡 𝐲 𝑓,𝑡
H
𝑓,𝑡 𝑀 𝑓,𝑡
𝑛 , 𝑀𝑓,𝑡
𝑛
was
computed by AT.
• Mask-based beamformer for robust ASR

Table 7: WERs (%) for the development and evaluation sets.
• ASR results
1. ATME provides significant improvements over Unprocessed.
2. Unsupervised ATME slightly underperforms supervised MMSE.

𝐺𝑆→𝑇 𝐺 𝑇→𝑆
𝐷 𝑇
Scalar: belongs to
domain T or not
𝐺 𝑇→𝑆 𝐺𝑆→𝑇
𝐷𝑆
Scalar: belongs to
domain S or not
Speech Enhancement (AFT)
• Cycle-GAN-based acoustic feature transformation (AFT)
[Mimura et al., ASRU 2017]
𝑉𝐹𝑢𝑙𝑙 = 𝑉𝐺𝐴𝑁 𝐺 𝑋→𝑌, 𝐷 𝑌 ＋𝑉𝐺𝐴𝑁 𝐺 𝑋→𝑌, 𝐷 𝑌
＋𝜆 𝑉𝐶𝑦𝑐(𝐺 𝑋→𝑌, 𝐺 𝑌→𝑋)
Noisy Enhanced Noisy
Clean Syn. Noisy Clean

• ASR results on noise robustness and style adaptation
Table 8: Noise robust ASR. Table 9: Speaker style adaptation.
1. 𝐺 𝑇→𝑆 can transform acoustic features and effectively improve
ASR results for both noisy and accented speech.
2. 𝐺𝑆→𝑇 can be used for model adaptation and effectively improve
ASR results for noisy speech.
S: Clean; 𝑇: Noisy JNAS: Read; CSJ-SPS: Spontaneous (relax);
CSJ-APS: Spontaneous (formal);
Speech Enhancement (AFT)

• Postfilter for synthesized or transformed speech
 Conventional postfilter approaches for G estimation include global variance
(GV) [Toda et al., IEICE 2007], variance scaling (VS) [Sil’en et al., Interpseech
2012], modulation spectrum (MS) [Takamichi et al., ICASSP 2014],DNN with
MSE criterion [Chen et al., Interspeech 2014; Chen et al., TASLP 2015].
 GAN is used a new objective function to estimate the parameters in G.
Postfilter
Synthesized
spectral texture
Natural
spectral texture
G Output
Objective function
Speech
synthesizer
Voice
conversion
Speech
enhancement

• GAN postfilter [Kaneko et al., ICASSP 2017]
 Traditional MMSE criterion results in statistical averaging.
 GAN is used as a new objective function to estimate the parameters in G.
 The proposed work intends to further improve the naturalness of
synthesized speech or parameters from a synthesizer.
Postfilter
Synthesized
Mel cepst. coef.
Natural
Mel cepst. coef.
D
Nature
or
Generated
Generated
Mel cepst. coef.
G

Fig. 4: Spectrograms of: (a) NAT (nature); (b) SYN (synthesized); (c) VS (variance
scaling); (d) MS (modulation spectrum); (e) MSE; (f) GAN postfilters.
Postfilter (GAN-based Postfilter)
GAN postfilter reconstructs spectral texture similar to the natural one.

Fig. 5: Mel-cepstral trajectories (GANv:
GAN was applied in voiced part).
Fig. 6: Averaging difference in
modulation spectrum per Mel-
cepstral coefficient.
• Objective evaluations

Table 10: Preference score (%). Bold font indicates the numbers over 30%.
• Subjective evaluations
1. GAN postfilter significantly improves the synthesized speech.
2. GAN postfilter is effective particularly in voiced segments.
3. GANv outperforms GAN and is comparable to NAT.

Postfilter (GAN-postfilter-SFTF)
• GAN post-filter for STFT spectrograms [Kaneko et al.,
Interspeech 2017]
 GAN postfilter was applied on high-dimensional STFT spectrograms.
 The spectrogram was partitioned into N bands (each band overlaps its
neighboring bands).
 The GAN-based postfilter was trained for each band.
 The reconstructed spectrogram from each band was smoothly connected.

Fig. 7: Spectrograms of: (1) SYN, (2) GAN, (3) Original (NAT)
Postfilter (GAN-postfilter-SFTF)

Speech Synthesis
• Input: linguistic features; Output: speech parameters
𝒄𝒄
𝑮 𝑺𝑺
Natural
speech
parameters
Generated
speech
parameters
Linguistic
features
sp sp
Objective function
Minimum
generation error
(MGE), MSE
• Speech synthesis with anti-spoofing verification (ASV)
[Saito et al., ICASSP 2017]
𝐿 𝐷 𝒄, 𝒄 = 𝐿 𝐷,1 𝒄 + 𝐿 𝐷,0 𝒄
𝐿 𝐷,1 𝒄 = −
1
𝑇 𝑡=1
𝑇
log( 𝐷 𝒄 𝑡 )…NAT
𝐿 𝐷,0 𝒄 = −
1
𝑇 𝑡=1
𝑇
log(1 − 𝐷 𝒄 𝑡 )…SYN
𝐿 𝒄, 𝒄 = 𝐿 𝐺 𝒄, 𝒄 + 𝜔 𝐷
𝐸 𝐿 𝐺
𝐸 𝐿 𝐷
𝐿 𝐷,1 𝒄
Minimum generation error (MGE)
with adversarial loss.
𝒄𝒄
𝑮 𝑺𝑺
Natural
speech
parameters
Generated
speech
parameters
Gen.
Nature
𝑫 𝑨𝑺𝑽𝝓(∙)
Linguistic
features
sp sp
MGE

Fig. 8: Averaged GVs of MCCs.
Speech Synthesis (ASV)
• Objective and subjective evaluations
1. The proposed algorithm generates MCCs similar to the natural ones.
Fig. 9: Scores of speech quality.
2. The proposed algorithm outperforms conventional MGE training.

• Speech synthesis with GAN (SS-GAN) [Saito et al., TASLP 2018]
𝐿 𝐷 𝒄, 𝒄 = 𝐿 𝐷,1 𝒄 + 𝐿 𝐷,0 𝒄
𝐿 𝐷,1 𝒄 = −
1
𝑇 𝑡=1
𝑇
log( 𝐷 𝒄 𝑡 )…NAT
𝐿 𝐷,0 𝒄 = −
1
𝑇 𝑡=1
𝑇
log(1 − 𝐷 𝒄 𝑡 )…SYN
𝐿 𝒄, 𝒄 = 𝐿 𝐺 𝒄, 𝒄 + 𝜔 𝐷
𝐸 𝐿 𝐺
𝐸 𝐿 𝐷
𝐿 𝐷,1 𝒄
Minimum generation error (MGE)
with adversarial loss.
𝒄𝒄
Speech Synthesis
𝑮 𝑺𝑺
Natural
speech
parameters
Generated
speech
parameters
Gen.
Nature
𝑫𝝓(∙)
Linguistic
features
𝑫 𝑨𝑺𝑽
sp, f0, duration sp, f0, duration
MGE

Fig. 11: Scores of speech quality
(sp and F0).
.
Speech Synthesis (SS-GAN)
Fig. 10: Scores of speech quality (sp).
The proposed algorithm works for both spectral parameters and F0.

Speech Synthesis
G
Natural
speech
parameters
Generated
speech
parameters
Gen.
Nature
𝑫
Acoustic
features
• Speech synthesis with GAN glottal waveform model
(GlottGAN) [Bollepalli et al., Interspeech 2017]
Glottal waveform Glottal waveform

• Objective evaluations
Speech Synthesis (GlottGAN)
The proposed GAN-based approach can generate glottal
waveforms similar to the natural ones.
Fig. 12: Glottal pulses generated by GANs.
G, D: DNN
G, D: conditional DNN
G, D: Deep CNN
G, D: Deep CNN + LS loss

• Speech synthesis with GAN & multi-task learning
(SS-GAN-MTL) [Yang et al., ASRU 2017]
𝒙𝒙
Speech Synthesis
𝑮 𝑺𝑺
Natural
speech
parameters
Generated
speech
parameters
Gen.
Nature
𝑫
Noise
𝒛
𝒚
𝑉𝐺𝐴𝑁 𝐺, 𝐷 = 𝐸 𝒙~𝑝 𝑑𝑎𝑡𝑎(𝑥) 𝑙𝑜𝑔𝐷 𝒙|𝒚
+ 𝐸𝒛~𝑝 𝑧, log(1 − 𝐷 𝐺 𝒛|𝒚 |𝒚
𝑉𝐿2 𝐺 = 𝐸𝒛~𝑝 𝑧,[𝐺 𝒛|𝒚 − 𝒙]2
Linguistic
features
MSE

• Speech synthesis with GAN & multi-task learning
(SS-GAN-MTL) [Yang et al., ASRU 2017]
Speech Synthesis (SS-GAN-MTL)
𝒙
𝑮 𝑺𝑺
Natural
speech
parameters
Generated
speech
parameters
Gen.
Nature
𝑫
Noise
𝒛
𝒚
Linguistic
features
MSE
𝒙
𝑫 𝑪𝑬
𝑉𝐺𝐴𝑁 𝐺, 𝐷 = 𝐸 𝒙~𝑝 𝑑𝑎𝑡𝑎(𝑥) 𝑙𝑜𝑔𝐷 𝒙|𝒚
+ 𝐸𝒛~𝑝 𝑧, log(1 − 𝐷 𝐺 𝒛|𝒚 |𝒚
𝒙𝒙
𝑮 𝑺𝑺
Natural
speech
parameters
Generated
speech
parameters
Gen.
Nature
Noise
𝒛
𝒚
Linguistic
features
MSE
𝑫 𝑪𝑬
𝑉𝐺𝐴𝑁 𝐺, 𝐷 = 𝐸 𝒙~𝑝 𝑑𝑎𝑡𝑎(𝑥) 𝑙𝑜𝑔𝐷 𝐶𝐸 𝒙|𝒚, 𝑙𝑎𝑏𝑒𝑙
+ 𝐸𝒛~𝑝 𝑧, log(1 − 𝐷 𝐶𝐸 𝐺 𝒛|𝒚 |𝒚, 𝑙𝑎𝑏𝑒𝑙
CE
True label

Table 11: Objective evaluation results. Fig. 13: The preference score (%).
Speech Synthesis (SS-GAN-MTL)
1. From objective evaluations, no remarkable difference is observed.
2. From subjective evaluations, GAN outperforms BLSTM and ASV,
while GAN-PC underperforms GAN.

• Convert (transform) speech from source to target
 Conventional VC approaches include Gaussian mixture model (GMM) [Toda
et al., TASLP 2007], non-negative matrix factorization (NMF) [Wu et al., TASLP
2014; Fu et al., TBME 2017], locally linear embedding (LLE) [Wu et al.,
Interspeech 2016], restricted Boltzmann machine (RBM) [Chen et al., TASLP
2014], feed forward NN [Desai et al., TASLP 2010], recurrent NN (RNN)
[Nakashika et al., Interspeech 2014].
Voice Conversion
G Output
Objective function
Target
speaker
Source
speaker

• VAW-GAN [Hsu et al., Interspeech 2017]
Conventional MMSE approaches often encounter the “over-smoothing” issue.
 GAN is used a new objective function to estimate G.
 The goal is to increase the naturalness, clarity, similarity of converted speech.
Voice Conversion
D
Real
or
Fake
G
Target
speaker
Source
speaker
𝑉 𝐺, 𝐷 = 𝑉𝐺𝐴𝑁 𝐺, 𝐷 + 𝜆 𝑉𝑉𝐴𝐸 𝒙|𝒚

Fig. 15: MOS on naturalness.Fig. 14: The spectral envelopes.
Voice Conversion (VAW-GAN)
VAW-GAN outperforms VAE in terms of objective and subjective
evaluations with generating more structured speech.

Voice Conversion
• Sequence-to-sequence VC with learned similarity metric
(LSM) [Kaneko et al., Interspeech 2017]
D
Real
or
Fake
𝑪
Target
speaker
Source
speaker
𝑉 𝐶, 𝐺, 𝐷 = 𝑉𝑆𝑉𝐶
𝐷 𝑙
𝐶, 𝐷 + 𝑉𝐺𝐴𝑁 (𝐶, 𝐺, 𝐷)
𝑮Noise
𝒛
𝑉𝑆𝑉𝐶
𝐷𝑙
𝐶, 𝐷 =
1
𝑀 𝑙
𝐷𝑙 𝒚 − 𝐷𝑙 𝐶(𝒙 ) 2
𝒚
Similarity metric

Fig. 16: Comparison of MCCs (upper) and STFT spectrograms (lower).
Voice Conversion (LSM)
Source Target FVC MSE(S2S) LSM(S2S)
The spectral textures of LSM are more similar to the target ones.

Table 12: Preference scores for naturalness.
Table 12: Preference scores for clarity.
Fig. 17: Similarity of TGT and SRC with VCs.
Voice Conversion (LSM)
LSM outperforms FVC and MSE in terms of subjective evaluations.
Target
speaker
Source
speaker

• CycleGAN-VC [Kaneko et al., arXiv 2017]
• used a new objective function to estimate G
𝑉𝐹𝑢𝑙𝑙 = 𝑉𝐺𝐴𝑁 𝐺 𝑋→𝑌, 𝐷 𝑌 ＋𝑉𝐺𝐴𝑁 𝐺 𝑋→𝑌, 𝐷 𝑌
＋𝜆 𝑉𝐶𝑦𝑐(𝐺 𝑋→𝑌, 𝐺 𝑌→𝑋)
Voice Conversion
𝑮 𝑺→𝑻 𝐺 𝑇→𝑆
𝑫 𝑻
Scalar: belongs to
domain T or not
Scalar: belongs to
domain S or not
𝐺 𝑇→𝑆 𝑮 𝑺→𝑻
𝑫 𝑺
Target Syn. Source Target
Source Syn. Target Source

Fig. 18: MOS for naturalness.
Fig. 19: Similarity of to source and
to target speakers. S: Source;
T:Target; P: Proposed; B:Baseline
Voice Conversion (CycleGAN-VC)
1. The proposed method uses non-parallel data.
2. For naturalness, the proposed method outperforms baseline.
3. For similarity, the proposed method is comparable to the baseline.
Target
speaker
Source
speaker

• Multi-target VC [Chou et al., arxiv 2018]
𝑒𝑛𝑐(𝒙)
𝒙
Voice Conversion
C
𝑬nc Dec
𝒚
𝒚 𝒚′····
𝑒𝑛𝑐(𝒙)
𝑬nc Dec
𝒚"
𝑮
𝒚"
D+C
Real
data
𝒙 𝑑𝑒𝑐(𝑒𝑛𝑐 𝒙 , 𝒚) 𝑑𝑒𝑐(𝑒𝑛𝑐 𝒙 , 𝒚′)
 Stage-1
 Stage-2
F/R
ID
···

Voice Conversion (Multi-target VC)
Fig. 20: Preference test results
1. The proposed method uses non-parallel data.
2. The multi-target VC approach outperforms one-stage only.
3. The multi-target VC approach is comparable to Cycle-GAN-VC in
terms of the naturalness and the similarity.

Speech Recognition
• Adversarial multi-task learning (AMT)
[Shinohara Interspeech 2016]
Output 1
Senone
Input
Acoustic feature
E
G𝑉𝑦
Output 2
Domain
D 𝑉𝑧
𝒛
𝒚
𝒙
GRL
𝑉𝑦=− 𝑖 log 𝑃(𝑦𝑖|𝑥𝑖; 𝜃 𝐸, 𝜃 𝐺)
𝑉𝑧=− 𝑖 log 𝑃(𝑧𝑖|𝑥𝑖; 𝜃 𝐸, 𝜃 𝐷)
𝜃 𝐺 ← 𝜃 𝐺 − ϵ
𝜕𝑉𝑦
𝜕𝜃 𝐺
𝜃 𝐸 ← 𝜃 𝐸 − ϵ
𝜕𝑉𝑦
𝜕𝜃 𝐺
𝜃 𝐷 ← 𝜃 𝐷 − ϵ
𝜕𝑉𝑧
𝜕𝜃 𝐷
Model update
Max
classification
accuracy
Max domain
accuracy
Max classification accuracy
Objective function
+𝛼
𝜕𝑉𝑧
𝜕𝜃 𝐺
and Min domain accuracy

• ASR results in known (k) and unknown (unk)
noisy conditions
Speech Recognition (AMT)
Table 13: WER of DNNs with single-task learning (ST) and AMT.
The AMT-DNN outperforms ST-DNN with yielding lower WERs.

Speech Recognition
• Domain adversarial training for accented ASR (DAT)
[Sun et al., ICASSP2018]
Output 2
Domain
Output 1
Senone
Input
Acoustic feature
E
G D
GRL
𝑉𝑧𝑉𝑦
𝒛
𝒚
𝒙
𝜃 𝐺 ← 𝜃 𝐺 − ϵ
𝜕𝑉𝑦
𝜕𝜃 𝐺
𝜃 𝐸 ← 𝜃 𝐸 − ϵ
𝜕𝑉𝑦
𝜕𝜃 𝐺
𝜃 𝐷 ← 𝜃 𝐷 − ϵ
𝜕𝑉𝑧
𝜕𝜃 𝐷
Model update
Max
classification
accuracy
Max domain
accuracy
Objective function
+𝛼
𝜕𝑉𝑧
𝜕𝜃 𝐺

• ASR results on accented speech
Speech Recognition (DAT)
1. With labeled transcriptions, ASR performance notably improves.
Table 14: WER of the baseline and adapted model.
2. DAT is effective in learning features invariant to domain differences
with and without labeled transcriptions.
STD: standard speech

Speech Recognition
• Robust ASR using GAN enhancer (GAN-Enhancer)
[Sriram et al., arXiv 2017]
𝐻 ℎ 𝒛 , 𝐲 + λ
𝒛 − 𝒛 1
𝒛 1 + 𝒛 1 + 𝜖
Cross entropy with L1 Enhancer:
Output
label
Clean data
E
GL1
𝒚
𝒙
Emb.
Noisy data
E
𝒙
Emb.
𝒛 = 𝑔(𝒙) 𝒛 = 𝑔( 𝒙)
𝑔(∙)𝑔(∙)
ℎ(∙)
Cross entropy with GAN Enhancer:
𝐻 ℎ 𝒛 , 𝐲 + λ 𝑉𝑎𝑑𝑣 (𝑔(𝒙) , 𝑔( 𝒙))
D

• ASR results on far-field speech:
Speech Recognition (GAN-Enhancer)
GAN Enhancer outperforms the Augmentation and L1-
Enhancer approaches on far-field speech.
Fig. 15: WER of GAN enhancer and the baseline methods.

Speaker Recognition
• Domain adversarial neural network (DANN)
[Wang et al., ICASSP 2018]
DANN
DANN
Pre-
processing
Pre-
processing
Scoring
Enroll
i-vector
Test
i-vector
Output 2
Domain
Output 1
Speaker
ID
Input
Acoustic feature
E
G D
GRL
𝑉𝑧𝑉𝑦
𝒛
𝒚
𝒙

• Recognition results of domain mismatched conditions
Table 16: Performance of DAT and the state-of-the-art methods.
Speaker Recognition (DANN)
The DAT approach outperforms other methods with
achieving lowest EER and DCF scores.

Emotion Recognition
• Adversarial AE for emotion recognition (AAE-ER)
[Sahu et al., Interspeech 2017]
AE with GAN :
𝐻 ℎ 𝒛 , 𝒙 + λ 𝑉𝐺𝐴𝑁 (𝒒, 𝑔(𝒙))
E
D
𝒙
Emb.
Syn.
𝒛 = 𝑔(𝒙)
𝑔(∙)
ℎ(∙)
𝒒
𝒙
G
The distribution of code vectors

• Recognition results of domain mismatched conditions:
Table 18: Classification results on real and synthesized features.
Emotion Recognition (AAE-ER)
Table 17: Classification results on different systems.
1. AAE alone could not yield performance improvements.
2. Using synthetic data from AAE can yield higher UAR.
Original
Training
data

Lip-reading
• Domain adversarial training for lip-reading (DAT-LR)
[Wand et al., arXiv 2017]
Output 1
Words
E
G𝑉𝑦
Output 2
Speaker
D
GRL
𝑉𝑧
𝒛
𝒚
𝒙
𝜃 𝐺 ← 𝜃 𝐺 − ϵ
𝜕𝑉𝑦
𝜕𝜃 𝐺
𝜃 𝐸 ← 𝜃 𝐸 − ϵ
𝜕𝑉𝑦
𝜕𝜃 𝐺
𝜃 𝐷 ← 𝜃 𝐷 − ϵ
𝜕𝑉𝑧
𝜕𝜃 𝐷
Model update
Max
classification
accuracy
Max domain
accuracy
Objective function
+𝛼
𝜕𝑉𝑧
𝜕𝜃 𝐺
~80% WAC

• Recognition results of speaker mismatched conditions
Lip-reading (DAT-LR)
Table 19: Performance of DAT and the baseline.
The DAT approach notably enhances the recognition
accuracies in different conditions.

More GANs in Speech
Diagnosis of autism spectrum
Jun Deng, Nicholas Cummins, Maximilian Schmitt, Kun Qian, Fabien Ringeval, and Björn Schuller, Speech-based
Diagnosis of Autism Spectrum Condition by Generative Adversarial Network Representations, ACM DH, 2017.
Emotion recognition
Jonathan Chang, and Stefan Scherer, Learning Representations of Emotional Speech with Deep Convolutional
Generative Adversarial Networks, ICASSP, 2017.
Robust ASR
Dmitriy Serdyuk, Kartik Audhkhasi, Philémon Brakel, Bhuvana Ramabhadran, Samuel Thomas, and Yoshua Bengio,
Invariant Representations for Noisy Speech Recognition, arXiv, 2016.
Speaker verification
Hong Yu, Zheng-Hua Tan, Zhanyu Ma, and Jun Guo, Adversarial Network Bottleneck Features for Noise Robust
Speaker Verification, arXiv, 2017.

References
Speech enhancement (conventional methods)
• Yuxuan Wang and Deliang Wang, Cocktail Party Processing via Structured Prediction, NIPS 2012.
• Yong Xu, Jun Du, Li-Rong Dai, and Chin-Hui Lee, An Experimental Study on Speech Enhancement Based on Deep
Neural Networks," IEEE SPL, 2014.
• Yong Xu, Jun Du, Li-Rong Dai, and Chin-Hui Lee, A Regression Approach to Speech Enhancement Based on Deep
Neural Networks, IEEE/ACM TASLP, 2015.
• Xugang Lu, Yu Tsao, Shigeki Matsuda, Chiori Hori, Speech Enhancement Based on Deep Denoising Autoencoder,
Interspeech 2012.
• Zhuo Chen, Shinji Watanabe, Hakan Erdogan, John R. Hershey, Integration of Speech Enhancement and
Recognition Using Long-short term Memory Recurrent Neural Network, Interspeech 2015.
• Felix Weninger, Hakan Erdogan, Shinji Watanabe, Emmanuel Vincent, Jonathan Le Roux, John R. Hershey, and
Bjorn Schuller, Speech Enhancement with LSTM Recurrent Neural Networks and Its Application to Noise-robust
ASR, LVA/ICA, 2015.
• Szu-Wei Fu, Yu Tsao, and Xugang Lu, SNR-aware Convolutional Neural Network Modeling for Speech
Enhancement, Interspeech, 2016.
• Szu-Wei Fu, Yu Tsao, Xugang Lu, and Hisashi Kawai, End-to-end Waveform Utterance Enhancement for Direct
Evaluation Metrics Optimization by Fully Convolutional Neural Networks, arXiv, IEEE/ACM TASLP, 2018.
Speech enhancement (GAN-based methods)
• Pascual Santiago, Bonafonte Antonio, and Serra Joan, SEGAN: Speech Enhancement Generative Adversarial
Network, Interspeech, 2017.
• Michelsanti Daniel, and Zheng-Hua Tan, Conditional Generative Adversarial Networks for Speech Enhancement
and Noise-robust Speaker Verification, Interspeech, 2017.
• Donahue Chris, Li Bo, and Prabhavalkar Rohit, Exploring Speech Enhancement with Generative Adversarial
Networks for Robust Speech Recognition, ICASPP, 2018.
• Higuchi Takuya, Kinoshita Keisuke, Delcroix Marc, and Nakatani Tomohiro, Adversarial Training for Data-driven
Speech Enhancement without Parallel Corpus, ASRU, 2017.

Postfilter (conventional methods)
• Toda Tomoki, and Tokuda Keiichi, A Speech Parameter Generation Algorithm Considering Global Variance for
HMM-based Speech Synthesis, IEICE Trans. Inf. Syst., 2007.
• Sil’en Hanna, Helander Elina, Nurminen Jani, and Gabbouj Moncef, Ways to Implement Global Variance in
Statistical Speech Synthesis, Interspeech, 2012.
• Takamichi Shinnosuke, Toda Tomoki, Neubig Graham, Sakti Sakriani, and Nakamura Satoshi, A Postfilter to
Modify the Modulation Spectrum in HMM-based Speech Synthesis, ICASSP, 2014.
• Ling-Hui Chen, Tuomo Raitio, Cassia Valentini-Botinhao, Junichi Yamagishi, and Zhen-Hua Ling, DNN-based
Stochastic Postfilter for HMM-based Speech Synthesis, Interspeech, 2014.
• Ling-Hui Chen, Tuomo Raitio, Cassia Valentini-Botinhao, Zhen-Hua Ling, and Junichi Yamagishi, A Deep
Generative Architecture for Postfiltering in Statistical Parametric Speech Synthesis, IEEE/ACM TASLP, 2015.
Postfilter (GAN-based methods)
• Kaneko Takuhiro, Kameoka Hirokazu, Hojo Nobukatsu, Ijima Yusuke, Hiramatsu Kaoru, and Kashino Kunio,
Generative Adversarial Network-based Postfilter for Statistical Parametric Speech Synthesis, ICASSP, 2017.
• Kaneko Takuhiro, Takaki Shinji, Kameoka Hirokazu, and Yamagishi Junichi, Generative Adversarial Network-
Based Postfilter for STFT Spectrograms, Interspeech, 2017.
• Saito Yuki, Takamichi Shinnosuke, and Saruwatari Hiroshi, Training Algorithm to Deceive Anti-spoofing
Verification for DNN-based Speech Synthesis, ICASSP, 2017.
• Saito Yuki, Takamichi Shinnosuke, Saruwatari Hiroshi, Saito Yuki, Takamichi Shinnosuke, and Saruwatari Hiroshi,
Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks, IEEE/ACM TASLP, 2018.
• Bajibabu Bollepalli, Lauri Juvela, and Alku Paavo, Generative Adversarial Network-based Glottal Waveform
Model for Statistical Parametric Speech Synthesis, Interspeech, 2017.
• Yang Shan, Xie Lei, Chen Xiao, Lou Xiaoyan, Zhu Xuan, Huang Dongyan, and Li Haizhou, Statistical Parametric
Speech Synthesis Using Generative Adversarial Networks Under a Multi-task Learning Framework, ASRU, 2017.
References

VC (conventional methods)
• Toda Tomoki, Black Alan W, and Tokuda Keiichi, Voice Conversion Based on Maximum Likelihood Estimation of
Spectral Parameter Trajectory, IEEE/ACM TASLP, 2007.
• Ling-Hui Chen, Zhen-Hua Ling, Li-Juan Liu, and Li-Rong Dai, Voice Conversion Using Deep Neural Networks with
Layer-wise Generative Training, IEEE/ACM TASLP, 2014.
• Srinivas Desai, Alan W Black, B. Yegnanarayana, and Kishore Prahallad, Spectral mapping Using artificial Neural
Networks for Voice Conversion, IEEE/ACM TASLP, 2010.
• Nakashika Toru, Takiguchi Tetsuya, Ariki Yasuo, High-order Sequence Modeling Using Speaker-dependent
Recurrent Temporal Restricted Boltzmann Machines for Voice Conversion, Interspeech, 2014.
• Takuhiro Kaneko, Hirokazu Kameoka, Kaoru Hiramatsu, and Kunio Kashino, Sequence-to-sequence Voice
Conversion with Similarity Metric Learned Using Generative Adversarial Networks, Interspeech, 2017.
• Zhizheng Wu, Tuomas Virtanen, Eng-Siong Chng, and Haizhou Li, Exemplar-based Sparse Representation with
Residual Compensation for Voice Conversion, IEEE/ACM TASLP, 2014.
• Szu-Wei Fu, Pei-Chun Li, Ying-Hui Lai, Cheng-Chien Yang, Li-Chun Hsieh, and Yu Tsao, Joint Dictionary Learning-
based Non-negative Matrix Factorization for Voice Conversion to Improve Speech Intelligibility After Oral
Surgery, IEEE TBME, 2017.
• Yi-Chiao Wu, Hsin-Te Hwang, Chin-Cheng Hsu, Yu Tsao, and Hsin-Min Wang, Locally Linear Embedding for
Exemplar-based Spectral Conversion, Interspeech, 2016.
VC (GAN-based methods)
• Chin-Cheng Hsu, Hsin-Te Hwang, Yi-Chiao Wu, Yu Tsao, and Hsin-Min Wang Voice Conversion from Unaligned
Corpora Using Variational Autoencoding Wasserstein Generative Adversarial Networks, arXiv, 2017.
• Takuhiro Kaneko, Hirokazu Kameoka, Kaoru Hiramatsu, and Kunio Kashino, Sequence-to-sequence Voice
Conversion with Similarity Metric Learned Using Generative Adversarial Networks, Interspeech, 2017.
• Takuhiro Kaneko, and Hirokazu Kameoka. Parallel-Data-Free Voice Conversion Using Cycle-Consistent Adversarial
Networks, arXiv, 2017.
References

ASR
• Yusuke Shinohara, Adversarial Multi-Task Learning of Deep Neural Networks for Robust Speech Recognition.
Interspeech, 2016.
• Sining Sun, Ching-Feng Yeh, Mei-Yuh Hwang, Mari Ostendorf, and Lei Xie, Domain Adversarial Training for
Accented Speech Recognition, ICASSP, 2018
• Masato Mimura, Shinsuke Sakai, and Tatsuya Kawahara, Cross-domain Speech Recognition Using Nonparallel
Corpora with Cycle-consistent Adversarial Networks, ASRU, 2017.
• Anuroop Sriram, Heewoo Jun, Yashesh Gaur, and Sanjeev Satheesh, Robust Speech Recognition Using
Generative Adversarial Networks, arXiv, 2017.
Speaker recognition
• Qing Wang, Wei Rao, Sining Sun, Lei Xie, Eng Siong Chng, and Haizhou Li, Unsupervised Domain Adaptation via
Domain Adversarial Training for Speaker Recognition, ICASSP, 2018.
Emotion recognition
• Saurabh Sahu, Rahul Gupta, Ganesh Sivaraman, Wael AbdAlmageed, and Carol Espy-Wilson, Adversarial Auto-
encoders for Speech Based Emotion Recognition. Interspeech, 2017.
Lipreading
• Michael Wand, and Jürgen Schmidhuber, Improving Speaker-Independent Lipreading with Domain-Adversarial
Training, arXiv, 2017.
References

• Saurabh Sahu, Rahul Gupta, Ganesh Sivaraman, Carol Espy-Wilson, Smoothing Model Predictions using
Adversarial Training Procedures for Speech Based Emotion Recognition
• Fuming Fang, Junichi Yamagishi, Isao Echizen, Jaime Lorenzo-Trueba, High-quality Nonparallel Voice Conversion
Based on Cycle-consistent Adversarial Network
• Lauri Juvela, Bajibabu Bollepalli, Xin Wang, Hirokazu Kameoka, Manu Airaksinen, Junichi Yamagishi, Paavo Alku,
Speech Waveform Synthesis from MFCC Sequences with Generative Adversarial Networks
• Zhong Meng, Jinyu Li, Yifan Gong, Biing-Hwang (Fred) Juang, Adversarial Teacher-Student Learning for
Unsupervised Domain Adaptation
• Zhong Meng, Jinyu Li, Zhuo Chen, Yong Zhao, Vadim Mazalov, Yifan Gong, Biing-Hwang (Fred) Juang, Speaker-
Invariant Training via Adversarial Learning
• Sen Li, Stephane Villette, Pravin Ramadas, Daniel Sinder, Speech Bandwidth Extension using Generative
Adversarial Networks
• Qing Wang, Wei Rao, Sining Sun, Lei Xie, Eng Siong Chng, Haizhou Li, Unsupervised Domain Adaptation via
Domain Adversarial Training for Speaker Recognition
• Hu Hu, Tian Tan, Yanmin Qian, Generative Adversarial Networks Based Data Augmentation for Noise Robust
Speech Recognition
• Yuki Saito, Shinnosuke Takamichi, Hiroshi Saruwatari, Text-to-speech Synthesis using STFT Spectra Based on Low-
/multi-resolution Generative Adversarial Networks
• Jiangyan Yi, Jianhua Tao, Zhengqi Wen, Ye Bai, Adversarial Multilingual Training for Low-resource Speech
Recognition
• Meet H. Soni, Neil Shah, Hemant A. Patil, Time-frequency Masking-based Speech Enhancement using Generative
Adversarial Network
• Taira Tsuchiya, Naohiro Tawara, Tetsuji Ogawa, Tetsunori Kobayashi, Speaker Invariant Feature Extraction for Zero-
resource Languages with Adversarial Learning
GANs in ICASSP 2018

• Jing Han, Zixing Zhang, Zhao Ren, Fabien Ringeval, Bjoern Schuller, Towards Conditional Adversarial Training
for Predicting Emotions from Speech
• Chenxing Li, Lei Zhu, Shuang Xu, Peng Gao, Bo Xu, CBLDNN-based Speaker-independent Speech Separation
via Generative Adversarial Training
• Anuroop Sriram, Heewoo Jun, Yashesh Gaur, Sanjeev Satheesh, Robust Speech Recognition using Generative
Adversarial Networks
• Cem Subakan, Paris Smaragdis, Generative Adversarial Source Separation,
• Ashutosh Pandey, Deliang Wang, On Adversarial Training and Loss Functions for Speech Enhancement
• Bin Liu, Shuai Nie, Yaping Zhang, Dengfeng Ke, Shan Liang, Wenju Liu, Boosting Noise Robustness of Acoustic
Model via Deep Adversarial Training
• Yang Gao, Rita Singh, Bhiksha Raj, Voice Impersonation using Generative Adversarial Networks
• Aditay Tripathi, Aanchan Mohan, Saket Anand, Maneesh Singh, Adversarial Learning of Raw Speech Features
for Domain Invariant Speech Recognition
• Zhe-Cheng Fan, Yen-Lin Lai, Jyh-Shing Jang, SVSGAN: Singing Voice Separation via Generative Adversarial
Network
• Santiago Pascual, Maruchan Park, Joan Serra, Antonio Bonafonte, Kang-Hun Ahn, Language and Noise
Transfer in Speech Enhancement Generative Adversarial Network
GANs in ICASSP 2018

A promising research direction and still has room for further
improvements in the speech signal processing domain
Thank You Very Much
Tsao, Yu Ph.D.
yu.tsao@citi.sinica.edu.tw
https://www.citi.sinica.edu.tw/pages/yu.tsao/contact_zh.html

Part III: Natural
Language Processing

Outline of Part III
Conditional Sequence Generation
• RL (human feedback)
• GAN (discriminator feedback)
Unsupervised Conditional Sequence Generation
• Text Style Transfer
• Unsupervised Abstractive Summarization
• Unsupervised Translation

Generator
機器學習
Generator
Machine Learning
Generator
How are you?
How are you I am fine.
ASR Translation Chatbot
The generator is a typical seq2seq model.
With GAN, you can train seq2seq model in another way.

Review: Sequence-to-sequence
• Chat-bot as example
Encoder Decoder
Input sentence c
output
sentence x
Training
data:
A: How are you ?
B: I’m good.
…………
How are you ?
I’m good.
Generator
Output: Not bad I’m John.
Maximize
likelihood
Training Criterion
Human better
better

Outline of Part III
Improving Supervised Seq-to-seq Model
Unsupervised Seq-to-seq Model

Introduction
• Machine obtains feedback from user
• Chat-bot learns to maximize the expected reward
https://image.freepik.com/free-vector/variety-
of-human-avatars_23-2147506285.jpg
How are
you?
Bye bye 
Hello
Hi 
-10 3
http://www.freepik.com/free-vector/variety-
of-human-avatars_766615.htm

Maximizing Expected Reward
Human
Input sentence c response sentence x
Chatbot
En De
response sentence x
Input sentence c
[Li, et al., EMNLP, 2016]
reward
𝑅 𝑐, 𝑥
Learn to maximize expected reward
Policy Gradient

Policy Gradient - Implemenation
𝜃 𝑡
𝑐1, 𝑥1
𝑐2
, 𝑥2
𝑐 𝑁, 𝑥 𝑁
……
𝑅 𝑐1, 𝑥1
𝑅 𝑐2, 𝑥2
𝑅 𝑐 𝑁, 𝑥 𝑁
……
1
𝑁
𝑖=1
𝑁
𝑅 𝑐 𝑖, 𝑥 𝑖 𝛻𝑙𝑜𝑔𝑃 𝜃 𝑡 𝑥 𝑖|𝑐 𝑖
𝜃 𝑡+1 ← 𝜃 𝑡 + 𝜂𝛻 𝑅 𝜃 𝑡
𝑅 𝑐 𝑖, 𝑥 𝑖 is positive
Updating 𝜃 to increase 𝑃 𝜃 𝑥 𝑖
|𝑐 𝑖
𝑅 𝑐 𝑖
, 𝑥 𝑖
is negative
Updating 𝜃 to decrease 𝑃 𝜃 𝑥 𝑖|𝑐 𝑖

Comparison
1
𝑁
𝑖=1
𝑁
𝑅 𝑐 𝑖, 𝑥 𝑖 𝛻𝑙𝑜𝑔𝑃 𝜃 𝑥 𝑖|𝑐 𝑖
1
𝑁
𝑖=1
𝑁
𝑙𝑜𝑔𝑃 𝜃 𝑥 𝑖|𝑐 𝑖
1
𝑁
𝑖=1
𝑁
𝛻𝑙𝑜𝑔𝑃 𝜃 𝑥 𝑖|𝑐 𝑖
1
𝑁
𝑖=1
𝑁
𝑅 𝑐 𝑖, 𝑥 𝑖 𝑙𝑜𝑔𝑃 𝜃 𝑥 𝑖|𝑐 𝑖
𝑅 𝑐 𝑖
, 𝑥 𝑖
= 1 obtained from interaction
weighted by 𝑅 𝑐 𝑖
, 𝑥 𝑖
Objective
Function
Gradient
Maximum
Likelihood
Reinforcement
Learning
Training
Data
𝑐1, 𝑥1 , … , 𝑐 𝑁, 𝑥 𝑁
𝑐1, 𝑥1 , … , 𝑐 𝑁, 𝑥 𝑁

Outline of Part III
Improving Supervised Seq-to-seq Model
Unsupervised Seq-to-seq Model
I am busy.

Conditional GAN
Discriminator
Input sentence c response sentence x
Real or fake
http://www.nipic.com/show/3/83/3936650kd7476069.html
human
dialogues
Chatbot
En De
response sentence x
Input sentence c
“reward”

Algorithm
• Initialize generator G (chatbot) and discriminator D
• In each iteration:
• Sample input c and response 𝑥 from training set
• Sample input 𝑐′ from training set, and generate
response 𝑥 by G(𝑐′)
• Update D to increase 𝐷 𝑐, 𝑥 and decrease 𝐷 𝑐′, 𝑥
• Update generator G (chatbot) such that
Discrimi
nator
scalar
update
Training data:
𝑐
Chatbot
En De
Pairs of conditional input c
and response x

A A
A
B
A
B
A
A
B
B
B
<BOS>
Can we use
gradient ascent?
Due to the sampling process, “discriminator+ generator”
is not differentiable
Discriminator
Discrimi
nator
scalar
update
scalarChatbot
En De
NO!
Update Parameters

Three Categories of Solutions
Gumbel-softmax
• [Matt J. Kusner, et al, arXiv, 2016]
Continuous Input for Discriminator
• [Sai Rajeswar, et al., arXiv, 2017][Ofir Press, et al., ICML workshop, 2017][Zhen
Xu, et al., EMNLP, 2017][Alex Lamb, et al., NIPS, 2016][Yizhe Zhang, et al., ICML,
2017]
“Reinforcement Learning”
• [Yu, et al., AAAI, 2017][Li, et al., EMNLP, 2017][Tong Che, et al, arXiv,
2017][Jiaxian Guo, et al., AAAI, 2018][Kevin Lin, et al, NIPS, 2017][William
Fedus, et al., ICLR, 2018]

A A
A
B
A
B
A
A
B
B
B
<BOS>
Use the distribution
as the input of
discriminator
Avoid the sampling
process
Discriminator
Discrimi
nator
scalar
update
scalarChatbot
En De
Update Parameters
We can do
backpropagation
now.

What is the problem?
• Real sentence
• Generated
1
0
0
0
0
0
1
0
0
0
0
0
1
0
0
0
0
0
1
0
0
0
0
0
1
0.9
0.1
0
0
0
0.1
0.9
0
0
0
0.1
0.1
0.7
0.1
0
0
0
0.1
0.8
0.1
0
0
0
0.1
0.9
Can never
be 1-of-N
Discriminator can
immediately find
the difference.
WGAN is helpful

Reinforcement Learning?
• Consider the output of discriminator as reward
• Update generator to increase discriminator = to get
maximum reward
• Using the formulation of policy gradient, replace reward
𝑅 𝑐, 𝑥 with discriminator output D 𝑐, 𝑥
• Different from typical RL
• The discriminator would update
Discrimi
nator
scalar
update
Chatbot
En De

𝜃 𝑡
𝑐1, 𝑥1
𝑐2
, 𝑥2
𝑐 𝑁, 𝑥 𝑁
……
……
g-step
d-step
discriminator
𝑐
𝑥
fake real
𝑅 𝑐1, 𝑥1
𝑅 𝑐2, 𝑥2
𝑅 𝑐 𝑁
, 𝑥 𝑁
𝐷 𝑐1
, 𝑥1
𝐷 𝑐2, 𝑥2
𝐷 𝑐 𝑁, 𝑥 𝑁
1
𝑁
𝑖=1
𝑁
𝑅 𝑐 𝑖, 𝑥 𝑖 𝛻𝑙𝑜𝑔𝑃 𝜃 𝑡 𝑥 𝑖|𝑐 𝑖
𝜃 𝑡+1
← 𝜃 𝑡
+ 𝜂𝛻 𝑅 𝜃 𝑡
𝑅 𝑐 𝑖, 𝑥 𝑖 is positive
Updating 𝜃 to increase 𝑃 𝜃 𝑥 𝑖|𝑐 𝑖
𝑅 𝑐 𝑖, 𝑥 𝑖 is negative
Updating 𝜃 to decrease 𝑃 𝜃 𝑥 𝑖
|𝑐 𝑖
𝐷 𝑐 𝑖
, 𝑥 𝑖
𝐷 𝑐 𝑖, 𝑥 𝑖
𝐷 𝑐 𝑖, 𝑥 𝑖
D

Reward for Every Generation Step
Method 2. Discriminator For Partially Decoded Sequences
𝑙𝑜𝑔𝑃 𝜃 𝑥 𝑖
|ℎ𝑖
= 𝑙𝑜𝑔𝑃 𝑥1
𝑖
|𝑐 𝑖
𝑖
|𝑐 𝑖
, 𝑥1
𝑖
𝑖
|𝑐 𝑖
, 𝑥1:2
𝑖
ℎ𝑖
= “What is your name?” 𝑥 𝑖 = “I don’t know”
𝑃 "𝑑𝑜𝑛′𝑡"|𝑐 𝑖, "𝐼" 𝑃 "𝑘𝑛𝑜𝑤"|𝑐 𝑖, "𝐼 𝑑𝑜𝑛′𝑡"
Method 1. Monte Carlo (MC) Search
𝛻 𝑅 𝜃 ≈
1
𝑁
𝑖=1
𝑁
𝑡=1
𝑇
𝑄 𝑐 𝑖
, 𝑥1:𝑡
𝑖
− 𝑏 𝛻𝑙𝑜𝑔𝑃 𝜃 𝑥𝑡
𝑖
|𝑐 𝑖
, 𝑥1:𝑡−1
𝑖
𝛻 𝑅 𝜃 ≈
1
𝑁
𝑖=1
𝑁
𝐷 𝑐 𝑖, 𝑥 𝑖 𝛻𝑙𝑜𝑔𝑃 𝜃 𝑥 𝑖|𝑐 𝑖
[Yu, et al., AAAI, 2017]

Find more comparison in the survey papers.
[Lu, et al., arXiv, 2018][Zhu, et al., arXiv, 2018]
Input We've got to look for another route.
MLE I'm sorry.
GAN You're not going to be here for a while.
Input You can save him by talking.
MLE I don't know.
GAN You know what's going on in there, you know what I
mean?
• MLE frequently generates “I’m sorry”, “I don’t know”, etc.
(corresponding to fuzzy images?)
• GAN generates longer and more complex responses
(however, no strong evidence shows that they are better)
Experimental Results

More Applications
• Supervised machine translation [Wu, et al., arXiv
2017][Yang, et al., arXiv 2017]
• Supervised abstractive summarization [Liu, et al., AAAI
2018]
• Image/video caption generation [Rakshith Shetty, et al., ICCV
2017][Liang, et al., arXiv 2017]
If you are using seq2seq models,
consider to improve them by GAN.

Text Style Transfer
Domain X Domain Y
male female
It is good.
It’s a good day.
I love you.
It is bad.
It’s a bad day.
I don’t love you.
positive sentences negative sentences

𝐷 𝑌𝐷 𝑋
negative sentence? positive sentence?
It is bad. It is good. It is bad.
I love you. I hate you. I love you.
positive
positive
positivenegative
negative negative

𝐷 𝑌𝐷 𝑋
negative sentence? positive sentence?
It is bad. It is good. It is bad.
I love you. I hate you. I love you.
positive
positive
positivenegative
negative negative
Word embedding
[Lee, et al., ICASSP, 2018]
Discrete?

• Negative sentence to positive sentence:
it's a crappy day → it's a great day
i wish you could be here → you could be here
it's not a good idea → it's good idea
i miss you → i love you
i don't love you → i love you
i can't do that → i can do that
i feel so sad → i happy
it's a bad day → it's a good day
it's a dummy day → it's a great day
sorry for doing such a horrible thing → thanks for doing a
great thing
my doggy is sick → my doggy is my doggy
my little doggy is sick → my little doggy is my little doggy
Title: SCALABLE SENTIMENT FOR SEQUENCE-TO-SEQUENCE CHATBOT RESPONSE WITH PERFORMANCE ANALYSIS
Session: Dialog Systems and Applications
Time: Wednesday, April 18, 08:30 - 10:30
Authors: Chih-Wei Lee, Yau-Shian Wang, Tsung-Yuan Hsu, Kuan-Yu Chen, Hung-Yi Lee, Lin-Shan Lee

𝐸𝑁 𝑋
𝐷𝐸 𝑋 𝐷 𝑋
𝐷 𝑌
Discriminator
of X domain
Discriminator
of Y domain

𝐸𝑁 𝑋
𝐷𝐸 𝑋 𝐷 𝑋
𝐷 𝑌
Discriminator
of X domain
Discriminator
of Y domain
Positive
Sentence
Positive
Sentence
Negative
Sentence
Negative
Sentence
Decoder hidden layer as discriminator input
[Shen, et al., NIPS, 2017]
From 𝐸𝑁𝑋 or 𝐸𝑁𝑌
Domain
Discriminator
𝐸𝑁𝑋 and 𝐸𝑁𝑌 fool the
domain discriminator
[Zhao, et al., arXiv, 2017]
[Fu, et al., AAAI, 2018]

Abstractive Summarization
• Now machine can do abstractive summary by
seq2seq (write summaries in its own words)
summary 1
summary 2
summary 3
Training Data
summary
seq2seq
(in its own words)
Supervised: We need lots of
labelled training data.

Unsupervised Abstractive
Summarization
1 2
Summary?
Seq2seq Seq2seq
long
document
long
document
short
document
The two seq2seq models are jointly learn to
minimize the reconstruction error.
Only need a lot
of documents to
train the model

Summarization
1 2
Summary?
Seq2seq Seq2seq
long
document
long
document
short
document
This is a seq2seq2seq auto-encoder.
Using a sequence of words as latent representation.
not readable …
Policy gradient is used.

Summarization
1 2
Summary?
Seq2seq Seq2seq
long
document
long
document
short
document
3
Human written summaries Real or not
Discriminator
Let Discriminator considers
my output as real

Semi-supervised Learning
(unpublished)
unsupervised
semi-supervised

Unsupervised Machine Translation
Domain X Domain Y
male female
positive sentences
[Alexis Conneau, et al., ICLR, 2018]
[Guillaume Lample, et al., ICLR, 2018]

Unsupervised learning
with 10M sentences
Supervised learning with
100K sentence pairs
=
supervised
unsupervised

Unsupervised Speech Recognition
The dog is ……
The cats are ……
The woman is ……
The man is ……
The cat is ……Cycle GAN
“The”=
Acoustic Pattern Discovery
Can we achieve
unsupervised speech
recognition?
p1
p1 p3 p2
p1 p4 p3 p5 p5
p1 p5 p4 p3
p1 p2 p3 p4
[Liu, et al., arXiv, 2018] [Chen, et al., arXiv, 2018]

Unsupervised Speech Recognition
• Phoneme recognition
Audio: TIMIT
Text: WMT
supervised
WGAN-GP
Gumbel-softmax

Concluding Remarks
Unsupervised Conditional Sequence Generation

To Learn More …
https://www.youtube.com/playlist?list=PLJV_el3uVTsMd2G9ZjcpJn1YfnM9wVOBf
You can learn more from the YouTube Channel
(in Mandarin)

Reference
• Conditional Sequence Generation
• Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, Dan
Jurafsky, Deep Reinforcement Learning for Dialogue Generation, EMNLP,
2016
• Jiwei Li, Will Monroe, Tianlin Shi, Sébastien Jean, Alan Ritter, Dan Jurafsky,
Adversarial Learning for Neural Dialogue Generation, EMNLP, 2017
• Matt J. Kusner, José Miguel Hernández-Lobato, GANS for Sequences of
Discrete Elements with the Gumbel-softmax Distribution, arXiv 2016
• Tong Che, Yanran Li, Ruixiang Zhang, R Devon Hjelm, Wenjie Li, Yangqiu
Song, Yoshua Bengio, Maximum-Likelihood Augmented Discrete Generative
Adversarial Networks, arXiv 2017
• Lantao Yu, Weinan Zhang, Jun Wang, Yong Yu, SeqGAN: Sequence
Generative Adversarial Nets with Policy Gradient, AAAI 2017

Reference
• Sai Rajeswar, Sandeep Subramanian, Francis Dutil, Christopher Pal, Aaron
Courville, Adversarial Generation of Natural Language, arXiv, 2017
• Ofir Press, Amir Bar, Ben Bogin, Jonathan Berant, Lior Wolf, Language
Generation with Recurrent Generative Adversarial Networks without Pre-
training, ICML workshop, 2017
• Zhen Xu, Bingquan Liu, Baoxun Wang, Chengjie Sun, Xiaolong Wang,
Zhuoran Wang, Chao Qi , Neural Response Generation via GAN with an
Approximate Embedding Layer, EMNLP, 2017
• Alex Lamb, Anirudh Goyal, Ying Zhang, Saizheng Zhang, Aaron
Courville, Yoshua Bengio, Professor Forcing: A New Algorithm for Training
Recurrent Networks, NIPS, 2016
• Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan
Shen, Lawrence Carin, Adversarial Feature Matching for Text Generation,
ICML, 2017
• Jiaxian Guo, Sidi Lu, Han Cai, Weinan Zhang, Yong Yu, Jun Wang, Long Text
Generation via Adversarial Training with Leaked Information, AAAI, 2018
• Kevin Lin, Dianqi Li, Xiaodong He, Zhengyou Zhang, Ming-Ting Sun,
Adversarial Ranking for Language Generation, NIPS, 2017
• William Fedus, Ian Goodfellow, Andrew M. Dai, MaskGAN: Better Text
Generation via Filling in the______, ICLR, 2018

Reference
• Sidi Lu, Yaoming Zhu, Weinan Zhang, Jun Wang, Yong Yu, Neural Text
Generation: Past, Present and Beyond, arXiv, 2018
• Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun
Wang, Yong Yu, Texygen: A Benchmarking Platform for Text Generation
Models, arXiv, 2018
• Zhen Yang, Wei Chen, Feng Wang, Bo Xu, Improving Neural Machine
Translation with Conditional Sequence Generative Adversarial Nets, NAACL,
2018
• Lijun Wu, Yingce Xia, Li Zhao, Fei Tian, Tao Qin, Jianhuang Lai, Tie-Yan Liu,
Adversarial Neural Machine Translation, arXiv 2017
• Linqing Liu, Yao Lu, Min Yang, Qiang Qu, Jia Zhu, Hongyan Li, Generative
Adversarial Network for Abstractive Text Summarization, AAAI 2018
• Rakshith Shetty, Marcus Rohrbach, Lisa Anne Hendricks, Mario Fritz, Bernt
Schiele, Speaking the Same Language: Matching Machine to Human
Captions by Adversarial Training, ICCV 2017
• Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, Eric P. Xing, Recurrent
Topic-Transition GAN for Visual Paragraph Generation, arXiv 2017

Reference
• Zhenxin Fu, Xiaoye Tan, Nanyun Peng, Dongyan Zhao, Rui Yan, Style
Transfer in Text: Exploration and Evaluation, AAAI, 2018
• Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola, Style Transfer
from Non-Parallel Text by Cross-Alignment, NIPS 2017
• Chih-Wei Lee, Yau-Shian Wang, Tsung-Yuan Hsu, Kuan-Yu Chen, Hung-Yi Lee,
Lin-shan Lee, Scalable Sentiment for Sequence-to-sequence Chatbot
Response with Performance Analysis, ICASSP, 2018
• Junbo (Jake) Zhao, Yoon Kim, Kelly Zhang, Alexander M. Rush, Yann LeCun,
Adversarially Regularized Autoencoders, arxiv, 2017

Reference
• Unsupervised Machine Translation
• Alexis Conneau, Guillaume Lample, Marc'Aurelio Ranzato, Ludovic
Denoyer, Hervé Jégou, Word Translation Without Parallel Data, ICRL 2018
• Guillaume Lample, Ludovic Denoyer, Marc'Aurelio Ranzato, Unsupervised
Machine Translation Using Monolingual Corpora Only, ICRL 2018
• Unsupervised Speech Recognition
• Da-Rong Liu, Kuan-Yu Chen, Hung-Yi Lee, Lin-shan Lee, Completely
Unsupervised Phoneme Recognition by Adversarially Learning Mapping
Relationships from Audio Embeddings, arXiv, 2018
• Yi-Chen Chen, Chia-Hao Shen, Sung-Feng Huang, Hung-yi Lee, Towards
Unsupervised Automatic Speech Recognition Trained by Unaligned Speech
and Text only, arXiv, 2018

ICASSP 2018 Tutorial: Generative Adversarial Network and its Applications to Signal Processing and Natural Language Processing

More Related Content

What's hot (20)

Similar to ICASSP 2018 Tutorial: Generative Adversarial Network and its Applications to Signal Processing and Natural Language Processing (20)

Recently uploaded (20)

ICASSP 2018 Tutorial: Generative Adversarial Network and its Applications to Signal Processing and Natural Language Processing