Sparsity Normalization: Stabilizing the Expected Outputs of Deep Networks
0 likes168 views
The document discusses the issue of variable sparsity in deep networks, highlighting that differing sparsity levels in input data can hinder training and affect output consistency. It proposes a method called sparsity normalization to stabilize outputs regardless of input sparsity, leading to improved performance across various datasets including collaborative filtering and electronic health records. Experimental results indicate that sparsity normalization enhances model accuracy and stability, outperforming existing techniques on multiple benchmark datasets.
Sparsity Normalization: Stabilizing the Expected Outputs of Deep Networks
- 1. SPARSITY NORMALIZATION: STABILIZING THE EXPECTED OUTPUTS OF DEEP NETWORKS 2019. 06. 07. JoonyoungYi [email protected]
- 2. 2 • Many benchmark datasets differ in the sparsity between the data instances. • Variable sparsity problem: the expected value of the output layer depends on the sparsity of the input data instance which makes the training difficult. • Varying outputs for data instances with similar characteristics under different sparsity. VARIABLE SPARSITY PROBLEM
- 3. 3 • Divide each input data instance by l0: • So that outputs are not dependent on sparsity (can be applied to CNN similarly). • Sparsity Normalization solves various sparsity problem (theoretically, experimentally). • Sparsity in a hidden layer is more stable after applying Sparsity Normalization. SPARSITY NORMALIZATION
- 4. 4 • Collaborative filtering datasets: Achieved states-of-the-arts performance on Movielens 100K & 1M by simply applying Sparsity Normalization to non-states-of-the-arts model. • Electronic health records (EHR) dataset: Better AUC & orthogonal to Dropout. • Vision datasets: Better accuracy with less capacity & orthogonal to BN. • 6 UCI datasets: better performance even compared to other missing handling techniques. EXPERIMENTAL RESULTS