Text Localization in Scientific Figures using Fully Convolutional Neural Networks on Limited Training Data
0 likes319 views
This document presents a supervised approach for text localization in scientific figures using fully convolutional neural networks, addressing the challenges posed by limited training data. The study emphasizes the effectiveness of pre-training and dataset augmentation, achieving superior results compared to previous unsupervised methods. The proposed method demonstrated the ability to generalize across diverse datasets of scientific figures.
![Introduction
Datasets
Overview: Supervised Approach
Results
Outlook
Our previous unsupervised Approach [MTAP’18]
Observation: text localization (1)-(4) is most challenging part
Propose a supervised approach for text localization that can
work with limited training data =⇒ DocEng’19
Morten Jessen, Falk B¨oschen, Ansgar Scherp 3 / 24](https://image.slidesharecdn.com/jessenboeschenscherp-nntextextraction-190926130309/85/Text-Localization-in-Scientific-Figures-using-Fully-Convolutional-Neural-Networks-on-Limited-Training-Data-3-320.jpg)
![Introduction
Datasets
Overview: Supervised Approach
Results
Outlook
Localization
Pre-Training
Artificial Dataset Extension
Recognition
Faster R-CNN
Figure: Faster R-CNN Architecture [Ren et al., 2015].
Morten Jessen, Falk B¨oschen, Ansgar Scherp 11 / 24](https://image.slidesharecdn.com/jessenboeschenscherp-nntextextraction-190926130309/85/Text-Localization-in-Scientific-Figures-using-Fully-Convolutional-Neural-Networks-on-Limited-Training-Data-11-320.jpg)
Text Localization in Scientific Figures using Fully Convolutional Neural Networks on Limited Training Data
- 1. Introduction Datasets Overview: Supervised Approach Results Outlook Text Localization in Scientific Figures using Fully Convolutional Neural Networks on Limited Training Data Morten Jessen, Falk B¨oschen, Ansgar Scherp DocEng, September 2019 Morten Jessen, Falk B¨oschen, Ansgar Scherp 1 / 24
- 2. Introduction Datasets Overview: Supervised Approach Results Outlook Motivation Figures are widely used in scientific papers, media, and other Figures often contain information that is not present in the surrounding text and transport core message(s) of a document Extracted text can be used for improving existing retrieval systems building (better) figure retrieval systems making figures available to visually impaired people . . . However, common Optical Character Recognition (OCR) engines have problems with processing figures So far the focus was on unsupervised approaches due to the lack of training data Morten Jessen, Falk B¨oschen, Ansgar Scherp 2 / 24
- 3. Introduction Datasets Overview: Supervised Approach Results Outlook Our previous unsupervised Approach [MTAP’18] Observation: text localization (1)-(4) is most challenging part Propose a supervised approach for text localization that can work with limited training data =⇒ DocEng’19 Morten Jessen, Falk B¨oschen, Ansgar Scherp 3 / 24
- 4. Introduction Datasets Overview: Supervised Approach Results Outlook Datasets of Scholarly Figures CHIME-R CHIME-S DeGruyter EconBiz DeTEXT Number of Images 115 85 120 121 192 Text elements 14 12 24 25 14 Words 18 18 34 35 20 Characters 76 69 149 151 120 Available datasets are quite small Makes training of supervised methods difficult Morten Jessen, Falk B¨oschen, Ansgar Scherp 4 / 24
- 5. Introduction Datasets Overview: Supervised Approach Results Outlook CHIME-R Examples 115 real images bar-, pie- and line-charts Morten Jessen, Falk B¨oschen, Ansgar Scherp 5 / 24
- 6. Introduction Datasets Overview: Supervised Approach Results Outlook CHIME-S Examples 85 synthetic images bar-, pie- and line-charts Morten Jessen, Falk B¨oschen, Ansgar Scherp 6 / 24
- 7. Introduction Datasets Overview: Supervised Approach Results Outlook DeGruyter Examples 120 figures from academic books Additional content: Scatter-, flow-/process-charts, histograms Morten Jessen, Falk B¨oschen, Ansgar Scherp 7 / 24
- 8. Introduction Datasets Overview: Supervised Approach Results Outlook EconBiz Examples 121 randomly extracted scholarly figures Additional content: maps Morten Jessen, Falk B¨oschen, Ansgar Scherp 8 / 24
- 9. Introduction Datasets Overview: Supervised Approach Results Outlook DeTEXT Examples 192 biomedical images Additional content: medical images (real and abstract) Morten Jessen, Falk B¨oschen, Ansgar Scherp 9 / 24
- 10. Introduction Datasets Overview: Supervised Approach Results Outlook Localization Pre-Training Artificial Dataset Extension Recognition Overview Focus on a neural network based approach for text localization in scientific figures Evaluate different approaches to address the challenge of limited training data Pre-Training on large datasets Artificial dataset extension We use a common Optical Character Recognition engine for text recognition (Tesseract) Morten Jessen, Falk B¨oschen, Ansgar Scherp 10 / 24
- 11. Introduction Datasets Overview: Supervised Approach Results Outlook Localization Pre-Training Artificial Dataset Extension Recognition Faster R-CNN Figure: Faster R-CNN Architecture [Ren et al., 2015]. Morten Jessen, Falk B¨oschen, Ansgar Scherp 11 / 24
- 12. Introduction Datasets Overview: Supervised Approach Results Outlook Localization Pre-Training Artificial Dataset Extension Recognition Pre-Training on COCO-Text COCO-Text: images from MS-COCO plus text annotations We use images with English, machine written and legible text 145, 000 text annotations on 63, 686 images (avg.: 2.28 annotations/image) Morten Jessen, Falk B¨oschen, Ansgar Scherp 12 / 24
- 13. Introduction Datasets Overview: Supervised Approach Results Outlook Localization Pre-Training Artificial Dataset Extension Recognition Artificial Dataset Extension Extend the figure datasets with transformed versions (rotation, noise, translation, flipping, rescaling) of each figure Morten Jessen, Falk B¨oschen, Ansgar Scherp 13 / 24
- 14. Introduction Datasets Overview: Supervised Approach Results Outlook Localization Pre-Training Artificial Dataset Extension Recognition Text Exraction with Tesseract 4.0 OCR engine using LSTM neural network Text extraction process Generate multiple input images from one bounding box (provded by Faster R-CNN) Stop when Tesseract’s confidence score is ≥ 96%, OR take best otherwise Morten Jessen, Falk B¨oschen, Ansgar Scherp 14 / 24
- 15. Introduction Datasets Overview: Supervised Approach Results Outlook Localization Pre-Training Artificial Dataset Extension Recognition Preprocessing for OCR Increase bounding box size by 5px Added white border, 25px Rotations: 0°, 90°, 270°, 45°, 315°, 30°, 60°, 300°, 330° Resizing shortest side to: 100px, 200px Binarization Morten Jessen, Falk B¨oschen, Ansgar Scherp 15 / 24
- 16. Introduction Datasets Overview: Supervised Approach Results Outlook Evaluation Measures Text localization: detection of bounding boxes Average Prevision (AP), AP50, AP75 over “Intersection over Union” (IoU) Precision, Recall Text recognition: extraction of text from bounding boxes Levenshtein Distance: number of edits needed to correct word Gestalt Pattern Matching: correctness of extraction in relation to word length Morten Jessen, Falk B¨oschen, Ansgar Scherp 16 / 24
- 17. Introduction Datasets Overview: Supervised Approach Results Outlook Intersection over Union (IoU) (a) Definition of IoU. (b) Examples for IoU. Morten Jessen, Falk B¨oschen, Ansgar Scherp 17 / 24
- 18. Introduction Datasets Overview: Supervised Approach Results Outlook Average Precision (AP) over IoU Figure: Visualization of different IoU Values. AP50: Percentage of predictions with IoU > 0.5 AP75: Percentage of predictions with IoU > 0.75 AP: Summary metric, combines ten equally spaced IoU thresholds (0.50, 0.55, 0.60, ..., 0.90, 0.95) Morten Jessen, Falk B¨oschen, Ansgar Scherp 18 / 24
- 19. Introduction Datasets Overview: Supervised Approach Results Outlook Effect of Pre-Training on COCO-Text Pretraining: none on COCO-Text AP50 91.35% 95.21% AP75 63.49% 76.33% AP 58.37% 65.98% Table: Comparison for training with and without pre-training on COCO-Text. Morten Jessen, Falk B¨oschen, Ansgar Scherp 19 / 24
- 20. Introduction Datasets Overview: Supervised Approach Results Outlook Effect of Dataset Augmentation AP AP75 AP 50 without augmentation 52.90% 53.02% 90.34% with augmentation 60.81% 67.57% 92.88% Table: Comparison: Effect of artificially extended dataset on ResNet101. Morten Jessen, Falk B¨oschen, Ansgar Scherp 20 / 24
- 21. Introduction Datasets Overview: Supervised Approach Results Outlook Generalization Experiments: Train on 4 + Test on Last Tested on AP50 AP75 AP CHIME-R 80.45% 45.19% 45.82% CHIME-S 87.73% 30.59% 41.12% DeGruyter 86.63% 35.88% 43.06% EconBiz 84.61% 15.88% 34.03% DeTEXT 70.32% 29.49% 34.46% Table: Generalization: Training on four of the datasets for 200, 000 iterations and testing on the fifth dataset. Morten Jessen, Falk B¨oschen, Ansgar Scherp 21 / 24
- 22. Introduction Datasets Overview: Supervised Approach Results Outlook Comparison to Unsupervised Approach: Localization Precision Recall F1 (STD) TX 0.66 0.55 0.56 (0.25) NN 0.86 0.83 0.87 (0.12) Table: Comparison of the unsupervised approach (TX) with our proposed supervised approach (NN) for text localization in scientific figures. Morten Jessen, Falk B¨oschen, Ansgar Scherp 22 / 24
- 23. Introduction Datasets Overview: Supervised Approach Results Outlook Comparison to Unsupervised Approach: Recognition Levenshteinavg (SD) Global Levenshtein (SD) TX 6.23 (4.93) 108.81 (108.53) NN 3.44 (4.42) 39.11 (41.75) Table: Comparison of text recognition of the unsupervised approach (TX) with our proposed supervised approach (NN). Morten Jessen, Falk B¨oschen, Ansgar Scherp 23 / 24
- 24. Introduction Datasets Overview: Supervised Approach Results Outlook Summary Proposed a supervised text extraction approach from scientific figures using neural networks Showed that dataset extension and pre-training with natural images alleviates problem of limited training data Supervised approach outperforms the previously known best unsupervised approach(es) Capable of handling different datasets: generalizes to new datasets if they contain figures of same type Thank you! Any questions? Email: [email protected] Morten Jessen, Falk B¨oschen, Ansgar Scherp 24 / 24