深層学習を用いた音源定位、音源分離、クラス分類の統合～環境音セグメンテーション手法の紹介～

Editor's Notes

• #2: My name is Yui Sudo from Tokyo Institute of Technology. I am going to talk about “Environmental sound セグメンテーション utilizing Mask U-Net”. 10s
• #3: Robots in real environment must recognize many kinds of sounds like speech in a noisy environment. Or sometimes not only speech but also music, bird singing and so on. So many methods have been developed for example noise reduction, sound source localization and separation. However, these conventional methods are used in カスケード like this block diagram. The biggest drawback of the カスケード system is that 誤差 which occurred at each function block are accumulated. Therefore it is necessary to develop end-to-end system in order to actualize overall optimized and more general method. Of simultaneous section detection classification and separation. 1’00
• #4: まず、弊社の紹介ですが、 ホンダリサーチインスティチュートは「21世紀の最先端技術で、価値の高い技術を最速で創造し、未来の社会に貢献」を狙い、 2003年に日本、アメリカ、ドイツの3拠点に設立されました。 その日本の拠点になりますのが、ホンダ・リサーチ・インスティチュート・ジャパンとなります。 以下HRI-JPと略させていただきます。
• #5: Robots in real environment must recognize many kinds of sounds like speech in a noisy environment. Or sometimes not only speech but also music, bird singing and so on. So many methods have been developed for example noise reduction, sound source localization and separation. However, these conventional methods are used in カスケード like this block diagram. The biggest drawback of the カスケード system is that 誤差 which occurred at each function block are accumulated. Therefore it is necessary to develop end-to-end system in order to actualize overall optimized and more general method. Of simultaneous section detection classification and separation. 1’00
• #6: Robots in real environment must recognize many kinds of sounds like speech in a noisy environment. Or sometimes not only speech but also music, bird singing and so on. So many methods have been developed for example noise reduction, sound source localization and separation. However, these conventional methods are used in カスケード like this block diagram. The biggest drawback of the カスケード system is that 誤差 which occurred at each function block are accumulated. Therefore it is necessary to develop end-to-end system in order to actualize overall optimized and more general method. Of simultaneous section detection classification and separation. 1’00
• #7: Robots in real environment must recognize many kinds of sounds like speech in a noisy environment. Or sometimes not only speech but also music, bird singing and so on. So many methods have been developed for example noise reduction, sound source localization and separation. However, these conventional methods are used in カスケード like this block diagram. The biggest drawback of the カスケード system is that 誤差 which occurred at each function block are accumulated. Therefore it is necessary to develop end-to-end system in order to actualize overall optimized and more general method. Of simultaneous section detection classification and separation. 1’00
• #9: Robots in real environment must recognize many kinds of sounds like speech in a noisy environment. Or sometimes not only speech but also music, bird singing and so on. So many methods have been developed for example noise reduction, sound source localization and separation. However, these conventional methods are used in カスケード like this block diagram. The biggest drawback of the カスケード system is that 誤差 which occurred at each function block are accumulated. Therefore it is necessary to develop end-to-end system in order to actualize overall optimized and more general method. Of simultaneous section detection classification and separation. 1’00
• #10: I’ll review some related work of 音響イベント検出. This slide shows a one of the popular approach of 音響イベント検出 which is a CNN based method. This method applies CRNN to spectrogram and detect on-set and off-set of each class sound event. However this SED method cannot remain frequency information by using frequency pooling. So this method cannot separate each sound event. 30s
• #11: I’ll review some related work of 音響イベント検出. This slide shows a one of the popular approach of 音響イベント検出 which is a CNN based method. This method applies CRNN to spectrogram and detect on-set and off-set of each class sound event. However this SED method cannot remain frequency information by using frequency pooling. So this method cannot separate each sound event. 30s
• #12: Subsequently, Let me introduce some more related works of sound source separation. One traditional approach is NMF based method which uses NMF. Computational cost is low, its performance is also low compared to DNN based approach. Additionally, it is difficult to deal with many classes like environmental sounds. Second is Deep learning-based approach. U-Net which is originally proposed for image semantic セグメンテーション has been proposed for vocal separation. This method predicts mask spectrograms for separating singing voice and is possible to train end-to-end. However the number of classes is also small, vocal and instrument. And additionally, it is pointed out in the previous study of image semantic セグメンテーション that when the object size is very huge, the performance will be low for example bed and pillow. So it is difficult to simply apply this method to envitonmental sound セグメンテーション which have a lot of classes. 1’15
• #13: Subsequently, Let me introduce some more related works of sound source separation. One traditional approach is NMF based method which uses NMF. Computational cost is low, its performance is also low compared to DNN based approach. Additionally, it is difficult to deal with many classes like environmental sounds. Second is Deep learning-based approach. U-Net which is originally proposed for image semantic セグメンテーション has been proposed for vocal separation. This method predicts mask spectrograms for separating singing voice and is possible to train end-to-end. However the number of classes is also small, vocal and instrument. And additionally, it is pointed out in the previous study of image semantic セグメンテーション that when the object size is very huge, the performance will be low for example bed and pillow. So it is difficult to simply apply this method to envitonmental sound セグメンテーション which have a lot of classes. 1’15
• #14: Subsequently, Let me introduce some more related works of sound source separation. One traditional approach is NMF based method which uses NMF. Computational cost is low, its performance is also low compared to DNN based approach. Additionally, it is difficult to deal with many classes like environmental sounds. Second is Deep learning-based approach. U-Net which is originally proposed for image semantic セグメンテーション has been proposed for vocal separation. This method predicts mask spectrograms for separating singing voice and is possible to train end-to-end. However the number of classes is also small, vocal and instrument. And additionally, it is pointed out in the previous study of image semantic セグメンテーション that when the object size is very huge, the performance will be low for example bed and pillow. So it is difficult to simply apply this method to envitonmental sound セグメンテーション which have a lot of classes. 1’15
• #18: This slide shows the complete architecture of environmental sound セグメンテーション consists of three blocks, 特徴抽出, セグメンテーション and reconstruction. In the 特徴抽出 block STFT is applied to the mixed waveforms and divided into spectral and 空間特徴s. These features are input into the セグメンテーション block. This block predicts a mask spectrogram for separating each class from the input spectrogram. Then an inverse STFT is applied to reconstruct the time domain signal using predicted amplitude spectrogram and phase spectrogram obtained from mixed waveform. The differences between the conventional method and our model are Input features are extended to multi-channel input Deeplabv3+ was applied instead of U-Net based method. These difference are expected that improve the performance on overlapping sound and robustness of large variation in sound event length. 1’10
• #19: This slide shows the complete architecture of environmental sound セグメンテーション consists of three blocks, 特徴抽出, セグメンテーション and reconstruction. In the 特徴抽出 block STFT is applied to the mixed waveforms and divided into spectral and 空間特徴s. These features are input into the セグメンテーション block. This block predicts a mask spectrogram for separating each class from the input spectrogram. Then an inverse STFT is applied to reconstruct the time domain signal using predicted amplitude spectrogram and phase spectrogram obtained from mixed waveform. The differences between the conventional method and our model are Input features are extended to multi-channel input Deeplabv3+ was applied instead of U-Net based method. These difference are expected that improve the performance on overlapping sound and robustness of large variation in sound event length. 1’10
• #20: We evaluate our method by conducting some simulation experiments using three custom データセットs. We created these custom データセットs using 10 corpuses contaning many classes of dry source. セグメンテーション 結果s are evaluated by calculating RMSE. Then I will show you the few example and discuss the effect of Deeplabv3+ and 空間特徴s. 30s
• #22: This figure and this table shows the experimental settings for the numerical simulations. Three dry sources are randomly selected from these 10 corpuses. and the impulse response was convolved like this mixed spectrogram. Then, diffuse noise were added to all time frames. We created 10,000 training set 1,000 評価 set. 30s
• #23: I’ll review some related work of 音響イベント検出. This slide shows a one of the popular approach of 音響イベント検出 which is a CNN based method. This method applies CRNN to spectrogram and detect on-set and off-set of each class sound event. However this SED method cannot remain frequency information by using frequency pooling. So this method cannot separate each sound event. 30s
• #24: I’ll review some related work of 音響イベント検出. This slide shows a one of the popular approach of 音響イベント検出 which is a CNN based method. This method applies CRNN to spectrogram and detect on-set and off-set of each class sound event. However this SED method cannot remain frequency information by using frequency pooling. So this method cannot separate each sound event. 30s
• #25: This table shows the まとめ of the simulation 結果. First, Let’s see the 結果 of データセット1 containing three classes of sound. Regarding deep learning models, Deeplabv3+ showed higher performance than conventional models. As for input features, by using sinIPD and cosIP, RMSE was obviously improved especially on CRNN and UNet. And these figure shows an example. This example contains overlapping sound as shown in blue spectra behind green and yellow spectra. As you can see these colored spectrogram, every model look good. However 1’00
• #26: 30s
• #27: 30s