Voice Separation with an Unknown Number of Multiple Speakers

ICML 2020  ·  Eliya Nachmani, Yossi Adi, Lior Wolf ·

We present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously. The new method employs gated neural networks that are trained to separate the voices at multiple processing steps, while maintaining the speaker in each output channel fixed. A different model is trained for every number of possible speakers, and the model with the largest number of speakers is employed to select the actual number of speakers in a given sample. Our method greatly outperforms the current state of the art, which, as we show, is not competitive for more than two speakers.

PDF Abstract ICML 2020 PDF

Results from the Paper

Task Dataset Model Metric Name Metric Value Global Rank Result Benchmark
Speech Separation WHAMR! VSUNOS SI-SDRi 12.2 # 8
Speech Separation WSJ0-2mix Gated DualPathRNN SI-SDRi 20.12 # 12
Speech Separation WSJ0-3mix Gated DualPathRNN SI-SDRi 16.85 # 7
Speech Separation WSJ0-4mix Gated DualPathRNN SI-SDRi 12.88 # 1
Speech Separation WSJ0-5mix Gated DualPathRNN SI-SDRi 10.56 # 4