Speech Restoration & Enhancement

How it works

• An STFT front end feeds a noise profiler that fingerprints the first few frames. A decision step picks fan noise (steady) or rain noise (broadband) and applies the matching filter before converting back with an inverse STFT.
• Spectral subtraction estimates a noise profile and subtracts it. It is simple and works well on steady fan or AC hum.
• On broadband rain it fails, because over-subtraction floors values to zero and leaves robotic 'musical noise'. The fix is a spectral-flooring term that keeps a little comfort noise.
• Wiener filtering instead weights each frequency by how likely speech is present, so it handles rain more gracefully. A 300 to 3400 Hz telephony bandpass and noise gating finish the voice. A U-Net that masks the spectrogram was also tried.

Results

• On steady fan noise with spectral subtraction, PSNR went from 30.9 to 35.5 dB and MSE dropped to 1.8e-5.
• On rain noise the Wiener filter reached about 30.6 dB PSNR, and the flooring fix removed the robotic artefacts.
• The takeaway is that the method has to match the noise. Subtraction assumes steady noise, and rain overlaps the consonants in speech.

In code

S = stft(x)                                  # noisy magnitude spectrum
noise = S[:, :n_ref].mean(axis=1, keepdims=True)   # profile from first frames
mag = np.abs(S) - alpha * noise
mag = np.maximum(mag, beta * np.abs(S))      # floor -> keep comfort noise
y = istft(mag * np.exp(1j * np.angle(S)))▊

Spectral subtraction with a flooring term, the fix that kills the robotic 'musical noise'.

Slides

The full project deck. You can read it right here.

Comments

Thoughts or questions on this project? Leave a note below.