Noise Cancellation

Using tens of thousands of hours of audio recordings, Krisp.ai has created a sophisticated Deep Neural Network that can differentiate between background sounds and the human voice. It's with this technology that we created the Krisp Audio Plugin for Twilio Video, a lightweight audio processor that can run inside your client application and create crystal clear audio.

The plugin needs to be loaded alongside the Twilio SDK and runs as part of the audio pipeline between the microphone and audio encoder in a preprocessing step. During this step, the AI-based noise cancellation algorithm removes unwanted sounds like barking dogs, construction noises, honking horns, and coffee shop chatter.

After the preprocessing step, the audio is encoded and delivered to the end user. Note that all of these steps happen on your device, with almost no latency, and no media sent to a server.

Twilio Video Noise Cancellation requires you to host and serve the Krisp audio plugin for Twilio as part of your web application. It also requires browser support of the WebAudio API (specifically Worklet.addModule). The table below lists the minimum browser version across major browsers:

If you try to use Krisp noise cancellation on a browser that is not supported, the SDK will return a non-Krisp enabled track to the application and your audio will follow the standard audio pipeline. In this case, you will get audioTrack with audioTrack.noiseCancellation set to null.

• Noise cancellation will increase CPU load on a device, as all the preprocessing happens on the device.
• You can only run one audio track through the audio plugin pipeline per Participant.
• When acquiring an audio track with Krisp noise cancellation enabled, we recommend that you disable browser's in-built noise cancellation by specifying noiseSuppression: false as one of the constraints. However, one caveat is if you disable Krisp noise cancellation at runtime, you would not get browser's noise cancellation.

The plugin is available on NPM.

You can install it with

npm install @twilio/krisp-audio-plugin

Once you install the plugin, you need to host the contents of ./node_modules/@twilio/krisp-audio-plugin/dist/ from your web server. We recommend that you add the plugin version number to the hosted path to ensure that the browser does not use stale version when it's updated.

If your application is using the default-src self content security policy directive, then you should add another directive unsafe-eval, which is required for the Krisp Audio Plugin to load successfully. In your application code, you will need to specify that you want to use Krisp in noiseCancellationOptions when you create the local audio track. You also need to specify the path where your application server is hosting Krisp files from step 1 above.

1
const { connect, createLocalAudioTrack } = require('twilio-video');
2

3
// Create a LocalAudioTrack with Krisp noise cancellation enabled.
4
const localAudioTrack = await createLocalAudioTrack({
5
  noiseCancellationOptions: {
6
    sdkAssetsPath: 'path/to/hosted/krisp/audio/plugin/dist',
7
    vendor: 'krisp'
8
  }
9
});
10

11
if (!localAudioTrack.noiseCancellation) {
12
  // If the Krisp audio plugin fails to load, then a warning message will be logged
13
  // in the browser console, and the "noiseCancellation" property will be set to null.
14
  // You can still use the LocalAudioTrack to join a Room. However, it will use the
15
  // browser's noise suppression instead of the Krisp noise cancellation. Make sure
16
  // the "sdkAssetsPath" provided in "noiseCancellationOptions" points to the correct
17
  // hosted path of the plugin assets.
18
} else {
19
  // Join a Room with the LocalAudioTrack.
20
  const room = await connect('token', {
21
    name: 'my-cool-room',
22
    tracks: [localAudioTrack]
23
  });
24
}
25

26
/**
27
 * Enable/disable noise cancellation.
28
 * @param {boolean} enable - whether noise cancellation should be enabled
29
 */
30
function setNoiseCancellation(enable) {
31
  const { noiseCancellation } = localAudioTrack;
32
  if (noiseCancellation) {
33
    if (enable) {
34
      // If enabled, then the LocalAudioTrack will use the Krisp noise
35
      // cancellation instead of the browser's noise suppression.
36
      noiseCancellation.enable();
37
    } else {
38
      // If disabled, then the LocalAudioTrack will use the browser's
39
      // noise suppression instead of the Krisp noise cancellation.
40
      noiseCancellation.disable();
41
    }
42
  }
43
}

• Safari is not a supported browser due to API unreliability on version 15.x. This will be reevaluated in future versions.
• Chrome on iOS is not supported and can unsuccessfully fail at returning a non-Krisp enabled track on certain iOS versions. We recommend to not apply Krisp noise suppression on iOS at this time.

Noise cancellation for iOS is available using an audio device called NoiseCancellationProcessor, which is provided in the TwilioAudioProcessors SDK.

This feature requires Twilio Video iOS SDK version 5.4 or higher.

The following code sample demonstrates how to enable the Twilio Video NoiseCancellationProcessor on iOS.

1

2
// import the SDK
3
import TwilioAudioProcessors
4

5
// initialize the audio device to use Krisp
6
TwilioVideoSDK.audioDevice = NoiseCancellationProcessor()
7

8
// disable noise cancellation while using Krisp audio device
9
guard let krispAudioDevice = TwilioVideoSDK.audioDevice as? NoiseCancellationProcessor else {
10
  fatalError("Current audio device is not of type NoiseCancellationProcessor")
11
}
12
krispAudioDevice.pauseProcessing = true
13

Noise cancellation for Android is available using a custom audio device called NoiseCancellationAudioDevice, which uses the Twilio Video Android SDK's AudioDevice framework. You can find the package on Maven Central and access its documentation here.

This feature requires Twilio Video Android SDK version 7.5.1 or higher.

The following code sample demonstrates how to enable the Twilio Video NoiseCancellationAudioDevice on Android.

1
// import the NoiseCancellationAudioDevice plugin
2
import com.twilio.video.NoiseCancellationAudioDevice
3

4
// initialize the audio device to use Krisp
5
Video.setAudioDevice(NoiseCancellationAudioDevice(getApplicationContext())); /* use either application context or activity context 'this' */
6

The following code can be used to disable the noise cancellation audio processing.

1
audioDevice = Video.getAudioDevice();
2
if (!audioDevice instanceof NoiseCancellationAudioDevice) {
3
    throw new RuntimeException("Current audio device is not of type NoiseCancellationAudioDevice");
4
}
5

6
// disable noise cancellation while using Krisp audio device
7
audioDevice.setPausedProcessing(true); 
8