Adding Virtual Backgrounds

Twilio Video supports background blur and replacement on the web using the optional Video Processors JavaScript library. The Video Processors library is a collection of video processing tools that can be used with the Twilio Video JavaScript SDK to apply transformations and filters, such as background blurring and virtual backgrounds, to a VideoTrack. You can also use the tools to create your own transformations or filters.

See a live demo with blurred backgrounds and virtual backgrounds in the browser using the Video Processors tools here.

View the Video Processors demo application and run the example application locally.

The recent changes to the library are documented here

• Twilio Video JavaScript SDK (v2.29+)
• Node.js (v18+)
• NPM (v10+, comes installed with newer Node versions)

You can install the Video Processors library using Node Package Manager (npm):

npm install @twilio/video-processors --save

Using this method, you can import twilio-video-processors like so:

import * as VideoProcessors from '@twilio/video-processors';

After installing with npm, you can also copy twilio-video-processors.js from the dist/build folder and include it directly in your web app using a <script> tag:

<script src="https://my-server-path/twilio-video-processors.js"></script>

Using this method, twilio-video-processors.js will set a browser global:

const VideoProcessor = Twilio.VideoProcessors;

Add the video processor to the Video track.

1
videoTrack.addProcessor(VideoProcessor, {
2
  inputFrameBufferType: 'videoframe',
3
  outputFrameBufferContextType: 'bitmaprenderer'
4
});

Video frame processing is handled by web workers on all major browsers preventing the main thread from blocking. Support for cross-domain worker hosting is now available. Below is an example of a cross-domain configuration:

1
import { GaussianBlurBackgroundProcessor } from '@twilio/video-processors';
2

3
/* Application is running at https://example.com/app */
4

5
const processor = new GaussianBlurBackgroundProcessor({
6
  assetsPath: "https://example.net/path/to/assets"
7
});

Note that this requires the Access-Control-Allow-Origin headers to properly point to your application domain.

View the latest Video Processors JavaScript library documentation here.

Documentation for version 2.x Video Processors JavaScript library is available here.

The following table lists the browsers and operating systems supported by the Video Processors JavaScript library.

The Video Processors library is CPU intensive as it processes each video frame individually. The library uses Web Assembly and will offload some video processing to the GPU; however, there is a minimum hardware requirement to ensure that the device isn't overpowered and has resources to support the ongoing video call.

The following is the minimum hardware specification:

• CPU: Intel i5-7200
• GPU: Intel HD Graphics 620
• RAM: 8GB
• OS: Windows 10 Pro

Example: HP ProBook 450 G4