Improve the Efficiency of Your Multi-Party Video Experiences

Improve the Efficiency of Your Multi-Party Video Experiences

A good multi-party video experience is critical for education, social, and workplace collaboration use cases, so that everyone can be part of the conversation. While the value is clear, building a performant multi-party video application for a browser or mobile device is hard. You need to create a video UI that is intuitive for your users that also delivers a great experience regardless of the device and type of network they are using.

In 2019, we introduced the Network Bandwidth Profile API to help developers allocate bandwidth to specific participants based on the use case. Today, we are delighted to introduce the new and improved Network Bandwidth Profile API to empower developers to create multi-party applications that are more efficient with both bandwidth and CPU. This efficiency prevents users' computer fans from going haywire, improves video quality, and reduces the support burden on developers. In addition, this update provides fine-grained control over video track switch-off and allows developers to create unique layouts.

Conserving CPU & Bandwidth

A video application consumes CPU resources in a number of ways. First, there is the general interaction and UI rendering of the application. Second, there are the media decode and encode activities. CPU cycles are expended on the decode of each of the incoming video tracks corresponding to the participants in the Room. The more video tracks and the higher the resolution, the more CPU will be consumed. The same can be said for the downstream bandwidth—the more video tracks and the higher the resolution, the more bandwidth will be consumed.

Video applications waste CPU and bandwidth when the rendered video is smaller than the encoded video. For example, if a 1280x720 video track is received by a client but is rendered into a region that is 176x144 pixels wide, then there is wasted CPU and bandwidth on the subscriber's computer.

With the original release of the Network Bandwidth API, a common mistake we saw developers make was to use renderDimensions values much larger than the actual video region in the UI. Based on this observation, we’ve updated the Network Bandwidth Profile API to solve for this pain point.

With this latest release of our Network Bandwidth Profile API, we are introducing two specific optimizations for our JavaScript, iOS, and Android SDKs:

1. Video tracks that are not visible are automatically switched off to conserve CPU and bandwidth.
2. The Twilio Media Server will deliver video tracks to the client that more closely match the rendering dimensions actually being used in the client application.

In addition, developers can take manual control of these features and use them to specify exactly which video tracks are switched on at a particular moment and what the desired resolution of each one is.  

Getting Started

These new features are enabled by default in the latest version of the Network Bandwidth Profile API but can be explicitly turned off if required.

JavaScript

The first setting is called clientTrackSwitchOffControl.  When set to auto, Twilio uses document visibility, track attachments, and the visibility of video elements to determine whether a RemoteVideoTrack should be switched on or off. A RemoteVideoTrack will be switched off when the document is no longer visible, when no video elements are attached to the track, or when the video elements attached to the track are not visible. When set to manual, developers can explicitly turn video tracks off or on using the RemoteVideoTrack.switchOff() / switchOn() methods.

The next setting is called contentPreferencesMode. When set to auto, the SDK automatically detects the layout dimensions of the video element and communicates that to the Media Server. The Media Server in turn delivers a video stream that more closely matches the provided layout dimensions. This ensures that as the layout of the UI changes, the relative size of the video tiles change, and the allocation of bandwidth and CPU resources associated with each video element also adjusts. When contentPreferencesMode is set to manual, developers can signal to the Media Server the desired resolution for each video track using the setContentPreferences() method.

Android

On Android, we have a new Network Bandwidth Profile setting called clientTrackSwitchOffControl, which can be set to auto or manual. In auto, the SDK determines whether tracks should be switched off based on the renderer attachment, view visibility, and application lifecycle. In manual, developers can explicitly turn tracks off/on using the RemoteVideoTrack.switchOff() / RemoteVideoTrack.switchOn() methods.

There is another new Network Bandwidth Profile option called VideoContentPreferencesMode. When set to auto, the Twilio Media Server delivers video content to the client that more closely aligns with the video view size. A RemoteVideoTrack rendered by a VideoView or VideoTextureView with larger dimensions will get a higher quality video compared to a RemoteVideoTrack rendered by a VideoView or VideoTextureView with smaller dimensions. Should a developer want more manual control over the dimensions of the video delivered, then manual mode should be selected and the RemoteVideoTrack.setContentPreferences() method can be used.  

iOS

The story is similar for the iOS SDK with the clientTrackSwitchOffControl and contentPreferencesMode fields added to the Network Bandwidth Profile options. When the clientTrackSwitchOffControl field is set to auto, the SDK determines whether tracks should be switched off based on the renderer attachment, view visibility, and application lifecycle. When set to  manual, developers can explicitly turn tracks off/on using the RemoteVideoTrack.switchOff() / RemoteVideoTrack.switchOn() methods.

When the contentPreferencesMode is set to auto, the Twilio Media Server delivers video content to the client that more closely aligns with the video view size. A RemoteVideoTrack rendered by a VideoView with larger dimensions will get a higher quality video compared to a RemoteVideoTrack rendered by a VideoView with smaller dimensions. Should a developer want more manual control over the dimensions of the video delivered, then manual mode should be selected and the RemoteVideoTrack.setContentPreferences() method can be used.

With this new Network Bandwidth Profile API functionality, you can create a great multi-party experience for your end users, safe in the knowledge that the Twilio Video platform is optimizing the bandwidth and CPU resources for you and ensuring that the video and audio quality is not being compromised.

For additional information and code samples, have a look at the documentation here. Don’t hesitate to reach out if you have any questions. We can't wait to see what you build!