Tutorial: Enabling Beauty Filters

Feature Description

TRTC can use the RTCBeautyPlugin to help developers easily implement basic beauty filter features. Users can adjust beauty filter parameters in the plugin to achieve natural beauty filter effects. You can also add portrait segmentation to the video through the plug-in to present the effect of background bokeh or virtual background during the call. Click here to experience the effect.

Note

• Only the listed platforms are supported.
• The calculation and rendering of beauty and portrait segmentation are performance-consuming, so if the user's device performance is not enough to support the calculation, it is recommended not to enable the beauty and portrait segmentation functions.
  • Performance data for a device with a 2.6 GHz hexa-core Intel Core i7 16G memory processor.
    • For a 480p 15 fps video, the CPU usage is about 10% more than a normal audio/video call when using the beauty process.
    • 480p 15 fps video with background blur processing uses about 25% more CPU than a normal audio/video call.
    • For 480p 15 fps video, using virtual background processing, the CPU usage is about 30% more than normal audio/video calls, which may lead to higher performance consumption if the resolution of the custom background is too high.

Integration Instructions

Prerequisites

To use the beauty capabilities of RTCBeautyPlugin, please upgrade the TRTC SDK to v4.11.1 and above. To use the portrait segmentation capabilities of RTCBeautyPlugin, please upgrade the TRTC SDK to v4.11.10 and above.

Install the RTCBeautyPlugin in the project:

npm install rtc-beauty-plugin

Step 1. Create an RTCBeautyPlugin instance

One RTCBeautyPlugin instance can be used to process only one local audio/video stream.

const beautyPlugin = new RTCBeautyPlugin();

Step 2. Use the RTCBeautyPlugin instance to process the stream to be published

const beautyStream = beautyPlugin.generateBeautyStream(localStream);
// Publish the retouched stream
await client.publish(beautyStream);

API Description

generateBeautyStream(localStream)

This API is used to convert a local stream (localStream) into a retouched stream (beautyStream).

// Initialize the RTCBeautyPlugin
const beautyPlugin = new RTCBeautyPlugin();
await localStream.initialize();
// Generate a retouched stream
const beautyStream = beautyPlugin.generateBeautyStream(localStream);
// Publish the retouched stream
await client.publish(beautyStream);

setBeautyParam(options)

Feature: adjust beauty filter parameters in the RTCBeautyPlugin.

To disable beautification, you only need to set all the three parameters listed in the table below to 0.

Parameters

beautyPlugin.setBeautyParam({ beauty: 0.5, brightness: 0.5, ruddy: 0.5 });
// To disable beautification, you only need to set all the three parameters to `0`. To enable beautification, you only need to set any one of the parameters to a value greater than 0.
beautyPlugin.setBeautyParam({ beauty: 0, brightness: 0, ruddy: 0 });

(async) loadResources()

Feature： If you need to use the virtual background capability, you can call the loadResources() method at the beginning to load the resources for the virtual background when computing.

const beautyPlugin = new RTCBeautyPlugin();
// load resources
await beautyPlugin.loadResources();

(async) generateVirtualStream({ localStream, type, img })

Processes the localStream into a virtualStream after virtual background processing.

Params:

const beautyPlugin = new RTCBeautyPlugin();
await beautyPlugin.loadResources();
await localStream.initialize();
const virtualStream = await beautyPlugin.generateVirtualStream({ 
  localStream: localStream,
  type: 'blur'
});
await client.publish(virtualStream);

const beautyPlugin = new RTCBeautyPlugin();
await beautyPlugin.loadResources();
await localStream.initialize();
const virtualStream = await beautyPlugin.generateVirtualStream({
  localStream: localStream,
  type: 'virtual',
  img: document.getElementById('img'),
});
await client.publish(virtualStream);

(async) generateStream({ localStream, type, img })

Generate stream with both virtual background processing and beauty processing.

Params:

const beautyPlugin = new RTCBeautyPlugin();
await beautyPlugin.loadResources();
await localStream.initialize();
const stream = await beautyPlugin.generateStream({ 
  localStream: this.localStream_,	  
  type: 'virtual',	  
  img: document.getElementById('img'),
});
await client.publish(stream);

setBackground(img)

功能： Used to dynamically change the background image when using the custom background feature of virtual backgrounds.

Params:

beautyPlugin.setBackground(document.getElementById('newImg'));

destroy()

Feature: terminate the RTCBeautyPlugin.

Usage: after publishing is completed, you can terminate the RTCBeautyPlugin to stop memory usage and performance consumption.

await client.leave();
beautyPlugin.destroy();

FAQs

1. One RTCBeautyPlugin instance can be used to process only one local audio/video stream.

2. The replaceTrack operation will cause the beauty filter effects of localStream to disappear. Please perform it with caution.