Tutorial: Enable Video Mixer Plugin

Enable Video Mixer Plugin

Function Description

This article introduces how to use the VideoMixer plugin to mix different input sources into a single track.

Prerequisites

  • TRTC Web SDK version >= 5.12.0.

  • Compatibility:

    Browser Compatibility
    Desktop Chrome
    Desktop Safari
    Desktop Firefox
    Desktop Edge
    Mobile Devices ❌Mixing feature is not supported yet

Implementation Steps

Install VideoMixer Plugin

import { VideoMixer } from 'trtc-sdk-v5/plugins/video-effect/video-mixer';
let trtc = TRTC.create({ plugins: [VideoMixer] });

Start VideoMixer Plugin

await trtc.startPlugin('VideoMixer', {
  view: 'local_preview_div', // Preview
  canvasInfo: {
    width: 1920,
    height: 1080
  },
  camera: [
    {
      id: 'camera1',
      layout: {
        x: 0,
        y: 0,
        width: 640,
        height: 480,
        zIndex: 1
      }
    }
  ]
  // ...
});

When starting the plugin, the canvas width and height must be set. Other parameters are optional.

Mix Various Input Sources

Camera

You can add a camera input source. The mixing plugin can control the camera's capture, and the camera parameter array is passed with an object that has a unique id to represent the captured camera source for mixing.

await trtc.updatePlugin('VideoMixer', {
  camera: [
    {
      id: 'camera1',
      profile: { width: 640, height: 480, frameRate: 15 },
      layout: {
        x: 100,
        y: 0,
        width: 640,
        height: 480,
        zIndex: 1
      }
    }
  ]
});

To update the camera parameters, pass the same id and update other parameters.

await trtc.updatePlugin('VideoMixer', {
  camera: [
    {
      id: 'camera1',
      layout: {
        x: 500,
        y: 500,
        width: 640,
        height: 480,
        zIndex: 1,
        rotation: 90,
        mirror: true
      }
    }
  ]
});

To turn off a camera, simply remove the object from the array.

await trtc.updatePlugin('VideoMixer', {
  camera: []
});

Note: To temporarily hide the camera instead of physically turning it off, use the hidden parameter:

await trtc.updatePlugin('VideoMixer', {
  camera: [
    {
      id: 'camera1',
      layout: {
        // ...
        hidden: true,
      },
    },
  ],
});

The same applies to other input sources.

Screen Sharing

You can add a screen sharing input source. The plugin can control screen sharing capture, and the screen parameter array is passed with an object that has a unique id to represent the shared screen capture for mixing.

await trtc.updatePlugin('VideoMixer', {
  screen: [{
    id: 'screen1',
    profile: { width: 1920, height: 1080, frameRate: 15 },
    layout: {
      x: 0,
      y: 0,
      width: 1920,
      height: 1080,
      zIndex: 0,      
    }
  }]
});

To update screen sharing parameters:

await trtc.updatePlugin('VideoMixer', {
  screen: [{
    id: 'screen1',
    layout: {
      x: 100,
      y: 100,
      width: 1000,
      height: 500,
      zIndex: 2,
      rotation: 180,
      mirror: true
    }
  }]
});

To stop screen sharing, remove the object from the array.

await trtc.updatePlugin('VideoMixer', {
  screen: []
});

Text

You can add a text input source. The text parameter array is passed with an object that has a unique id to represent the text source for mixing.

await trtc.updatePlugin('VideoMixer', {
  text: [
    {
      id: 'text1',
      content: 'MultiLine\nTest',
      font: 'bold 60px SimHei',
      color: 'red',
      layout: {
        x: 200,
        y: 300,
        width: 300,
        height: 150,
        zIndex: 6,
      }
    }
  ]
});

Note: If the text content exceeds the layout area, the overflowing portion will be clipped. The application side should adjust the layout width and height according to the text content.

To update text parameters:

await trtc.updatePlugin('VideoMixer', {
  text: [
    {
      id: 'text1',
      content: 'TRTC🥳',
      font: 'bold 120px Georgia',
      color: 'blue',
      layout: {
        x: 100,
        y: 200,
        width: 600,
        height: 150,
        zIndex: 7,
        mirror: true
      }
    }
  ]
});

To delete text:

await trtc.updatePlugin('VideoMixer', {
  text: []
});

Image

You can add an image input source. The image parameter array is passed with an object that has a unique id to represent the image source for mixing.

await trtc.updatePlugin('VideoMixer', {
  image: [{
    id: 'img1',
    url: './image.png',
    layout: {
      x: 0,
      y: 500,
      width: 800,
      height: 400,
      zIndex: 4,
    }
  }]
});

To update image parameters:

await trtc.updatePlugin('VideoMixer', {
  image: [{
    id: 'img1',
    url: './another-img.png',
    layout: {
      x: 100,
      y: 100,
      width: 600,
      height: 500,
      zIndex: 4,
      fillMode: 'fill'
    }
  }]
});

To delete image:

await trtc.updatePlugin('VideoMixer', {
  image: []
});

Video

You can add a video input source. The video parameter array is passed with an object that has a unique id to represent the video source for mixing.

await trtc.updatePlugin('VideoMixer', {
  video: [{
    id: 'video1',
    url: './video.mp4',
    layout: {
      x: 0,
      y: 0,
      width: 1000,
      height: 500,
      zIndex: 5,
    }
  }]
});

To update video parameters:

await trtc.updatePlugin('VideoMixer', {
  video: [{
    id: 'video1',
    url: './another-video.mp4',
    layout: {
      x: 100,
      y: 100,
      width: 1280,
      height: 720,
      zIndex: 5,
    }
  }]
});

To delete video:

await trtc.updatePlugin('VideoMixer', {
  video: []
});

Stop Mixing

Remove all input sources and stop mixing:

await trtc.stopPlugin('VideoMixer');

Publish Mixed Stream

Streaming is not controlled by the Mixing Plugin, but the startPlugin will return the mixed video track, which can be passed as a parameter to interfaces like startLocalVideo to control publishing stream.

const { track } = await trtc.startPlugin('VideoMixer', {
  canvasInfo: {
    width: 1920,
    height: 1080
  },
  // ...
});
// Start publishing
await trtc.startLocalVideo({
  option: {
    videoTrack: track,
    profile: {
      width: 1920,
      height: 1080,
      bitrate: 2000
    }
  }
});

Specify Frame Rate

Specify the frame rate of the mixed canvas:

await trtc.startPlugin('VideoMixer', {
  canvasInfo: {    
    width: 1920,
    height: 1080,
    frameRate: 20
  },
});

Note: Manually setting the mixed frame rate is not recommended, as the plugin automatically calculates the optimal frame rate——when a camera or screen is present, it uses the maximum frame rate between the camera and screen-sharing video; otherwise, it defaults to 15 fps. Manual settings may result in performance degradation.

Error Handling

Since the mixing plugin accepts a large number of parameters when calling start/updatePlugin, to ensure that the failure of some input sources does not affect the normal mixing of other sources, the plugin's error handling strategy is as follows:

  • When some input source parameters are successfully applied and others fail, the interface does not throw an error. Instead, it returns the success and failure information to the business side.
  • When all input source parameters fail, or there are other errors that may interrupt the mixing process, the interface will throw an error with error details.

Example of business logic:

try {
  const { result } = await trtc.updatePlugin('VideoMixer', {
    camera: [/**/],
    screen: [/**/],
    text: [/**/],
    image: [/**/],
    video: [/**/],
  });
  // Error handling when some fail
  result.failedDetails.forEach(({id, error}: {id: string, error: any}) => {
    console.error(`${id} mix failed`, error);
  })
  console.log(result.successOptions) // Information on successfully applied parameters
} catch (error: any) {
  // Error handling when all fail
  console.error(error);
  error?.data?.failedDetails.forEach(({ id, error }: { id: string, error: any }) => {
    console.error(`${id} mix failed`, error);
  })  
}

Detailed Explanation of successOptions

successOptions returns the parameters that were actually applied after the update to the mixing process. Specifically, it means:

  • When no errors occur, successOptions will be the same as the parameters passed in the options.
  • When some errors occur, successOptions will include the successfully updated parameters, and the failed ones will retain the previous values.

For example, suppose the mixing already has a camera and screen input source:

await trtc.startPlugin('VideoMixer', {
  camera: [{/*zIndex = 1*/}],
  screen: [{/*zIndex = 0*/}]
});

Now, if you attempt to update the camera's zIndex to 0 (which will fail because zIndex is duplicated) and update the screen's mirror to true (which will succeed):

await trtc.updatePlugin('VideoMixer', {
  camera: [{/*zIndex = 0*/}],
  screen: [{/*mirror = true*/}]
});

In this case, it’s a partial failure (camera update fails, screen update succeeds), and the returned successOptions will look like this:

{
  camera: [{/*zIndex = 1*/}], // Update failed, so the previous zIndex is still applied
  screen: [{/*mirror = true*/}]  // Update succeeded, so the new parameter is applied
}

In other words, successOptions tells the business side the actual parameters applied after the update, allowing the business side to synchronize the UI or data accordingly.

API Description

trtc.startPlugin('VideoMixer', options)

Used to start the mixing plugin.

options: VideoMixerOptions

Name Type Required Description
view string | HTMLElement | null Optional The HTMLElement instance or id for video preview. If not provided or null, mixed video will not be played.
canvasInfo CanvasInfo Required Parameters for setting up the mixing canvas.
camera CameraSource[] Optional Parameters to control the camera input source.
screen ScreenSource[] Optional Parameters to control the screen share input source.
text TextSource[] Optional Parameters to control the text input source.
image ImageSource[] Optional Parameters to control the image input source.
video VideoSource[] Optional Parameters to control the video input source.
onScreenShareStop (id: string) => {} Optional Callback triggered when screen sharing stops. This may be used when the user stops screen sharing via the browser button instead of the SDK. The callback parameter is the ID of the screen that stopped being captured.

CanvasInfo

Name Type Required Description
canvasColor string | CanvasGradient | CanvasPattern Optional Background color for the mixing canvas. Refer to the canvas fillStyle property.
width number Required Width of the mixing canvas
height number Required Height of the mixing canvas
frameRate number Optional Manually specify the mixing canvas frame rate. Normally, it's better to let the plugin calculate the frame rate automatically to avoid performance loss.

CameraSource

Name Type Required Description
id string Required The unique id of the input source.
layout LayerOption Required Layout parameters for the input source.
cameraId string Optional The camera id to specify which camera to capture.
videoTrack MediaStreamTrack Optional The custom video track for capture.
profile VideoProfile Optional Video capture parameters. Default: 480p_2.

ScreenSource

Name Type Required Description
id string Required The unique id of the input source.
layout LayerOption Required Layout parameters for the input source.
profile ScreenShareProfile Optional Screen capture parameters. Default: 1080p.
captureElement HTMLElement Optional Capture a specific HTMLElement from the current page. Supported on Chrome 104+
preferDisplaySurface 'current-tab' | 'tab' | 'window' | 'monitor' Optional Controls the priority of the screen capture options in the screen sharing dialog. Default is monitor. If current-tab is chosen, only the current page will be displayed. Supported on Chrome 94+.

TextSource

Name Type Required Description
id string Required The unique id of the input source.
layout LayerOption Required Layout parameters for the text source.
content string Required The text content to be displayed.
color string | CanvasGradient | CanvasPattern Optional The color of the text. Refer to the canvas fillStyle property.
font string Optional The font for the text. Refer to the canvas font property.

ImageSource

Name Type Required Description
id string Required The unique id of the input source.
layout LayerOption Required Layout parameters for the image source.
url string Required The URL of the image to be mixed.

VideoSource

Name Type Required Description
id string Required The unique id of the input source.
layout LayerOption Required Layout parameters for the video source.
url string Required The URL of the video to be mixed.

LayerOption

Name Type Required Description
x number Required The horizontal coordinate relative to the top-left corner of the canvas. Can be negative.
y number Required The vertical coordinate relative to the top-left corner of the canvas. Can be negative.
width number Required The layout width of the input source. Must be greater than 0.
height number Required The layout height of the input source. Must be greater than 0.
zIndex number Required The layer index of the input source. The layer index must be unique for each source.
fillMode 'contain' | 'cover' | 'fill' Optional The mode to fill the layout, where cameras default to cover and other sources default to contain. See the CSS object-fit property for more details.
mirror boolean Optional Whether to mirror the input source.
rotation 0 | 90 | 180 | 270 Optional The rotation angle of the source (0, 90, 180, or 270 degrees).
hidden boolean Optional Whether to temporarily hide the input source on the canvas. Use case: To temporarily hide the camera/screen without physically turning it off or destroying the resources.

Returns: Promise<MixParseResult>

MixParseResult

Name Type Description
track MediaStreamTrack The mixed output video track.
result object Contains information about the success or failure of the parameters passed in this update. The structure is as follows:
Name Type Description
successOptions VideoMixerOptions | UpdateVideoMixerOptions The options successfully applied during this update.
failedDetails Array<MixFailedDetail> Details about the options that failed to be applied during this update.

MixFailedDetail

Name Type Description
id string The id of the input source that failed to update.
error Error The reason the update failed.

Example:

// Example 1: Start mixing with only the canvas settings
await trtc.startPlugin('VideoMixer', {
  view: 'local_preview_div',
  canvasInfo: {
    canvasColor: 'green',
    width: 1920,
    height: 1080
  },
});
// Example 2: Start mixing with input sources and log the reason for failure
try {
  const { result } = await trtc.startPlugin('VideoMixer', {
    view: `local_preview_div`,
    canvasInfo: {
      width: 1920,
      height: 1080,
    },
    screen: [
      {
        id: 'screen1',
        profile: { width: 1920, height: 1080, frameRate: 15 },
        preferDisplaySurface: 'tab',
        layout: {
          x: 0,
          y: 0,
          width: 1920,
          height: 1080,
          zIndex: 0,
        },
      },
    ],
    // ...
    onScreenShareStop: (id: string) => {
      console.log(`screen share stop, id: ${id}`);
    },
  });
  result.failedDetails.forEach(({ id, error }: { id: string; error: any }) => {
    console.error(`${id} mix failed`, error);
  });
} catch (error: any) {
  console.error(error);
  error?.data?.failedDetails.forEach(({ id, error }: { id: string, error: any }) => {
    console.error(`${id} mix failed`, error);
  })  
}

trtc.updatePlugin('VideoMixer', options)

options: UpdateVideoMixerOptions

Name Type Required Description
view string | HTMLElement | null Optional The HTMLElement instance or Id for the video preview. If not provided or null, mixed video will not be played.
canvasInfo CanvasInfo Optional Parameters to set up the mixing canvas.
camera CameraSource[] Optional Parameters to control the camera input sources.
screen ScreenSource[] Optional Parameters to control the screen sharing input sources.
text TextSource[] Optional Parameters to control the text input sources.
image ImageSource[] Optional Parameters to control the image input sources.
video VideoSource[] Optional Parameters to control the video input sources.

Returns

The return value is the same as startPlugin.

Example

// Example 1: Remove all cameras
await trtc.updatePlugin('VideoMixer', {
  camera: []
});
// Example 2: Update mixing parameters and log the reason for failure
try {
  const { result } = await trtc.updatePlugin('VideoMixer', {
    camera: [
      {
        id: 'camera1',
        profile: { width: 640, height: 480, frameRate: 30 },
        layout: {
          x: 100,
          y: 0,
          width: 640,
          height: 480,
          zIndex: 1,
        },
      },
    ],
    // ...
  });
  result.failedDetails.forEach(({id, error}: {id: string, error: any}) => {
    console.error(`${id} mix failed`, error);
  })
} catch (error: any) {
  console.error(error);
  error?.data?.failedDetails.forEach(({ id, error }: { id: string, error: any }) => {
    console.error(`${id} mix failed`, error);
  })  
}

trtc.stopPlugin('VideoMixer')

Stops the mixing plugin.

Example:

await trtc.stopPlugin('VideoMixer');

Q&A

1. How to set resolution and bitrate?

  • The resolution refers to the width and height of the canvas.
  • The bitrate is not controlled by the mixing plugin. Instead, it is set during publishing by calling start/updateLocalVideo on the business side.

2. Why are input source parameters passed as arrays, and how can I update a specific type of input source in the array?

  • Each input source type is fully updated in order to support batch updates and deletions (by comparing the arrays before and after).
  • All input source parameters are optional. For example, if you only want to update the camera, you can pass only the camera array in updatePlugin without passing arrays for other types of input sources, or pass them along while keeping the original parameters. Example:
await trtc.startPlugin('VideoMixer', { // Assume camera and screen have already been mixed
  camera: [/* config */],
  screen: [/* config */]
});

Now if you only want to update the camera without updating the screen, there are two ways:

  1. Only pass the camera and omit screen:
await trtc.updatePlugin('VideoMixer', {
  camera: [/* newConfig */], // Only pass the camera array
});
  1. Pass the old screen parameters along with the new camera parameters:
await trtc.updatePlugin('VideoMixer', {
  camera: [/* newConfig */],
  screen: [/* config */] // Keep the parameters for screen unchanged
});

The first approach is generally preferred as it involves fewer parameters for the mixing plugin to process. Although the second approach achieves the same result, it involves redundant parameter parsing.

3. How can I update only one input source when passing an array?

For example, if you want to update a specific camera (id = 1), you should modify only the parameters of the element with id = 1 in the camera array. Other elements that do not require updates should have their original parameters passed along unchanged.

4. Why are the bitrate settings for the camera and screen capture ignored?

The bitrate settings for camera and screen capture are ignored. The upstream bitrate is set by the business side when calling the streaming interface (start/updateLocalVideo).

5. Why isn't the rotation angle set around the center?

The mixing rotation works by rotating the image and keeping the top-left corner of the image at the coordinates provided in the layout. If you want center-based rotation, the business side needs to adjust the coordinates accordingly.

6. How can the business side detect if a user stops screen sharing by clicking the browser button instead of using the SDK?

The onScreenShareStop callback is provided in the startPlugin parameters. Example usage:

await trtc.startPlugin('VideoMixer', {
  // ...
  onScreenShareStop: (id: string) => {
    console.log(`screen share stop, id: ${id}`);
  }
});

Best Practices

  1. Since the start/updatePlugin interface of the mixing plugin accepts many parameters, it is recommended to minimize the transmission of parameters that have not changed and are not required during each update. For example, when updating the camera coordinates frequently, there is no need to pass the capture-related parameters (such as profile, cameraId, etc.), as these only need to be passed once. This helps reduce the overhead of parsing parameters in the plugin's underlying layer.
  2. The larger the canvas resolution and the more input sources, the greater the rendering overhead. The business side needs to consider the device's capabilities to determine the appropriate mixing parameters.