Tutorial: Enable Video Mixer Plugin

Function Description

This article introduces how to use the VideoMixer plugin to mix different input sources into a single track.

Prerequisites

• TRTC Web SDK version >= 5.12.0.

• Compatibility:

  Browser	Compatibility
  Desktop Chrome	✅
  Desktop Safari	✅
  Desktop Firefox	✅
  Desktop Edge	✅
  Mobile Devices	❌Mixing feature is not supported yet

Implementation Steps

Install VideoMixer Plugin

import { VideoMixer } from 'trtc-sdk-v5/plugins/video-effect/video-mixer';
let trtc = TRTC.create({ plugins: [VideoMixer] });

Start VideoMixer Plugin

await trtc.startPlugin('VideoMixer', {
  view: 'local_preview_div', // Preview
  canvasInfo: {
    width: 1920,
    height: 1080
  },
  camera: [
    {
      id: 'camera1',
      layout: {
        x: 0,
        y: 0,
        width: 640,
        height: 480,
        zIndex: 1
      }
    }
  ]
  // ...
});

When starting the plugin, the canvas width and height must be set. Other parameters are optional.

Mix Various Input Sources

Camera

You can add a camera input source. The mixing plugin can control the camera's capture, and the camera parameter array is passed with an object that has a unique id to represent the captured camera source for mixing.

await trtc.updatePlugin('VideoMixer', {
  camera: [
    {
      id: 'camera1',
      profile: { width: 640, height: 480, frameRate: 15 },
      layout: {
        x: 100,
        y: 0,
        width: 640,
        height: 480,
        zIndex: 1
      }
    }
  ]
});

To update the camera parameters, pass the same id and update other parameters.

await trtc.updatePlugin('VideoMixer', {
  camera: [
    {
      id: 'camera1',
      layout: {
        x: 500,
        y: 500,
        width: 640,
        height: 480,
        zIndex: 1,
        rotation: 90,
        mirror: true
      }
    }
  ]
});

To turn off a camera, simply remove the object from the array.

await trtc.updatePlugin('VideoMixer', {
  camera: []
});

Note: To temporarily hide the camera instead of physically turning it off, use the hidden parameter:

await trtc.updatePlugin('VideoMixer', {
  camera: [
    {
      id: 'camera1',
      layout: {
        // ...
        hidden: true,
      },
    },
  ],
});

The same applies to other input sources.

Screen Sharing

You can add a screen sharing input source. The plugin can control screen sharing capture, and the screen parameter array is passed with an object that has a unique id to represent the shared screen capture for mixing.

await trtc.updatePlugin('VideoMixer', {
  screen: [{
    id: 'screen1',
    profile: { width: 1920, height: 1080, frameRate: 15 },
    layout: {
      x: 0,
      y: 0,
      width: 1920,
      height: 1080,
      zIndex: 0,      
    }
  }]
});

To update screen sharing parameters:

await trtc.updatePlugin('VideoMixer', {
  screen: [{
    id: 'screen1',
    layout: {
      x: 100,
      y: 100,
      width: 1000,
      height: 500,
      zIndex: 2,
      rotation: 180,
      mirror: true
    }
  }]
});

To stop screen sharing, remove the object from the array.

await trtc.updatePlugin('VideoMixer', {
  screen: []
});

Text

You can add a text input source. The text parameter array is passed with an object that has a unique id to represent the text source for mixing.

await trtc.updatePlugin('VideoMixer', {
  text: [
    {
      id: 'text1',
      content: 'MultiLine\nTest',
      font: 'bold 60px SimHei',
      color: 'red',
      layout: {
        x: 200,
        y: 300,
        width: 300,
        height: 150,
        zIndex: 6,
      }
    }
  ]
});

Note: If the text content exceeds the layout area, the overflowing portion will be clipped. The application side should adjust the layout width and height according to the text content.

To update text parameters:

await trtc.updatePlugin('VideoMixer', {
  text: [
    {
      id: 'text1',
      content: 'TRTC🥳',
      font: 'bold 120px Georgia',
      color: 'blue',
      layout: {
        x: 100,
        y: 200,
        width: 600,
        height: 150,
        zIndex: 7,
        mirror: true
      }
    }
  ]
});

To delete text:

await trtc.updatePlugin('VideoMixer', {
  text: []
});

Image

You can add an image input source. The image parameter array is passed with an object that has a unique id to represent the image source for mixing.

await trtc.updatePlugin('VideoMixer', {
  image: [{
    id: 'img1',
    url: './image.png',
    layout: {
      x: 0,
      y: 500,
      width: 800,
      height: 400,
      zIndex: 4,
    }
  }]
});

To update image parameters:

await trtc.updatePlugin('VideoMixer', {
  image: [{
    id: 'img1',
    url: './another-img.png',
    layout: {
      x: 100,
      y: 100,
      width: 600,
      height: 500,
      zIndex: 4,
      fillMode: 'fill'
    }
  }]
});

To delete image:

await trtc.updatePlugin('VideoMixer', {
  image: []
});

Video

You can add a video input source. The video parameter array is passed with an object that has a unique id to represent the video source for mixing.

await trtc.updatePlugin('VideoMixer', {
  video: [{
    id: 'video1',
    url: './video.mp4',
    layout: {
      x: 0,
      y: 0,
      width: 1000,
      height: 500,
      zIndex: 5,
    }
  }]
});

To update video parameters:

await trtc.updatePlugin('VideoMixer', {
  video: [{
    id: 'video1',
    url: './another-video.mp4',
    layout: {
      x: 100,
      y: 100,
      width: 1280,
      height: 720,
      zIndex: 5,
    }
  }]
});

To delete video:

await trtc.updatePlugin('VideoMixer', {
  video: []
});

Stop Mixing

Remove all input sources and stop mixing:

await trtc.stopPlugin('VideoMixer');

Publish Mixed Stream

Streaming is not controlled by the Mixing Plugin, but the startPlugin will return the mixed video track, which can be passed as a parameter to interfaces like startLocalVideo to control publishing stream.

const { track } = await trtc.startPlugin('VideoMixer', {
  canvasInfo: {
    width: 1920,
    height: 1080
  },
  // ...
});
// Start publishing
await trtc.startLocalVideo({
  option: {
    videoTrack: track,
    profile: {
      width: 1920,
      height: 1080,
      bitrate: 2000
    }
  }
});

Specify Frame Rate

Specify the frame rate of the mixed canvas:

await trtc.startPlugin('VideoMixer', {
  canvasInfo: {    
    width: 1920,
    height: 1080,
    frameRate: 20
  },
});

Note: Manually setting the mixed frame rate is not recommended, as the plugin automatically calculates the optimal frame rate——when a camera or screen is present, it uses the maximum frame rate between the camera and screen-sharing video; otherwise, it defaults to 15 fps. Manual settings may result in performance degradation.

Error Handling

Since the mixing plugin accepts a large number of parameters when calling start/updatePlugin, to ensure that the failure of some input sources does not affect the normal mixing of other sources, the plugin's error handling strategy is as follows:

• When some input source parameters are successfully applied and others fail, the interface does not throw an error. Instead, it returns the success and failure information to the business side.
• When all input source parameters fail, or there are other errors that may interrupt the mixing process, the interface will throw an error with error details.

Example of business logic:

try {
  const { result } = await trtc.updatePlugin('VideoMixer', {
    camera: [/**/],
    screen: [/**/],
    text: [/**/],
    image: [/**/],
    video: [/**/],
  });
  // Error handling when some fail
  result.failedDetails.forEach(({id, error}: {id: string, error: any}) => {
    console.error(`${id} mix failed`, error);
  })
  console.log(result.successOptions) // Information on successfully applied parameters
} catch (error: any) {
  // Error handling when all fail
  console.error(error);
  error?.data?.failedDetails.forEach(({ id, error }: { id: string, error: any }) => {
    console.error(`${id} mix failed`, error);
  })  
}

Detailed Explanation of successOptions

successOptions returns the parameters that were actually applied after the update to the mixing process. Specifically, it means:

• When no errors occur, successOptions will be the same as the parameters passed in the options.
• When some errors occur, successOptions will include the successfully updated parameters, and the failed ones will retain the previous values.

For example, suppose the mixing already has a camera and screen input source:

await trtc.startPlugin('VideoMixer', {
  camera: [{/*zIndex = 1*/}],
  screen: [{/*zIndex = 0*/}]
});

Now, if you attempt to update the camera's zIndex to 0 (which will fail because zIndex is duplicated) and update the screen's mirror to true (which will succeed):

await trtc.updatePlugin('VideoMixer', {
  camera: [{/*zIndex = 0*/}],
  screen: [{/*mirror = true*/}]
});

In this case, it’s a partial failure (camera update fails, screen update succeeds), and the returned successOptions will look like this:

{
  camera: [{/*zIndex = 1*/}], // Update failed, so the previous zIndex is still applied
  screen: [{/*mirror = true*/}]  // Update succeeded, so the new parameter is applied
}

In other words, successOptions tells the business side the actual parameters applied after the update, allowing the business side to synchronize the UI or data accordingly.

API Description

trtc.startPlugin('VideoMixer', options)

Used to start the mixing plugin.

options: VideoMixerOptions

CanvasInfo

CameraSource

ScreenSource

TextSource

ImageSource

VideoSource

LayerOption

Returns: Promise<MixParseResult>

MixParseResult

MixFailedDetail

Example:

// Example 1: Start mixing with only the canvas settings
await trtc.startPlugin('VideoMixer', {
  view: 'local_preview_div',
  canvasInfo: {
    canvasColor: 'green',
    width: 1920,
    height: 1080
  },
});
// Example 2: Start mixing with input sources and log the reason for failure
try {
  const { result } = await trtc.startPlugin('VideoMixer', {
    view: `local_preview_div`,
    canvasInfo: {
      width: 1920,
      height: 1080,
    },
    screen: [
      {
        id: 'screen1',
        profile: { width: 1920, height: 1080, frameRate: 15 },
        preferDisplaySurface: 'tab',
        layout: {
          x: 0,
          y: 0,
          width: 1920,
          height: 1080,
          zIndex: 0,
        },
      },
    ],
    // ...
    onScreenShareStop: (id: string) => {
      console.log(`screen share stop, id: ${id}`);
    },
  });
  result.failedDetails.forEach(({ id, error }: { id: string; error: any }) => {
    console.error(`${id} mix failed`, error);
  });
} catch (error: any) {
  console.error(error);
  error?.data?.failedDetails.forEach(({ id, error }: { id: string, error: any }) => {
    console.error(`${id} mix failed`, error);
  })  
}

trtc.updatePlugin('VideoMixer', options)

options: UpdateVideoMixerOptions

Returns

The return value is the same as startPlugin.

Example

// Example 1: Remove all cameras
await trtc.updatePlugin('VideoMixer', {
  camera: []
});
// Example 2: Update mixing parameters and log the reason for failure
try {
  const { result } = await trtc.updatePlugin('VideoMixer', {
    camera: [
      {
        id: 'camera1',
        profile: { width: 640, height: 480, frameRate: 30 },
        layout: {
          x: 100,
          y: 0,
          width: 640,
          height: 480,
          zIndex: 1,
        },
      },
    ],
    // ...
  });
  result.failedDetails.forEach(({id, error}: {id: string, error: any}) => {
    console.error(`${id} mix failed`, error);
  })
} catch (error: any) {
  console.error(error);
  error?.data?.failedDetails.forEach(({ id, error }: { id: string, error: any }) => {
    console.error(`${id} mix failed`, error);
  })  
}

trtc.stopPlugin('VideoMixer')

Stops the mixing plugin.

Example:

await trtc.stopPlugin('VideoMixer');

Q&A

1. How to set resolution and bitrate?

• The resolution refers to the width and height of the canvas.
• The bitrate is not controlled by the mixing plugin. Instead, it is set during publishing by calling start/updateLocalVideo on the business side.

2. Why are input source parameters passed as arrays, and how can I update a specific type of input source in the array?

• Each input source type is fully updated in order to support batch updates and deletions (by comparing the arrays before and after).
• All input source parameters are optional. For example, if you only want to update the camera, you can pass only the camera array in updatePlugin without passing arrays for other types of input sources, or pass them along while keeping the original parameters. Example:

await trtc.startPlugin('VideoMixer', { // Assume camera and screen have already been mixed
  camera: [/* config */],
  screen: [/* config */]
});

Now if you only want to update the camera without updating the screen, there are two ways:

1. Only pass the camera and omit screen:

await trtc.updatePlugin('VideoMixer', {
  camera: [/* newConfig */], // Only pass the camera array
});

2. Pass the old screen parameters along with the new camera parameters:

await trtc.updatePlugin('VideoMixer', {
  camera: [/* newConfig */],
  screen: [/* config */] // Keep the parameters for screen unchanged
});

The first approach is generally preferred as it involves fewer parameters for the mixing plugin to process. Although the second approach achieves the same result, it involves redundant parameter parsing.

3. How can I update only one input source when passing an array?

For example, if you want to update a specific camera (id = 1), you should modify only the parameters of the element with id = 1 in the camera array. Other elements that do not require updates should have their original parameters passed along unchanged.

4. Why are the bitrate settings for the camera and screen capture ignored?

The bitrate settings for camera and screen capture are ignored. The upstream bitrate is set by the business side when calling the streaming interface (start/updateLocalVideo).

5. Why isn't the rotation angle set around the center?

The mixing rotation works by rotating the image and keeping the top-left corner of the image at the coordinates provided in the layout. If you want center-based rotation, the business side needs to adjust the coordinates accordingly.

6. How can the business side detect if a user stops screen sharing by clicking the browser button instead of using the SDK?

The onScreenShareStop callback is provided in the startPlugin parameters. Example usage:

await trtc.startPlugin('VideoMixer', {
  // ...
  onScreenShareStop: (id: string) => {
    console.log(`screen share stop, id: ${id}`);
  }
});

Best Practices

1. Since the start/updatePlugin interface of the mixing plugin accepts many parameters, it is recommended to minimize the transmission of parameters that have not changed and are not required during each update. For example, when updating the camera coordinates frequently, there is no need to pass the capture-related parameters (such as profile, cameraId, etc.), as these only need to be passed once. This helps reduce the overhead of parsing parameters in the plugin's underlying layer.
2. The larger the canvas resolution and the more input sources, the greater the rendering overhead. The business side needs to consider the device's capabilities to determine the appropriate mixing parameters.