Tutorial: Enable Video Mixer Plugin

Overview

This document provides a detailed guide on how to use the TRTC Web SDK's VideoMixer plugin to composite multiple input sources including camera, screen sharing, text, images, and videos into a single video track for streaming with custom layouts. Developers can flexibly control the layout (position, size, layer zIndex) of each source.

Prerequisites

• TRTC Web SDK version >= 5.12.0.

• Compatibility:

  Browser	Compatibility
  PC Chrome	✅
  PC Safari	✅
  PC Firefox	✅
  PC Edge	✅
  Mobile	❌ Mixing not supported yet

Implementation

1. Import and Register Plugin

import { VideoMixer } from 'trtc-sdk-v5/plugins/video-effect/video-mixer';
const trtc = TRTC.create({ plugins: [VideoMixer] });

2. Start Mixing

Start the mixing plugin by calling trtc.startPlugin('VideoMixer', options). You must set the canvas width and height canvasInfo in options, all other parameters are optional.

const options = {
  view: 'local_preview_div', // Preview
  canvasInfo: { width: 1920, height: 1080 },
  camera: [{ // Add a camera input source
      id: 'camera1',
      layout: { x: 0, y: 0, width: 640, height: 480, zIndex: 1 }
    }]
  // ...
}
const { track } = await trtc.startPlugin('VideoMixer', options);
// track is the mixed output MediaStreamTrack, which can be used for streaming

3. Publish After Mixing

The mixing plugin is only responsible for generating the mixed video track. Publishing operation is still controlled by the TRTC SDK. Pass the track returned by startPlugin as a parameter to the startLocalVideo interface to publish.

await trtc.startLocalVideo({ option: { 
  videoTrack: track,
  profile: { width: 1920, height: 1080, bitrate: 2000 } // Set encoding parameters
}});

4. Manage Input Sources (Update and Remove)

Dynamically add, update, or remove mixing input sources by calling trtc.updatePlugin('VideoMixer', options).

Update Rules:

• Input Source Description: Input sources are divided into: camera, screen, text, image, video. All are arrays, each array element represents an input source, uniquely identified by the id attribute.
• Update Input Source: Pass the same id and modify other parameters.
• Add Input Source: Pass a new id object.
• Remove Input Source: Remove the object from the corresponding input source array.

Camera

Control camera capture and layout.

// Update camera layout with id 'camera1'  
await trtc.updatePlugin('VideoMixer', {
  camera: [
    {
      id: 'camera1',
      profile: { width: 640, height: 480, frameRate: 15 }, // Update camera capture parameters
      layout: { x: 100, y: 0, width: 640, height: 480, zIndex: 1, rotation: 90, mirror: true } // Update camera layout parameters
    }
  ]
});
// Remove all cameras  
await trtc.updatePlugin('VideoMixer', { camera: [] });
// If you only want to temporarily hide the input source on the canvas without physically closing capture, you can set hidden: true in the layout parameter. The same applies to other input sources.

Screen Sharing

// Add screen sharing
await trtc.updatePlugin('VideoMixer', {
  screen: [{
    id: 'screen1',
    profile: { width: 1920, height: 1080, frameRate: 15 }, // Screen sharing capture parameters
    layout: {
      x: 0,
      y: 0,
      width: 1920,
      height: 1080,
      zIndex: 0,      
    }
  }],
  // If the user stops screen sharing via browser button instead of SDK, you can listen through this callback
  onScreenShareStop: (id: string) => console.log(`Screen sharing stopped, id: ${id}`);
});

Text

Add text input sources by passing an object with a unique id in the text parameter array to represent a text source in the mix:

Note: When text content exceeds the layout area, the overflowing part will be clipped. You need to adjust the layout width and height according to text content.

await trtc.updatePlugin('VideoMixer', {
  text: [
    {
      id: 'text1',
      content: 'MultiLine\nTest',
      font: 'bold 60px SimHei',
      color: 'red',
      layout: {
        x: 200,
        y: 300,
        width: 300,
        height: 150,
        zIndex: 6,
      }
    }
  ]
});

Images and Videos

Add images (such as logos) or local video files as mixing input sources.

await trtc.updatePlugin('VideoMixer', {
  image: [{ 
    id: 'img1', 
    url: './image.png',
    layout: { x: 0, y: 500, width: 800, height: 400, zIndex: 4 }
  }],
  video: [{ 
    id: 'video1', 
    url: './video.mp4',
    layout: { x: 0, y: 0, width: 1000, height: 500, zIndex: 5 }
  }]
});

5. Stop Mixing

Remove all input sources and stop mixing:

await trtc.stopPlugin('VideoMixer');

Error Handling

The error handling strategy of the mixing plugin is designed to ensure that errors in some input sources do not affect the mixing of other normal input sources.

Business logic example:

try {
  const { result } = await trtc.updatePlugin('VideoMixer', {
    camera: [/* ... */],
    screen: [/* ... */],
    text: [/* ... */],
    image: [/* ... */],
    video: [/* ... */],
  });
  // Scenario 1: Error handling for partial failures
  result.failedDetails.forEach(({id, error}: {id: string, error: any}) => {
    console.error(`${id} mix failed`, error);
  })
  console.log(result.successOptions) // Successfully applied options for this update
} catch (error: any) {
  // Scenario 2: Error catching for all failures (catching thrown errors)  
  console.error(error);
  error?.data?.failedDetails.forEach(({ id, error }: { id: string, error: any }) => {
    console.error(`${id} mix failed`, error);
  })  
}

Detailed Explanation of successOptions

This return value informs the business side of the parameters actually applied by the mixing underlying layer after this update. When some updates fail, successOptions will include the successfully updated parts and the failed parts that retain the old configuration, making it convenient for the business side to synchronize UI or data status.

API Reference

trtc.startPlugin('VideoMixer', options)

Used to start the mixing plugin and initialize canvas and input sources.

Options

Returns: Promise<MixParseResult>

MixParseResult

MixFailedDetail

trtc.updatePlugin('VideoMixer', options)

Update mixing input sources and layout. The options structure is the same as startPlugin, canvasInfo is not required when updating.

trtc.stopPlugin('VideoMixer')

Stop mixing.

Core Parameter Structures

CanvasInfo

CameraSource

ScreenSource

TextSource

ImageSource

VideoSource

LayerOption

Best Practices and FAQs

1. Frame Rate and Performance Optimization

• Frame Rate Settings:
  • Not Recommended: manually setting canvasInfo.frameRate.
  • Recommended: let the plugin automatically calculate internally: when camera or screen sharing exists, take the maximum frame rate of the two; otherwise fixed at 15 fps. Manual specification may cause performance loss.
• Performance Considerations:
  • The larger the canvas resolution and the more mixing input sources, the greater the rendering overhead. The business side should determine appropriate canvas parameters and number of input sources based on user device performance.

2. How to Set Resolution and Bitrate?

• Resolution is the canvas width and height
• Bitrate is not controlled by the mixing plugin, it's set by the business side when publishing through startLocalVideo/updateLocalVideo.

3. Why Are Input Source Parameters in Array Form? How to Update Only One Type of Input Source in the Array?

• Each type of input source is updated in full. The purpose is to support batch updates and deletions (by comparing arrays passed before and after).

• Each input source parameter is optional. For example, if you only want to update camera, only pass the camera array when calling updatePlugin, and don't pass arrays for other types of input sources, or pass them together but keep the original parameters. Example:

  await trtc.startPlugin('VideoMixer', { // Assume camera and screen are already mixed
    camera: [/* config */],
    screen: [/* config */]
  });
  

  At this point, if you only want to update camera and not screen, you can pass only camera without passing screen

  await trtc.updatePlugin('VideoMixer', {
    camera: [/* newConfig */], // Only pass camera array
  });
  

4. Given an Array, How to Update Only One Input Source?

For example, to update a camera with id = 1, only modify the parameters of the element with id = 1 in the camera array passed in. Although other elements don't need to be updated, their parameters need to be passed in as is.

5. Why Is the Bitrate Set When Capturing Camera and Screen Invalid?

The bitrate set when capturing camera and screen will be ignored. The uplink bitrate is set by the interface (start/updateLocalVideo) called when the business side publishes.

6. Why Doesn't Rotation Happen Around the Center?

The principle of mixing rotation is to rotate the image and then draw it while keeping the upper left corner of the image at the coordinates passed in by layout. If you want to achieve center rotation, the business side needs to adjust the coordinates accordingly.

7. If User Stops Screen Sharing via Browser Button Instead of SDK During Mixing Screen Sharing, How Can the Business Side Detect This?

The onScreenShareStop callback is provided in the parameters of startPlugin. Usage example:

await trtc.startPlugin('VideoMixer', {
  // ...
  onScreenShareStop: (id: string) => {
    console.log(`screen share stop, id: ${id}`);
  }
});