Tutorial: Screen Share Annotation

Feature Description

This article describes how to annotate on a shared screen in real time. After the sharer (the host) starts screen sharing, the host and the viewers (the guests) can draw on the shared screen together. Remote users receive a composited "screen + annotation" picture.

• Omitting userId means the local user is the host. Passing userId means the local user is a guest opening that user's annotation.
• Screen share annotation and the Interactive Whiteboard are mutually exclusive. Only one of them can run at a time.
• Demo RoomKit

Core Advantages

Conventional annotation solutions usually require a separate annotation SDK and service: strokes are synchronized over an independent signaling channel, separate from the audio/video path, so viewing endpoints, recording and playback all need extra adaptation. TRTC Screen Share Annotation composites the annotation directly into the screen share picture on the host side, and what is published to remote users is an ordinary video stream (the sub stream). As a result:

Prerequisites

• TRTC Web SDK version >= 5.21.0

• Compatibility:

  OS	Browser	Host (sharer)	Guest (viewer)
  Windows / macOS	Chrome 86+, Edge 86+ and other Chromium 86+ based browsers	✅	✅
  macOS	Safari 15.4+	✅	✅
  Windows / macOS	Firefox 117+	✅	✅
  Android	Chrome 86+, WeChat built-in browser and other Chromium 86+ based browsers	❌ Screen sharing and mixing are not supported on mobile	✅
  iOS 15.4+	Safari, WeChat built-in browser, etc.	❌ Screen sharing and mixing are not supported on mobile	✅

  Users who only pull streams and do not take part in annotation are not limited by the table above: the annotation is composited into the sub stream picture, so it is visible in any environment that supports TRTC stream pulling. See Browsers Supported.

• The role must be anchor (the audience role cannot call annotation APIs).

Implementation

Import and Register Plugins

Annotation depends on VideoMixer (composites "screen + annotation") and Whiteboard (the annotation plugin), and requires enableSEI.

import TRTC from 'trtc-sdk-v5';
import { VideoMixer } from 'trtc-sdk-v5/plugins/video-effect/video-mixer';
import { Whiteboard } from 'trtc-sdk-v5/plugins/whiteboard';
import AudioPlayer from 'trtc-sdk-v5/plugins/audio-player'; // optional, see below
const trtc = TRTC.create({
  plugins: [VideoMixer, Whiteboard, AudioPlayer],
  enableSEI: true
});

• If VideoMixer / Whiteboard are not registered, startScreenShare works as a plain screen share and is not affected at all. Calling annotation APIs explicitly throws an error.
• AudioPlayer is optional. It plays the audio of videos inside windows. Without it, those videos are silent; annotation is not affected.

Host: Start Screen Sharing

With the plugins registered and enableSEI enabled, startScreenShare starts annotation automatically, and remote users receive the composited "screen + annotation" picture. At this point the host is read-only locally: no toolbar and no drawing. Guests can draw once they open annotation.

await trtc.startScreenShare({
  view: 'local-screen',
  option: {
    profile: '1080p_2',  // recommended to set explicitly
    fillMode: 'contain'  // contain / cover recommended
  }
});

Host: Turn Local Annotation On / Off

// Turn on local annotation: shows the toolbar and allows drawing
await trtc.startAnnotation();
// Turn off local annotation: hides the toolbar and disables drawing.
// The annotation stays in the composited picture, and guests can still draw.
await trtc.stopAnnotation();

stopAnnotation() only turns off the host's own interaction. It does not end the annotation session. To end it, call stopScreenShare().

Guest: Open the Annotation

A guest must subscribe to the host's screen share (sub stream) first, then open the annotation. Whether it succeeds depends on the remote side, so wrap it in try/catch.

await trtc.startRemoteVideo({
  userId: 'host_user_id',
  streamType: TRTC.TYPE.STREAM_TYPE_SUB,
  view: 'remote-screen'
});
try {
  await trtc.startAnnotation({ userId: 'host_user_id' });
} catch (error) {
  // The remote user hasn't started screen sharing / the sub stream isn't subscribed / timed out, etc.
  console.warn(error.message);
}

• You must subscribe to the sub stream with startRemoteVideo before calling startAnnotation({ userId }). The reverse order throws an error.
• Either the host or the guest can start first. The SDK waits for the remote side to be ready (up to 10s).
• A guest can open annotation for only one host at a time. Passing a new userId switches the target host.

Update Parameters as Needed

// Host: switch the theme
await trtc.updateAnnotation({ option: { theme: 'dark' } });
// Guest: open in read-only mode (watch without drawing, keep annotation open)
await trtc.updateAnnotation({ userId: 'host_user_id', option: { interactive: false } });
// Host: switch the preview container; the annotation moves with the video
await trtc.updateScreenShare({ view: 'another-view' });
// Guest: switch the playback container; the annotation moves with the video
await trtc.updateRemoteVideo({ userId: 'host_user_id', streamType: TRTC.TYPE.STREAM_TYPE_SUB, view: 'another-view' });

End the Annotation

// Guest: close the annotation
await trtc.stopAnnotation({ userId: 'host_user_id' });
// Host: stop screen sharing; the annotation ends with it
await trtc.stopScreenShare();

Leaving and Switching Rooms

exitRoom() ends the annotation session but keeps screen sharing, following the SDK's existing behavior: the local preview is played back into your view, and the stream republished after re-entering the room is the raw screen capture (without annotation). The same applies when switchRoom() falls back to "exit room + enter room".

To keep annotating after leaving, the host must call startAnnotation() again after re-entering, guests must call startAnnotation({ userId }) again, and guests that were following this host need to start following it again.

Screen capture keeps running while you are out of the room, so there is no need to call startScreenShare() again. Only stopScreenShare() actually stops it.

API Description

trtc.startAnnotation(options)

• Host: turns on local annotation and shows the toolbar (idempotent).
• Guest: opens the specified user's annotation.

options

Example:

// host
await trtc.startAnnotation({ option: { theme: 'dark' } });
// guest
await trtc.startAnnotation({ userId: 'host_user_id' });

trtc.updateAnnotation(options)

Updates annotation parameters incrementally. Only the fields passed in are changed.

options

trtc.stopAnnotation(options)

• Host: turns off local annotation and hides the toolbar. The annotation stays in the composited picture, and guests can still draw.
• Guest: closes the annotation.

options

Events

Annotation state changes are notified via TRTC.EVENT.ANNOTATION_STATE_CHANGED. The callback receives { userId, type, payload }:

Handle only the three type values above and ignore any others.

available: Annotation Is Available

Show / enable your annotation entry when you receive it. Values of payload.reason:

trtc.on(TRTC.EVENT.ANNOTATION_STATE_CHANGED, ({ userId, type, payload }) => {
  if (type !== 'available') return;
  // Show / enable the annotation entry
  showAnnotationEntry(userId);
  if (payload.reason === 'resume') {
    hideBanner(); // dismiss notices such as "Connection lost"
  }
});

unavailable: Annotation Is Unavailable

Hide / disable your annotation entry when you receive it. Whether to notify the user or call an API again depends on payload.reason:

• The host only receives 'end'.
• A session may receive unavailable several times (e.g. 'host-lost' and then 'end'). Use the most recent one.

trtc.on(TRTC.EVENT.ANNOTATION_STATE_CHANGED, ({ userId, type, payload }) => {
  if (type !== 'unavailable') return;
  disableAnnotationEntry(userId);
  switch (payload.reason) {
    case 'end':
      showToast('The presenter ended annotation');
      break;
    case 'local-stop':
      // The local user closed the annotation / switched hosts; no notice needed
      break;
    case 'silence':
      showBanner('Waiting for the presenter to start annotation');
      break;
    case 'host-lost':
      showBanner('Connection lost, reconnecting');
      break;
    case 'queue-full':
    case 'ack-timeout':
      showBanner('Syncing, please wait');
      break;
  }
});

error: Annotation Runtime Error

An asynchronous error during runtime. payload is an Error object whose message carries the reason. A failed automatic annotation start with startScreenShare also emits this event, including missing plugins, a failed VideoMixer start, or a failed whiteboard plugin start. Screen sharing is still published in that case, just without annotation; if annotation is needed, ask the user to restart screen sharing.

Failed API calls (e.g. startAnnotation rejected) are handled with try/catch; see "Error Handling" below.

trtc.on(TRTC.EVENT.ANNOTATION_STATE_CHANGED, ({ userId, type, payload }) => {
  if (type !== 'error') return;
  console.error('[annotation] error', payload.message);
  reportError(payload);
});

Error Handling

The error code for annotation is TRTC.ERROR_CODE.INVALID_OPERATION (5100), distinguished by extraCode:

try {
  await trtc.startAnnotation({ userId: 'host_user_id' });
} catch (error) {
  if (error.code === TRTC.ERROR_CODE.INVALID_OPERATION && error.extraCode === 5110) {
    console.warn(error.message);
  }
}

FAQ

1. The host cannot draw after starting screen sharing?

The annotation started automatically by startScreenShare is read-only on the host side. Call trtc.startAnnotation() to turn on local interaction.

2. Remote users cannot see the annotation after screen sharing starts?

Check whether enableSEI: true is set in TRTC.create. Without it, screen sharing is published as usual, but the annotation content is hidden.

3. Annotation is misaligned in the local preview but correct for remote users?

Do not size canvas / video inside the container with descendant selectors (especially with !important), because they also match the plugin's internal canvases. Constrain only the container's own layer:

#local-screen > .trtc-wb-video-layer video {
  width: 100%;
  height: 100%;
  /* no object-fit; it is determined by fillMode */
}

4. Do I need to call an API after the container size changes?

No. When the container itself is resized, or a parent's size change propagates to the container (percentage / flex layout), the annotation re-lays out automatically. To switch containers, use updateScreenShare({ view }) (host) or updateRemoteVideo({ view }) (guest).

Applying transform: scale() to the container or its ancestors is not recommended, as windows, text and other content will be misaligned with the annotation. To change the display size, change the container's width / height directly.

5. The shared window was resized during sharing?

The composited picture keeps the size and aspect ratio from when sharing started, and the screen is placed in it according to fillMode: contain adds margins, cover crops. Existing strokes keep their position relative to the picture but may no longer match the window content; the host can clear them with the clear button in the toolbar. To fill the picture again, stop sharing and start again.

6. Switching between the whiteboard and screen sharing fails with OPERATION_ABORT (already start) instead of 5110?

Screen sharing and the Interactive Whiteboard share the same sub stream. If you call one while the previous startScreenShare() / startWhiteboard() has not returned yet (for example, the screen picker is still open), the call fails with OPERATION_ABORT. Only after the previous call has finished starting does the call fail with INVALID_OPERATION (5110). Wait for the previous call to return (whether it succeeds or fails) before switching, or disable the switch entry while it is in progress:

await trtc.startScreenShare(); // wait for it to return before switching
await trtc.stopScreenShare();
await trtc.startWhiteboard({ view: 'local-board' });