Tutorial: Adding Music and Audio Effects

Feature Description

This document describes the features that AudioMixerPlugin can implement.

• Feature 1: it can play back the background music in TRTC. Click here to try out the audio mixing demo.
• Feature 2: it can capture the sound of both the mic and system/tab at the same time.

Feature 1. Playing back Background Music in TRTC Call

Supported platforms

Step 1. Create a music instance

Install the RTCAudioMixer in the project.

npm install rtc-audio-mixer

The AudioMixerPlugin plugin needs to be imported together with TRTC in the same scope.

import TRTC from 'trtc-js-sdk';
import AudioMixerPlugin from 'rtc-audio-mixer';

Call the AudioMixerPlugin.createAudioSource(params) method to create an AudioSource music instance with the following parameters:

• url: it is the music file address, which is required and of String type. You can use an online file address or use a blob URL generated by the object selected from the local system.
• loop: it indicates whether to loop the music, which is optional and of Boolean type. The default value is false, indicating not to loop.
• volume: it is the volume, which is optional and of Number type. The value range is 0–1, and the default value is 1.

// You can use an online file address or use a blob URL generated by a local file to load. For more information, please see the API overview at the end of this document.
let audioSourceA = AudioMixerPlugin.createAudioSource({url: 'https://audioSourceA.mp3'});

Notes

• When an online sound effect file is played back, it must support CORS, and the access protocol must be https.
• Supported formats include MP3, AAC, and other audio formats supported by the used browser.
• When there is no user interaction on the webpage, the browser will block the playback of any medias with sound on the webpage. We recommend you prompt the user to click on the webpage first before performing relevant operations. For more information, please see Autoplay_guide.

Step 2. Publish mixed audio stream

Call the AudioMixerPlugin.mix(params) method to get mixedAudioTrack with background music added, which can replace the audio track in the stream to be published.

• targetTrack: audio track to which background music to be added. This parameter is optional. If it is empty, a track containing only the background music will be generated.
• sourceList: music instance created in step 1, which is required and of Array type, such as [ audioSourceA, audioSourceB ].

Scenario 1. Replace before release

Before publishing localStream, replace the audio track in localStream with mixedAudioTrack.

let audioSourceA = AudioMixerPlugin.createAudioSource({url: 'https://audioSourceA.mp3'});
let audioSourceB = AudioMixerPlugin.createAudioSource({url: 'https://audioSourceB.mp3'});
const localStream = TRTC.createStream({userId: 'user', audio: true, video: true});
await localStream.initialize();
// 1. Get the mic track
let originAudioTrack = localStream.getAudioTrack();
// 2. Mix it with `audioSourceA` and `audioSourceB` to generate `mixedAudioTrack`
mixedAudioTrack = await AudioMixerPlugin.mix({targetTrack: originAudioTrack, sourceList: [audioSourceA, audioSourceB]});
// 3. Replace the mic track
await localStream.replaceTrack(mixedAudioTrack);
// 4. Publish
client.publish(localStream);
// 5. Call the `play` method to play back the track, and both parties in the call can hear the background music
audioSourceA.play();

Scenario 2. Replacing after publish

let audioSourceA = AudioMixerPlugin.createAudioSource({url: 'https://audioSourceA.mp3'});
let audioSourceB = AudioMixerPlugin.createAudioSource({url: 'https://audioSourceB.mp3'});
// 1. Get the mic track
let originAudioTrack = localStream.getAudioTrack();
// 2. Mix the mic track with `audioSourceA` and `audioSourceB`
mixedAudioTrack = await AudioMixerPlugin.mix({targetTrack: originAudioTrack, sourceList: [audioSourceA, audioSourceB]});
// 3. Replace the audio track in `localStream`
await localStream.replaceTrack(mixedAudioTrack);
// 4. Call the `play` method to play back the track, and both parties in the call can hear the background music
audioSourceA.play();
audioSourceB.play();

Scenario 3. Mute mic and play back background music

let audioSourceA = AudioMixerPlugin.createAudioSource({url: 'https://audioSourceA.mp3'});
let audioSourceB = AudioMixerPlugin.createAudioSource({url: 'https://audioSourceB.mp3'});
// It temporarily stores the track containing the mic sound and background music, which can be used to unmute the mic
let microAndBgmTrack = localStream.getAudioTrack();
// 1. Generate the background music containing only `audioSourceA` and `audioSourceB`
let bgmTrack = await AudioMixerPlugin.mix({sourceList: [audioSourceA, audioSourceB]});
// 2. Replace the audio track in `localStream`. At this point, only the background music exists, and mic sound does not exist
await localStream.replaceTrack(bgmTrack);
// If you want to unmute the mic and retain the background music, you can simply restore the track containing the mic sound and background music
localStream.replaceTrack(microAndBgmTrack);

Feature 2. Capturing the Sound of Both Mic and System/Tab at the Same Time

Supported platforms

The system sound can be captured only with Chrome 74+. On Windows and Chrome OS, the sound of the entire system can be captured. On Linux and macOS, only the sound of the active tab can be captured.

Step1. Create a video stream for screen sharing and an audio stream containing only the mic sound

// The `AudioMixerPlugin` plugin needs to be imported together with `TRTC` in the same scope.
import TRTC from 'trtc-js-sdk';
import AudioMixerPlugin from 'rtc-audio-mixer';

Capture the screen sharing stream and the mic audio stream.

// Create a screen sharing stream
const screenStream = TRTC.createStream({
  screenAudio: true,
  screen: true,
  ...
});
// Create a mic audio stream
const localStream = TRTC.createStream({
  audio: false,
  video: true,
  ...
});
await screenStream.initialize();
await localStream.initialize();

Step 2. Mix the system audio track with the mic audio track and publish the mixed track

const screenAudioTrack = screenStream.getAudioTrack();
const microphoneTrack = localStream.getAudioTrack();
// Mix the system audio track with the mic audio track
const mixedTrack = await AudioMixerPlugin.mix({targetTrack: microphoneTrack, trackList: [screenAudioTrack]});
// Replace the audio track in `screenStream`
await screenStream.replaceTrack(mixedTrack);
// Publish the stream
await client.publish(screenStream);

Feature 3: Frequent change of music in the mix

If you need to update the background music frequently during the call, you can refer to the following code snippet.

// Create a custom audioElement
let audioElementA = document.createElement('audio');
audioElementA.src = 'music-001.mp3'; // use a format supported by the browser
audioElementA.load();
audioElementA.play(); // call the play method when you need to play background music
// createAudioSourceFromElement(HTMLAudioElement) can pass in an AudioElement that you created yourself, or an AudioElement that you get from the page using getElementById
let audioSourceA = AudioMixerPlugin.createAudioSourceFromElement(audioElementA);
// Add background music
let originAudioTrack = localStream.getAudioTrack();
mixedAudioTrack = await AudioMixerPlugin.mix({targetTrack: originAudioTrack, sourceList: [audioSourceA]});
await localStream.replaceTrack(mixedAudioTrack);
audioSourceA.play();
// Update music url
audioElementA.src = 'music-002.mp3';
audioElementA.load();
audioElementA.play();

Feature 4: Release resources

await client.unpublish(localStream); // unpush the stream
await client.leave(); // check out
// Destroy the generated mixedAudioTrack, and the previously referenced audio resource 
AudioMixerPlugin.destroySource(mixedAudioTrack);
// If you don't need to use it you can
audioSourceA.destroy();
audioSourceB.destroy();

If you are using a custom AudioElement, you can use the following code to release the resource.

audioElement.removeAttribute('src'); // if the src attribute is used
audioElement.removeAttribute('srcObject'); // if the srcObject attribute is used 
audioElement.load();
audioElement = null;

API Description

AudioMixerPlugin

The AudioMixerPlugin plugin is used to implement the background music and audio effect features, which can be accessed through TRTC.AudioMixerPlugin.

• isSupported() is used to check whether the current browser environment supports adding background music to the audio track.
• createAudioMixer(params) is used to create a music instance.
• mix(params) is used to generate an audio track with background music.

isSupported()

This API is used to check whether the current browser supports the feature of adding background music. The returned value is of Boolean type. true indicates that the feature is supported, while false indicates not supported.

const result = AudioMixerPlugin.isSupported();
if (!result) {
  alert('Your browser is not compatible with AudioMixerPlugin');
}

createAudioSource(params)

This API is used to create an AudioSource music instance to add background music to the audio track.

Params

Example:

Example 1. Create an instance through a file address

// Online file address
let audioSourceA = AudioMixerPlugin.createAudioSource({url: 'https://XXXX.mp3', loop: true, volume: 0.5});
// Relative file address
let audioSourceB = AudioMixerPlugin.createAudioSource({url: '../XXXX.mp3'});
audioSourceA.play();

Example 2. Select the blob URL generated by a local file to load the music file

// Use `<input type="file" id="fileInput">` to select the blob URL generated by a local file to load the music file
fileInput.addEventListener('change', function () {
  var file = fileInput.files[0];
  var reader = new FileReader();
  reader.onload = function (e) {
    let result = e.target.result;
    let audioSourceC = AudioMixerPlugin.createAudioSource({url: result, loop: true, volume: 0.5});
  };
  reader.readAsDataURL(file);
});

createAudioSourceFromElement(element)

This API is used to create an AudioSource music instance to add background music to the audio track.

Params:

Example:

let audioSourceA = AudioMixerPlugin.createAudioSourceFromElement(document.getElementById('audioElementA'));
audioSourceA.play();

AudioSource methods

Event list

The events that can be listened on are the same as the standard events of HTMLAudioElement. For more information, please see Events.

// Listen on the playback event
audioSourceA.on('play', function play(event) {
  console.log('play event trigger', event);
});
// Cancel listening on all events
audioSourceA.off('*', function cancel(event) {
  console.log('Cancel listening on all events', event);
});
// Destroy audioSourceA
audioSourceA.destroy();

mix(params)

This API is used to mix the created music instance with the audio track. targetTrack is the audio track to be mixed, and sourceList is the music instance array created by createAudioSource, such as [ audioSourceA, audioSourceB ].

If targetTrack is not passed in, an audio track containing only the background music will be generated.

Parameters

Example:

Both targetTrack and sourceList are passed in:

// 1. Get the mic track
let originAudioTrack = localStream.getAudioTrack();
// 2. Mix the mic track with `audioSourceA` and `audioSourceB`
mixedAudioTrack = await AudioMixerPlugin.mix({targetTrack: originAudioTrack, sourceList: [audioSourceA, audioSourceB]});
// 3. Replace the mic track
await localStream.replaceTrack(mixedAudioTrack);

Only sourceList array is passed in:

// Generate the background music containing only `audioSourceA` and `audioSourceB`
let bgmTrack = await AudioMixerPlugin.mix({sourceList: [audioSourceA, audioSourceB]});

destroySource(mixedAudioTrack)

When you unpublish or call leave to check out, you can use destroySource to clean up the resource.

client.leave();
audioSource.destroy();
AudioMixerPlugin.destroySource(mixedAudioTrack);

FAQs

1. What should I do if a CORS error occurs?

Example: Access to audio at XXX from origin XXX has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Solution: you need to configure the CORS protocol of the online music file.

2. What should I do if the audio format is incorrect and cannot be played back by the browser?

Example: NotSupportedError: The operation is not supported.

Solution: you need to use an audio format supported by the browser.

3. When you want to replace the background music after publishing it

Example: You want to replace the background music during the call after posting the background music.

Solution.

1. You can re-capture a new audioTrack, use the same process to add new background music after, use LocalStream's replaceTrack method to replace the audio track being pushed to the stream, [click reference](https://web.sdk.qcloud.com/trtc/webrtc/ doc/zh-cn/tutorial-22-advanced-audio-mixer.html#h3-7).
2. You can add multiple background sounds at the beginning, when you need a certain background music, call the background music audioSource.play() method to play it, if you don't need to play it, call the background music audioSource.pause() method to stop playing it.