Stream Mixing
1 Introduction
Streaming mixing is a technology that combines multiple media streams from the cloud into one single stream. ZEGO SDK is capable of mixing audio-video streams, and also audio-only streams.
1.1 Why Stream Mixing?
1.2 Application Scenarios
1. It reduces the complexity of development. For instance, when there are x number of hosts co-hosting an interactive live broadcast, it saves developers a lot of effort to pull just one single mixed stream rather than to pull x number of individual streams and perform the layout of streams on the client side.
2. It reduces the performance requirements on client devices and network bandwidth. The more streams to process, the more system and bandwidth
resources are required. With stream mixing, the burden of processing multiple streams is removed from the client side.
3. It makes publishing multiple streams to multiple CDNs much easier. Such a requirement can be achieved easily by setting up multiple output streams to the corresponding CDN URLs for the stream mixing task.
4. When it is required to record multiple streams for future playback, the recording can be done on the CDN to which the mixed stream is published.
5. Stream mixing also makes obscene content detection easier because it can be done by checking the mixed stream, rather than checking multiple individual streams.
1.3 Timing of Stream Mixing
Stream mixing should be started after the streams are pushed/pulled successfully.
For example, in a case where the host (user A) links up a viewer (user B) during a live broadcast, stream mixing can be initiated by the host (user A) after the stream of the user B is pulled successfully. Stream mixing can also be started at other timing according to the actual situation.
1.4 System Architecture
When stream mixing is enabled, the SDK pushes the streams to the ZEGO server, then ZEGO's stream mixing service mixes the streams into one single stream and then pushes the mixed stream to CDN. The viewers can then pull the mixed stream from CDN and play it.
2 Download the Sample Codes
Please refer to Sample Codes to get the sample codes.
The sample code package contains many code examples of different features of the SDK. Please refer to the source files in the directory
"/ZegoExpressExample/Mixer" for the code examples of how to use the APIs of the stream mixing module.
3 Prerequisites
Before implementing the "Stream Mixing" functions, please make sure:
ZEGO Express SDK has been integrated in the project. For details, please refer to Quick Starts - Integration.
4 Implementation Steps
4.1 Initialization and Room Login
Please refer to the following documents for how to initialize the SDK and log in to a room.
4.2 Set up a Stream Mixing Task
ZegoMixerTask is a class defined in the SDK for configuring a stream mixing task. The configuration parameters include the layout of input streams, the output streams, and others.
4.2.1 Create a Stream Mixing Task Object
Create a stream mixing task object using the constructor ZegoMixerTask, and then call the instance methods to set up the input and output parameters.
ZegoMixerTask task = new ZegoMixerTask("task1");
4.2.2 (Optional) Set up the Video Configurations Set up the Video Configurations
4.2.3 (Optional) Set up the Audio Configurations Set up the Audio Configurations
Quick Start - Initializing the SDK Quick Start - Logging in to a Room
The prerequisite for steam mixing is that streams already exist in the room.
4.2.4 Set up the Input Streams
By default, it supports up to 9 streams. If you need to input more streams, please contact ZEGO technical support for confirmation and configuration.
When the mixed stream is an audio stream (that is, the "ContentType" parameter is set to the audio mixed stream type), the SDK does not process the layout field, and there is no need to pay attention to the "layout" parameter.
The code example below demonstrates mixing two streams with a top-bottom output layout.
/** Create an input stream list*/
ArrayList<ZegoMixerInput> inputList = new ArrayList<>();
/** Configue the first input stream */
ZegoMixerInput input_1 = new ZegoMixerInput(); input_1.streamID = "streamID_1";
input_1.contentType = ZegoMixerInputContentType.VIDEO;
input_1.layout = new Rect(0,0,videoConfig.resolutionWidth, videoConfig.resolutionHeigh t/2);
inputList.add(input_1);
/** Configue the second input stream */
ZegoMixerInput input_2 = new ZegoMixerInput(); input_2.streamID = "streamID_2";
input_2.contentType = ZegoMixerInputContentType.VIDEO;
input_2.layout = new Rect(0,videoConfig.resolutionHeight/2,videoConfig.resolutionWidt h, videoConfig.resolutionHeight);
inputList.add(input_2);
/** Set up the input steam list for the stream mixing task */
task.setInputList(inputList);
4.2.5 Set up the Output Streams
Up to 3 output streams can be configured.
/** Create an output stream object*/
ZegoMixerOutput mixerOutput = new ZegoMixerOutput("output_streamid_1");
/** Construct the output stream list */
ArrayList<ZegoMixerOutput> mixerOutputList = new ArrayList<>(); mixerOutputList.add(mixerOutput);
/** Set up the output stream list for the stream mixing task */
task.setOutputList(mixerOutputList);
4.2.6 (Optional) Set up the Watermark Set up the Watermark
4.2.7 (Optional) Set up the Background Image Set up the Background Image
4.2.8 (Optional) Set Advanced Configuration Set Advanced Configuration
4.3 Start the Stream Mixing Task
After the ZegoMixerTask object is created and configured, call the API
startMixerTask to start the stream mixing task and handle the failure of task startup in the callback block.
/**
* Start the stream mixing task *
* In consideration of the performance of client devices, stream mixing is performed o n ZEGO's cloud servers.
* After this API is called, ZegoExpressEngine will send a stream-mixing request to ZE GO's backend service running on the cloud, which will then start mixing the streams co nfigured in the task and the one being published by the current user (if any). * If any exception occurs in starting up the task, for example, the specified stream does not exist, the related error code will be provided in the callback. Refer to the document [https://doc-en.zego.im/en/5548.html] for details of the error codes.
* If a certain input stream gets lost when the stream mixing is still in progress, th e stream mixing task will automatically retry to pull that stream with a timeout of 90
seconds, after which no more retries will be attempted. * @param task: The stream mixing task object.
* @param callback: The callback to report the result of task startup. */
public void startMixerTask(ZegoMixerTask task, IZegoMixerStartCallback callback)
engine.startMixerTask(task, new IZegoMixerStartCallback() { @Override
public void onMixerStartResult(int errorCode, JSONObject extendedData) { if (errorCode != 0) {
//Failed to start or update the stream mixing task. Task update failure wi ll affect the current task that is in progress.
} else {
//The stream mixing task is started or updated successfully.
} } });
4.4 Update the Configurations of the Stream Mixing Task
When changes need to made to the stream mixing task, such as the number of input streams increases or decrease, or the bitrate of the output stream needs to be adjusted, you can update the stream mixing task object with the new
configurations and then call the API startMixerTask again.
4.5 Stop the Stream Mixing Task
/**
* Stop the stream mixing task *
* Similar to [startMixerTask], after this API is called, ZegoExpressEngine will send a request to ZEGO's backend service running on the cloud to stop the specified stream mixing task.
* If the developer starts the next stream mixing task without stopping the previous s tream mixing task, the previous stream mixing task will not stop automatically, but it
API Call Example:
will stop 90 seconds after all its input streams no longer exist.
* It's worth reminding that developers should stop the previous stream mixing task be fore starting the next one, to prevent that the viewers are still pulling the output s tream of the previous stream mixing task event after the host already started a new ta sk.
* @param task: The stream mixing task object.
* @param callback: The callback to report the result of ending the task. */
public void stopMixerTask(ZegoMixerTask task, IZegoMixerStopCallback callback);
/** Pass in the taskID of the steam mixing task to be stopped. */
engine.stopMixerTask("task1");
5 API Reference
Method Description
startMixerTask Starts a stream mixing task. stopMixerTask Stops a stream mixing task.
6 FAQ
API Call Example:
1. Can the mixed stream be pushed to a third-party CDN? How to push the mixed stream to multi-CDNs?
Yes, the mixed stream can be pushed to a third-party CDN. If you need to do so, you can assign the corresponding CDN URL to the "target" property of the "ZegoMixerOutput" object. The URL needs to be in the RTMP
format:"rtmp://xxxxxxxx".
To push the mixed stream to multiple CDNs, you need to create the
2. How to lay out the input streams in the view of the output stream?
The layout of the input streams can be done by setting up the "layout" property of the "ZegoMixerInput" object. Here is a simple example for illustration.
Suppose the resolution of the mixed stream (view A) is 375x667, and the input stream (View B) needs to be placed at the position with left-top coordinates (50, 330) and right-bottom coordinates (200, 450).
The layout can be illustrated as the picture below:
Set the "width" and "height" properties of the "videoConfig" parameter of the ZegoMixerTask object to "width=375; height = 667;".
Set the "layout" property of the "ZegoMixerInput" parameter of the
3. When the aspect ratio of an input stream's layout property does not match the stream's actual resolution, how will the picture be cropped?
The SDK will do proportional scaling to fill the defined view. For example, suppose the resolution of an input stream is "720x1280", that is an aspect ratio of "9:16", and the "layout" property of "ZegoMixerInput" of this stream is "[left:0 top:0 right:100 bottom:100]", which means an aspect ratio of "1:1", then the middle part of the stream will be displayed, and the upper and lower parts will be cropped out.
The hosts can do their own layout and then initiate a stream mixing task respectively.
For example, host A can set up the view of stream A (pushed by him/herself) to be larger than the view of stream B (published by host B), and then initiate a stream mixing task to output a stream A_Mix; While host B can set up the view of stream B to be the larger one, and initiate another stream mixing task to output a stream B_Mix.
5. There are two ways to do stream mixing in an interactive live broadcast:"To start stream mixing immediately after the live broadcast begins when there is only one host" and "To start stream mixing only when the second host is linked up". What are the advantages and disadvantages?
The advantage of starting stream mixing from the beginning when there is only one host is that it is easier to implement. The downside is that there will be some extra cost for doing stream mixing for just one stream.
The second approach can save some costs, but the implementation is relatively more complicated. The viewers need to pull the individual stream of the first host at the beginning, and then when the second host joins and the stream mixing is started, the viewers have to stop pulling the stream of the first host, and then switch to pull the mixed stream. While for the first approach mentioned above, the viewers do not need to do such a switch.
6. Does stream mixing support setting up a round or rectangular output view?
