Scalable Video Coding (SVC) Extension for WebRTC

Abstract

This document defines a set of ECMAScript APIs in WebIDL to extend the WebRTC 1.0 API to enable user agents to support scalable video coding (SVC).

4. Operational model

This specification extends [WEBRTC] to enable configuration of encoding parameters for SVC, as well as discovery of the SVC capabilities of both an encoder and decoder, by extending the RTCRtpEncodingParameters and RTCRtpCodecCapability dictionaries.

Since this specification does not change the behavior of WebRTC objects and methods, restrictions relating to Offer/Answer negotiation and encoding parameters remain, as described in [WEBRTC] Section 5.2: "setParameters() does not cause SDP renegotiation and can only be used to change what the media stack is sending or receiving within the envelope negotiated by Offer/Answer."

The configuration of SVC-capable codecs implemented in browsers fits within this restriction. Codecs such as VP8 [RFC6386], VP9 [VP9] and AV1 [AV1] mandate support for SVC and require a compliant decoder to be able to decode any compliant encoding that an encoder can send. Therefore, for these codecs there is no need to configure the decoder or to negotiate SVC support within Offer/Answer, enabling encoding parameters to be used for SVC configuration.

4.1 Error handling

[WEBRTC] Section 5.2 describes error handling in setParameters(), including use of RTCError to indicate a "hardware-encoder-error" due to an unsupported encoding parameter, as well as OperationError for other errors. Implementations of this specification utilize RTCError and OperationError in the prescribed manner when an invalid scalabilityMode value is provided to setParameters() or addTransceiver().

When the addTransceiver() and setCodecPreferences() methods are called prior to conclusion of the Offer/Answer negotiation, the negotiated codec and its capabilities may not be known. In this situation the scalabilityMode values configured in sendEncodings may not be supported by the eventually negotiated codec. However, an error will result only if the requested scalabilityMode value is invalid for any supported codec. To determine whether the requested scalabilityMode values have been applied, an application can call the RTCRtpSender.getParameters() method after negotiation has completed and the sending codec has been determined. If the configuration is not satisfactory, the setParameters() method can be used to change it.

Note that where SVC support is negotiated in SDP Offer/Answer, setParameters() can only change scalabilityMode values within the envelope negotiated by Offer/Answer, resulting in an error if the requested scalabilityMode value is outside this envelope. When sendEncodings is used to request the sending of multiple simulcast streams using addTransceiver(), it is not possible to configure the sending of "S" scalability modes. The browser may only be configured to send simulcast encodings with multiple SSRCs and RIDs, or alternatively, to send all simulcast encodings on a single RTP stream. Attempting to simultaneously utilize both simulcast transport techniques MUST return OperationError in setParameters() or addTransceiver().

4.2 Negotiation

So as to ensure that the desired scalabilityMode values can be applied, setCodecPreferences() can be used to limit the negotiated codecs to those supporting the desired configuration. For example, if temporal scalability is desired along with spatial simulcast, when addTransceiver() is called, sendEncodings can be configured to send multiple simulcast streams with different resolutions, with each stream utilizing temporal scalability. If only the VP8, VP9 and AV1 codec implementations support temporal scalability, setCodecPreferences() can be used to remove the H.264/AVC codec from the Offer, guaranteeing that a codec supporting temporal scalability is negotiated.

There are situations where a peer may only support reception of a subset of codecs and scalability modes. For example, an SFM that parses codec payloads may only support the H.264/AVC codec without scalability and the H.264/SVC codec with temporal scalability. A browser that can decode any VP8 or VP9 scalability mode may not support H.264/SVC or AV1. In these situations, the RTCRtpReceiver's getCapabilities method can be used to determine the scalability modes supported by the RTCRtpReceiver, and the RTCRtpSender's getCapabilities method can be used to determine the scalability modes supported by the RTCRtpSender. After exchanging capabilities, the application can compute which codecs and scalabilityMode values are supported by both the browser and SFM. The intersection of codecs and scalability modes supported by the browser's RTCRtpSender and the SFM's receiver can then be used to determine the arguments passed to the browser's addTransceiver() and setParameters() methods.

Since sending simulcast encodings on a single stream is not negotiated within Offer/Answer, an application using SDP signaling needs to determine whether single stream simulcast transport is supported prior to the Offer/Answer negotiation. This can be handled by having the SFM send it's receiver capabilities to the application prior to Offer/Answer. This allows the application to determine whether single stream simulcast is supported, and if so, what scalability modes the SFM can handle. For example, an SFM that can only support reception of a maximum of 2 simulcast encodings on a single SSRC with the AV1 codec would only indicate support for the "S2T1" and "S2T1h" scalability modes in its receiver capabilities.

5. Dictionary extensions

5.1 RTCRtpEncodingParameters Dictionary Extensions

WebIDLpartial dictionary RTCRtpEncodingParameters {
             DOMString scalabilityMode;
};

Dictionary `RTCRtpEncodingParameters` Members

scalabilityMode of type DOMString: A case-sensitive identifier of the scalability mode to be used for this stream. The scalabilityMode selected MUST be one of the scalability modes supported for the codec, as indicated in RTCRtpCodecCapability. Scalability modes are defined in Section 6.

5.2 `RTCRtpCodecCapability` Dictionary Extensions

WebIDLpartial dictionary RTCRtpCodecCapability {
             sequence<DOMString> scalabilityModes;
};

Dictionary `RTCRtpCodecCapability` Members

scalabilityModes of type sequence<DOMString>

An sequence of the scalability modes (defined in Section 6) supported by the encoder implementation.

In response to a call to RTCRtpSender.getCapabilities(kind), conformant implementations of this specification MUST return a sequence of scalability modes supported by each codec of that kind. If a codec does not support encoding of any scalability modes, then the scalabilityModes member is not provided.

In response to a call to RTCRtpReceiver.getCapabilities(kind), decoders that do not support decoding of scalability modes or that are required to decode any scalability mode (such as compliant VP8, VP9 and AV1 decoders) omit the scalabilityModes member. However, decoders that only support decoding of a subset of scalability modes MUST return a sequence of the scalability modes supported by that codec.

Note

The scalabilityModes sequence represents the scalability modes supported by a user agent. For an SFM the supported scalabilityModes may depend on the negotiated RTP header extensions. For example, if the SFM cannot parse codec payloads (either because it is not designed to do so, or because the payloads are encrypted), then negotiation of an RTP header extension (such as the AV1 Dependency Descriptor defined in Appendix A of [AV1-RTP]) may be required by the SFM in order to forward scalabilityModes. As a result, the scalabilityModes supported by an SFM may not be known until completion of the Offer/Answer negotiation.

6. Scalability modes

The scalability modes supported in this specification, as well as their associated identifiers and characteristics, are provided in the table below. The names of the scability modes (which are case sensitive) are provided, along with the scalability mode identifiers assigned in [AV1] Section 6.7.5, and links to dependency diagrams provided in Section 9.

While [AV1] and VP9 [VP9-PAYLOAD] implementations can support all the modes defined in the table, other codecs cannot. For example, VP8 [RFC7741] only supports temporal scalability (e.g. "L1T2", "L1T3"); H.264/SVC [RFC6190], which supports both temporal and spatial scalability, only permits transport of simulcast on distinct SSRCs, so that it does not support the "S" modes, where multiple encodings are transported on a single RTP stream.

Scalability Mode Identifier	Spatial Layers	Resolution Ratio	Temporal Layers	Inter-layer dependency	AV1 scalability_mode_idc
"L1T2"	1		2		SCALABILITY_L1T2
"L1T3"	1		3		SCALABILITY_L1T3
"L2T1"	2	2:1	1	Yes	SCALABILITY_L2T1
"L2T2"	2	2:1	2	Yes	SCALABILITY_L2T2
"L2T3"	2	2:1	3	Yes	SCALABILITY_L2T3
"L3T1"	3	2:1	1	Yes	SCALABILITY_L3T1
"L3T2"	3	2:1	2	Yes	SCALABILITY_L3T2
"L3T3"	3	2:1	3	Yes	SCALABILITY_L3T3
"L2T1h"	2	1.5:1	1	Yes	SCALABILITY_L2T1h
"L2T2h"	2	1.5:1	2	Yes	SCALABILITY_L2T2h
"L2T3h"	2	1.5:1	3	Yes	SCALABILITY_L2T3h
"S2T1"	2	2:1	1	No	SCALABILITY_S2T1
"S2T2"	2	2:1	2	No	SCALABILITY_S2T2
"S2T3"	2	2:1	3	No	SCALABILITY_S2T3
"S2T1h"	2	1.5:1	1	No	SCALABILITY_S2T1h
"S2T2h"	2	1.5:1	2	No	SCALABILITY_S2T2h
"S2T3h"	2	1.5:1	3	No	SCALABILITY_S2T3h
"S3T1"	3	2:1	1	No	SCALABILITY_S3T1
"S3T2"	3	2:1	2	No	SCALABILITY_S3T2
"S3T3"	3	2:1	3	No	SCALABILITY_S3T3
"S3T1h"	3	1.5:1	1	No	SCALABILITY_S3T1h
"S3T2h"	3	1.5:1	2	No	SCALABILITY_S3T2h
"S3T3h"	3	1.5:1	3	No	SCALABILITY_S3T3h
"L2T2_KEY"	2	2:1	2	Yes	SCALABILITY_L3T2_KEY
"L2T2_KEY_SHIFT"	2	2:1	2	Yes	SCALABILITY_L3T2_KEY_SHIFT
"L2T3_KEY"	2	2:1	3	Yes	SCALABILITY_L3T3_KEY
"L2T3_KEY_SHIFT"	2	2:1	3	Yes	SCALABILITY_L3T3_KEY_SHIFT
"L3T2_KEY"	3	2:1	2	Yes	SCALABILITY_L4T5_KEY
"L3T2_KEY_SHIFT"	3	2:1	2	Yes	SCALABILITY_L4T5_KEY_SHIFT
"L3T3_KEY"	3	2:1	3	Yes	SCALABILITY_L4T7_KEY
"L3T3_KEY_SHIFT"	3	2:1	3	Yes	SCALABILITY_L4T7_KEY_SHIFT

6.1 Guidelines for addition of `scalabilityMode` values

When proposing a scalabilityMode value, the following principles should be followed:

The proposed scalabilityMode MUST define entries to the table in Section 6, including values for the Scalabilty Mode Identifier, spatial and temporal layers, Resolution Ratio, Inter-layer dependency and the corresponding AV1 scalability_mode_idc value (if assigned).
The Scalability Mode Identifier SHOULD be consistent with the existing naming scheme, which utilizes LxTy to denote a scalabilityMode with x spatial layers using a 2:1 resolution ratio and y temporal layers. LxTyh denotes x spatial layers with a 1.5:1 resolution ratio and y temporal layers. SxTy denotes a scalabilityMode with x simulcast encodings with a 2:1 resolution ratio, with each simulcast encoding containing y temporal layers. SxTyh denotes a 1.5:1 resolution ratio. LxTy_KEY denotes a scalabilityMode with x spatial layers using a 2:1 resolution ratio and y temporal layers in which spatial layers only depend on lower spatial layers at a key frame. LxTy_KEY_SHIFT modes denotes a scalabilityMode with x spatial layers using a 2:1 resolution ratio and y temporal layers in which spatial layers only depend on lower spatial layers at a key frame and subsequent frames have their temporal identifier shifted upward.
A dependency diagram MUST be supplied, in the format provided in Section 9.

7. Examples

7.1 Spatial Simulcast and Temporal Scalability

This section is non-normative.

This example extends [WEBRTC] Section 7.1 (Example 1) to demonstrate sending three spatial simulcast layers each with three temporal layers, using an SSRC and RID for each simulcast layer. Only the "sendEncodings" attribute is changed from the original example.

Example 1

const signaling = new SignalingChannel(); // handles JSON.stringify/parse
const constraints = {audio: true, video: true};
const configuration = {'iceServers': [{'urls': 'stun:stun.example.org'}]};
let pc;

// call start() to initiate
async function start() {
  pc = new RTCPeerConnection(configuration);

  // let the "negotiationneeded" event trigger offer generation
  pc.onnegotiationneeded = async () => {
    try {
      await pc.setLocalDescription();
      // send the offer to the other peer
      signaling.send({description: pc.localDescription});
    } catch (err) {
      console.error(err);
    }
  };

  try {
    // get a local stream, show it in a self-view and add it to be sent
    const stream = await navigator.mediaDevices.getUserMedia(constraints);
    selfView.srcObject = stream;
    pc.addTransceiver(stream.getAudioTracks()[0], {direction: 'sendonly'});
    pc.addTransceiver(stream.getVideoTracks()[0], {
      direction: 'sendonly',
      sendEncodings: [
        {rid: 'q', scaleResolutionDownBy: 4.0, scalabilityMode: 'L1T3'}
        {rid: 'h', scaleResolutionDownBy: 2.0, scalabilityMode: 'L1T3'},
        {rid: 'f', scalabilityMode: 'L1T3'},
      ]    
    });
  } catch (err) {
    console.error(err);
  }
}

signaling.onmessage = async ({data: {description, candidate}}) => {
  try {
    if (description) {
      await pc.setRemoteDescription(description);
      // if we got an offer, we need to reply with an answer
      if (description.type == 'offer') {
        await pc.setLocalDescription();
        signaling.send({description: pc.localDescription});
      }
    } else if (candidate) {
      await pc.addIceCandidate(candidate);
    }
  } catch (err) {
    console.error(err);
  }
};

This is an example with two spatial layers (with a 2:1 ratio) and three temporal layers.

Example 2

let sendEncodings = [
  {scalabilityMode: 'L2T3'}
];

This is an example with three spatial simulcast layers each with three temporal layers on a single SSRC.

Example 3

let sendEncodings = [
  {scalabilityMode: 'S3T3'}
]

7.2 SVC Encoder Capabilities

This section is non-normative.

This is an example of RTCRtpSender.getCapabilities}}('video').codecs[] returned by a browser implementing [WEBRTC]. Only the scalabilityModes attribute is defined in this specification.

Example 4

  "codecs": [
    {
      "clockRate": 90000,
      "mimeType": "video/VP8",
      "scalabilityModes": [
        "L1T2",
        "L1T3"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=96"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/VP9",
      "scalabilityModes": [
        "L1T2",
        "L1T3",
        "L2T1",
        "L2T2",
        "L2T3",
        "L3T1",
        "L3T2",
        "L3T3",
        "L1T2h",
        "L1T3h",
        "L2T1h",
        "L2T2h",
        "L2T3h"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=98"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "packetization-mode=1;profile-level-id=42001f;level-asymmetry-allowed=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=100"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "packetization-mode=0;profile-level-id=42001f;level-asymmetry-allowed=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=102"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "level-asymmetry-allowed=1;profile-level-id=42e01f;packetization-mode=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=104"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "level-asymmetry-allowed=1;profile-level-id=42e01f;packetization-mode=0"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=106"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "level-asymmetry-allowed=1;profile-level-id=4d0032;packetization-mode=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=108"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "level-asymmetry-allowed=1;profile-level-id=640032;packetization-mode=1"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=110"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/red"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=112"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/ulpfec"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/AV1",
      "scalabilityModes": [
        "L1T2",
        "L1T3",
        "L2T1",
        "L2T2",
        "L2T3",
        "L3T1",
        "L3T2",
        "L3T3",
        "L1T2h",
        "L1T3h",
        "L2T1h",
        "L2T2h",
        "L2T3h",
        "S2T1",
        "S2T2",
        "S2T3",
        "S3T1",
        "S3T2",
        "S3T3",
        "S2T1h",
        "S2T2h",
        "S2T3h",
        "S3T1h",
        "S3T2h",
        "S3T3h"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=113"
    }
]

7.3 SFM Capabilities

This section is non-normative.

This is an example of RTCRtpReceiver.getCapabilities('video').codecs[] returned by an SFM that only supports forwarding of VP8, VP9 and AV1 temporal scalability modes.

Example 5

 "codecs": [
    {
      "clockRate": 90000,
      "mimeType": "video/VP8",
      "scalabilityModes": [
        "L1T2",
        "L1T3"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/VP9",
      "scalabilityModes": [
        "L1T2",
        "L1T3",
        "L1T2h",
        "L1T3h"
      ]
    },
    {
      "clockRate": 90000,
      "mimeType": "video/AV1",
      "scalabilityModes": [
        "L1T2",
        "L1T3",
        "L1T2h",
        "L1T3h"
      ]
    }
]

7.4 SVC Decoder Capabilities

This section is non-normative.

This is an example of RTCRtpReceiver.getCapabilities('video').codecs[] returned by a browser that can support all scalability modes of the VP8 and VP9 codecs.

Example 6

  "codecs": [
    { 
      "clockRate": 90000,
      "mimeType": "video/VP8"
    },
    { 
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=96"
    },
    { 
      "clockRate": 90000,
      "mimeType": "video/VP9"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/rtx",
      "sdpFmtpLine": "apt=98"
    },
    {
      "clockRate": 90000,
      "mimeType": "video/H264",
      "sdpFmtpLine": "packetization-mode=1;profile-level-id=42001f;level-asymmetry-allowed=1"
    },

    ...
]

8. Privacy and Security Considerations

This section is non-normative.

This section is non-normative; it specifies no new behaviour, but instead summarizes information already present in other parts of the specification. WebRTC protocol security considerations are described in [RTCWEB-SECURITY-ARCH] and the security and privacy considerations for the WebRTC APIs are described in [WEBRTC] Section 13.

8.1 Persistent information

The WebRTC API exposes information about the underlying media system via the RTCRtpSender.getCapabilities() and RTCRtpReceiver.getCapabilities methods, including detailed and ordered information about the codecs that the system is able to produce and consume. The WebRTC-SVC extension adds the scalabilityModes supported by the RTCRtpSender to that information, which is persistent across time, therefore increasing the fingerprint surface. Since for SVC codecs implemented in WebRTC browsers compliant decoders are required to be able to decode all scalability modes, additional information is not provided relating to the RTCRtpReceiver.

Since for SVC codecs implemented in WebRTC the use of scalable coding tools is not negotiated and is independent of the supported profiles, and since SVC is rarely supported in hardware encoders, knowledge of the scalabilityModes supported by the RTCRtpSender does not provide additional information on the underlying hardware. However, since browsers may differ in their support for SVC modes, the supported scalabilityModes may permit differentiation between browsers. This additional fingerprint surface is expected to decrease over time as this specification is more widely implemented.