draft-ietf-payload-vp9-03.txt   draft-ietf-payload-vp9-04.txt 
Payload Working Group J. Uberti Payload Working Group J. Uberti
Internet-Draft S. Holmer Internet-Draft S. Holmer
Intended status: Standards Track M. Flodman Intended status: Standards Track M. Flodman
Expires: September 14, 2017 Google Expires: January 4, 2018 Google
J. Lennox J. Lennox
D. Hong D. Hong
Vidyo Vidyo
March 13, 2017 July 3, 2017
RTP Payload Format for VP9 Video RTP Payload Format for VP9 Video
draft-ietf-payload-vp9-03 draft-ietf-payload-vp9-04
Abstract Abstract
This memo describes an RTP payload format for the VP9 video codec. This memo describes an RTP payload format for the VP9 video codec.
The payload format has wide applicability, as it supports The payload format has wide applicability, as it supports
applications from low bit-rate peer-to-peer usage, to high bit-rate applications from low bit-rate peer-to-peer usage, to high bit-rate
video conferences. It includes provisions for temporal and spatial video conferences. It includes provisions for temporal and spatial
scalability. scalability.
Status of This Memo Status of This Memo
skipping to change at page 1, line 38 skipping to change at page 1, line 38
Internet-Drafts are working documents of the Internet Engineering Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet- working documents as Internet-Drafts. The list of current Internet-
Drafts is at http://datatracker.ietf.org/drafts/current/. Drafts is at http://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress." material or to cite them other than as "work in progress."
This Internet-Draft will expire on September 14, 2017. This Internet-Draft will expire on January 4, 2018.
Copyright Notice Copyright Notice
Copyright (c) 2017 IETF Trust and the persons identified as the Copyright (c) 2017 IETF Trust and the persons identified as the
document authors. All rights reserved. document authors. All rights reserved.
This document is subject to BCP 78 and the IETF Trust's Legal This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents Provisions Relating to IETF Documents
(http://trustee.ietf.org/license-info) in effect on the date of (http://trustee.ietf.org/license-info) in effect on the date of
publication of this document. Please review these documents publication of this document. Please review these documents
skipping to change at page 2, line 14 skipping to change at page 2, line 14
to this document. Code Components extracted from this document must to this document. Code Components extracted from this document must
include Simplified BSD License text as described in Section 4.e of include Simplified BSD License text as described in Section 4.e of
the Trust Legal Provisions and are provided without warranty as the Trust Legal Provisions and are provided without warranty as
described in the Simplified BSD License. described in the Simplified BSD License.
Table of Contents Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 3 2. Conventions, Definitions and Acronyms . . . . . . . . . . . . 3
3. Media Format Description . . . . . . . . . . . . . . . . . . 3 3. Media Format Description . . . . . . . . . . . . . . . . . . 3
4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 4 4. Payload Format . . . . . . . . . . . . . . . . . . . . . . . 5
4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 5 4.1. RTP Header Usage . . . . . . . . . . . . . . . . . . . . 5
4.2. VP9 Payload Description . . . . . . . . . . . . . . . . . 6 4.2. VP9 Payload Description . . . . . . . . . . . . . . . . . 6
4.2.1. Scalability Structure (SS): . . . . . . . . . . . . . 10 4.2.1. Scalability Structure (SS): . . . . . . . . . . . . . 10
4.3. VP9 Payload Header . . . . . . . . . . . . . . . . . . . 12 4.3. VP9 Payload Header . . . . . . . . . . . . . . . . . . . 12
4.4. Frame Fragmentation . . . . . . . . . . . . . . . . . . . 12 4.4. Frame Fragmentation . . . . . . . . . . . . . . . . . . . 12
4.5. Examples of VP9 RTP Stream . . . . . . . . . . . . . . . 12 4.5. Scalable encoding considerations . . . . . . . . . . . . 12
5. Feedback Messages . . . . . . . . . . . . . . . . . . . . . . 12 4.6. Examples of VP9 RTP Stream . . . . . . . . . . . . . . . 13
5.1. Reference Picture Selection Indication (RPSI) . . . . . . 12 4.6.1. Reference picture use for scalable structure . . . . 13
5.2. Slice Loss Indication (SLI) . . . . . . . . . . . . . . . 13 5. Feedback Messages and Header Extensions . . . . . . . . . . . 14
5.3. Full Intra Request (FIR) . . . . . . . . . . . . . . . . 13 5.1. Reference Picture Selection Indication (RPSI) . . . . . . 14
5.4. Layer Refresh Request (LRR) . . . . . . . . . . . . . . . 14 5.2. Slice Loss Indication (SLI) . . . . . . . . . . . . . . . 14
6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 14 5.3. Full Intra Request (FIR) . . . . . . . . . . . . . . . . 15
6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 15 5.4. Layer Refresh Request (LRR) . . . . . . . . . . . . . . . 15
6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 16 5.5. Frame Marking . . . . . . . . . . . . . . . . . . . . . . 16
6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 16 6. Payload Format Parameters . . . . . . . . . . . . . . . . . . 16
6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 17 6.1. Media Type Definition . . . . . . . . . . . . . . . . . . 16
7. Security Considerations . . . . . . . . . . . . . . . . . . . 17 6.2. SDP Parameters . . . . . . . . . . . . . . . . . . . . . 18
8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 17 6.2.1. Mapping of Media Subtype Parameters to SDP . . . . . 18
9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 17 6.2.2. Offer/Answer Considerations . . . . . . . . . . . . . 18
10. References . . . . . . . . . . . . . . . . . . . . . . . . . 18 7. Security Considerations . . . . . . . . . . . . . . . . . . . 19
10.1. Normative References . . . . . . . . . . . . . . . . . . 18 8. Congestion Control . . . . . . . . . . . . . . . . . . . . . 19
10.2. Informative References . . . . . . . . . . . . . . . . . 19 9. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 19
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 19 10. References . . . . . . . . . . . . . . . . . . . . . . . . . 19
10.1. Normative References . . . . . . . . . . . . . . . . . . 20
10.2. Informative References . . . . . . . . . . . . . . . . . 21
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 21
1. Introduction 1. Introduction
This memo describes an RTP payload specification applicable to the This memo describes an RTP payload specification applicable to the
transmission of video streams encoded using the VP9 video codec transmission of video streams encoded using the VP9 video codec
[VP9-BITSTREAM]. The format described in this document can be used [VP9-BITSTREAM]. The format described in this document can be used
both in peer-to-peer and video conferencing applications. both in peer-to-peer and video conferencing applications.
TODO: VP9 description. Please see [VP9-BITSTREAM]. TODO: VP9 description. Please see [VP9-BITSTREAM].
skipping to change at page 3, line 42 skipping to change at page 3, line 42
qualities (and thus with different amounts of coding error). VP9 qualities (and thus with different amounts of coding error). VP9
supports quality layers as spatial layers without any resolution supports quality layers as spatial layers without any resolution
changes; hereinafter, the term "spatial layer" is used to represent changes; hereinafter, the term "spatial layer" is used to represent
both spatial and quality layers. both spatial and quality layers.
This payload format specification defines how such temporal and This payload format specification defines how such temporal and
spatial scalability layers can be described and communicated. spatial scalability layers can be described and communicated.
Temporal and spatial scalability layers are associated with non- Temporal and spatial scalability layers are associated with non-
negative integer IDs. The lowest layer of either type has an ID of negative integer IDs. The lowest layer of either type has an ID of
0. 0, and is sometimes referred to as the "base" temporal or spatial
layer.
Layers are designed (and MUST be encoded) such that if any layer, and Layers are designed (and MUST be encoded) such that if any layer, and
all higher layers, are removed from the bitstream along any of the all higher layers, are removed from the bitstream along either of the
two dimensions, the remaining bitstream is still correctly decodable. two dimensions, the remaining bitstream is still correctly decodable.
For terminology, this document uses the term "layer frame" to refer For terminology, this document uses the term "frame" to refer to a
to a single encoded VP9 frame for a particular resolution/quality, single encoded VP9 frame for a particular resolution/quality, and
and "super frame" to refer to all the representations (layer frames) "picture" to refer to all the representations (frames) at a single
at a single instant in time. A super frame thus consists of one or instant in time. A picture thus consists of one or more frames,
more layer frames, encoding different spatial layers. encoding different spatial layers.
Within a super frame, a layer frame with spatial layer ID equal to S, Within a picture, a frame with spatial layer ID equal to S, where S >
where S > 0, can depend on a layer frame of the same super frame with 0, can depend on a frame of the same picture with a lower spatial
a lower spatial layer ID. This "inter-layer" dependency can result layer ID. This "inter-layer" dependency can result in additional
in additional coding gain compared to the case where only traditional coding gain compared to the case where only traditional "inter-
"inter-picture" dependency is used, where a frame depends on picture" dependency is used, where a frame depends on previously
previously coded frame in time. For simplicity, this payload format coded frame in time. For simplicity, this payload format assumes
assumes that, within a super frame and if inter-layer dependency is that, within a picture and if inter-layer dependency is used, a
used, a spatial layer S frame can only depend on spatial layer S-1 spatial layer S frame can only depend on spatial layer S-1 frame when
frame when S > 0. Additionally, if inter-picture dependency is used, S > 0. Additionally, if inter-picture dependency is used, spatial
spatial layer S frame is assumed to only depend on previously coded layer S frame is assumed to only depend on a previously coded spatial
spatial layer S frame. layer S frame.
Given above simplifications for inter-layer and inter-picture Given above simplifications for inter-layer and inter-picture
dependencies, a flag (the D bit described below) is used to indicate dependencies, a flag (the D bit described below) is used to indicate
whether a spatial layer S frame depends on spatial layer S-1 frame. whether a spatial layer S frame depends on spatial layer S-1 frame.
Given the D bit, a receiver only needs to additionally know the Given the D bit, a receiver only needs to additionally know the
inter-picture dependency structure for a given spatial layer frame in inter-picture dependency structure for a given spatial layer frame in
order to determine its decodability. Two modes of describing the order to determine its decodability. Two modes of describing the
inter-picture dependency structure are possible: "flexible mode" and inter-picture dependency structure are possible: "flexible mode" and
"non-flexible mode". An encoder can only switch between the two on "non-flexible mode". An encoder can only switch between the two on
the very first packet of a key frame with temporal layer ID equal to the first packet of a key frame with temporal layer ID equal to 0.
0.
In flexible mode, each packet can contain up to 3 reference indices, In flexible mode, each packet can contain up to 3 reference indices,
which identify all frames referenced by the frame transmitted in the which identify all frames referenced by the frame transmitted in the
current packet for inter-picture prediction. This (along with the D current packet for inter-picture prediction. This (along with the D
bit) enables a receiver to identify if a frame is decodable or not bit) enables a receiver to identify if a frame is decodable or not
and helps it understand the temporal layer structure. Since this is and helps it understand the temporal layer structure. Since this is
signaled in each packet it makes it possible to have very flexible signaled in each packet it makes it possible to have very flexible
temporal layer hierarchies and patterns which are changing temporal layer hierarchies and patterns which are changing
dynamically. dynamically.
In non-flexible mode, the inter-picture dependency (the reference In non-flexible mode, the inter-picture dependency (the reference
indices) of a group of frames (GOF) MUST be pre-specified as part of indices) of a Picture Group (PG) MUST be pre-specified as part of the
the scalability structure (SS) data. In this mode, each packet MUST scalability structure (SS) data. In this mode, each packet MUST have
have an index to refer to one of the described frames in the GOF, an index to refer to one of the described pictures in the PG, from
from which the frames referenced by the frame transmitted in the which the pictures referenced by the picture transmitted in the
current packet for inter-picture prediction can be identified. current packet for inter-picture prediction can be identified.
(Editor's Note: A "Picture Group", as used in this document, is not
the same thing as a "Group of Pictures" as traditionally used in
video coding. Suggestions for better terminology are welcome.)
The SS data can also be used to specify the resolution of each The SS data can also be used to specify the resolution of each
spatial layer present in the VP9 stream for both flexible and non- spatial layer present in the VP9 stream for both flexible and non-
flexible modes. flexible modes.
4. Payload Format 4. Payload Format
This section describes how the encoded VP9 bitstream is encapsulated This section describes how the encoded VP9 bitstream is encapsulated
in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is in RTP. To handle network losses usage of RTP/AVPF [RFC4585] is
RECOMMENDED. All integer fields in the specifications are encoded as RECOMMENDED. All integer fields in the specifications are encoded as
unsigned integers in network octet order. unsigned integers in network octet order.
skipping to change at page 5, line 46 skipping to change at page 5, line 51
The VP9 payload descriptor and VP9 payload header will be described The VP9 payload descriptor and VP9 payload header will be described
in Section 4.2 and Section 4.3. OPTIONAL RTP padding MUST NOT be in Section 4.2 and Section 4.3. OPTIONAL RTP padding MUST NOT be
included unless the P bit is set. The figure specifically shows the included unless the P bit is set. The figure specifically shows the
format for the first packet in a frame. Subsequent packets will not format for the first packet in a frame. Subsequent packets will not
contain the VP9 payload header, and will have later octets in the contain the VP9 payload header, and will have later octets in the
frame payload. frame payload.
Figure 1 Figure 1
Marker bit (M): MUST be set to 1 for the final packet of the highest Marker bit (M): MUST be set to 1 for the final packet of the highest
spatial layer frame (the final packet of the super frame), and 0 spatial layer frame (the final packet of the picture), and 0
otherwise. Unless spatial scalability is in use for this super otherwise. Unless spatial scalability is in use for this picture,
frame, this will have the same value as the E bit described below. this will have the same value as the E bit described below. Note
Note this bit MUST be set to 1 for the target spatial layer frame this bit MUST be set to 1 for the target spatial layer frame if a
if a stream is being rewritten to remove higher spatial layers. stream is being rewritten to remove higher spatial layers.
Payload Type (TP): In line with the policy in Section 3 of Payload Type (PT): In line with the policy in Section 3 of
[RFC3551], applications using the VP9 RTP payload profile MUST [RFC3551], applications using the VP9 RTP payload profile MUST
assign a dynamic payload type number to be used in each RTP assign a dynamic payload type number to be used in each RTP
session and provide a mechanism to indicate the mapping. See session and provide a mechanism to indicate the mapping. See
Section 6.2 for the mechanism to be used with the Session Section 6.2 for the mechanism to be used with the Session
Description Protocol (SDP) [RFC4566]. Description Protocol (SDP) [RFC4566].
Timestamp: The RTP timestamp indicates the time when the input frame Timestamp: The RTP timestamp indicates the time when the input frame
was sampled, at a clock rate of 90 kHz. If the input frame is was sampled, at a clock rate of 90 kHz. If the input picture is
encoded with multiple layer frames, all of the layer frames of the encoded with multiple layer frames, all of the frames of the
super frame MUST have the same timestamp. picture MUST have the same timestamp.
The remaining RTP Fixed Header Fields (V, P, X, CC, sequence number, The remaining RTP Fixed Header Fields (V, P, X, CC, sequence number,
SSRC and CSRC identifiers) are used as specified in Section 5.1 of SSRC and CSRC identifiers) are used as specified in Section 5.1 of
[RFC3550]. [RFC3550].
4.2. VP9 Payload Description 4.2. VP9 Payload Description
In flexible mode (with the F bit below set to 1), The first octets In flexible mode (with the F bit below set to 1), The first octets
after the RTP header are the VP9 payload descriptor, with the after the RTP header are the VP9 payload descriptor, with the
following structure. following structure.
skipping to change at page 7, line 31 skipping to change at page 7, line 31
V: | SS | V: | SS |
| .. | | .. |
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
Figure 3 Figure 3
I: Picture ID (PID) present. When set to one, the OPTIONAL PID MUST I: Picture ID (PID) present. When set to one, the OPTIONAL PID MUST
be present after the mandatory first octet and specified as below. be present after the mandatory first octet and specified as below.
Otherwise, PID MUST NOT be present. Otherwise, PID MUST NOT be present.
P: Inter-picture predicted layer frame. When set to zero, the layer P: Inter-picture predicted picture. When set to zero, the picture
frame does not utilize inter-picture prediction. In this case, does not utilize inter-picture prediction. In this case, up-
up-switching to current spatial layer's frame is possible from switching to a current spatial layer's frame is possible from
directly lower spatial layer frame. P SHOULD also be set to zero directly lower spatial layer frame. P SHOULD also be set to zero
when encoding a layer synchronization frame in response to an LRR when encoding a layer synchronization frame in response to an LRR
[I-D.ietf-avtext-lrr] message (see Section 5.4). When P is set to [I-D.ietf-avtext-lrr] message (see Section 5.4). When P is set to
zero, the T bit (described below) MUST also be set to 0 (if zero, the T field (described below) MUST also be set to 0 (if
present). present). Note that the P bit does not forbid intra-picture,
inter-layer prediction from earlier frames of the same picture, if
any.
L: Layer indices present. When set to one, the one or two octets L: Layer indices present. When set to one, the one or two octets
following the mandatory first octet and the PID (if present) is as following the mandatory first octet and the PID (if present) is as
described by "Layer indices" below. If the F bit (described described by "Layer indices" below. If the F bit (described
below) is set to 1 (indicating flexible mode), then only one octet below) is set to 1 (indicating flexible mode), then only one octet
is present for the layer indices. Otherwise if the F bit is set is present for the layer indices. Otherwise if the F bit is set
to 0 (indicating non-flexible mode), then two octets are present to 0 (indicating non-flexible mode), then two octets are present
for the layer indices. for the layer indices.
F: Flexible mode. F set to one indicates flexible mode and if the P F: Flexible mode. F set to one indicates flexible mode and if the P
bit is also set to one, then the octets following the mandatory bit is also set to one, then the octets following the mandatory
first octet, the PID, and layer indices (if present) are as first octet, the PID, and layer indices (if present) are as
described by "Reference indices" below. This MUST only be set to described by "Reference indices" below. This MUST only be set to
1 if the I bit is also set to one; if the I bit is set to zero, 1 if the I bit is also set to one; if the I bit is set to zero,
then this MUST also be set to zero and ignored by receivers. The then this MUST also be set to zero and ignored by receivers. The
value of this F bit CAN ONLY CHANGE on the very first packet of a value of this F bit MUST only change on the first packet of a key
key picture. This is a packet with the P bit equal to zero, S or picture. A key picture is a picture whose base spatial layer
D bit (described below) equal to zero, and B bit (described below) frame is a key frame, and which thus completely resets the encoder
equal to 1. state. This packet will have its P bit equal to zero, S or D bit
(described below) equal to zero, and B bit (described below) equal
to 1.
B: Start of a layer frame. MUST be set to 1 if the first payload B: Start of a frame. MUST be set to 1 if the first payload octet of
octet of the RTP packet is the beginning of a new VP9 layer frame, the RTP packet is the beginning of a new VP9 frame, and MUST NOT
and MUST NOT be 1 otherwise. Note that this layer frame might not be 1 otherwise. Note that this frame might not be the first frame
be the very first layer frame of a super frame. of a picture.
E: End of a layer frame. MUST be set to 1 for the final RTP packet E: End of a frame. MUST be set to 1 for the final RTP packet of a
of a VP9 layer frame, and 0 otherwise. This enables a decoder to VP9 frame, and 0 otherwise. This enables a decoder to finish
finish decoding the layer frame, where it otherwise may need to decoding the frame, where it otherwise may need to wait for the
wait for the next packet to explicitly know that the layer frame next packet to explicitly know that the frame is complete. Note
is complete. Note that, if spatial scalability is in use, more that, if spatial scalability is in use, more frames from the same
layer frames from the same super frame may follow; see the picture may follow; see the description of the M bit above.
description of the M bit above.
V: Scalability structure (SS) data present. When set to one, the V: Scalability structure (SS) data present. When set to one, the
OPTIONAL SS data MUST be present in the payload descriptor. OPTIONAL SS data MUST be present in the payload descriptor.
Otherwise, the SS data MUST NOT be present. Otherwise, the SS data MUST NOT be present.
-: Bit reserved for future use. MUST be set to zero and MUST be -: Bit reserved for future use. MUST be set to zero and MUST be
ignored by the receiver. ignored by the receiver.
The mandatory first octet is followed by the extension data fields The mandatory first octet is followed by the extension data fields
that are enabled: that are enabled:
skipping to change at page 8, line 50 skipping to change at page 9, line 6
on the M bit. This is a running index of the pictures. The field on the M bit. This is a running index of the pictures. The field
MUST be present if the I bit is equal to one. If M is set to MUST be present if the I bit is equal to one. If M is set to
zero, 7 bits carry the PID; else if M is set to one, 15 bits carry zero, 7 bits carry the PID; else if M is set to one, 15 bits carry
the PID in network byte order. The sender may choose between a 7- the PID in network byte order. The sender may choose between a 7-
or 15-bit index. The PID SHOULD start on a random number, and or 15-bit index. The PID SHOULD start on a random number, and
MUST wrap after reaching the maximum ID. The receiver MUST NOT MUST wrap after reaching the maximum ID. The receiver MUST NOT
assume that the number of bits in PID stay the same through the assume that the number of bits in PID stay the same through the
session. session.
In the non-flexible mode (when the F bit is set to 0), this PID is In the non-flexible mode (when the F bit is set to 0), this PID is
used as an index to the GOF specified in the SS data bleow. In used as an index to the picture group (PG) specified in the SS
this mode, the PID of the key frame corresponds to the very first data below. In this mode, the PID of the key frame corresponds to
specified frame in the GOF. Then subsequent PIDs are mapped to the first specified frame in the PG. Then subsequent PIDs are
subsequently specified frames in the GOF (modulo N_G, specified in mapped to subsequently specified frames in the PG (modulo N_G,
the SS data below), respectively. specified in the SS data below), respectively.
Layer indices: This information is optional but recommended whenever Layer indices: This information is optional but recommended whenever
encoding with layers. For both flexible and non-flexible modes, encoding with layers. For both flexible and non-flexible modes,
one octet is used to specify a layer frame's temporal layer ID (T) one octet is used to specify a layer frame's temporal layer ID (T)
and spatial layer ID (S) as shown both in Figure 2 and Figure 3. and spatial layer ID (S) as shown both in Figure 2 and Figure 3.
Additionally, a bit (U) is used to indicate that the current frame Additionally, a bit (U) is used to indicate that the current frame
is a "switching up point" frame. Another bit (D) is used to is a "switching up point" frame. Another bit (D) is used to
indicate whether inter-layer prediction is used for the current indicate whether inter-layer prediction is used for the current
layer frame. frame.
In the non-flexible mode (when the F bit is set to 0), another In the non-flexible mode (when the F bit is set to 0), another
octet is used to represent temporal layer 0 index (TL0PICIDX), as octet is used to represent temporal layer 0 index (TL0PICIDX), as
depicted in Figure 3. The TL0PICIDX is present so that all depicted in Figure 3. The TL0PICIDX is present so that all
minimally required frames - the base temporal layer frames - can minimally required frames - the base temporal layer frames - can
be tracked. be tracked.
The T and S fields indicate the temporal and spatial layers and The T and S fields indicate the temporal and spatial layers and
can help middleboxes and and endpoints quickly identify which can help middleboxes and and endpoints quickly identify which
layer a packet belongs to. layer a packet belongs to.
T: The temporal layer ID of current frame. In the case of non- T: The temporal layer ID of current frame. In the case of non-
flexible mode, if PID is mapped to a frame in a specified GOF, flexible mode, if PID is mapped to a picture in a specified PG,
then the value of T MUST match the corresponding T value of the then the value of T MUST match the corresponding T value of the
mapped frame in the GOF. mapped picture in the PG.
U: Switching up point. If this bit is set to 1 for the current U: Switching up point. If this bit is set to 1 for the current
frame with temporal layer ID equal to T, then "switch up" to a picture with temporal layer ID equal to T, then "switch up" to
higher frame rate is possible as subsequent higher temporal a higher frame rate is possible as subsequent higher temporal
layer frames will not depend on any frame before the current layer pictures will not depend on any picture before the
frame (in coding time) with temporal layer ID greater than T. current picture (in coding order) with temporal layer ID
greater than T.
S: The spatial layer ID of current frame. Note that frames with S: The spatial layer ID of current frame. Note that frames with
spatial layer S > 0 may be dependent on decoded spatial layer spatial layer S > 0 may be dependent on decoded spatial layer
S-1 frame within the same super frame. S-1 frame within the same picture.
D: Inter-layer dependency used. MUST be set to one if current D: Inter-layer dependency used. MUST be set to one if current
spatial layer S frame depends on spatial layer S-1 frame of the spatial layer S frame depends on spatial layer S-1 frame of the
same super frame. MUST only be set to zero if current spatial same picture. MUST only be set to zero if current spatial
layer S frame does not depend on spatial layer S-1 frame of the layer S frame does not depend on spatial layer S-1 frame of the
same super frame. For the base layer frame with S equal to 0, same picture. For the base layer frame (with S equal to 0),
this D bit MUST be set to zero. this D bit MUST be set to zero.
TL0PICIDX: 8 bits temporal layer zero index. TL0PICIDX is only TL0PICIDX: 8 bits temporal layer zero index. TL0PICIDX is only
present in the non-flexible mode (F = 0). This is a running present in the non-flexible mode (F = 0). This is a running
index for the temporal base layer frames, i.e., the frames with index for the temporal base layer pictures, i.e., the pictures
T set to 0. If T is larger than 0, TL0PICIDX indicates which with T set to 0. If T is larger than 0, TL0PICIDX indicates
temporal base layer frame the current frame depends on. which temporal base layer picture the current picture depends
TL0PICIDX MUST be incremented when T is equal to 0. The index on. TL0PICIDX MUST be incremented when T is equal to 0. The
SHOULD start on a random number, and MUST restart at 0 after index SHOULD start on a random number, and MUST restart at 0
reaching the maximum number 255. after reaching the maximum number 255.
Reference indices: When P and F are both set to one, indicating a Reference indices: When P and F are both set to one, indicating a
non-key frame in flexible mode, then at least one reference index non-key frame in flexible mode, then at least one reference index
has to be specified as below. Additional reference indices (total has to be specified as below. Additional reference indices (total
of up to 3 reference indices are allowed) may be specified using of up to 3 reference indices are allowed) may be specified using
the N bit below. When either P or F is set to zero, then no the N bit below. When either P or F is set to zero, then no
reference index is specified. reference index is specified.
P_DIFF: The reference index (in 7 bits) specified as the relative P_DIFF: The reference index (in 7 bits) specified as the relative
PID from the current frame. For example, when P_DIFF=3 on a PID from the current picture. For example, when P_DIFF=3 on a
packet containing the frame with PID 112 means that the frame packet containing the picture with PID 112 means that the
refers back to the frame with PID 109. This calculation is picture refers back to the picture with PID 109. This
done modulo the size of the PID field, i.e., either 7 or 15 calculation is done modulo the size of the PID field, i.e.,
bits. either 7 or 15 bits.
N: 1 if there is additional P_DIFF following the current P_DIFF. N: 1 if there is additional P_DIFF following the current P_DIFF.
4.2.1. Scalability Structure (SS): 4.2.1. Scalability Structure (SS):
The scalability structure (SS) data describes the resolution of each The scalability structure (SS) data describes the resolution of each
layer frame within a super frame as well as the inter-picture frame within a picture as well as the inter-picture dependencies for
dependencies for a group of frames (GOF). If the VP9 payload a picture group (PG). If the VP9 payload descriptor's "V" bit is
descriptor's "V" bit is set, the SS data is present in the position set, the SS data is present in the position indicated in Figure 2 and
indicated in Figure 2 and Figure 3. Figure 3.
+-+-+-+-+-+-+-+-+ +-+-+-+-+-+-+-+-+
V: | N_S |Y|G|-|-|-| V: | N_S |Y|G|-|-|-|
+-+-+-+-+-+-+-+-+ -\ +-+-+-+-+-+-+-+-+ -\
Y: | WIDTH | (OPTIONAL) . Y: | WIDTH | (OPTIONAL) .
+ + . + + .
| | (OPTIONAL) . | | (OPTIONAL) .
+-+-+-+-+-+-+-+-+ . - N_S + 1 times +-+-+-+-+-+-+-+-+ . - N_S + 1 times
| HEIGHT | (OPTIONAL) . | HEIGHT | (OPTIONAL) .
+ + . + + .
skipping to change at page 11, line 33 skipping to change at page 11, line 33
Figure 4 Figure 4
N_S: N_S + 1 indicates the number of spatial layers present in the N_S: N_S + 1 indicates the number of spatial layers present in the
VP9 stream. VP9 stream.
Y: Each spatial layer's frame resolution present. When set to one, Y: Each spatial layer's frame resolution present. When set to one,
the OPTIONAL WIDTH (2 octets) and HEIGHT (2 octets) MUST be the OPTIONAL WIDTH (2 octets) and HEIGHT (2 octets) MUST be
present for each layer frame. Otherwise, the resolution MUST NOT present for each layer frame. Otherwise, the resolution MUST NOT
be present. be present.
G: GOF description present flag. G: PG description present flag.
-: Bit reserved for future use. MUST be set to zero and MUST be -: Bit reserved for future use. MUST be set to zero and MUST be
ignored by the receiver. ignored by the receiver.
N_G: N_G indicates the number of frames in a GOF. If N_G is greater N_G: N_G indicates the number of pictures in a Picture Group (PG).
than 0, then the SS data allows the inter-picture dependency If N_G is greater than 0, then the SS data allows the inter-
structure of the VP9 stream to be pre-declared, rather than picture dependency structure of the VP9 stream to be pre-declared,
indicating it on the fly with every packet. If N_G is greater rather than indicating it on the fly with every packet. If N_G is
than 0, then for N_G pictures in the GOF, each frame's temporal greater than 0, then for N_G pictures in the PG, each picture's
layer ID (T), switch up point (U), and the R reference indices temporal layer ID (T), switch up point (U), and the R reference
(P_DIFFs) are specified. indices (P_DIFFs) are specified.
The very first frame specified in the GOF MUST have T set to 0. The first picture specified in the PG MUST have T set to 0.
G set to 0 or N_G set to 0 indicates that either there is only one G set to 0 or N_G set to 0 indicates that either there is only one
temporal layer or no fixed inter-picture dependency information is temporal layer or no fixed inter-picture dependency information is
present going forward in the bitstream. present going forward in the bitstream.
Note that for a given super frame, all layer frames follow the Note that for a given picture, all frames follow the same inter-
same inter-picture dependency structure. However, the frame rate picture dependency structure. However, the frame rate of each
of each spatial layer can be different from each other and this spatial layer can be different from each other and this can be
can be controlled with the use of the D bit described above. The controlled with the use of the D bit described above. The
specified dependency structure in the SS data MUST be for the specified dependency structure in the SS data MUST be for the
highest frame rate layer. highest frame rate layer.
In a scalable stream sent with a fixed pattern, the SS data SHOULD be In a scalable stream sent with a fixed pattern, the SS data SHOULD be
included in the first packet of every key frame. This is a packet included in the first packet of every key frame. This is a packet
with P bit equal to zero, S or D bit equal to zero, and B bit equal with P bit equal to zero, S or D bit equal to zero, and B bit equal
to 1. The SS data MUST only be changed on the frame that corresponds to 1. The SS data MUST only be changed on the picture that
to the very first frame specified in the previous SS data's GOF (if corresponds to the first picture specified in the previous SS data's
the previous SS data's N_G was greater than 0). PG (if the previous SS data's N_G was greater than 0).
4.3. VP9 Payload Header 4.3. VP9 Payload Header
TODO: need to describe VP9 payload header. TODO: need to describe VP9 payload header.
4.4. Frame Fragmentation 4.4. Frame Fragmentation
VP9 frames are fragmented into packets, in RTP sequence number order, VP9 frames are fragmented into packets, in RTP sequence number order,
beginning with a packet with the B bit set, and ending with a packet beginning with a packet with the B bit set, and ending with a packet
with the RTP marker bit M set. There is no mechanism for finer- with the E bit set. There is no mechanism for finer-grained access
grained access to parts of a VP9 frame. to parts of a VP9 frame.
4.5. Examples of VP9 RTP Stream 4.5. Scalable encoding considerations
TODO In addition to the use of reference frames, VP9 has several
additional forms of inter-frame dependencies, largely involving
probability tables for the entropy and tree encoders. In VP9 syntax,
the syntax element "error_resilient_mode" resets this additional
inter-frame data, allowing a frame's syntax to be decoded
independently.
5. Feedback Messages Due to the requirements of scalable streams, a VP9 encoder producing
a scalable stream needs to ensure that a frame does not depend on a
previous frame (of the same or a previous picture) that can
legitimately be removed from the stream. Thus, a frame that follows
a removable frame (in full decode order) MUST be encoded with
"error_resilient_mode" to true.
5.1. Reference Picture Selection Indication (RPSI) For spatially-scalable streams, this means that
"error_resilient_mode" needs to be turned on for the base spatial
layer; it can however be turned off for higher spatial layers,
assuming they are sent with inter-layer dependency (i.e. with the "D"
bit set). For streams that are only temporally-scalable without
spatial scalability, "error_resilient_mode" can additionally be
turned off for any picture that immediately follows a temporal layer
0 frame.
TODO: Update to indicate which frame within the picture. 4.6. Examples of VP9 RTP Stream
TODO: Examples of packet layouts
4.6.1. Reference picture use for scalable structure
As discussed in Section 3, the VP9 codec can maintain up to eight
reference frames, of which up to three can be referenced or updated
by any new frame. This section illustrates one way that a scalable
structure (with three spatial layers and three temporal layers) can
be constructed using these reference frames.
+----------+---------+------------+---------+
| Temporal | Spatial | References | Updates |
+----------+---------+------------+---------+
| 0 | 0 | 0 | 0 |
| | | | |
| 0 | 1 | 0,1 | 1 |
| | | | |
| 0 | 2 | 1,2 | 2 |
| | | | |
| 2 | 0 | 0 | 6 |
| | | | |
| 2 | 1 | 1,6 | 7 |
| | | | |
| 2 | 2 | 2,7 | - |
| | | | |
| 1 | 0 | 0 | 3 |
| | | | |
| 1 | 1 | 1,3 | 4 |
| | | | |
| 1 | 2 | 2,4 | 5 |
| | | | |
| 2 | 0 | 3 | 6 |
| | | | |
| 2 | 1 | 4,6 | 7 |
| | | | |
| 2 | 2 | 5,7 | - |
+----------+---------+------------+---------+
Example scalability structure
This structure is constructed such that the "U" bit can always be
set.
5. Feedback Messages and Header Extensions
5.1. Reference Picture Selection Indication (RPSI)
The reference picture selection index is a payload-specific feedback The reference picture selection index is a payload-specific feedback
message defined within the RTCP-based feedback format. The RPSI message defined within the RTCP-based feedback format. The RPSI
message is generated by a receiver and can be used in two ways. message is generated by a receiver and can be used in two ways.
Either it can signal a preferred reference picture when a loss has Either it can signal a preferred reference picture when a loss has
been detected by the decoder -- preferably then a reference that the been detected by the decoder -- preferably then a reference that the
decoder knows is perfect -- or, it can be used as positive feedback decoder knows is perfect -- or, it can be used as positive feedback
information to acknowledge correct decoding of certain reference information to acknowledge correct decoding of certain reference
pictures. The positive feedback method is useful for VP9 used for pictures. The positive feedback method is useful for VP9 used for
point to point (unicast) communication. The use of RPSI for VP9 is point to point (unicast) communication. The use of RPSI for VP9 is
preferably combined with a special update pattern of the codec's two preferably combined with a special update pattern of the codec's two
special reference frames -- the golden frame and the altref frame -- special reference frames -- the golden frame and the altref frame --
in which they are updated in an alternating leapfrog fashion. When a in which they are updated in an alternating leapfrog fashion. When a
receiver has received and correctly decoded a golden or altref frame, receiver has received and correctly decoded a golden or altref frame,
and that frame had a PictureID in the payload descriptor, the and that frame had a PictureID in the payload descriptor, the
receiver can acknowledge this simply by sending an RPSI message back receiver can acknowledge this simply by sending an RPSI message back
to the sender. The message body (i.e., the "native RPSI bit string" to the sender. The message body (i.e., the "native RPSI bit string"
in [RFC4585]) is simply the PictureID of the received frame. in [RFC4585]) is simply the PictureID of the received frame.
Note: because all frames of the same picture must have the same
inter-picture reference structure, there is no need for a message to
specify which frame is being selected.
5.2. Slice Loss Indication (SLI) 5.2. Slice Loss Indication (SLI)
TODO: Update to indicate which frame within the picture. TODO: Update to indicate which frame within the picture.
The slice loss indication is another payload-specific feedback The slice loss indication is another payload-specific feedback
message defined within the RTCP-based feedback format. The SLI message defined within the RTCP-based feedback format. The SLI
message is generated by the receiver when a loss or corruption is message is generated by the receiver when a loss or corruption is
detected in a frame. The format of the SLI message is as follows detected in a frame. The format of the SLI message is as follows
[RFC4585]: [RFC4585]:
skipping to change at page 13, line 48 skipping to change at page 15, line 21
an SLI message, it can make use of the knowledge from the latest an SLI message, it can make use of the knowledge from the latest
received RPSI message. Knowing that the last golden or altref frame received RPSI message. Knowing that the last golden or altref frame
was successfully received, it can encode the next frame with was successfully received, it can encode the next frame with
reference to that established reference. reference to that established reference.
5.3. Full Intra Request (FIR) 5.3. Full Intra Request (FIR)
The Full Intra Request (FIR) [RFC5104] RTCP feedback message allows a The Full Intra Request (FIR) [RFC5104] RTCP feedback message allows a
receiver to request a full state refresh of an encoded stream. receiver to request a full state refresh of an encoded stream.
Upon receipt of an FIR request, a VP9 sender MUST send a super frame Upon receipt of an FIR request, a VP9 sender MUST send a picture with
with a keyframe for its spatial layer 0 layer frame, and then send a keyframe for its spatial layer 0 layer frame, and then send frames
frames without inter-picture prediction (P=0) for any higher layer without inter-picture prediction (P=0) for any higher layer frames.
frames.
5.4. Layer Refresh Request (LRR) 5.4. Layer Refresh Request (LRR)
The Layer Refresh Request [I-D.ietf-avtext-lrr] allows a receiver to The Layer Refresh Request [I-D.ietf-avtext-lrr] allows a receiver to
request a single layer of a spatially or temporally encoded stream to request a single layer of a spatially or temporally encoded stream to
be refreshed, without necessarily affecting the stream's other be refreshed, without necessarily affecting the stream's other
layers. layers.
+---------------+---------------+ +---------------+---------------+
|0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7| |0|1|2|3|4|5|6|7|0|1|2|3|4|5|6|7|
+-------------+-----------------+ +---------------+---------+-----+
| T |R| S | RES | | RES | T | RES | S |
+-------------+-----------------+ +---------------+---------+-----+
Figure 6 Figure 6
Figure 6 shows the format of LRR's layer index field for VP9 streams. Figure 6 shows the format of LRR's layer index fields for VP9
This is designed to follow the same layout as the "L" byte of the VP9 streams. The two "RES" fields MUST be set to 0 on transmission and
payload header, which carries the stream's layer information. The ingnored on reception. See Section 4.2 for details on the T and S
"R" and "RES" fields MUST be set to 0 on transmission and ingnored on fields.
reception. See Section 4.2 for details on the T and S fields.
Identification of a layer refresh frame can be derived from the Identification of a layer refresh frame can be derived from the
reference IDs of each frame by backtracking the dependency chain reference IDs of each frame by backtracking the dependency chain
until reaching a point where only decodable frames are being until reaching a point where only decodable frames are being
referenced. Therefore it's recommended for both the flexible and the referenced. Therefore it's recommended for both the flexible and the
non-flexible mode that, when upgrade frames are being encoded in non-flexible mode that, when upgrade frames are being encoded in
response to a LRR, those packets should contain layer indices and the response to a LRR, those packets should contain layer indices and the
reference fields so that the decoder or an MCU can make this reference fields so that the decoder or an MCU can make this
derivation. derivation.
skipping to change at page 14, line 46 skipping to change at page 16, line 18
LRR {1,0}, {2,1} is sent by an MCU when it is currently relaying LRR {1,0}, {2,1} is sent by an MCU when it is currently relaying
{1,0} to a receiver and which wants to upgrade to {2,1}. In response {1,0} to a receiver and which wants to upgrade to {2,1}. In response
the encoder should encode the next frames in layers {1,1} and {2,1} the encoder should encode the next frames in layers {1,1} and {2,1}
by only referring to frames in {1,0}, or {0,0}. by only referring to frames in {1,0}, or {0,0}.
In the non-flexible mode, periodic upgrade frames can be defined by In the non-flexible mode, periodic upgrade frames can be defined by
the layer structure of the SS, thus periodic upgrade frames can be the layer structure of the SS, thus periodic upgrade frames can be
automatically identified by the picture ID. automatically identified by the picture ID.
5.5. Frame Marking
The Frame Marking RTP header extension [I-D.ietf-avtext-framemarking]
is a mechanism to provide information about frames of video streams
in a largely codec-independent manner. However, for its extension
for scalable codecs, the specific manner in which codec layers are
identified needs to be specified specifically for each codec. This
section defines how frame marking is used with VP9.
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| ID=2 | L=2 |S|E|I|D|B| T |0|0|0|0|0| S | TL0PICIDX |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 7
When this header extension is used with VP9, the T and S fields MUST
match the values in the packet which the header extension is attached
to; see Section 4.2 for details on these fields.
See [I-D.ietf-avtext-framemarking] for explanations of the other
fields, which are generic.
6. Payload Format Parameters 6. Payload Format Parameters
This payload format has two optional parameters. This payload format has two optional parameters.
6.1. Media Type Definition 6.1. Media Type Definition
This registration is done using the template defined in [RFC6838] and This registration is done using the template defined in [RFC6838] and
following [RFC4855]. following [RFC4855].
Type name: video Type name: video
skipping to change at page 18, line 6 skipping to change at page 20, line 4
order to reduce network congestion. Note that discarding of non- order to reduce network congestion. Note that discarding of non-
reference frames cannot be done if the stream is encrypted (because reference frames cannot be done if the stream is encrypted (because
the non-reference marker is encrypted). the non-reference marker is encrypted).
9. IANA Considerations 9. IANA Considerations
The IANA is requested to register the following values: The IANA is requested to register the following values:
- Media type registration as described in Section 6.1. - Media type registration as described in Section 6.1.
10. References 10. References
10.1. Normative References 10.1. Normative References
[I-D.ietf-avtext-framemarking]
Berger, E., Nandakumar, S., and M. Zanaty, "Frame Marking
RTP Header Extension", draft-ietf-avtext-framemarking-04
(work in progress), March 2017.
[I-D.ietf-avtext-lrr] [I-D.ietf-avtext-lrr]
Lennox, J., Hong, D., Uberti, J., Holmer, S., and M. Lennox, J., Hong, D., Uberti, J., Holmer, S., and M.
Flodman, "The Layer Refresh Request (LRR) RTCP Feedback Flodman, "The Layer Refresh Request (LRR) RTCP Feedback
Message", draft-ietf-avtext-lrr-03 (work in progress), Message", draft-ietf-avtext-lrr-06 (work in progress),
July 2016. June 2017.
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/ Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/
RFC2119, March 1997, RFC2119, March 1997,
<http://www.rfc-editor.org/info/rfc2119>. <http://www.rfc-editor.org/info/rfc2119>.
[RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V. [RFC3550] Schulzrinne, H., Casner, S., Frederick, R., and V.
Jacobson, "RTP: A Transport Protocol for Real-Time Jacobson, "RTP: A Transport Protocol for Real-Time
Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550, Applications", STD 64, RFC 3550, DOI 10.17487/RFC3550,
July 2003, <http://www.rfc-editor.org/info/rfc3550>. July 2003, <http://www.rfc-editor.org/info/rfc3550>.
 End of changes. 51 change blocks. 
144 lines changed or deleted 249 lines changed or added

This html diff was produced by rfcdiff 1.45. The latest version is available from http://tools.ietf.org/tools/rfcdiff/