• Home   /  
  • Archive by category "1"

H 264 Header Format For Essay

Introduction to H.264: (1) NAL Unit

Author: Yumi Chan2014-11-12

H.264 / MPEG-4 Part 10, Advanced Video Coding (MPEG-4 AVC) is a common video compression format developed by ITU-T Video Coding Experts Group (VCEG) and ISO/IEC JTC1 Moving Picture Experts Group (MPEG). Network Abstraction Layer (NAL) and Video Coding Layer (VCL) are the two main concepts in H.264. A H.264 file consists of a number of NAL units (NALU) and each NALU can be classified as VCL or non-VCL. Video data is processed by the codec and packed into NAL units.

NALU in Packet-Transport Protocol V.S. Byte-Stream Format

There are two ways to pack a NAL unit for different systems, Packet-Transport System and Byte-Stream Format. For Packet-Transport systems like RTP, the transport system protocol frames the coded data into different pieces. Hence, the system can easily identify the boundaries of NAL units and we don’t need to add extra start code, which is a waste of resources. This method is usually used in streaming.

However, in other systems, there is no such protocol to separate NAL units. For example, you want to store a H.264 file and decode it on another computer. The decoder has no idea on how to search the boundaries of the NAL units. So, a three-byte or four-byte start code, 0x000001 or 0x00000001, is added at the beginning of each NAL unit. They are called Byte-Stream Format. Hence, the decoder can now identify the boundaries easily. According to ITU-T Recommendation, this format is usaully used in Rec. ITU-T H.222.0 | ISO/IEC 13818-1 systems or Rec. ITU-T H.320 systems.

In this series of articles, I will use byte-stream format as an example to introduce H.264. Here is a sample of a H.264 byte-stream. You can see several start codes 0x00000001 here, which means there are several NAL units in this image.

Figure 1: A sample of raw H.264 in byte stream format

But there are chances that 0x000001 or 0x00000001 exists in the bitstream of a NAL unit. So a emulation prevention bytes, 0x03, is presented when there is 0x000000, 0x000001, 0x000002 and 0x000003 to make them become 0x00000300, 0x00000301, 0x00000302 and 0x00000303 respectively. This ensures that no sequence of consecutive byte-aligned bytes in the NAL unit contains a start code prefix. Line 1 and 2 in the image demonstrate two examples. You may also read the next article for details.

First Byte of NALU: NAL Unit Header

In a NALU, the first byte is a header byte indicating the type of data contained in it and other information. The rest of bytes are the payload of a NAL unit. For the header byte, it can be parsed into 3 parts as shown. Let’s take 0x67 on Line 1 of Figure 1 as an example.

HexBinary
0x670110 0111

Figure 2: Parse the First Byte of a NAL unit

(Note: If you don’t understand the descriptor, read this article: Explanation of Descriptors in the ITU-T Publication on H.264 Coding Standard/Recommendation (with example) first)

So, for 0x67, we have:
forbidden_zero_bit = 0,
nal_ref_idc= 3,
nal_unit_type = 7

The 1st bit is forbidden_zero_bit which is used to check whether there is any error occurred during the transmission. 0 means that it is normal while 1 indicates a syntax violation. So, we should always find that forbidden_zero_bit equals 0.

The next 2 bits are nal_ref_idc indicating whether this NAL unit is a reference field / frame / picture. On one hand, if it is a reference field / frame / picture, nal_ref_idc is not equal to 0. According to the Recommendation, non-zero nal_ref_idc specifies that the content of the NAL unit contains a sequence parameter set (SPS), a SPS extension, a subset SPS, a picture parameter set (PPS), a slice of a reference picture, a slice data partition of a reference picture, or a prefix NAL unit preceding a slice of a reference picture. On the other hand, if it is a non-reference field / frame / picture, nal_ref_idc is equal to 0. For any non-zero value, the larger the value, the more the importance of the NAL unit. In this case, it is equal to 3 (0x11) and is a reference field / frame / picture (In fact, it’s a SPS. I’ll tell you more later).

The following 5 bits specify the nal_unit_type. It specifies the type of RBSP data structure contained in the NAL unit as specified in Table 7-1. VCL NAL units are specified as those NAL units having nal_unit_type equal to 1 to 5, inclusive. All remaining NAL units are called non-VCL NAL units.

More about NAL Unit Type

The following table lists all NAL unit types and their properties. (Normally, we follow Annex A.)

Table 7-1 – NAL unit type codes, syntax element categories, and NAL unit type classes

nal_unit_typeContent of NAL unit &
RBSP syntax structure
CNAL unit type class
[Annex A]
NAL unit type class
[Annex G & H]
NAL unit type class
[Annex I]
0Unspecifiednon-VCLnon-VCLnon-VCL
1Coded slice of a non-IDR picture
slice_layer_without_partitioning_rbsp( )
2, 3, 4VCLVCLVCL
2Coded slice data partition A
slice_data_partition_a_layer_rbsp( )
2VCLnot applicablenot applicable
3Coded slice data partition B
slice_data_partition_b_layer_rbsp( )
3VCLnot applicablenot applicable
4Coded slice data partition C
slice_data_partition_c_layer_rbsp( )
4VCLnot applicablenot applicable
5Coded slice of an IDR picture
slice_layer_without_partitioning_rbsp( )
2, 3VCLVCLVCL
6Supplemental enhancement information (SEI)
sei_rbsp( )
5non-VCLnon-VCLnon-VCL
7Sequence parameter set
seq_parameter_set_rbsp( )
0non-VCLnon-VCLnon-VCL
8Picture parameter set
pic_parameter_set_rbsp( )
1non-VCLnon-VCLnon-VCL
9Access unit delimiter
access_unit_delimiter_rbsp( )
6non-VCLnon-VCLnon-VCL
10End of sequence
end_of_seq_rbsp( )
7non-VCLnon-VCLnon-VCL
11End of stream
end_of_stream_rbsp( )
8non-VCLnon-VCLnon-VCL
12Filler data
filler_data_rbsp( )
9non-VCLnon-VCLnon-VCL
13Sequence parameter set extension
seq_parameter_set_extension_rbsp( )
10non-VCLnon-VCLnon-VCL
14Prefix NAL unit
prefix_nal_unit_rbsp( )
2non-VCLsuffix dependentsuffix dependent
15Subset sequence parameter set
subset_seq_parameter_set_rbsp( )
0non-VCLnon-VCLnon-VCL
16 – 18Reservednon-VCLnon-VCLnon-VCL
19Coded slice of an auxiliary coded picture without partitioning
slice_layer_without_partitioning_rbsp( )
2, 3, 4non-VCLnon-VCLnon-VCL
20Coded slice extension
slice_layer_extension_rbsp( )
2, 3, 4non-VCLVCLVCL
21Coded slice extension for depth view components
slice_layer_extension_rbsp( )
(specified in Annex I)
2, 3, 4non-VCLnon-VCLVCL
22 – 23Reservednon-VCLnon-VCLVCL
24 – 31Unspecifiednon-VCLnon-VCLnon-VCL

The column marked “C” lists the categories of the syntax elements that may be present in the NAL unit. In addition, syntax elements with syntax category “All” may be present, as determined by the syntax and semantics of the RBSP data structure. The presence or absence of any syntax elements of a particular listed category is determined from the syntax and semantics of the associated RBSP data structure. nal_unit_type shall not be equal to 3 or 4 unless at least one syntax element is present in the RBSP data structure having a syntax element category value equal to the value of nal_unit_type and not categorized as “All”.

The entry “suffix dependent” for nal_unit_type equal to 14 is specified as follows:
–  If the NAL unit directly following in decoding order is a NAL unit with type 1 or 5, it is a VCL NAL unit.
–  Otherwise (the NAL unit directly following in decoding order is a NAL unit with type not equal to 1 or 5), it is a non-VCL NAL unit.
Decoders shall ignore (remove from the bitstream and discard) the NAL unit with type 14 and the NAL unit directly following (in decoding order) the NAL unit with type 14.

To pack into byte-stream format:

byte_stream_nal_unit( NumBytesInNALunit ) { // Descriptor while(next_bits(24) != 0x000001 && next_bits(32) != 0x00000001) leading_zero_8bits /* equal to 0x00 */ // f(8) if(next_bits(24) != 0x000001) zero_byte /* equal to 0x00 */ // f(8) start_code_prefix_one_3bytes /* equal to 0x000001 */ // f(24) nal_unit( NumBytesInNALunit ) while(more_data_in_byte_stream() && next_bits(24) != 0x000001 && next_bits(32) != 0x00000001) trailing_zero_8bits /* equal to 0x00 */ // f(8) }

Relationship betweem nal_ref_idc and NAL Unit Type

Nal Unit TypePossible nal_ref_idc value
1 – 4If one of the NALU is 0, all NAL units with Type in the tang of 1 – 4, inclusive, of the picture are 0
5 Coded slice of an IDR picturenon-zero
7 Sequence parameter setnon-zero
8 Picture parameter setnon-zero
13 Sequence parameter set extensionnon-zero
15 Subset sequence parameter setnon-zero
6, 9, 10, 11 or 120

Comments

The H.264/AVC bitstream consists of a hierarchy of layers.

Fig.1 H.264/AVC syntax – a hierarchy of layers

The first layer is the Network Abstraction Layer (NAL) where the bitstream is divided into a set of NAL units. Some NAL units signal common control parameters to the decoder, such as the Sequence Parameter Sets (SPS) and Picture Parameter Sets (PPS). Others contain video data. The Video Coding Layer (VCL) NAL units contain slices of coded video. A coded frame or field is called an access unit and can be encoded as one or more slices.

A coded video sequence starts with an Instantaneous Decoder Refresh (IDR). All following video frames or fields are coded as slices. A new IDR signals that the previous video sequence is ended, and a new one is beginning.

Each NAL unit begins with a one byte header followed by the Raw Byte Sequence Payload (RBSP). The RBSP contains encoded slices. Slices are binary coded, so they may be padded with zero bits to ensure that the length is an integer number of bytes.

In the slice layer, each slice consists of a slice header and slice data. Slice data are specified as a series of macroblocks (MB) which includes the skip macroblock indicator (signal that macroblock position does not have data).

The MB layer specifies the MB structure. Each MB consists of:

  • MB type (I,P or B)
  • Prediction information that includes prediction mode for I-macroblock, reference frames and motion vectors for P and B macroblocks
  • Coded Block Pattern (CBP) that indicates luma and choma blocks that have non-zero residual coefficients
  • Quantization Parameter (QP) for non-zero macroblocks
  • Residual data, for non-zero macroblocks

More Information

One thought on “H 264 Header Format For Essay

Leave a comment

L'indirizzo email non verrà pubblicato. I campi obbligatori sono contrassegnati *