News Column

Researchers Submit Patent Application, "Method of Multiplexing H.264 Elementary Streams without Timing Information Coded", for Approval

July 24, 2014

By a News Reporter-Staff News Editor at Politics & Government Week -- From Washington, D.C., VerticalNews journalists report that a patent application by the inventor Karegoudar, Malateshgouda V. (Bangalore, IN), filed on March 4, 2014, was made available online on July 10, 2014.

The patent's assignee is Tektronix International Sales GmbH.

News editors obtained the following quote from the background information supplied by the inventors: "The digital broadcast industry is currently embracing multiple technology advancements including High Definition, Internet Protocol Television (IPTV), and Mobile Video. The key enabler for these technologies is the compression efficiencies and therefore lower bandwidth requirements of the H.264/AVC (Advanced Video Codec/MPEG-4, part 10) video codec. MPEG-2 ('Moving Picture Experts Group') and 'H.264,' both of which are international standards for moving image encoding processes, have defined an image frame or an image field as a unit of compression, called a 'picture.' Each 'picture' is used as an access unit (AU) in encoding and decoding processes.

"Extensions to the MPEG-2 standard to incorporate H.264/AVC as an additional Elementary Stream (ES) type within an MPEG-2 Transport Stream (TS) have made it accessible and attractive to broadcasters and network operators

"An elementary stream (ES) is basically a raw compressed representation of one audio or video signal. The format of the elementary stream depends upon the codec or data carried in the stream, and carries a common header when packetized into a packetized elementary stream.

"In MPEG-2 systems, a video elementary stream is assembled into a packetized elementary stream (PES). Presentation Time Stamps (PTS) are carried in headers of the packetized elementary stream. Decoding time stamps (DTS) are also carried in PES headers of an I- or P-picture when bi-directional predictive coding is enabled.

"An H.264 Video elementary stream is generally stored as a file while multiplexing in deferred mode. The H.264 Video elementary stream consists of a sequence of Network Abstraction Layer (NAL) unit syntax structures. NAL Units are classified into Video Coding Later (VCL) and non-VCL NAL Units. The VCL NAL units contain data that represent video pictures; non-VCL NAL Units contain additional information such as Sequence Parameter Set (SPS) and Picture Parameter Set (PPS) which apply to a large number of VCL NAL Units. Parameters in the Sequence Parameter Set (SPS) include identifier limits on the frame number, picture order count (POC), and frame rate of the stream. SEI NAL Units are also coded to enhance the usability, however are not necessary for decoding the VCL NAL units.

"Typically, one or more sequence parameter set(s) and picture parameter set(s) are sent to the decoder prior to the decoding of the slice headers and slice data. A coded slice header refers to the picture parameter set and this activates the particular picture parameter set. The activated picture parameter set then remains active until a different picture parameter set is activated by being referred to in another slice header. Similarly, picture parameter set refers to the identifier which activates the sequence parameter set. The activated SPS remains active until a different sequence parameter set is activated.

"An H.264 video elementary stream can be considered as a coded video sequence. A coded video sequence consists of one or more Access Units. Each Access Unit (AU) is a set of NAL units that always contain exactly one primary coded picture. In addition to the primary coded picture, an AU may also contain one or more redundant coded pictures or other NAL units not containing slices or slice data partitions of a coded picture. The decoding of an AU always results in a decoded picture. The general structure of a conventional access unit is shown in FIG. 1.

"Packetized Elementary Streams (PES): For practical purposes, the endless continuous elementary streams carrying audio or video from compressors need to be broken into PES packets. These packets are identified by headers that contain time stamps (PTS and DTS) for synchronizing. PES packets can be used to create Program Streams or Transport Streams. After compression, pictures are sent out of sequence because of bidirectional coding. They require a variable amount of data and are subject to variable delay due to multiplexing and transmission. In order to keep the audio and video locked together, time stamps are periodically incorporated in each picture.

"A time stamp is a 33-bit number that is a sample of a counter driven by a 90-kHz clock. This clock is obtained by dividing the 27-MHz program clock by 300. Since presentation times are evenly spaced, it is not essential to include a time stamp in every presentation unit. Instead, time stamps can be interpolated by the decoder, but they must not be more than 700 ms apart in either program streams or transport streams.

"Time stamps indicate where a particular access unit belongs in time. Lip sync is obtained by incorporating time stamps into the headers in both video and audio PES packets. When a decoder receives a selected PES packet, it decodes each access unit and buffers it into RAM. When the time line count reaches the value of the time stamp, the RAM is read out. This operation has two desirable results. First, effective time base correction is obtained in each elementary stream. Second, the video and audio elementary streams can be synchronized together to make a program.

"PES Packets are broken into fixed 188 byte TS packets. Transport Stream (TS) packets always begin with a header. The remainder of the packet carries data referred to as the payload. Each PES packet forms the payload of the TS packet. The TS header is relatively small, but for special purposes (e.g., to carry PCR) the header may be extended. In this case the payload gets smaller so that the overall size of the TS packet is unchanged.

"FIG. 2 depicts a conventional method of multiplexing audio and video streams into a single transport stream. Packetizers 202a and 202b receive video and audio streams from video encoder 201a and audio encoder 201b respectively. These streams are multiplexed in a TS multiplexer 203 to obtain a single transport stream at the output. Each access unit contains the primary coded picture. The primary coded picture can be an I slice (Intra Slice in case of IDR, Slice contains integer number of macroblocks), P slice (Predictive slice), or B-slice (a slice that may be decoded using intra prediction). The type of picture carried is obtained by decoding the slice_type of the slice header. When the elementary stream contains access units with B-frames, the number of consecutive coded B-frames is variable and unbounded. The order of the coded frames in the bit-stream, also called the coded order, is the order in which a decoder reconstructs them. The order of the reconstructed frames at the output of the decoding process, also called the display order, is not always the same as the codec order. Presentation Time Stamp and Decode Time Stamp are coded in the PES headers which determine decode and display order of the pictures.

"A Sequence Parameter Set also contains video usability information (VUI). HRD (Hypothetical Reference Decoder) parameters are signaled through video usability information. HRD parameters contain the bit-rate information and values of coded picture buffer and decoded picture buffer. The HRD is used to check conformance of bit-streams and decoders. Also, the HRD parameters along with the optional SEI messages, when present, help to calculate the PTS and DTS values of the pictures coded in the stream. See Annex C of the H.264 standard for more information.

"When the H.264 Video Elementary stream do not have the HRD parameters in video usability information (VUI) coded, the number of reference parameter list list0 and list1 may be used for prediction. However, this can be a cumbersome process.

"Thus there is required a method that eliminates the need of constructing the reference list list0 and list1 (as described in the standard) to obtain the timing information for those elementary streams that do not have timing information coded."

As a supplement to the background information on this patent application, VerticalNews correspondents also obtained the inventor's summary information for this patent application: "It is an object of the present invention to provide a mechanism that can be applied in deferred time multiplexing of high definition video elementary streams which do not have timing parameters coded.

"Another object of the present invention is to provide a method for generating valid multiplex of video elementary streams to generate the Transport Stream (TS).

"Accordingly, embodiments of the present invention provide an apparatus and methods of multiplexing elementary streams without timing information coded.

"In some embodiments, 'n' number of access units (AU) are constructed and pushed into a list one by one. Each access unit is capable of looking ahead to obtain information on timing for the next access unit. A header is constructed by calculating timing information from the topmost access unit of the list. Said header is written in the TS and the remaining bytes of TS are filled with data from the said elementary stream. Thus the transport stream is constructed.

"In some embodiments, primary picture types coded in each of the access units are checked iteratively and accordingly timing information including presentation time stamp (PTS) and decode time stamp (DTS) for each are calculated. Also, variable frame rate (tc) is derived for the elementary streams.

"If the primary picture type coded is a B frame, then dts and pts are calculated as follows:

"dts(n)=dts(n-1)+tc, and


"The B frames are decoded and presented at the same time without including any offset between pts and dts of the B frames.

"For P and I type picture frames the dts and pts are calculated as follows:

"dts(n)=dts(n-1)+tc, and


"Next, access units are checked for the number of B frames (pts_offset) that precede before the next I or P frames occurs.

"Other features of the present invention will be apparent from the drawings, and from the detailed description that follows below.


"Reference will be made to embodiments of the invention, examples of which may be illustrated in the accompanying figures. These figures are intended to be illustrative, not limiting. Although the invention is generally described in the context of these embodiments, it should be understood that it is not intended to limit the scope of the invention to these particular embodiments.

"FIG. 1 illustrates the general structure of a conventional access unit.

"FIG. 2 depicts a conventional method of multiplexing audio and video into a single transport stream.

"FIG. 3 depicts a flowchart of a look-ahead method according to an embodiment of the present invention.

"FIG. 4 depicts a method of multiplexing high definition video elementary streams to transport streams according to an embodiment of the present invention.

"FIG. 5 depicts a block diagram of a multiplexing apparatus according to an embodiment of the present invention."

For additional information on this patent application, see: Karegoudar, Malateshgouda V. Method of Multiplexing H.264 Elementary Streams without Timing Information Coded. Filed March 4, 2014 and posted July 10, 2014. Patent URL:

Keywords for this news article include: Tektronix International Sales GmbH.

Our reports deliver fact-based news of research and discoveries from around the world. Copyright 2014, NewsRx LLC

For more stories covering the world of technology, please see HispanicBusiness' Tech Channel

Source: Politics & Government Week

Story Tools Facebook Linkedin Twitter RSS Feed Email Alerts & Newsletters