Parse the H.264/HEVC frame_packing_arrangement SEI so frame-packed 3D is detected without a container tag #3419

[REQUIRED] Use case description

Media3 detects stereoscopic video from container-level signalling only. Frame-packed 3D that carries its layout in the elementary stream is invisible to it.

What is present today:

• MatroskaExtractor reads the Matroska StereoMode element (private static final int ID_STEREO_MODE = 0x53B8;).
• BoxParser reads Apple's vexu box, and HevcConfig parses the 3D Reference Display Information SEI, for MV-HEVC spatial video.
• H264Reader's SEI handling is documented as "An SEI reader for consuming closed caption channels", and wires up only CEA-608/708.

What is absent: the classic frame_packing_arrangement SEI — ITU-T H.264 Annex D, payload type 45, and its HEVC equivalent. Searching this repository for frame_packing, FramePacking or frame packing returns no hits.

The consequence is narrow but real: an H.264 stream whose only stereoscopic signal is that SEI plays as a flat double-width (or double-height) picture, with no way for an app to know it is 3D. That is the normal case for:

• MPEG-TS delivery, where container-level stereoscopic tags do not exist at all. In the DVB frame-compatible 3DTV specification (ETSI TS 101 547-2) the in-stream SEI is the signal, and it takes precedence over container signalling.
• Any remux from Matroska to TS, which drops StereoMode silently.
• Broadcast-era and disc-derived material generally, which is where most frame-packed content in existence comes from.

This was requested once before, on the ExoPlayer tracker: google/ExoPlayer#7869 (September 2020), answered with "as you are the first one to ask for it, we will probably not look into it in the near future" and labelled low priority. This is the second ask, five years on, with measurements and with the rest of the ecosystem having moved.

Where the ecosystem now stands, in case it helps weigh the cost:

• GStreamer already does the whole loop: gst_h264_parser_parse_frame_packing() decodes payload type 45, h264parse maps it to GstVideoMultiviewMode and publishes it on caps, and x264enc derives its write-side parameter from those caps automatically.
• FFmpeg exposes the same signal as AV_FRAME_DATA_STEREO3D, so anything built on libavcodec can read it in a few lines.
• VLC parses payload type 45 into multiview_mode in its H.264 packetizer and uses it for stereo rendering.
• HandBrake merged a patch to write the SEI when encoding (PR #8100, September 2026).
• Universal Media Server has an approved PR doing the same for its transcodes (PR #6330).

Why it is worth doing now rather than in 2020. Stereoscopic playback stopped being a niche the moment headsets became ordinary again, and Android is the platform most of them run on. Media3 already grew MV-HEVC support for Apple spatial video (#2011); this is the same class of feature for the far larger body of existing content, which is frame-packed rather than multi-view.

The behaviour is measurable on hardware, not theoretical. On two 2011/2012 televisions I own, an MP4 whose only stereoscopic signal is this SEI switches the set into 3D automatically; the identical file with the SEI removed — a difference of 791 bytes, picture data untouched — plays flat; and a Matroska carrying only the container StereoMode tag is ignored. Method and files: https://github.com/danielcamposramos/sony-bravia-linux/blob/main/docs/3d-signalling-ecosystem.md

Proposed solution

Parse SEI payload type 45 where the elementary stream is already being walked (H264Reader/SeiReader, and the HEVC equivalent), and map it onto the Format.stereoMode values that already exist:

content_interpretation_type carries eye order and would select between left-first and right-first. A container tag, where one exists, should presumably still win, so this would only fill in stereoMode when the container did not state it.

One adjacent observation, since it is in the same area and may be a separate small fix: WebmConstants defines STEREO_MODE = 0x53B8 and lists it in the element table, but WebmWriter, WebmMuxer and WebmElements contain no reference to it, so Media3's own muxer does not appear to write the tag back out on transcode.

Alternatives considered

• Remux to add a container tag. Fails for the MPEG-TS and streaming cases, which is where this signal is most common, and requires knowing the layout in the first place.
• Leave it to the app. An app can only guess from resolution and aspect, or ask the user to pick a layout by hand, which is what players without this support do today.

Happy to help with the implementation if that would be useful — I understand a code contribution would need the Google CLA, and I am willing to do that if you would take the patch.