Parse frame packing SEI in MP4 and fragmented MP4, right-eye-first stereo modes, StereoMode in WebmMuxer - #3439

Builds on 58f396c (thank you, @microkatz), for the MP4 case in #3419.

What

• Mp4Extractor and FragmentedMp4Extractor read the stereo mode from H.264 and H.265 frame_packing_arrangement SEI messages, through the NalUnitUtil.parseSeiStereoMode added in 58f396c, unchanged. Only for H.264/H.265 tracks whose container declares no stereo mode, so a container value still wins.
  • The SEI NAL units are read while the sample is copied, as the MPEG-TS readers do, and as FragmentedMp4Extractor already does for CEA-608/708 (it now also buffers them for these tracks, and feeds the caption path only when caption outputs exist). No extra peeking.
  • The refined format is emitted ahead of the sample whose SEI carried it, like the sgpd-in-moof case in FragmentedMp4Extractor. As in the MPEG-TS readers, a cancellation or 2D arrangement only ends a stereo layout read earlier.
  • FLAG_OMIT_TRACK_SAMPLE_TABLE and init-segment-only reads are unaffected: track formats are output when the header is parsed, as before.
• Right eye first. The SEI's content_interpretation_type 2 means constituent frame 0 (the left or top half) is the right view, and Matroska has StereoMode 11 (side by side) and 2 (top-bottom) for the same layouts. Media3 had no constants for them, so these layouts were read as left eye first. This adds C.STEREO_MODE_RIGHT_LEFT and C.STEREO_MODE_BOTTOM_TOP: NalUnitUtil.parseSeiStereoMode reads content_interpretation_type, MatroskaExtractor maps 11 and 2, and the spherical video renderer (ProjectionRenderer) swaps the eyes for them.
• WebmMuxer writes the Matroska StereoMode element: 1 and 11 for side by side, 3 and 2 for top-bottom (left and right eye first), the inverse of MatroskaExtractor's mapping, so a transcode keeps the layout. Mono is the element's default and isn't written.

Why MP4

The case measured in #3419 is an MP4: a file whose only stereo signal is this SEI switches two 2011/2012 Sony televisions into 3D, and the same file with the SEI removed (791 bytes, picture data untouched) plays flat. MP4 files from common encoders carry the frame-packing layout only in the SEI: x264 and FFmpeg write it there (-x264-params frame-packing=…), and HandBrake does since HandBrake#8100. Measurement method and files: 3d-signalling-ecosystem.md.

Sources

• SEI syntax: ITU-T H.264 and H.265, Annex D, frame_packing_arrangement (payloadType 45), as parsed by 58f396c.
• Matroska StereoMode: RFC 9559, section 5.1.4.1.28.3, table 5 (0x53B8; 1 and 11 side by side, 3 and 2 top-bottom, left and right eye first).
• Eye order: content_interpretation_type semantics in the same Annex D clause (1: frame 0 is the left view; 2: frame 0 is the right view).

Tests

• Mp4ExtractorParameterizedTest (sample_h264_frame_packing_side_by_side.mp4, sample_h264_frame_packing_top_bottom.mp4) and FragmentedMp4ExtractorParameterizedTest (sample_fragmented_h264_frame_packing_side_by_side.mp4), with setDeduplicateConsecutiveFormats(true) for the refined format.
• DumpableFormat doesn't print stereoMode, so Mp4ExtractorNonParameterizedTest and FragmentedMp4ExtractorNonParameterizedTest assert it directly (side by side, top-bottom, and an H.264 file without the SEI staying unset).
• NalUnitUtilTest: content_interpretation_type 2 for side by side and top-bottom, and 1 staying left eye first.
• WebmMuxerEndToEndTest: all four layouts muxed and read back with MatroskaExtractor.
• The existing dump files are unchanged; lib-common, lib-container, lib-extractor and lib-muxer unit tests pass.

The three assets (about 8 KB each, 128x64, 5 frames) were made with FFmpeg 9.0.1 and libx264, for example:

ffmpeg -f lavfi -i testsrc2=size=128x64:rate=30 -frames:v 5 -c:v libx264 -preset ultrafast \
  -pix_fmt yuv420p -x264-params frame-packing=3 -movflags +faststart \
  sample_h264_frame_packing_side_by_side.mp4

(frame-packing=4 for top-bottom; -movflags frag_keyframe+empty_moov+default_base_moof for the fragmented file.)

Prepared with AI assistance (Claude), directed, reviewed and tested by me.