Matroska: mark H.264/H.265 samples not depended on, to speed up seeking #3437

[SUGGESTION] Use case description

We play local-network MKV files (H.264 High, muxed with mkvmerge) with Media3 1.11.1 on a Lenovo TB336ZU tablet (MediaTek Dimensity 6300, Android 16). An exact seek costs about 100 ms plus 5.1 ms for every frame decoded from the previous keyframe up to the target (r = 0.99 over nine in-buffer seeks, measured with DecoderCounters deltas between the seek and the first rendered frame). With keyframes 4–10 s apart, a +15 s skip regularly took 500–600 ms, and time-bar seeks up to about 1.2 s (122–146 frames decoded before the target).

Media3 already speeds up this catch-up for MP4. In 1.11.1, Mp4Extractor and FragmentedMp4Extractor mark H.264/H.265 samples that nothing depends on with C.BUFFER_FLAG_NOT_DEPENDED_ON (experimentalSetCodecsToParseWithinGopSampleDependencies, default VIDEO_CODEC_FLAG_H264 | VIDEO_CODEC_FLAG_H265). MediaCodecVideoRenderer.shouldDiscardDecoderInputBuffer then skips them while catching up to the seek target. AV1 gets the same benefit in any container through Av1SampleDependencyParser. MatroskaExtractor never sets the flag, so for H.264/H.265 in MKV every frame between the keyframe and the target is decoded. In a 12 s sample of one of our files, about half of the slice NAL units had nal_ref_idc == 0, so roughly half of that catch-up could be skipped.

Proposed solution

Parse within-GOP sample dependencies in MatroskaExtractor for H.264/H.265 video, as the MP4 extractors do. For example:

• honour DefaultExtractorsFactory.experimentalSetCodecsToParseWithinGopSampleDependencies through a new Matroska flag;
• check the first NAL unit header of each length-prefixed sample with NalUnitUtil.isDependedOn;
• set C.BUFFER_FLAG_NOT_DEPENDED_ON on the sample.

MatroskaExtractor already reads the NAL length prefixes of AVC/HEVC samples (nalUnitLengthFieldLength) when it rewrites them to start codes, so the header byte is at hand.

Alternatives considered

• SeekParameters that snap to a keyframe. We now use this for relative skips (the approach mpv takes by default). It gives up accuracy, so it doesn't fit time-bar seeks.
• DefaultRenderersFactory.experimentalSetEnableMediaCodecBufferDecodeOnlyFlag(true). On this MediaTek decoder it made no measurable difference to exact seeks: 1135 vs 1206 ms for 158 frames decoded before the target.
• Scrubbing mode. It suits dragging, not single seeks during playback, because it suppresses playback and audio.
• Remuxing the files to MP4. That would work, but it isn't practical for an existing library.

We haven't measured MP4 remuxes of the same files, so the size of the gain on this device is an estimate.

🤖 Written by Claude Code (Claude Opus 5.5) and filed through the owner's GitHub account.