[Feature request] Support for MP4 track header's alternate_group

Use case description

We have a single mp4 containing groups of audio tracks. One group represents voice in different languages and the other group represents background audio in stereo or multi channel format. Each group has a corresponding alternate_group in the track header atom.

# Movie Info - 4 tracks - TimeScale 1000
Duration 00:04:55.680
Fragmented: no
Major Brand isom - version 512 - compatible brands: isom dby1 iso2 mp41
Created: UNKNOWN DATE

Meta-Data Tags:
        ...

# Track 1 Info - ID 1 - TimeScale 44100
Media Duration 00:03:35.813
Media Language: English (eng)
<b>Alternate Group ID 1
...

# Track 2 Info - ID 2 - TimeScale 44100
Media Duration 00:03:28.013
Media Language: Spanish; Castilian (spa)
Alternate Group ID 1
...

# Track 3 Info - ID 3 - TimeScale 48000
Media Duration 00:04:55.680
Alternate Group ID 2
Media Type: soun:ec-3
        EC-3 stream - Sample Rate 48000 - 5.1 channel(s) - bitrate 768000
        ...

# Track 4 Info - ID 4 - TimeScale 44100
Media Duration 00:04:12.368
Alternate Group ID 2
Media Type: soun:mp4a
        MPEG-1 Audio - 2 Channel(s) - SampleRate 44100 - Layer 3
        ...

Proposed solution

Request is for BoxParser.parseTkhd to extract alternate_group and save it in TkhdData. This can eventually be used by ProgressiveMediaPeriod to create TrackGroup corresponding to each logical group. Considering Mp4 parsing and usage of track group as an existing functionality, new request could be made as an opt-in.

private static final class TkhdData {

    private final int id;
    private final long duration;
    private final int rotationDegrees;
    private final int alternateGroup;
    
  }

Alternatives considered

N/A

FYI. Here is the doc for alternate_group. It offers grouping of tracks based on language (even though the actual content would be different)

However, androidx.media3.common.TrackGroup documentation claims otherwise

 * <p>Note that audio tracks whose languages differ are not grouped, because content in different
 * languages is not considered to be the same. Conversely, audio tracks in the same language that
 * only differ in properties such as bitrate, sampling rate, channel count and so on can be grouped.
 * This also applies to text tracks.

@rohitjoins

However, androidx.media3.common.TrackGroup documentation claims otherwise

This is the key point. I think this means that TrackGroup is not an appropriate representation for the MP4 grouping implied by alternate_group - because it will not interact correctly with all of ExoPlayer's track selection logic (which assumes that language of every track in a TrackGroup matches, and they only vary in "non content" ways like codec, bitrate, resolution, etc).

~~I'm a bit confused by the last sentence in the MP4 spec though:~~

~~A group may have only one member.~~

This makes it sound like it's not actually permitted to have multiple tracks with the same alternate_group value? Which seems surprising, because it seems the rest of the description it discussion groups/collections of tracks that do have the same alternate_group value.

@rohitjoins has pointed out the interpretation of this sentence is presumably meant to be "a group is permitted to have only one member (but is also permitted to have more than one)", rather than my interpretation of "a group must only have one member" - which makes sense.

Do you have an example file with the grouping structure you are trying to play? Please either upload it here or send it to android-media-github@google.com with the subject Issue #2242. Please also update this issue to indicate you’ve done this.

It would also be helpful if you could describe why you want these represented in the same TrackGroup - what playback/track selection behaviour of ExoPlayer are you trying to configure? We might be able to guide you towards achieving that without grouping these into a TrackGroup (see x-y problem)

This is the key point. I think this means that TrackGroup is not an appropriate representation for the MP4 grouping implied by alternate_group - because it will not interact correctly with all of ExoPlayer's track selection logic (which assumes that language of every track in a TrackGroup matches, and they only vary in "non content" ways like codec, bitrate, resolution, etc).

Sure, from TrackGroup's doc it would seem to apply to language based grouping but doc also claims tracks in same language but differing in channel count can be grouped. So an MP4 with two audio tracks (stereo and multi channel) could still use alternate_group and in this case TrackGroup still wouldn't group those tracks.

Conversely, audio tracks in the same language that only differ in properties such as bitrate, sampling rate, channel count and so on can be grouped.

It would also be helpful if you could describe why you want these represented in the same TrackGroup - what playback/track selection behaviour of ExoPlayer are you trying to configure?

Final objective is to be able to use a single ExoPlayer to play two audio tracks (one based on language and other based on channel count). Hope is also to leverage the existing logic of DefaultTrackSelector as much as possible without having to reinvent the wheel when using custom track selector (like track selection based on language score and track selection based on platform support for channels)

Final objective is to be able to use a single ExoPlayer to play two audio tracks (one based on language and other based on channel count).

Can you add a bit more detail here - are you trying to play both audio tracks at the same time? Can you add some detail on how the TrackGroup structure you are proposing will help with that?

TrackGroup is designed for grouping tracks that can be automatically/adaptively switched between. Examples of this are bitrate switching when streaming, or audio channel-count switching when different output devices are connected (e.g. spatial audio capable headset is connected -> switch from stereo to 5.1 audio). This is why the expectation is that the content of every track in the group has matching content.

Can you add a bit more detail here - are you trying to play both audio tracks at the same time? Can you add some detail on how the TrackGroup structure you are proposing will help with that?

Yes, one from each group at the same time (via single ExoPlayer instance). With grouping, the idea was to pick desired ones from each group via custom track selector. (TrackGroupArray having couple of groups). This is inspired from google/ExoPlayer#10050. However, it uses Unstable APIs and doesn't leverage default track selection logic (like language score based selection etc), default video track selection etc.

public class CustomTrackSelector extends TrackSelector {

  @Override
  public TrackSelectorResult selectTracks(
      RendererCapabilities[] rendererCapabilities,
      TrackGroupArray trackGroups,
      MediaSource.MediaPeriodId periodId,
      Timeline timeline
  ) throws ExoPlaybackException {
    Queue<Integer> audioRenderers = new ArrayDeque<>();
    RendererConfiguration[] configs = new RendererConfiguration[rendererCapabilities.length];
    ExoTrackSelection[] selections = new ExoTrackSelection[rendererCapabilities.length];

    for (int i = 0; i < rendererCapabilities.length; i++) {
      if (rendererCapabilities[i].getTrackType() == C.TRACK_TYPE_AUDIO) {
        audioRenderers.add(i);
        configs[i] = RendererConfiguration.DEFAULT;
      }
    }
    for (int i = 0; i < trackGroups.length; i++) {
      if (MimeTypes.isAudio(trackGroups.get(i).getFormat(0).sampleMimeType)) {
        Integer index = audioRenderers.poll();
        if (index != null) {
          selections[index] = new FixedTrackSelection(trackGroups.get(i), 0);
        }
      }
    }

    return new TrackSelectorResult(configs, selections, Tracks.EMPTY, new Object());
  }

  @Override
  public void onSelectionActivated(@Nullable Object info) {

  }
}

Examples of this are bitrate switching when streaming, or audio channel-count switching when different output devices are connected (e.g. spatial audio capable headset is connected -> switch from stereo to 5.1 audio). This is why the expectation is that the content of every track in the group has matching content.

So would that make a MP4 container with stereo and atmos track to be a valid candidate for grouping?

I discussed this with others on my team, we concluded:

TrackGroup exists to support seamless adaptive bitrate switching. This isn't relevant for playing tracks from the same MP4 container. Therefore we don't plan to group even 'compatible' MP4 alternate_group tracks into a TrackGroup.
We agree that alternate_group is currently not parsed at all by Mp4Extractor. We plan to parse this and expose it as a custom entry in Format.metadata, similar to the existing Mp4TimestampData.
We are not sure we understand your track selection implementation proposal. In particular, we think that if you want to map to two audio renderers (without clear 'format support' differences between them) then you will need to extend TrackSelector and not MappingTrackSelector and DefaultTrackSelector, because you need to write the mapping logic yourself.
- This is all quite custom, and I'm afraid we are not able to offer high touch support (but are happy to answer specific questions).
- You should expect to definitely need to use @UnstableApi annotated symbols. It will not be possible to implement what you want without this.
- We think you may be able to use two DefaultTrackSelector instances from inside your TrackSelector subclass, to help you score each 'group' of tracks. You will need to group the tracks yourself by inspecting the Format.metadata entry described in (2) above. We haven't tested this.

It would be good to update TrackGroup's doc to be explicit about just bitrate switching intent. Out of curiosity, could you point me to the code which has this restriction?
Yes, additional group info in Format.metadata would help in track selection.
Thanks for the hat tip, will explore using DefaultTrackSelector instance in custom TrackSelector.

It would be good to update TrackGroup's doc to be explicit about just bitrate switching intent.

This is kind of already covered in the second paragraph. It is also more a property of how AdaptiveTrackSelection (which is used by DefaultTrackSelector) uses TrackGroup.

Yes, additional group info in Format.metadata would help in track selection.

This change has been sent for internal review.

MP4: Parse `alternate_group` and expose it in `Format.metadata`

The parsing of alternate_group is implemented by the commit linked above.

MP4: Parse `alternate_group` and expose it in `Format.metadata`

Navigation Menu

Repository navigation

[Feature request] Support for MP4 track header's alternate_group #2242

Use case description

Proposed solution

Alternatives considered

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Feature request] Support for MP4 track header's alternate_group #2242

Description

Use case description

Proposed solution

Alternatives considered

Activity

Ethan1983 commented on Mar 19, 2025

icbaker commented on Mar 19, 2025

Ethan1983 commented on Mar 20, 2025

icbaker commented on Mar 21, 2025

Ethan1983 commented on Mar 22, 2025

icbaker commented on Mar 25, 2025

Ethan1983 commented on Mar 26, 2025

icbaker commented on Mar 26, 2025

icbaker commented on Mar 26, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions