Repository navigation
Significant frame drop while playing 1920x1080 HLS video tracks on AFTSSS and AFTSS #9565
Description
[REQUIRED] Issue description
Fire TV Stick - 3rd Gen (2020) (AFTSSS) and Fire TV Stick Lite - 1st Gen (2020) (AFTSS) consistently drop frames while playing 1920x1080 video tracks in HLS streams.
This bug report is a copy of a bug report I created on the Amazon ExoPlayer port repo: amzn/exoplayer-amazon-port#115
I haven't received any response on that repo, so I am re-posting this here.
Our production app uses the ExoPlayer Amazon port, but the issue occurs in both the ExoPlayer and the ExoPlayer Amazon port. This issue started when we upgraded our ExoPlayer Amazon port version from v2.10.6 to v2.11.3. So, I tested a few different ExoPlayer versions and confirmed the issue occurs on the following versions:
- v2.11.3 (ExoPlayer and ExoPlayer Amazon Port)
- v2.12.1 (ExoPlayer and ExoPlayer Amazon Port)
- v2.15.1 (ExoPlayer; there is no Amazon port for this version yet)
After some debugging, I have some insight as to why this started in v2.11:
In both v2.10 and v2.11, ExoPlayer's MediaCodecVideoRenderer.getDecoderInfos() method call returns a list of decoders available on the device. On v2.10, the order of the codecs was always returned in the following order, on both of these Fire Stick devices:
[OMX.MTK.VIDEO.DECODER.AVC, OMX.google.h264.decoder]
So, to decode our HLS stream, the ExoPlayer would choose the first decoder in the list, OMX.MTK.VIDEO.DECODER.AVC (the hardware decoder). When the ExoPlayer checks the Format of our 1920x1080 video track in the HLS stream against this hardware codec, it returns that the hardware codec does not support the 1920x1080 video track. So, it would never attempt to play the 1920x1080 video track.
Starting in ExoPlayer v2.11, the ExoPlayer's MediaCodecVideoRenderer.getDecoderInfos() implementation changed. This method started sorting the list of decoders by 'Format' support.
The sorting recognizes that the hardware decoder (OMX.MTK.VIDEO.DECODER.AVC) does not support the 1920x1080 video track in our HLS stream, but the software decoder (OMX.google.h264.decoder) does, so the sorting method call sorts the list of available decoders into the following order: [OMX.google.h264.decoder, OMX.MTK.VIDEO.DECODER.AVC]. The ExoPlayer then chooses the first decoder in the list, the software decoder, to decode the 1920x1080 video track. However, during playback, there is nearly constant frame drop causing a bad experience for the user.
I don't know what's causing the software codec to perform so poorly, but there are logs indicating frames being dropped:
2021-05-05 16:25:59.404 319-319/? E/FrameEvents: updateAcquireFence: Did not find frame.
2021-05-05 16:25:59.436 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 2, PTS = 800000
2021-05-05 16:25:59.478 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 3, PTS = 840000
2021-05-05 16:25:59.486 7423-7439/? I/exoplayer2.dem: Background concurrent copying GC freed 12043(757KB) AllocSpace objects, 4(272KB) LOS objects, 43% free, 7MB/13MB, paused 897us total 147.805ms
2021-05-05 16:25:59.521 319-319/? E/FrameEvents: updateAcquireFence: Did not find frame.
2021-05-05 16:25:59.571 319-319/? E/FrameEvents: updateAcquireFence: Did not find frame.
2021-05-05 16:25:59.584 513-601/? W/BestClock: java.time.DateTimeException: Missing NTP fix
2021-05-05 16:25:59.592 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 6, PTS = 960000
2021-05-05 16:25:59.602 315-496/? D/AudioFlinger: mixer(0xa5183780) throttle end: throttle time(3)
2021-05-05 16:25:59.638 319-319/? E/FrameEvents: updateAcquireFence: Did not find frame.
2021-05-05 16:25:59.644 513-601/? W/BestClock: java.time.DateTimeException: Missing NTP fix
2021-05-05 16:25:59.647 513-602/? W/BestClock: java.time.DateTimeException: Missing NTP fix
2021-05-05 16:25:59.655 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 7, PTS = 1000000
2021-05-05 16:25:59.704 319-319/? E/FrameEvents: updateAcquireFence: Did not find frame.
2021-05-05 16:25:59.708 315-496/? D/AudioFlinger: mixer(0xa5183780) throttle end: throttle time(2)
2021-05-05 16:25:59.745 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 1, PTS = 1080000
2021-05-05 16:25:59.785 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 2, PTS = 1120000
2021-05-05 16:25:59.836 315-496/? D/AudioFlinger: mixer(0xa5183780) throttle end: throttle time(3)
2021-05-05 16:25:59.838 319-319/? E/FrameEvents: updateAcquireFence: Did not find frame.
2021-05-05 16:25:59.879 315-496/? D/AudioFlinger: mixer(0xa5183780) throttle end: throttle time(3)
2021-05-05 16:25:59.888 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 4, PTS = 1200000
2021-05-05 16:25:59.919 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 5, PTS = 1240000
2021-05-05 16:25:59.971 319-319/? E/FrameEvents: updateAcquireFence: Did not find frame.
2021-05-05 16:26:00.032 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 7, PTS = 1320000
2021-05-05 16:26:00.072 319-319/? E/FrameEvents: updateAcquireFence: Did not find frame.
2021-05-05 16:26:00.082 513-601/? W/BestClock: java.time.DateTimeException: Missing NTP fix
2021-05-05 16:26:00.092 315-496/? D/AudioFlinger: mixer(0xa5183780) throttle end: throttle time(3)
2021-05-05 16:26:00.108 7423-7618/? I/CodecNameUnknown-MediaCodecVideoRenderer: dropOutputBuffer: bufferIndex = 1, PTS = 1400000
I also don't know why these two Fire Sticks are reporting that their hardware codec does not support the 1920x1080 video track. Seemingly, the same OMX.MTK.VIDEO.DECODER.AVC hardware codec can play 1920x1080 video tracks on the Fire TV Stick - 2nd Gen (2016-2019) (AFTT), but not on the newer models.
Finally, this may be related to amzn/exoplayer-amazon-port#42 but I'm not sure.
[REQUIRED] Reproduction steps
This does not reproduce with the Apple HLS samples in the demo app. However, you can reproduce it in the demo app using this sample I found on Bitmovin's website: https://bitdash-a.akamaihd.net/content/MI201109210084_1/m3u8s/f08e80da-bf1d-4e3d-8899-f0f6155f6efa.m3u8
Notice the choppiness of the video and the logs being printed. If the issue does not reproduce the first time, try:
- Backing out of the video, then starting it again.
- Manually selecting the 1920x1080 video track.
[REQUIRED] Link to test content
I can provide our production HLS streams if more than this sample is needed.
[REQUIRED] A full bug report captured from the device
Fire TV Stick Lite - 1st Gen (2020) (AFTSS):
bugreport-sheldon-PS7234-2021-05-05-16-39-26.zip
[REQUIRED] Version of ExoPlayer being used
- v2.11.3 (ExoPlayer and ExoPlayer Amazon Port)
- v2.12.1 (ExoPlayer and ExoPlayer Amazon Port)
- v2.15.1 (ExoPlayer)
It seems to reproduce in all ExoPlayer versions (Amazon port or not) starting with v2.11.
[REQUIRED] Device(s) and version(s) of Android being used
- Fire TV Stick - 3rd Gen (2020) (AFTSSS) - Fire OS 7 (Android API 28)
- Fire TV Stick Lite - 1st Gen (2020) (AFTSS) - Fire OS 7 (Android API 28)
https://developer.amazon.com/docs/fire-tv/fire-os-overview.html
Activity
claincly commentedon Oct 15, 2021
The bug report you linked to doesn't seem to contain enough information about the codec setup.
I'm surprised 1080p AVC is reported to be not supported by the decoder, but ExoPlayer is not able to change that. Furthermore, I wouldn't be surprised that a software decoder performs poorly on a fire stick.
ojw28 commentedon Oct 18, 2021
If you look at the
logcatoutput at the start of playback, you should see some logging that uses tagMediaCodecInfoand explains why ExoPlayer thinks that the hardware codec does not support playback. As @claincly notes, the bug report doesn't seem to include this, so it was likely not captured soon enough after the start of playback.Please could you try reproducing again using v2.15.1, and capturing the
logcatthat's output at the start of playback, which should include theMediaCodecInfolog lines (and other useful information).Koster35 commentedon Oct 18, 2021
Sorry about that, here is another bug report using the v2.15.1 demo app:
bugreport-sheldon-PS7242-2021-10-18-15-40-47.zip
Within it, I found one log with the
MediaCodecInfotag:And here is one more run just capturing logcat output:
AFTSS_logcat.txt
claincly commentedon Oct 25, 2021
Thanks for providing the detail.
ExoPlayer queries
android.media.MediaCodecInfofor whether a format is supported by a specific decoder. If the platform reports that the format is not supported, ExoPlayer will not try to play with that decoder.As such, we cannot provide further help if the platform reports it does not support hardware decoding AVC.
ojw28 commentedon Oct 25, 2021
If the device claims to not support a profile/level that it does actually support, then we should probably consider adding a device specific workaround for that. Particularly given it's effectively a regression, presumably introduced when we started to check the profile/level.
10 remaining items
gregni2 commentedon Dec 2, 2021
Wow. Great details. Your fix sounds perfect for us at least. Those steps would fix our current issue very well.
One more question though. I was impressed to see the adaptive algorithm (TrackSelection I think?) measure the bitrate and basically say 'we can handle more' and then to go to the higher resolution/bitrate/profile level. The player seems great about going up. But when the stuttering occurred I saw logging from the exoplayer that said it was dropping packets. But I never saw the adaptive algorithm go back down to a lower resolutions/bitrates.
Was there a reason that it didn't step down to a lower resolution once it realized it was dropping packets? It looked to me like the code was setup to step down as well.
I promise that's my last question. :) You folks have been awesome and playing with the exoplayer has given me a real appreciation for the great work you folks are doing.
ojw28 commentedon Dec 2, 2021
Adaptive bitrate is based on available network bandwidth. In this case there is sufficient network bandwidth, and it's CPU that's the limiting factor, so the logic to step down wont be triggered.
We've discussed looking at additional metrics such as how frequently frames are being dropped (many frame drops is indicative of being CPU bound), and even things like the thermal state of the device, and factoring those into the adaptive bitrate logic as well. It's something we should look at at some point, and is particularly useful for devices that only ship with software decoders. In the case discussed here though, it's better just to avoid switching up to the 1080p stream in the first place :).
Don't sort decoders by format support in supportsFormat
Add capability flags for hardware and decoder support
Don't sort decoders by format support in supportsFormat
Add capability flags for hardware and decoder support
Rollback of 0aa23b0
Rollback of bf12241
tidoemanuele commentedon Dec 20, 2021
Same problem here with LENOVO TB-X306X running the latest exoplayer version (2.16.1).
Here how I'm temporary solving it:
For all devices running Android 10 or above I use the VideoCapabilities to understand if the device is able to play the highest quality track. If not I limit the track list by framerate or bitrate for all the play sessions.
For all devices running Android 9 and below I'm looking to the Video Frame Processing Offset values during the play session and decide to limit the track list by framerate or bitrate for the next play sessions.
I'm not satisfied with my approach for devices running android 9 or below, because I need a training period to decide if the device can play smoothly or not. Waiting for a better solution or some advice.
Rollback of 3c4c1f4
Rollback of f713ca6
Limit adaptive selections to same level of decoder support
Limit adaptive selections to same level of decoder support