<div dir="ltr">I've done some time measurements to try to identify possible bottlenecks.<div><br></div><div>The av_hwframe_transfer_data implementation basically consists of 2 steps: vaapi_transfer_data_from, then av_copy_frame. On average, about 2/3 to 3/4 of the time is spent in vaapi_transfer_data_from.</div><div><br></div><div>Then vaapi_transfer_data_from consists of about4 steps: vaSyncSurface, vaCreateImage, vaGetImage and vaMapBuffer. The average times spent on each steps are as follows, when decoding UHD stream (<a href="https://storage.googleapis.com/wvmedia/clear/h264/tears/tears_uhd.mpd">https://storage.googleapis.com/wvmedia/clear/h264/tears/tears_uhd.mpd</a>') on my Baytrail platform:</div><div><br></div><div>vaSyncSurface: 7ms</div><div>vaCreateImage: 0.05ms</div><div>vaGetImage: 4ms</div><div>vaMapBuffer: 14ms</div><div><br></div><div>These are average numbers and some steps have large deviations. But one that is consistently long is vaMapBuffer.</div><div><br></div><div>I've then done some timing inside intel-vaapi-driver for the buffer mapping step. On my platform, this mapping basically consists of i965 driver calling dri_bo_map, which translates into drm_intel_gem_bo_map (in libdrm_intel). And within that function, most of the time (like 99%) is spent on drmIoctl(DRM_IOCTL_I915_GEM_SET_DOMAIN).</div><div><br></div><div>The vaSyncSurface also introduces quite a significant latency. After some Googling, I found this reference [1], which states that on Intel platform vaSyncSurface is unnecessary if it's followed by a vaMapBuffer. However this is in the context of encoding, so this might not really apply to me use case. Nevertheless, I've tried to remove the vaSyncSurface step from FFMPEG, and it did decrease the overall latency, without any visible effect on screen. Although it doesn't feel like the right to do.</div><div><br></div><div>What I've also tried is to map the hwframe directly with AV_PIX_FMT_DRM_PRIME, which is fast. But the content is Y-tiled, so that's not really helping if I need to do the detiling in software.</div><div><br></div><div>I'm not entirely sure where to go from there, I would appreciate some feedback or hints from the devs. I'm also not sure whether this is the right place for that kind of things, let me know if there's a better place for that type of support.</div><div><br></div><div>Thanks,</div><div>Michael.</div><div><br></div><div>[1] <a href="https://chromium.googlesource.com/chromium/src/+/main/media/gpu/vaapi/vaapi_wrapper.cc#2828">https://chromium.googlesource.com/chromium/src/+/main/media/gpu/vaapi/vaapi_wrapper.cc#2828</a></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Oct 6, 2022 at 10:20 PM Michael Goffioul <<a href="mailto:michael.goffioul@gmail.com">michael.goffioul@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">HI,<div><br></div><div>I'm using FFMPEG (5.1.2) on an Android (x86) platform running on an Intel Baytrail chipset. Video decoding uses VA-API for mpeg2 and h264. The implementation follows the same pattern as the hw_decode.c example.</div><div><br></div><div>Everything works fine, but when timing the various parts of the decoding loop (send packet, receive frame, hw-transfer, scale/pixfmt conversion), it appears that most of the time is spent in av_hwframe_transfer_data. E.g. these are typical numbers for various resolution:</div><div>- 1280x720: ~10ms</div><div>- 1920x1080: ~20ms</div><div>- 3840x1714: ~40ms</div><div><br></div><div>My question is then whether there's anything I do or look into in order to reduce that latency? Is it at all possible, or is it just the way it is and I can't squeeze more performance out of that platform?</div><div><br></div><div>Not sure whether it's helpful, but here's some VA-API related log output from FFMPEG:</div><div>10-06 22:18:00.162 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Opened VA display via Android device android.<br>10-06 22:18:00.162 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] libva: VA-API version 1.4.0<br>10-06 22:18:00.162 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] libva: va_getDriverName() returns 0<br>10-06 22:18:00.162 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] libva: Trying to open /vendor/lib64/dri/i965_drv_video.so<br>10-06 22:18:00.163 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] libva: Found init function __vaDriverInit_1_4<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] libva: va_openDriver() returns 0<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Initialised VAAPI connection: version 1.4<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Format 0x32315659 -> yuv420p.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Format 0x30323449 -> yuv420p.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Format 0x3231564e -> nv12.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Format 0x32595559 -> yuyv422.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Format 0x59565955 -> uyvy422.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Format 0x48323234 -> yuv422p.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Format 0x58424752 -> rgb0.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Format 0x58524742 -> bgr0.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Format 0x30313050 -> p010le.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] VAAPI driver: Intel i965 driver for Intel(R) Bay Trail - 2.4.0.pre1 (a9d2c1f).<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [AVHWDeviceContext @ 0x760040312900] Driver not found in known nonstandard list, using standard behaviour.<br>10-06 22:18:00.165 3805 3853 I HWACCEL : hw codec h264 enabled: s=0x7600e0316700 pix_fmts=44<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [h264 @ 0x7600e0316700] Format vaapi chosen by get_format().<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [h264 @ 0x7600e0316700] Format vaapi requires hwaccel initialisation.<br>10-06 22:18:00.165 3805 3853 I FFMPEG : [h264 @ 0x7600e0316700] Considering format 0x3231564e -> nv12.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [h264 @ 0x7600e0316700] Picked nv12 (0x3231564e) as best match for yuv420p.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [AVHWFramesContext @ 0x760060312bc0] Created surface 0x4000000.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [AVHWFramesContext @ 0x760060312bc0] Direct mapping disabled: derived image format 3231564e does not match expected format 32315659.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [AVHWFramesContext @ 0x760060312bc0] Created surface 0x4000001.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [AVHWFramesContext @ 0x760060312bc0] Created surface 0x4000002.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [AVHWFramesContext @ 0x760060312bc0] Created surface 0x4000003.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [AVHWFramesContext @ 0x760060312bc0] Created surface 0x4000004.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [AVHWFramesContext @ 0x760060312bc0] Created surface 0x4000005.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [AVHWFramesContext @ 0x760060312bc0] Created surface 0x4000006.<br>10-06 22:18:00.166 3805 3853 I FFMPEG : [AVHWFramesContext @ 0x760060312bc0] Created surface 0x4000007.<br></div><div><br></div><div>Thanks,</div><div>Michael.</div><div><br></div></div>
</blockquote></div>