<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body style="overflow-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;">I have a fork of ffmpeg 5.4 where I implemented the scale_vt filte from ffmpeg 6r:<div><div style="display: block;"><div style="-webkit-user-select: all; -webkit-user-drag: element; display: inline-block;" class="apple-rich-link" draggable="true" role="link" data-url="https://pastebin.com/ZCe9T0Zz"><a style="border-radius:10px;font-family:-apple-system, Helvetica, Arial, sans-serif;display:block;-webkit-user-select:none;width:300px;user-select:none;-webkit-user-modify:read-only;user-modify:read-only;overflow:hidden;text-decoration:none;" class="lp-rich-link" rel="nofollow" href="https://pastebin.com/ZCe9T0Zz" dir="ltr" role="button" draggable="false" width="300"><table style="table-layout:fixed;border-collapse:collapse;width:300px;background-color:#E5E6E9;font-family:-apple-system, Helvetica, Arial, sans-serif;" class="lp-rich-link-emailBaseTable" cellpadding="0" cellspacing="0" border="0" width="300"><tbody><tr><td vertical-align="center"><table bgcolor="#E5E6E9" cellpadding="0" cellspacing="0" width="300" style="font-family:-apple-system, Helvetica, Arial, sans-serif;table-layout:fixed;background-color:rgba(229, 230, 233, 1);" class="lp-rich-link-captionBar"><tbody><tr><td style="padding:8px 0px 8px 0px;" class="lp-rich-link-captionBar-textStackItem"><div style="max-width:100%;margin:0px 16px 0px 16px;overflow:hidden;" class="lp-rich-link-captionBar-textStack"><div style="word-wrap:break-word;font-weight:500;font-size:12px;overflow:hidden;text-overflow:ellipsis;text-align:left;" class="lp-rich-link-captionBar-textStack-topCaption-leading"><a rel="nofollow" href="https://pastebin.com/ZCe9T0Zz" style="text-decoration: none" draggable="false"><font color="#272727" style="color: rgba(0, 0, 0, 0.847059);">/* * Copyright (c) 2023 Zhao Zhili <zhilizhao@tencent.com> * * This file i</font></a></div><div style="word-wrap:break-word;font-weight:400;font-size:11px;overflow:hidden;text-overflow:ellipsis;text-align:left;" class="lp-rich-link-captionBar-textStack-bottomCaption-leading"><a rel="nofollow" href="https://pastebin.com/ZCe9T0Zz" style="text-decoration: none" draggable="false"><font color="#808080" style="color: rgba(0, 0, 0, 0.498039);">pastebin.com</font></a></div></div></td><td style="padding:6px 12px 6px 0px;" class="lp-rich-link-captionBar-rightIconItem" width="36"><a rel="nofollow" href="https://pastebin.com/ZCe9T0Zz" draggable="false"><img style="pointer-events:none !important;display:inline-block;width:36px;height:36px;border-radius:3px;" width="36" height="36" draggable="false" class="lp-rich-link-captionBar-rightIcon" alt="favicon.ico" src="cid:345D288B-2F82-4F34-A4C3-044A0C408603"></a></td></tr></tbody></table></td></tr></tbody></table></a></div></div>I am trying to pass 4k frames using gpu (videotoolbox) to this scale filter to be able to use them later.</div><div>However I am noticing that my CPU usage is very high so the frames are probably downloaded to the cpu and then reuploaded again.</div><div>What is wrong with my current implementation?</div><div>codec_context is the decoder codec context wich is already opened, however it seems like no hw_device_ctx is assigned to it so I had to allocate it myself.</div><div>I had to use <span style="caret-color: rgb(78, 201, 176); color: rgb(78, 201, 176); font-family: Menlo, Monaco, "Courier New", monospace; white-space: pre; background-color: rgb(31, 31, 31);">AVBufferSrcParameters </span> because the scale_vt filter would crash on configuration if the input link didn’t have a hw_frames_ctx.</div><div><br></div><div><div style="color: rgb(204, 204, 204); background-color: rgb(31, 31, 31); font-family: Menlo, Monaco, "Courier New", monospace; line-height: 18px; white-space: pre;"><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(212, 212, 212);">!</span><span style="color: rgb(156, 220, 254);">use_hw_resize_</span>)</div><div> {</div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><div> <span style="color: rgb(78, 201, 176);">AVBufferRef</span> <span style="color: rgb(212, 212, 212);">*</span><span style="color: rgb(156, 220, 254);">hw_frames_ref</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">av_hwframe_ctx_alloc</span>(<span style="color: rgb(156, 220, 254);">codec_context_</span>-><span style="color: rgb(156, 220, 254);">hw_device_ctx</span>);</div><div> <span style="color: rgb(156, 220, 254);">codec_context_</span>-><span style="color: rgb(156, 220, 254);">hw_frames_ctx</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(156, 220, 254);">hw_frames_ref</span>;</div><div> <span style="color: rgb(78, 201, 176);">AVPixelFormat</span> <span style="color: rgb(156, 220, 254);">input_format</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(79, 193, 255);">AV_PIX_FMT_CUDA</span>;</div><div><span style="color: rgb(106, 153, 85);"> // Create new graph</span></div><div> <span style="color: rgb(156, 220, 254);">filter_graph_</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_graph_alloc</span>();</div><div> <span style="color: rgb(78, 201, 176);">AVFilterContext</span> <span style="color: rgb(212, 212, 212);">*</span><span style="color: rgb(156, 220, 254);">filter_ctx</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(86, 156, 214);">nullptr</span>;</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(86, 156, 214);">this</span>-><span style="color: rgb(156, 220, 254);">hw_type_</span> <span style="color: rgb(212, 212, 212);">==</span> <span style="color: rgb(79, 193, 255);">AV_HWDEVICE_TYPE_VIDEOTOOLBOX</span>)</div><div> {</div><div> <span style="color: rgb(86, 156, 214);">const</span> <span style="color: rgb(78, 201, 176);">AVFilter</span> <span style="color: rgb(212, 212, 212);">*</span><span style="color: rgb(156, 220, 254);">filter</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_get_by_name</span>(<span style="color: rgb(206, 145, 120);">"scale_vt"</span>);</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(156, 220, 254);">filter</span>)</div><div> {</div><div> <span style="color: rgb(220, 220, 170);">snprintf</span>(<span style="color: rgb(156, 220, 254);">strbuf_</span>, <span style="color: rgb(86, 156, 214);">sizeof</span>(<span style="color: rgb(156, 220, 254);">strbuf_</span>), <span style="color: rgb(206, 145, 120);">"w=</span><span style="color: rgb(156, 220, 254);">%d</span><span style="color: rgb(206, 145, 120);">:h=</span><span style="color: rgb(156, 220, 254);">%d</span><span style="color: rgb(206, 145, 120);">"</span>,</div><div> <span style="color: rgb(156, 220, 254);">output_hw_width_</span>, <span style="color: rgb(156, 220, 254);">output_hw_height_</span>);</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(220, 220, 170);">avfilter_graph_create_filter</span>(<span style="color: rgb(212, 212, 212);">&</span><span style="color: rgb(156, 220, 254);">filter_ctx</span>, <span style="color: rgb(156, 220, 254);">filter</span>, <span style="color: rgb(86, 156, 214);">NULL</span>,</div><div> <span style="color: rgb(156, 220, 254);">strbuf_</span>, <span style="color: rgb(86, 156, 214);">NULL</span>, <span style="color: rgb(156, 220, 254);">filter_graph_</span>) <span style="color: rgb(212, 212, 212);"><</span> <span style="color: rgb(181, 206, 168);">0</span>)</div><div> {</div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><div> <span style="color: rgb(156, 220, 254);">input_format</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(79, 193, 255);">AV_PIX_FMT_VIDEOTOOLBOX</span>;</div><div> }</div><div> }</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(212, 212, 212);">!</span><span style="color: rgb(156, 220, 254);">filter_ctx</span>)</div><div> {</div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><div> <span style="color: rgb(86, 156, 214);">const</span> <span style="color: rgb(78, 201, 176);">AVFilter</span> <span style="color: rgb(212, 212, 212);">*</span><span style="color: rgb(156, 220, 254);">buffer</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_get_by_name</span>(<span style="color: rgb(206, 145, 120);">"buffer"</span>);</div><div> <span style="color: rgb(86, 156, 214);">const</span> <span style="color: rgb(78, 201, 176);">AVFilter</span> <span style="color: rgb(212, 212, 212);">*</span><span style="color: rgb(156, 220, 254);">buffersink</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_get_by_name</span>(<span style="color: rgb(206, 145, 120);">"buffersink"</span>);</div><div> <span style="color: rgb(86, 156, 214);">int</span> <span style="color: rgb(156, 220, 254);">err</span>;</div><div><span style="color: rgb(106, 153, 85);"> // Create buffer filter</span></div><div> <span style="color: rgb(220, 220, 170);">snprintf</span>(<span style="color: rgb(156, 220, 254);">strbuf_</span>, <span style="color: rgb(86, 156, 214);">sizeof</span>(<span style="color: rgb(156, 220, 254);">strbuf_</span>), <span style="color: rgb(206, 145, 120);">"video_size=</span><span style="color: rgb(156, 220, 254);">%d</span><span style="color: rgb(206, 145, 120);">x</span><span style="color: rgb(156, 220, 254);">%d</span><span style="color: rgb(206, 145, 120);">:pix_fmt=</span><span style="color: rgb(156, 220, 254);">%d</span><span style="color: rgb(206, 145, 120);">:time_base=</span><span style="color: rgb(156, 220, 254);">%d</span><span style="color: rgb(206, 145, 120);">/</span><span style="color: rgb(156, 220, 254);">%d</span><span style="color: rgb(206, 145, 120);">"</span>,</div><div> <span style="color: rgb(156, 220, 254);">input_hw_width_</span>, <span style="color: rgb(156, 220, 254);">input_hw_height_</span>,</div><div> <span style="color: rgb(156, 220, 254);">input_format</span>,</div><div> <span style="color: rgb(181, 206, 168);">1</span>,<span style="color: rgb(106, 153, 85);"> // TODO -> codec_context_->time_base.num</span></div><div> <span style="color: rgb(86, 156, 214);">AV_TIME_BASE</span><span style="color: rgb(106, 153, 85);"> // TODO</span></div><div> );</div><div> <span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_graph_create_filter</span>(<span style="color: rgb(212, 212, 212);">&</span><span style="color: rgb(156, 220, 254);">buffer_ctx_</span>, <span style="color: rgb(156, 220, 254);">buffer</span>,</div><div> <span style="color: rgb(206, 145, 120);">"in"</span>, <span style="color: rgb(156, 220, 254);">strbuf_</span>, <span style="color: rgb(86, 156, 214);">NULL</span>, <span style="color: rgb(156, 220, 254);">filter_graph_</span>);</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);"><</span> <span style="color: rgb(181, 206, 168);">0</span>)</div><div> {</div><div><span style="color: rgb(106, 153, 85);"> // Unable to create buffer filter</span></div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><div> <span style="color: rgb(78, 201, 176);">AVBufferSrcParameters</span> <span style="color: rgb(212, 212, 212);">*</span><span style="color: rgb(156, 220, 254);">src_params</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">av_buffersrc_parameters_alloc</span>();</div><div> <span style="color: rgb(156, 220, 254);">src_params</span>-><span style="color: rgb(156, 220, 254);">hw_frames_ctx</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">av_buffer_ref</span>(<span style="color: rgb(156, 220, 254);">hw_frames_ref</span>);<span style="color: rgb(106, 153, 85);"> // Set GPU context</span></div><div> <span style="color: rgb(156, 220, 254);">src_params</span>-><span style="color: rgb(156, 220, 254);">format</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(156, 220, 254);">input_format</span>;</div><div> <span style="color: rgb(156, 220, 254);">src_params</span>-><span style="color: rgb(156, 220, 254);">width</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(156, 220, 254);">input_hw_width_</span>;</div><div> <span style="color: rgb(156, 220, 254);">src_params</span>-><span style="color: rgb(156, 220, 254);">height</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(156, 220, 254);">input_hw_height_</span>;</div><div> <span style="color: rgb(156, 220, 254);">src_params</span>-><span style="color: rgb(156, 220, 254);">time_base</span> <span style="color: rgb(220, 220, 170);">=</span> <span style="color: rgb(156, 220, 254);">input_timebase</span>;</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(220, 220, 170);">av_buffersrc_parameters_set</span>(<span style="color: rgb(156, 220, 254);">buffer_ctx_</span>, <span style="color: rgb(156, 220, 254);">src_params</span>) <span style="color: rgb(212, 212, 212);"><</span> <span style="color: rgb(181, 206, 168);">0</span>)</div><div> {</div><div> <span style="color: rgb(220, 220, 170);">av_freep</span>(<span style="color: rgb(212, 212, 212);">&</span><span style="color: rgb(156, 220, 254);">src_params</span>);</div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><div> <span style="color: rgb(220, 220, 170);">av_freep</span>(<span style="color: rgb(212, 212, 212);">&</span><span style="color: rgb(156, 220, 254);">src_params</span>);</div><div><span style="color: rgb(106, 153, 85);"> // Create buffersink filter</span></div><div> <span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_graph_create_filter</span>(<span style="color: rgb(212, 212, 212);">&</span><span style="color: rgb(156, 220, 254);">buffersink_ctx_</span>, <span style="color: rgb(156, 220, 254);">buffersink</span>,</div><div> <span style="color: rgb(206, 145, 120);">"out"</span>, <span style="color: rgb(86, 156, 214);">NULL</span>, <span style="color: rgb(86, 156, 214);">NULL</span>, <span style="color: rgb(156, 220, 254);">filter_graph_</span>);</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);"><</span> <span style="color: rgb(181, 206, 168);">0</span>)</div><div> {</div><div><span style="color: rgb(106, 153, 85);"> // Unable to create buffersink filter</span></div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><div> <span style="color: rgb(86, 156, 214);">this</span>-><span style="color: rgb(156, 220, 254);">filter_ctx_list_</span>.<span style="color: rgb(220, 220, 170);">push_back</span>(<span style="color: rgb(156, 220, 254);">filter_ctx</span>);</div><br><div><span style="color: rgb(106, 153, 85);"> // Connect inputs and outputs</span></div><div> <span style="color: rgb(78, 201, 176);">AVFilterContext</span> <span style="color: rgb(212, 212, 212);">*</span><span style="color: rgb(156, 220, 254);">last_context</span>;</div><div> <span style="color: rgb(197, 134, 192);">for</span> (<span style="color: rgb(86, 156, 214);">int</span> <span style="color: rgb(156, 220, 254);">x</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(181, 206, 168);">0</span>; <span style="color: rgb(156, 220, 254);">x</span> <span style="color: rgb(212, 212, 212);"><</span> <span style="color: rgb(156, 220, 254);">filter_ctx_list_</span>.<span style="color: rgb(220, 220, 170);">size</span>(); <span style="color: rgb(156, 220, 254);">x</span><span style="color: rgb(212, 212, 212);">++</span>)</div><div> {</div><div> <span style="color: rgb(78, 201, 176);">AVFilterContext</span> <span style="color: rgb(212, 212, 212);">*</span><span style="color: rgb(156, 220, 254);">current_context</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(156, 220, 254);">filter_ctx_list_</span>.<span style="color: rgb(220, 220, 170);">at</span>(<span style="color: rgb(156, 220, 254);">x</span>);</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(156, 220, 254);">x</span> <span style="color: rgb(212, 212, 212);">==</span> <span style="color: rgb(181, 206, 168);">0</span>)</div><div> {</div><div> <span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_link</span>(<span style="color: rgb(156, 220, 254);">buffer_ctx_</span>, <span style="color: rgb(181, 206, 168);">0</span>, <span style="color: rgb(156, 220, 254);">current_context</span>, <span style="color: rgb(181, 206, 168);">0</span>);</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);"><</span> <span style="color: rgb(181, 206, 168);">0</span>)</div><div> {</div><div><span style="color: rgb(106, 153, 85);"> // Linking error</span></div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><div> <span style="color: rgb(156, 220, 254);">last_context</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(156, 220, 254);">current_context</span>;</div><div> <span style="color: rgb(197, 134, 192);">continue</span>;</div><div> }</div><div> <span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_link</span>(<span style="color: rgb(156, 220, 254);">last_context</span>, <span style="color: rgb(181, 206, 168);">0</span>, <span style="color: rgb(156, 220, 254);">current_context</span>, <span style="color: rgb(181, 206, 168);">0</span>);</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);"><</span> <span style="color: rgb(181, 206, 168);">0</span>)</div><div> {</div><div><span style="color: rgb(106, 153, 85);"> // Linking error</span></div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><div> <span style="color: rgb(156, 220, 254);">last_context</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(156, 220, 254);">current_context</span>;</div><div> }</div><br><div> <span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_link</span>(<span style="color: rgb(156, 220, 254);">last_context</span>, <span style="color: rgb(181, 206, 168);">0</span>, <span style="color: rgb(156, 220, 254);">buffersink_ctx_</span>, <span style="color: rgb(181, 206, 168);">0</span>);</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);"><</span> <span style="color: rgb(181, 206, 168);">0</span>)</div><div> {</div><div><span style="color: rgb(106, 153, 85);"> // Linking error</span></div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><br><div> <span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(220, 220, 170);">avfilter_graph_config</span>(<span style="color: rgb(156, 220, 254);">filter_graph_</span>, <span style="color: rgb(86, 156, 214);">NULL</span>);</div><div> <span style="color: rgb(197, 134, 192);">if</span> (<span style="color: rgb(156, 220, 254);">err</span> <span style="color: rgb(212, 212, 212);"><</span> <span style="color: rgb(181, 206, 168);">0</span>)</div><div> {</div><div><span style="color: rgb(106, 153, 85);"> // Error configuring the filter graph</span></div><div> <span style="color: rgb(197, 134, 192);">return</span> <span style="color: rgb(86, 156, 214);">false</span>;</div><div> }</div><br><div> <span style="color: rgb(156, 220, 254);">hw_filter_is_valid_</span> <span style="color: rgb(212, 212, 212);">=</span> <span style="color: rgb(86, 156, 214);">true</span>;</div></div></div><div></div></body></html>
<br>
<span style="font-family:'Courier New',Courier,mono;font-size:13px;background-color:rgb(255,255,255)">This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This message contains confidential information and is intended only for the individual named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by e-mail if you have received this e-mail by mistake and delete this e-mail from your system. If you are not the intended recipient you are notified that disclosing, copying, distributing or taking any action in reliance on the contents of this information is strictly prohibited.</span>