<div dir="ltr">Hello libav world,<br><br>Currently I am pulling pixels off of the GPU (specifically via Windows DXGI Desktop Duplication API), and then H264 encoding them using <font face="monospace">nvenc_h264 </font><font face="arial, sans-serif">for hardware acceleration</font><font face="monospace">.</font> Currently, that involves a download of the pixels into CPU memory (in an <font face="monospace">AVFrame</font>), and then an immediate re-upload of them onto the GPU for NVENC. <b>My question is: is it possible to cut out this step and go straight from GPU/d3d11 memory to NVENC encoding?</b><div><b><br></b></div><div>I noticed that there is a supported pixel format for <font face="monospace">AV_PIX_FMT_D3D11</font>, which takes a pointer to a <font face="monospace">ID3D11Texture2D</font> in the <font face="monospace">data[0]</font> slot. However, configuring my code to use this pixel format (including all the necessary hwdevice config, hwframe context and allocation, etc) still results in an error:<br><br><pre class="gmail-lang-cpp gmail-s-code-block" style="margin-top:0px;padding:12px;border:0px;font-variant-numeric:inherit;font-variant-east-asian:inherit;font-stretch:inherit;line-height:1.30769;font-size:13px;vertical-align:baseline;box-sizing:inherit;width:auto;max-height:600px;overflow:auto;border-radius:5px"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit"><span class="gmail-hljs-function" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit"><span class="gmail-hljs-type" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">int</span> <span class="gmail-hljs-title" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">write_gpu_video_frame</span><span class="gmail-hljs-params" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">(ID3D11Texture2D* gpuTex, AVFormatContext* oc, OutputStream* ost)</span> </span>{<br></code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">    AVFrame *hw_frame = ost->hw_frame;</code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">
</code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">    <span class="gmail-hljs-built_in" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">printf</span>(<span class="gmail-hljs-string" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">"gpuTex address = 0x%x\n"</span>, &gpuTex);</code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">
</code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">    hw_frame->data[<span class="gmail-hljs-number" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">0</span>] = (<span class="gmail-hljs-type" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">uint8_t</span> *) gpuTex;<br></code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">    hw_frame->data[<span class="gmail-hljs-number" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">1</span>] = (<span class="gmail-hljs-type" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">uint8_t</span> *) (<span class="gmail-hljs-type" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">intptr_t</span>) <span class="gmail-hljs-number" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">0</span>;<br></code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">    hw_frame->pts     = ost->next_pts++;</code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">
</code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">    <span class="gmail-hljs-keyword" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">return</span> <span class="gmail-hljs-built_in" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">write_frame</span>(oc, ost->enc, ost->st, hw_frame);<br></code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">    <span class="gmail-hljs-comment" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">// write_frame is identical to sample code in ffmpeg repo<br></span></code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">}</code></blockquote><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit"><font face="inherit">
</font><font face="arial, sans-serif">Resulting error:</font><font face="inherit">

</font><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">gpuTex address = <span class="gmail-hljs-number" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">0x4582f6d0<br></span></code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">[h264_nvenc @ <span class="gmail-hljs-number" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">00000191233e1</span>bc0] Error registering an input resource: invalid <span class="gmail-hljs-built_in" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">call</span> (<span class="gmail-hljs-number" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">9</span>):<br></code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">[h264_nvenc @ <span class="gmail-hljs-number" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">00000191233e1</span>bc0] Could <span class="gmail-hljs-keyword" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">not</span> <span class="gmail-hljs-keyword" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit">register</span> an input HW frame<br></code><code class="gmail-hljs gmail-language-cpp" style="margin:0px;padding:0px;border:0px;font-style:inherit;font-variant:inherit;font-weight:inherit;font-stretch:inherit;line-height:inherit;font-family:inherit;vertical-align:baseline;box-sizing:inherit;background-color:transparent;white-space:inherit">Error sending a frame to the encoder: Unknown error occurred</code></blockquote><font face="inherit">
</font><font face="arial, sans-serif">I can post the other setup code if it's helpful to diagnose the problem.

Any immediate intuitions about this type of approach?

Best regards,
- Nathan</font></code></pre></div></div>