[Ffmpeg-devel] swscale and 64 bit problem (mmx code)

Giancarlo Formicuccia giancarlo.formicuccia
Thu Mar 1 14:18:12 CET 2007

Hi *,

libswscale seems to be unsafe on 64 bit platforms. The problem is 
the 32-bit chrMmxFilter array of SwsContext, because it stores
pointers to allocated memory (lumMmxFilter has probably the same

For example this testcase crashes on my 64 bit P4:

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>

#include <ffmpeg/avcodec.h>
#include <ffmpeg/avutil.h>
#include <ffmpeg/swscale.h>

#define W1 640
#define H1 480
#define W2 352
#define H2 288

static void wait_for_alloc64()
  while(1) {
    void *addr = av_malloc(1024*32);
    if (addr>=(void *) 0x100000000ULL) {
      fprintf(stderr, "Heap reached %p\n", addr);

static AVPicture *alloc_picture(int w, int h)
  AVPicture *ret = av_malloc(sizeof(*ret));
  int sz = avpicture_get_size(PIX_FMT_YUV420P, w, h);
  uint8_t *buf = av_malloc(sz);
  avpicture_fill(ret, buf, PIX_FMT_YUV420P, w, h);
  return ret;

int main()
  AVPicture *p1 = alloc_picture(W1, H1);
  AVPicture *p2 = alloc_picture(W2, H2);
  struct SwsContext *ss;

  /* Wait for the allocator to return >32 bit addressess */

  ss = sws_getContext(W1, H1, PIX_FMT_YUV420P, W2, H2, PIX_FMT_YUV420P,

  sws_scale(ss, p1->data, p1->linesize, 0,
    H1, p2->data, p2->linesize);

  return 0;

The crash occurs at
0x00002b00be3e2914 in swScale_MMX (c=0x1000082a0, src=0x501010, srcStride=0x7fffeccfb4f0, srcSliceY=0, srcSliceH=480, dst=0x501050, dstStride=0x7fffeccfb500)
    at swscale_template.c:944
944                             YSCALEYUV2YV12X(   0, CHR_MMX_FILTER_OFFSET, uDest, chrDstW)

RIP is at swscale_template.c:20:
"movq " #x "(%%"REG_S", %%"REG_a", 2), %%mm2\n\t" /* srcData */

The heap is at 00501000-100021000, but c->chrMmxFilter[0] is 0x14c70 at this point
(the low dword of 100014c70).

Fixing the code is not trivial, due to the many hardcoded values in the asm code...


More information about the ffmpeg-devel mailing list