[Ffmpeg-devel] swscale and 64 bit problem (mmx code)

Giancarlo Formicuccia giancarlo.formicuccia
Thu Mar 1 14:18:12 CET 2007


Hi *,

libswscale seems to be unsafe on 64 bit platforms. The problem is 
the 32-bit chrMmxFilter array of SwsContext, because it stores
pointers to allocated memory (lumMmxFilter has probably the same
issue).

For example this testcase crashes on my 64 bit P4:

#include <stdio.h>
#include <stdlib.h>
#include <assert.h>

#include <ffmpeg/avcodec.h>
#include <ffmpeg/avutil.h>
#include <ffmpeg/swscale.h>

#define W1 640
#define H1 480
#define W2 352
#define H2 288

static void wait_for_alloc64()
{
  while(1) {
    void *addr = av_malloc(1024*32);
    assert(addr!=NULL);
    if (addr>=(void *) 0x100000000ULL) {
      fprintf(stderr, "Heap reached %p\n", addr);
      break;
    }
  }
}

static AVPicture *alloc_picture(int w, int h)
{
  AVPicture *ret = av_malloc(sizeof(*ret));
  int sz = avpicture_get_size(PIX_FMT_YUV420P, w, h);
  uint8_t *buf = av_malloc(sz);
  avpicture_fill(ret, buf, PIX_FMT_YUV420P, w, h);
  return ret;
}

int main()
{
  AVPicture *p1 = alloc_picture(W1, H1);
  AVPicture *p2 = alloc_picture(W2, H2);
  struct SwsContext *ss;

  /* Wait for the allocator to return >32 bit addressess */
  wait_for_alloc64();

  ss = sws_getContext(W1, H1, PIX_FMT_YUV420P, W2, H2, PIX_FMT_YUV420P,
    SWS_FAST_BILINEAR, NULL, NULL, NULL);

  sws_scale(ss, p1->data, p1->linesize, 0,
    H1, p2->data, p2->linesize);

  return 0;
}

The crash occurs at
0x00002b00be3e2914 in swScale_MMX (c=0x1000082a0, src=0x501010, srcStride=0x7fffeccfb4f0, srcSliceY=0, srcSliceH=480, dst=0x501050, dstStride=0x7fffeccfb500)
    at swscale_template.c:944
944                             YSCALEYUV2YV12X(   0, CHR_MMX_FILTER_OFFSET, uDest, chrDstW)

RIP is at swscale_template.c:20:
"movq " #x "(%%"REG_S", %%"REG_a", 2), %%mm2\n\t" /* srcData */

The heap is at 00501000-100021000, but c->chrMmxFilter[0] is 0x14c70 at this point
(the low dword of 100014c70).

Fixing the code is not trivial, due to the many hardcoded values in the asm code...
Ideas?

Giancarlo




More information about the ffmpeg-devel mailing list