Never match MD5 for gzipped file types

[alanjds]

It only works for non-gzipped files, as the MD5 is calculated from .read() of original file, and S3 ETAG if for the uploaded file, gzip-encoded if GZIP=True.

I tried to patch for both cases, but had not worked for some unknown reason (for me at least). I tried to hashlib.md5(self.storage._compress_content(source_storage.open(path).read())).hexdigest(), but it never give me the same answer as the ETAG.

Weird. I give up for now. Better luck next coder.

[alanjds]

Hi again. Have someone else at least a clue about why this occurs?

[mjtorn]

You can simulate this with the shell!

In [81]: ret = stor._compress_content(File(open('setup.py', 'rb')))

In [82]: ret2 = stor._compress_content(File(open('setup.py', 'rb')))

In [83]: retc = ret.read() ; ret2c = ret2.read()

In [84]: for i in xrange(len(retc)):
    if retc[i] != ret2c[i]:
        print i, ord(retc[i]), ord(ret2c[i])
   ....:         
4 214 221

So the byte where the screwup happens is the mtime: http://www.dotnetperls.com/gzip-header

The proper solution would be to figure out why django-storages doesn't do mtime properly. My best guess so far is that the storages backend doesn't allow passing the mtime around, as it seems to deal with contents. I'll let you know if I find anything.

[mjtorn]

https://bitbucket.org/david/django-storages/pull-request/83/converting-s3-last-modified-time-to-local/diff

That pull request contains a fix against matching by mtime, though the md5s still differ.

[mjtorn]

Based on that I'm 90% sure the local md5 calculation isn't done with the current timestamp, but with the local timezone. This is backed by debugging fasts3collectstatic, as the local md5 is always the same. If it was the current timestamp it'd change on every run.

[olofsj]

What version of boto and django-storages are you using? And what settings are you using to enable compression?

When I do some quick testing with boto==2.13.3 and django-storages==1.1.6 it seems to work for me. Just turning on the setting AWS_IS_GZIPPED gzips the files, and doesn't copy them again when re-running fasts3collectstatic. However, when turning on or off AWS_IS_GZIPPED it doesn't notice that the files should be re-uploaded so the ETAG seems to be of the file content before compression.