extmod/uzlib: Add gzip compression support. by andrewleech · Pull Request #5613 · micropython/micropython

andrewleech · 2020-02-06T00:28:31Z

Exposes basic compression support from uzlib. I originally wrote this nearly a year ago so don't really remember too much about it.

Pushed up to support #5590

Doesn't have any unit tests written... probably needs some first. Possibly some #define's to enable/disable the compress functionality?

stinos · 2020-02-06T09:02:24Z

+    memset(comp, 0, sizeof(*comp));
+
+    comp->dict_size = 32768;
+    comp->hash_bits = 12;


This and previsou line could use a comment stating why this is chosen I think (I know nothing about gzip though)

Would it make sense to expose these settings in the API?

stinos · 2020-02-06T09:02:44Z

+    uzlib_compress(comp, bufinfo.buf, len);
+    zlib_finish_block(&comp->out);
+
+    printf("compressed from %u to %u raw bytes\n", len, comp->out.outlen);


Use DEBUG_printf ?

stinos · 2020-02-06T09:03:14Z

+
+    printf("compressed from %u to %u raw bytes\n", len, comp->out.outlen);
+
+    mp_uint_t dest_buf_size = (comp->out.outlen + 6 + (3*sizeof(int)));


Same remark as earlier, what are these magic constants?

stinos · 2020-02-06T09:05:56Z

There's MICROPY_PY_UZLIB already, but I guess splitting that up in separate compress/decompress doesn't hurt and can be used to keep the original behavior (i.e. not enabling compression by default).

A bunch of tests would be nice indeed, and fairly easy to write (assuming asserting that decompress(compress(x)) == x is sufficient for proving it works)

codefreax · 2020-02-13T17:05:48Z

+    int mtime = 0;
+    memcpy(&dest_buf[i], &mtime, sizeof(mtime));
+    i += sizeof(mtime);


According to Wikipedia, mtime must be 4 bytes. Use uint32_t here?

codefreax · 2020-02-13T17:06:54Z

+    struct uzlib_comp *comp = m_new_obj(struct uzlib_comp);
+    memset(comp, 0, sizeof(*comp));


nit: Merge into m_new0(struct uzlib_comp, 1);?

codefreax · 2020-02-13T17:07:49Z

+    dest_buf[i++] = 0x1f;
+    dest_buf[i++] = 0x8b;
+    dest_buf[i++] = 0x08;
+    dest_buf[i++] = 0x00; // FLG


nit: Can you add a few more comments here and below where there are more magic numbers?

codefreax · 2020-02-13T17:09:35Z

+    unsigned int crc = ~uzlib_crc32(bufinfo.buf, len, ~0);
+    memcpy(&dest_buf[i], &crc, sizeof(crc));
+    i += sizeof(crc);


uint32_t?

codefreax · 2020-02-13T17:10:36Z

+    mp_obj_t data = args[0];
+    mp_buffer_info_t bufinfo;
+    mp_get_buffer_raise(data, &bufinfo, MP_BUFFER_READ);
+    int len = bufinfo.len;


This is written to the output; should this be uint32_t (or whatever the right size is)?

QAMU · 2020-04-27T23:26:34Z

hello,
for this commit, how to compress data with micropython.
can I use uzlib.compress(data)?

andrewleech · 2020-04-28T01:00:15Z

Hi @QAMU I think it worked like uzlib.gzip(data)
I haven't used it for some time though, not sure it's ever likely to get completed enough to merge.

QAMU · 2020-04-28T01:09:27Z

Hi @andrewleech,
thanks for your reply, can I use uzlib.decompress() to decompress data = uzlib.gzip()?

andrewleech · 2020-04-28T01:14:29Z

yep the two should work together well

harbaum · 2021-02-22T14:23:29Z

It seems compression and decompression don't work with each other:

>>> uzlib.decompress(uzlib.gzip("Hello World!"))
compressed from 12 to 14 raw bytes
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: -3

harbaum · 2021-02-27T11:35:42Z

It seems compression and decompression don't work with each other:

They do if one removes the gzip header, crc and length fields and uses the correct wbits:

>>> uzlib.decompress(uzlib.gzip("Hello World!")[10:-8], -15)
compressed from 12 to 14 raw bytes
bytearray(b'Hello World!')

harbaum · 2021-03-01T07:41:26Z

Updated attempt here.

QAMU · 2021-03-01T08:05:07Z

to compress use:

import uzlib 
def compress(buffer):
    encoded = = uzlib.gzip(buffer)
    return encoded

to decompress:

import uzlib 
import uio
def decompress(buffer):
    file = uio.BytesIO(buffer)
    decoded = uzlib.DecompIO(file, 31)
    return decoded.read()

harbaum

to decompress:

import uzlib 
import uio
def decompress(buffer):
    file = uio.BytesIO(buffer)
    decoded = uzlib.DecompIO(file, 31)
    return decoded.read()

Why not:

def decompress(buffer):
  return uzlib.decompress(buffer[10:-8], -15)

Seems more lightweight for an embedded solution.

harbaum · 2021-02-27T18:02:19Z

+    memset(comp->hash_table, 0, hash_size);
+
+    zlib_start_block(&comp->out);
+    uzlib_compress(comp, bufinfo.buf, len);


uzlib_compress as well as zlib_start_block internally (re-)allocate a buffer at comp->out which is never free'd. As a result the system runs out of memory after a few compression runs.

A free(comp->out.outbuf); after the subsequent memcpy solved this.

uzlib_compress as well as zlib_start_block internally (re-)allocate a buffer at comp->out which is never free'd. As a result the system runs out of memory after a few compression runs.

A free(comp->out.outbuf); after the subsequent memcpy solved this.

Im trying to encode a string to get into the shorter sms data possible.,
questio is : how can I add this gzip uzlib extension to my current 1.14 toolchain for esp32?

andrewleech · 2021-07-13T08:06:52Z

@harbaum Thanks for your additions in #6972, there is some great work there. I've rebased your changes on my original branch as a separate commit to keep your attribution.

In addition, I've added support for gzip header in the decompress(bytes, 31) function to allow it to work directly on the gzip compressed data.

The docs have been updated and I've added a basic compress unit test.

andrewleech · 2021-07-13T08:40:06Z

Ok I've broken the unit tests with the last additions of gzip decompress and rebase onto current master... and it's too big for tiny ports - so probably does need a new feature flag. Either that, or just build & distribute in the dynamic C module version of the uzlib.

clear out interrupt when freeing the timer

dpgeorge · 2023-07-21T13:32:04Z

Superseded by #11905.

stinos reviewed Feb 6, 2020

View reviewed changes

dpgeorge added the extmod Relates to extmod/ directory in source label Feb 6, 2020

codefreax reviewed Feb 13, 2020

View reviewed changes

harbaum mentioned this pull request Mar 1, 2021

extmod/uzlib: Add gzip compression support. (update to #5613) #6972

Closed

harbaum reviewed Mar 1, 2021

View reviewed changes

br0kenpixel mentioned this pull request May 6, 2021

Compression support #7228

Closed

andrewleech force-pushed the gzip branch 3 times, most recently from f69e550 to 106f843 Compare July 13, 2021 08:06

andrewleech force-pushed the gzip branch from 106f843 to c1fa8c7 Compare July 13, 2021 08:23

andrewleech force-pushed the gzip branch 2 times, most recently from 26c07f9 to 5fff6e9 Compare July 14, 2021 00:11

andrewleech force-pushed the gzip branch 2 times, most recently from 4da7fe2 to 8378ac1 Compare July 28, 2021 23:04

andrewleech force-pushed the gzip branch from 8378ac1 to e5c047d Compare October 6, 2021 22:00

andrewleech force-pushed the gzip branch from e5c047d to 27b3a2a Compare November 17, 2021 04:29

andrewleech force-pushed the gzip branch from 27b3a2a to ec57098 Compare November 25, 2021 03:14

tannewt added a commit to tannewt/circuitpython that referenced this pull request Dec 29, 2021

Merge pull request micropython#5613 from jepler/issue5418

08d09ac

clear out interrupt when freeing the timer

harbaum mentioned this pull request Jan 20, 2022

extmod/uzlib: Add gzip compression support. (update to #5613 and #6972) #8195

Closed

pi-anl and others added 3 commits January 24, 2022 09:00

extmod/uzlib: Add gzip header support to decompress().

77688de

extmod/uzlib: Add gzip compression support.

f692413

extmod/uzlib: Fix gzip compress buffer handling.

26c8d0e

andrewleech force-pushed the gzip branch from ec57098 to 26c8d0e Compare January 24, 2022 00:54

jimmo mentioned this pull request Oct 19, 2022

extmod: support compression in uzlib module #5590

Closed

dpgeorge added this to the release-1.21.0 milestone May 8, 2023

This was referenced Jun 27, 2023

extmod/modgzip.c: Add gzip.GzipFile providing compression and decompression. #11879

Closed

extmod/moddeflate.c: Add deflate.DeflateIO providing compression and decompression. #11905

Merged

dpgeorge closed this Jul 21, 2023


		printf("compressed from %u to %u raw bytes\n", len, comp->out.outlen);

		mp_uint_t dest_buf_size = (comp->out.outlen + 6 + (3*sizeof(int)));

		struct uzlib_comp *comp = m_new_obj(struct uzlib_comp);
		memset(comp, 0, sizeof(*comp));

Uh oh!

Conversation

andrewleech commented Feb 6, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codefreax Feb 13, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stinos commented Feb 6, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

QAMU commented Apr 27, 2020

Uh oh!

andrewleech commented Apr 28, 2020

Uh oh!

QAMU commented Apr 28, 2020

Uh oh!

andrewleech commented Apr 28, 2020

Uh oh!

harbaum commented Feb 22, 2021

Uh oh!

harbaum commented Feb 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

harbaum commented Mar 1, 2021

Uh oh!

QAMU commented Mar 1, 2021

Uh oh!

harbaum left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewleech commented Jul 13, 2021

Uh oh!

andrewleech commented Jul 13, 2021

Uh oh!

dpgeorge commented Jul 21, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

8 participants

codefreax Feb 13, 2020 •

edited

Loading

harbaum commented Feb 27, 2021 •

edited

Loading