Repository navigation
Serialization format #5
Description
Bug's data are stored using git Commit, Tree and Blob. Inside a Blob is serialized an OperationPack, that is an array of edit operation on the bug's state.
This OperationPack is currently serialized using golang's gob, which is neat because it just works. However, it might not be the best option for interoperability with other tools in the future.
How should that be serialized ? Json ? In any case, git will compress the data using zlib so a text format might not be that terrible.
Feel free to argue a case here.
Activity
daurnimator commentedon Aug 17, 2018
I just found the project via hackernews. I'd love to give this sort of thing a try, and integrate it into other tools. However using go serialisation rules out all my languages of choice.
I'd say use something JSON based, or if that's not enough, CBOR.
MichaelMure commentedon Aug 17, 2018
For the record, it would be an easy change to use something else, it's only a few lines of code: https://github.com/MichaelMure/git-bug/blob/d959acc29dcbc467790ae87389f9569bb830c8c6/bug/operation_pack.go#L25-L52
lukechampine commentedon Aug 17, 2018
My two cents:
gobis convenient and efficient, but not a great choice if you want interop with other languages. Unfortunately there just aren't many binary formats that are widely supported, except perhaps protobufs.JSON is probably your best bet. As you noted, it will be compressed anyway, and if performance is an issue you can always switch to a faster JSON encoder. The only big downside to JSON that I'm aware of is poor support for encoding binary blobs (
encoding/jsonencodes[]byteas a base-64 string). But ifOperationPackis almost entirely textual data anyway, there's little reason to worry about that.avar commentedon Aug 17, 2018
Git does its own delta-compression on top of zlib. You should decide this using a combination of whatever format needs you have (can you add more fields, is it extensible etc.), and how git manages to compress this using both delta compression and zlib, which you can figure out using a large enough set of realistic test data.
MichaelMure commentedon Aug 17, 2018
To give more details about the requirement, OperationPack hold currently very simple data (string, int, array..), and it's likely to stay the same even when adding new operations. For instance, embedded files are stored in git blobs and then linked in the git tree.
The only tricky part is that an OperationPack is an mixed array of Operation so the decoder need to support that and match the correct go struct for each operation.
MichaelMure commentedon Aug 19, 2018
With all these format that could fit the bill, the best way to choose would be a benchmark for both performance and blob size for several format (at least JSON and CBOR). Who knows how the git compression behave on something that is already binary.
Maybe the git people could do an educated guess.
j-f1 commentedon Aug 22, 2018
MessagePack is another option, but I feel like MessagePack and CBOR are both designed for getting the smallest possible representation of data, whereas JSON is designed to be human-readable, ASCII-compatible, and simple to parse. Compare JSON’s spec (the sidebar) with the CBOR and MessagePack specs.
MichaelMure commentedon Sep 6, 2018
I wrote some throwaway code to test the resulting blob size for various format. Here is one run:
As expected, there is not that much differences after encoding + compression. CBOR consistently win the size contest though.
MichaelMure commentedon Sep 6, 2018
Note: each serialization format is tested on the same set of randomly generated OperationPack with one
Createand 4AddCommentops.avar commentedon Sep 6, 2018
@MichaelMure This test case really isn't meaningful. You're just testing how a given payload compresses with zlib when creating loose objects, since when you add a new object it's compressed, a header is added to it, and it's added to the object store.
Instead, you should after every addition do
git add && git commit && git gc. Then measure the total size of the now-packed .git/objects directory, not individual objects.At that point, these objects will be delta-compressed, so you can see how the size of the repo grows as they're added.
The size of individual objects is pretty much irrelevant. You can have 10 objects that are all 1GB, but delta-compress down to 1GB + 1MB or whatever, or 10GB if they don't delta-compress at all.
avar commentedon Sep 6, 2018
@MichaelMure Also in reply to:
I'm sure there's some obscure edge case where the compression is tweaked for textual content in some way that'll prove me wrong, but in general this doesn't matter at all.
Git's just as good at delta-compressing binary data and non-binary data. What it's not good at compressing (and this goes for any compression), is data that's wildly different from one object to the next.
It just so happens that generally binary data is less delta-compressible, think say two *.mp3s with different songs v.s. a *.txt change to its lyrics.
But for these sort of pack formats I wouldn't expect them to delta-compress any worse than say JSON. It's going to be other things that matter, e.g. let's say you use a JSON encoder where the keys of the payload aren't sorted, and thus are different every time, that'll compress worse than if they're sorted, same for doing the same in some binary key-value format.
I do think that for UI purposes it makes sense to pick a widely implemented & used text format like JSON for introspection purposes and the availability of tooling (e.g.
jq), if the compression numbers for it aren't much worse that is.MichaelMure commentedon Sep 6, 2018
That's a good point, I'll check the repo size as well, before and after a
git gc.MichaelMure commentedon Sep 6, 2018
Alright, another run with the size of the repo before and after a
git gc(initial empty size substracted), 1000 OperationPack serialized:Whatever the format, there is no compression taking advantage of the similarity between each OperationPack. The packed repo is actually bigger by 47 bytes, and a
git gc --agressivedoes nothing.7 remaining items
MichaelMure commentedon Sep 12, 2018
With 60fcfcd, I changed the serialization format for Json.
Here are a few measurement with 10k random bugs and 10op/bug (100k ops total, same as the previous test):
Quite happy with these results! Note that the cache building is currently mono-processor. There is still perf to gain.
Also, now that the blobs are connected in a chain of commit,
git gcstart to actually compress them. 21Mo for 10k bugs is nice.MichaelMure commentedon Sep 13, 2018
With no sign of troubles after various tests, let's consider the matter resolved :-)
andyl commentedon Sep 13, 2018
Is there a CLI command to generate a JSON dump from the issues?
MichaelMure commentedon Sep 13, 2018
@andyl there is not. What's your usecase ?
andyl commentedon Sep 13, 2018
@MichaelMure I'm working on a project that allows people to post auction-style bids for issues (see bugmark.net). We'd very much like to integrate with git-bug. To do this, we need to be able to poll the issue repository and grab a json-like representation. JSON would be simple for us, but if there is another way to integrate open to that too.
MichaelMure commentedon Sep 13, 2018
@andyl that's certainly doable and should be supported by the CLI tools.
Could you open a new issue where we can discuss that ?
andyl commentedon Sep 13, 2018
@MichaelMure see #45
codereview git-bug#5: reverse-map and ImportWarning
codereview git-bug#5: reverse-map and ImportWarning
codereview git-bug#5: reverse-map and ImportWarning