Lightweight ZSTD with support for Android

[u/G3nghisKang]

Hello, I noticed the lack of a proper Java-only implementation of ZSTD (I know zstd-jni is fine, but it uses native libraries and is kind of heavy too) that also supports Android

So meet the murine-aircompressor library

After checking out version 2.0 of aircompressor, which is exclusively Java-only and used JDK 8, I saw the maintainer claimed Unsafe was not available for Android and therefore Android support was out of scope for it; however, that turns out not to be the case, Unsafe can be used on Android and there was only one single method and a static field which are missing on the Android implementation of Unsafe, I just made a helper method for the former, choosing the right implementation, while the static field was just a call to an existing method, and Bob's your uncle.

But not satisfied, I made a few changes, a lot of which are kanged from other forks the maintainer ignored:

- improved compression a lot (original fork that implemented this change here) - added an option to switch to the former behaviour too

- added a ring buffer mode and set as degault (also kanged from another fork, seems to improve performance over largely incompressible data)

- exposed levels for the streaming compressor

- added FAST algorithm to also support levels 1-2 on top of 3-4 which were the only supported ones in the original (level 2 is also pretty good and way faster; Claude is, however, mostly to praise [or blame] for this one)

- Improved overall performance on ARM (again, Claude is mostly to praise/blame here, but I did a lot of tests)

- added parallel compression support because fuck it

- added a wittle tar (with PAX) implementation because fuck it again, I wanted to make .tar.zst without qdding extra libraries

Using WindowSlideMode.HIGH_SPEED resets the library's (ORIGINAL) shit compression ratio for the stream, but makes it faster, I suppose; I default to HIGH_COMPRESSION which is still decently fast, and if you want it to be faster, it's much much better to use level 2 with HIGH_COMPRESSION than the default level 3 with HIGH_SPEED in my experience

Overall, the library is a little slower than zstd-jni with the (now) default parameters, but now matches its compression ratio, and may be a good compromise you'd rather want a lightweight full-java implementation than the heavy .so files in your apk

EDIT: I also noticed this may sound like I'm trashing the maintainer, I'm not lol, just stating facts, the maintainer is awesome for creating the library on the first place

[u/AD-LB]

Wait, why would making it more lightweight cause it to be slower in compression?

The compression to this file format is quite slow already, no? Does it at least reach the exact (or better) same level of compression, meaning files are at most as large as when using the original library?

[u/G3nghisKang → u/AD-LB]

Actually, ZSTD is quite fast when compared to other forms of compression, definitely faster when compared to the Gzip compression that already comes with Android, both this library and the JNI library

The advantage of the zstd-jni library is that it uses native code compiled in C, in the forms of .so libraries, the downside is that these libraries are heavier in raw file size, and also each .so library must be shipped inside the apk for each architecture (so the .apk will end up containing a .so for x86, one for x86_64, one for armv8 etc.), but it executes C code which is better for low level operations

This library is fully written in Java, so it's not dependant on architecture, it's pretty small in size, it uses the Unsafe library to do unsafe memory operation within Java, but it's a little slower than executing C code (the difference is not huge though), and it's the most portable since it doesn't depend on architecture (although it requires little endian, but I doubt that's an issue nowadays)

The original library (at least version 2.0, which is the one compatible with JDK 8) claimed to be faster than zstd-jni, but it also wasn't compressing the whole thing at the compression ratio it should have (at least when using streaming compression) so I guess it might tell a different story if you picked the library and just applied the compression fix

The compression ratio of this library is more or less on par with ztsd-jni, but not the one of the original 2.0 library I forked this from (which is just called "aircompressor")

[u/AD-LB → u/G3nghisKang]

Really? I thought that ZSTD is good at compression itself, but slow at it.

So if it's superior in both, why isn't it more popular? Decompression has downsides?

[u/G3nghisKang → u/AD-LB]

It's fast at compression and even faster at decompression, but maybe it's lacking support across operating systems, programming languages and archiving formats, it's called zstandard but it hasn't quite become a standard yet :P

Most browsers support it though, and some filesystems (e.g. btrfs and zfs) also support it for seamless disk compression

[u/AD-LB → u/G3nghisKang]

Weird.