r/java • u/za3faran_tea • 6d ago
Question about Native Image vs JIT
I was watching an interview with Thomas Wuerthinger of GraalVM, where he basically says that JIT should only be used when necessary (as opposted to native compilation). Where does this leave the work being done for Hotspot JIT (including projects like Leyden and others). Should we all plan to use native image while they address any performance gaps in the mean time (he does mention PGO bridging the gap). Is there work being done to speed up native compilation of Java code?
My understanding is that JIT will always have a memory overhead due to running threads that do compilation, optimization, deoptimization, code cache, etc. compared to native executables, so full parity even with projects that will improve memory usage in Java code will not be reached. Maybe that is not an issue on long running programs on large servers, but we've seen discussions here where another member worte a tool in golang to run alongside Spring Boot applications to gather statistics, so obviously small efficient apps have their place.
32
u/Deep_Age4643 6d ago
Thomas Wuerthinger is a clever guy, with a number of achievements in the Java world to his name. But he is also the founder and project lead of GraalVM, as the video title says. Naturally, he presents his own product – and what makes it unique – in a favorable light. Always consider the interests behind someone’s comments, and verify independently and objectively whether these claim are valid for your own use case.
Personally, I believe that JIT is the best option in most cases. Native compilation is particularly useful for microservices where start-up time needs to be instantaneous, but compiling to native code has its limitations and is considerably slower than JIT compilation.
3
u/reallynotfred 6d ago
Exactly. He’s pushing towards the Oracle paid product (it’s good, not saying it isn’t) and away from the open source alternative.
2
u/OddEstimate1627 6d ago
IMO AOT is also very nice for GUIs and CLIs
6
u/nekokattt 5d ago
Last time I tried to get GraalVM native images to work with AWT I lost an entire afternoon to adding arbitrary nonsense to JSON files just to get it to instrument correctly in places the JVM agent couldn't reach as part of unit and integration test runs. I still reserve the unpopular opinion that having to perform tracing of a runtime product as part of automation suites just to enable the correct runtime behaviour is far from ideal.
5
u/OddEstimate1627 5d ago
Yes, the metadata is the most common issue. That's why I open sourced https://www.reddit.com/r/java/comments/1uy79pg/reachability_annotations_for_generating_graalvm/
It generates the JSON data directly from annotated Java source code and won't go stale.
That being said, I think JavaFX probably still has much better AOT support than AWT/Swing.
2
9
u/pokeapoke 6d ago
So what is performance? Is it throughput? Low latency? Balance between both? G1 GC is not available in community GraalVM. ZGC was not available at all the last time I checked.
10
u/v4ss42 6d ago
This seems to miss the primary benefit of JIT - that it can optimize the code based on actual runtime conditions / data / load patterns, as well as re-optimize it if those patterns change. Legacy (AOT) compilation can't do that, since it has zero insight into the runtime environment and can't make assumptions about it.
And yes obviously this is only relevant for long-running processes (like server apps). For brief, one-shot processes (command line utilities etc.), then yes, legacy compilation makes sense.
10
u/za3faran_tea 6d ago
I think he touched at this point. He was saying that a lot of these benefits can be reached using PGO, and re-optimization is very rare in practice.
7
u/Thirty_Seventh 6d ago
unfortunately, PGO isn't available in Community Edition and therefore does not exist as far as I'm concerned
3
u/koflerdavid 6d ago
I'm very doubtful that PGO can effectively capture the effects of, say, changing runtime configuration parameters. Those are effectively constant for most of the runtime of the application until they, well, change. Perfect for the JIT who can now specialize on them. Meanwhile for PGO if you have more than, say, three booleans then the effort of collecting profiling data for all combinations becomes prohibitive.
6
u/v4ss42 6d ago
And I'm saying I disagree with his extraordinary assertion that AOT optimization is as good as JIT.
4
u/koflerdavid 6d ago
It can be good. After all AOT is what languages like C++ and Rust have been doing all along. Of course Java has a few features that make it harder to optimize.
4
u/v4ss42 6d ago
You'll notice that I was careful to make a comparative statement that AOT optimization can't be as good as JIT optimization. What I didn't say is that AOT optimization can't be "good" in some absolute sense - yes it can be quite adequate, and not only on the JVM, but the reality is that AOT optimization can't touch JIT optimization because it has zero visibility into the runtime behavior of the code.
2
u/koflerdavid 6d ago
That's where Profile Guided Optimization comes in, which works as long as the real workload doesn't drift too much from the profile workload. That's its biggest disadvantage of course - a JIT compiler can adapt to changing circumstances.
2
u/Electrical_Being_813 4d ago
If you are forced to use PGO, you can as well use JIT. It will be less messy and will do a better job.
3
u/notyouryyy 6d ago
I think Thomas is claiming that in practice, deoptimisation doesn’t occur unless some uncommon trap is sprung, and that the runtime guided optimisation in AOT is pretty good.
5
u/v4ss42 6d ago edited 6d ago
Right, but that's just as likely to be due to runtime patterns not tending to change much and therefore reoptimization not being necessary that often. The point is that AOT can't even identify what runtime patterns exist in the first place, and then tune optimization specifically for them. JIT can (and does).
And yes there are (or were) attempts to close that loop over the decades - for example back in the 90s I worked on a system that used an IBM C/C++ Compiler where we could enable instrumentation of our production code, and feed the outputs of that instrumentation back into the compiler to optimize subsequent builds. But that was a manual, slow, and clumsy feedback loop, and the JIT blows it out of the water in both reaction time and developer ergonomics.
2
u/notyouryyy 6d ago
Yeah idk, graal offers this too. I’m not a fan of the slowdowns when getting pushed back to interpreted mode, but I’d argue that the biggest problem with graal aot is that it doesn’t support the more sophisticated lowlatency GCs yet 🤷♀️
1
u/koflerdavid 6d ago
The point is that AOT can't even identify what runtime patterns exist in the first place, and then tune optimization specifically for them.
That's what PGO is. I agree that it's a manual and quite brittle version of what the JIT is doing, but in situation where you really don't want the overhead of the JIT it's the way to go.
2
u/v4ss42 6d ago
And what situations are those? Short-lived processes (like command line tools)?
1
u/koflerdavid 6d ago edited 6d ago
Yes, or generally being so resource constrained that the overhead of the JIT cannot be justified. Like on embedded systems. Of course the OpenJDK team works hard on making it more valuable having the JIT than not, but there is a limit.
The issue can be somewhat improved by generating a compiled code cache at shutdown. That might achieve a similar effect as PGO assuming the C2 JIT has run at all.
3
u/v4ss42 6d ago
JIT has the ability to produce better results than AOT (including those guided by PGO), precisely because it’s dynamic and can take the runtime environment into account in a way AOT can’t (even with PGO).
And JIT code caches, while useful and a bit of a no-brainer, only provide relatively niche benefits (i.e. cutting one-time startup and initial JIT costs) compared to the overall benefits of the JIT itself.
1
u/koflerdavid 6d ago
Too bad if the workload is too small for the C2 JIT to ever run; C1 merely stitches native code versions of each instruction together and does not do other optimizations.
1
u/rbygrave 2d ago
Fwiw I built some apps as 2 containers, with one being JVM and the other graalvm native image (no PGO). Ran synthetic benchmark tests against those 2 containers.
So for these apps (rest apis, helidon se webserver, postgres jdbc) it did show AOT to be very impressive. These apps have now been running in production for 8 months as native image.
Imo it's maybe something you have to try and benchmark with your own apps etc. For myself, I'd expect you'll see some cases of AOT slightly faster and some slightly slower peak performance in throughput (noting that this is using G1GC for both).
The AOT compile does use ML (Machine Learning) so perhaps it is more sophisticated than you might first think (so yes that does mean higher build times for that analysis).
3
u/koflerdavid 6d ago
A solution would be something similar to OpenJ9's compile server that is shared by multiple VMs and caches build results. Across a Deployment (in the k9s sense) the nodes all do approximately the same thing, therefore there is no reason for them to all JIT optimize the same code in almost identical ways.
3
u/samcarlberg 5d ago edited 4d ago
Another problem with AOT is that you can't cross-compile. So if you do development on a mac, you can't build native images that run on a linux machine; you either need a machine with the same OS (and possibly CPU arch) or a VM.
The niche that I'm in (competitive educational robotics) would love AOT to reduce timing jitter and have more deterministic behavior, especially since matches are so short that nothing hits the JIT compiler thresholds and programs execute almost entirely in interpreted mode. Unfortunately the target is arm64 Linux and almost all users are on Windows so AOT isn't an option.
2
u/OddEstimate1627 5d ago edited 5d ago
Yeah if users write and deploy their own Java code I can see that being a problem. For us there is usually a CI layer that creates the binaries for all platforms.
I guess you could setup docker with emulation, but I generally wouldn't recommend forcing the native image experience on beginners.
One project I'm working on is an annotation processor that takes Java code, compiles it into a native image C ABI, and generates idiomatic wrappers for various languages and platforms. That way users can call AOT compiled Java code from eg Python, C++, and MATLAB. That might actually be interesting for WPILib.
Here is a JavaFX-based project I created with it that implements fully decoupled rendering with near zero overhead on the caller thread, so you could do low-overhead robot visualizations from inside a control loop: HebiRobotics/hebi-charts (video: C++ examples and Python performance)
The next version will also support running on linux-aarch64, but I haven't released it yet.
Btw I'm also the author of QuickBuffers, which is used in WPILib for zero-allocation protobuf.
2
u/samcarlberg 3d ago
The annotation processor project certainly seems interesting, though I don't know if that's something we'd use in WPILib. Most of the development is done in C++ that's later ported to Java or exposed via JNI; I think I'm the only core developer working primarily in Java.
Big fan of QuickBuffers, btw. It's made telemetry so much easier
1
u/OddEstimate1627 21h ago
Thanks! I've never been involved with FRC, so I'm not very familiar with the stack. It looks like I was thinking of PathPlannerLib rather than WPILib. My understanding is that it's primarily Java based and gets ported to the other languages, and that use case could be automated and share a single implementation. You could also efficiently convert from Java to raw C structs.
2
u/samcarlberg 19h ago
Many third-party libraries for FRC are Java-only or Java-first and then ported to C++ later; about 93% of FRC teams use Java (3398 out of 3565 total). Established libraries like PathPlanner try to support both Java and C++ for parity, and they're often small teams or just a single developer. A tool that can automate bindings for C++ and python for them from their already-written Java code seems like it could be very useful for them. Probably not so much for WPILib, though, since most parts of the library are written in C++ first (eg all the linear algebra and controls math) and later ported to Java.
The few places that are Java-first are mostly things I've contributed - a units API, annotation-based telemetry, a javac plugin, and a continuation-based command framework - all of which are either Java-specific or provide parity with what was already offered in C++ (eg nholthaus units)
2
u/Bahatur 5d ago edited 5d ago
Great Scott, you are doing a robotics competition in Java? What a glorious challenge!
Am I at liberty to assume you have done all the JVM-on-Linux stability tricks, or is it something where everyone is constrained to an identical default setup to keep the playing field level?
Edit: can’t you get a Linux on Arm64 VM up, like with Qemu? That should work on Windows.
1
u/samcarlberg 4d ago
There've been a few tricks, but not many. Teams are constrained to the same hardware, but can do what they like at the OS and software level (they just need to use the standard heartbeat messages etc for safety reasons). Most of the JVM configuration work is for performance on the limited hardware (2-core 667/866Mhz arm32 softfp CPU, 256/512MB of RAM depending on revision): inline string concat, some GC tweaks. The next hardware platform we're getting will be much stronger (essentially a raspberry pi 5 compute module) so we can use ZGC and run faster control loops.
VMs would be a no-go, I think, unless that's something we'd be able to automatically install and maintain. Teams who really care about performance could certainly do it themselves, but most teams aren't performance-sensitive enough to put in that effort
3
u/Electrical_Being_813 4d ago
I would still stick to jit. The only place where I use AOT is for "optional" k8s services that scale down to 0 pods when not used.
1
u/SuspiciousDepth5924 5d ago
Honestly I sort of agree with the guy, at least in principle. The JIT can make your application run faster than it would with AOT, but in my experience it's rarely CPU bound work that is the constraining factor, rather it's usually either memory or network. AOT can't do anything about the network but it does help with memory.
That being said my experience with AOT is mixed, and it's often a bit of a pain to build and get running, I do wish the Java ecosystem would stop using runtime reflection as the magic bullet for every problem (looking at you Spring).
37
u/k-mcm 6d ago edited 6d ago
Well, I'd expect someone from the project say you should use the project.
I honestly haven't had any JIT problems since the Java 5 era. For me, AOT isn't worth the extra effort.
AOT and JIT both have a place in Java, and I hope they both continue to evolve.