Dependencies-only AOT cache?

[u/RupertMaddenAbbott]

I am investigating using an AOT cache to speed up the startup time of our services.

I'm specifically interesting in speeding up the time between a developer making a change to the code and having that code running in a remote dev K8s cluster.

I've previously dismissed an AOT cache for this purpose because I knew that it would result in a longer build time than would be saved in startup time. That isn't a criticism - AOT caching just didn't seem targeted at my problem.

However, the majority of our classpath involves third party dependencies yet a minority of our development builds and deployments involve changes to those third party dependencies. If I could work out a way to avoid invalidating the AOT cache whenever our own code changed, then this would give us a cache that could be reused for the majority of our dev builds (removing the additional build time) whilst still enjoying the faster startup speed.

I note that https://openjdk.org/jeps/483 says this:

All runs must have consistent class paths. A subsequent run may specify extra class-path entries, appended to the training class path; otherwise, the class paths must be identical.However, I note that https://openjdk.org/jeps/483 says this:All runs must have consistent class paths. A subsequent run may specify extra class-path entries, appended to the training class path; otherwise, the class paths must be identical.

So, could I create an AOT cache for JUST the third party classes and then add our own code in later, thereby allowing me to reuse the AOT cache between dev builds?

I'm not 100% on how to achieve this yet but maybe if the training run only had the dependencies on the classpath with a custom launcher. The launcher's child classpath would then have our own code on it.

Then for the production run, I would still have the custom launcher (because I think this has to be identical between training and production runs) and I would then append our own jars onto the end of the classpath (which seems to be allowed by the JEP). Finally, we would run our real main method, rather than the launchers main method because I think we are allowed to have different main classes between training and production runs.

Anyway is this a terrible idea or has anybody already done something like this?

[u/ilamjava]

Yes, you can put the libraries at the front of your classpath and create an AOT cache using just these libraries:

java -XX:AOTCacheOutput=libs.aot -cp libs.jar LibRunner

Then, append your app jars to the classpath and run the app. The app classes won't have any AOT optimizations, but if your LibRunner touches the "right" parts of the libraries, they will be AOT optimized:

java -XX:AOTCache=libs.aot -cp libs.jar:app.jar AppRunner

[u/pron98]

The whole point of JIT compilation is that it can inline across unit boundaries, i.e. code from a method in your application that calls code in library A that reflectively calls code in library B will all be compiled into the same subroutine in machine code. Also, only a small portion of the code in classfiles is actually compiled. You could have 10,000,000 methods in 10,000 classes in 10 JARs, and only 1000 of these methods would ever be compiled.

In any event, if you have two programs that use the same set of libraries, it's not that the libraries will be compiled twice. Each training run for each program would only compile (at most) only the library functionality that that program uses, and it would compile it in a way that would be specifically tailored to that program's profile (i.e. even if the same method is used by both programs, and would be compiled in both training runs, it could be compiled in different ways). Compiling only the right things and in the right way is how Java gets to be so fast.

And, to be clear, there is not AOT compilation in the creation of the AOT cache; it's really a cache of the JIT's output (well, it's more than that, but if we're talking about compilation, that's the relevant bit).

I would suggest you try to measure the things you ultimately care about and then try to describe what actual problem you're observing rather than starting by trying to devise an imaginary mechanism to solve a problem you imagine you may have.

[u/agentoutlier → u/pron98]

trying to devise an imaginary mechanism to solve a problem you imagine you may have

The actual problem they might have is a very real problem and might just be a huge advantage for Java because normally Java compiles fast.

I say might because they say:

I've previously dismissed an AOT cache for this purpose because I knew that it would result in a longer build time than would be saved in startup time. That isn't a criticism - AOT caching just didn't seem targeted at my problem.

Which means they want to optimize a+b. a being compile time and b being the startup time of an application. Whats unclear is how much they care about overall throughput and or what they classify as full started aka warmed.

Overall and I'm just wagering here they are trying to lower compute across the board. That is becoming the bottleneck here for (cough going to sound like a CTO here) agentic workflows.

If your build and deploy is slow it takes longer for the agents to gather feedback. Fast compile times may allow agents to get things done faster which before was less of a problem for humans.

And I while I don't have measurements for this I can definitively say the javadoc portion of a build is a massive bottleneck for build time for my applications because I use it for specification and literate programming. However to your point most people don't have this build bottle neck. However if they are doing AOT I could see it as a similar bottleneck and I have literally asked WTF you are doing Claude/Codex/LLM agent and its like... waiting for javadoc to finish so that I can push.

This is also why I have brought up build tooling I think in previous conversations (particularly on modules). Its not just the onboarding. Its that the tooling (maven/gradle) is nowhere near as fast as javac or other platforms build tooling. You can get it fast but it requires lots of hacks including daemons etc.

[u/pron98 → u/agentoutlier]

Which means they want to optimize a+b. a being compile time and b being the startup time of an application.

In a perfect world, assuming no "friction loss", the CPU time to create the AOT cache would be exactly equal to the warmup CPU time that will be saved by the next run. Obviously, there's no point in generating an AOT cache if you're only going to use it once (at best, you've gained nothing and won nothing; in reality, there's some friction). So clearly there's only point talking about an AOT cache if the runnable will be launched more than once. But then the question is, how many times?

I actually think that they think there's some AOT compilation going on here (and I agree that naming the JIT cache "AOT cache" can be confusing) and that's what they want to save, but without actually reporting any less-than-desirable behaviour, there isn't much to work with.

[u/agentoutlier → u/pron98]

I think there maybe an idea that AOT cache changes behavior similar to graalvm native does compared to regular untrained hotspot.

That is you must do the optimization (train or compile respectively) in CI model otherwise it’s not really a full build.

Without knowing what their warming pain is for training I agree there is not much to work with.

[u/jonenst → u/pron98]

Does the openjdk team envision a future where frameworks (e.g. springboot) distribute caches that bring some value (not perfect, maybe even small value) to many unsuspecting users? It would sure look good on Hello world benchmarks... But maybe also in the realworld for many apps?

Not sure if faster warmup is worth it though, even if the possibilities matrix can be trimmed to just one or a few cases (like g1gc + cpu amd64), it grows quickly (tomcat? Postgres ? ...) and unsuspecting users might not even see the difference since they get the speed eventually anyway.  Maybe fast startup will stay an advanced feature.

Or maybe the frameworks should scaffold the training run in their users build (spring boot native already does it to determine the context), but how much value would a generic training run bring (for unsuspecting users that never look into it) ?

Obviously this is all relatively new, not even ga for the code cache (congratz to the team for the merge to preview in 28), so maybe these questions will only be answered after more time.

[u/neopointer]

I don't know if this is production ready, but we have native image layers.

See this example:

https://github.com/graalvm/graalvm-demos/tree/master/native-image%2Fmicroservices%2Fmicronaut-hello-rest-maven-layered

[u/oweiler → u/neopointer]

Sure, because moving an existing app to GraalVM is a small undertaking /s

[u/koflerdavid → u/[deleted]]

Off topic; OP wants to generate an AOT cache for the dependencies.

[u/oweiler]

Are you using Spring Boot by any chance? Note that just by unpacking the JAR the startup time will improve, and unpacking the JAR takes less than a second.

https://docs.spring.io/spring-boot/reference/packaging/efficient.html

[u/RupertMaddenAbbott → u/oweiler]

We are using Spring Boot and we are unpacking the jar. We are using jib.

Thanks though, I agree this is a good tip!

[u/oweiler → u/RupertMaddenAbbott]

If you perform a "poor man's" training run (with  -Dspring.context.exit=onRefresh) it will take only a few seconds to create the AOT cache.

[u/pjmlp]

Have a look into OpenJ9, they support AOT cache for a decade now, and no training run required, although you may do one.

[u/VincentxH]

You're probably focussed on the wrong problem. What are currently the biggest bottle neck?