There is a smaller, older idea that needs none of it: test impact analysis from a saved dependency map, stored wherever you already store build state. This post shows a complete self-hosted CI architecture for it with blastradius, using either the GitHub Actions cache or a private S3 bucket. I ran both paths end to end on fresh clones so the setup, and the places it breaks, are tested rather than described.

What actually gets stored

Before you decide where it can live, look at what it is. After a base-branch build, blastradius writes an index that maps each test to the production classes it loaded. A single entry looks like this:

{
  "test": { "className": "demo.ParserCase34Test", "methodName": "applies" },
  "dependsOnClasses": ["demo.ParserCase34Test", "demo.Parser"]
}

The whole file also holds the commit SHA the index was built from, a timestamp, direct-invocation records, and a list of "ambient" classes (JDK and framework classes, mostly). It is class names and test names. It is not source code, test output or data. For my 120-test fixture the index was about 175 KB, and about 53 KB compressed in a cache tarball.

That is still metadata about your codebase's structure, so treat it as internal. Two practical rules from the project docs: do not put credentials or other secrets under .blastradius/, and remember that GitHub Actions caches are readable by pull-request workflows.

On "nothing leaves your infrastructure": the tool reads and writes a local directory or the S3 bucket you configure. I searched the plugin, core and S3 modules for hard-coded service endpoints and found none beyond the AWS SDK itself. That is a grep, not an audit, so do your own review if your policy requires one.

The architecture

 push to main                                   pull request
 ─────────────                                  ────────────
 mvn verify                                     restore index
   └─ TRACK: full suite + agent                   └─ SELECT: diff vs merge base
        │                                              │
        ▼                                              ▼
   .blastradius/<main-sha>/index.json ──store──▶  run only matching tests
        (GitHub Actions cache  or  S3 bucket)         (missing index → full suite)

Two jobs, one artifact:

1. Trunk job (TRACK). Runs the full suite with a Java agent attached and writes the index for that commit.
2. PR job (SELECT). Restores the newest trunk index, diffs the PR against its Git merge base, and narrows Surefire/Failsafe to the tests whose recorded classes changed. If no usable index exists, it runs the full suite.

Everything else is plumbing: how the index travels from job 1 to job 2.

Option 1: GitHub Actions cache (no extra infrastructure)

For most teams this is the right default. No bucket, no IAM, nothing outside GitHub.

name: build
on:
  push:
    branches: [main]
  pull_request:

jobs:
  verify:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0            # the merge base must be reachable

      - uses: actions/setup-java@v4
        with:
          distribution: temurin
          java-version: '21'
          cache: maven

      - name: Restore Blastradius index
        id: blastradius-index
        uses: actions/cache/restore@v4
        with:
          path: .blastradius
          key: blastradius-index-${{ runner.os }}-${{ github.sha }}
          restore-keys: |
            blastradius-index-${{ runner.os }}-

      - run: mvn -B --no-transfer-progress verify

      - name: Save Blastradius index (main only)
        if: ${{ github.ref == 'refs/heads/main' && success() && steps.blastradius-index.outputs.cache-hit != 'true' }}
        uses: actions/cache/save@v4
        with:
          path: .blastradius
          key: blastradius-index-${{ runner.os }}-${{ github.sha }}

Why it is shaped this way:

• Save only from successful main builds. PRs read an index made by the trusted base branch and never write one.
• Key by commit SHA. Each main build saves an immutable snapshot. A PR has a new SHA, so the exact lookup misses and restore-keys falls back to the newest main snapshot, which is what you want.
• success() before saving. A failed build should not become tomorrow's baseline.

And the plugin, with the base reference set to origin/main (see the first gotcha below for why):

<plugin>
  <groupId>io.github.baokhang83.blastradius</groupId>
  <artifactId>blastradius-maven-plugin</artifactId>
  <version>0.4.0</version>
  <executions>
    <execution>
      <phase>process-test-classes</phase>
      <goals><goal>select</goal></goals>
    </execution>
  </executions>
  <configuration>
    <baseRef>origin/main</baseRef>
  </configuration>
</plugin>

Option 2: a private S3 bucket

Use S3 when your runners cannot share a reliable workspace cache: self-hosted runners on different machines, several CI systems, or a cache you do not trust to stay warm. The index is written to and read from the bucket directly, so no cache steps are needed:

<configuration>
  <baseRef>origin/main</baseRef>
  <indexStore>s3</indexStore>
  <s3Bucket>ci-dependency-indexes</s3Bucket>
  <s3Prefix>blastradius</s3Prefix>
  <s3Region>eu-central-1</s3Region>
  <!-- <s3Endpoint>https://minio.example.com</s3Endpoint>  optional, S3-compatible stores -->
</configuration>

The object key is derived from the commit: blastradius/.blastradius/<commit-sha>/index.json. Credentials come from the standard AWS credential chain, so nothing secret goes in the POM.

For GitHub Actions, prefer OIDC and a short-lived role over long-lived access keys. The plugin needs only s3:GetObject and s3:PutObject on the prefix, and does not need to list the bucket:

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["s3:GetObject", "s3:PutObject"],
    "Resource": "arn:aws:s3:::<BUCKET>/<PREFIX>/*"
  }]
}

Give the main job a role that can write, and give PR jobs a separate read-only role (s3:GetObject only) if PRs may read the bucket at all. Do not grant either to untrusted fork pull requests; let those fall back to the full suite.

permissions:
  contents: read
  id-token: write

steps:
  - uses: actions/checkout@v4
    with:
      fetch-depth: 0
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::<ACCOUNT_ID>:role/blastradius-index-writer
      aws-region: eu-central-1
  - run: mvn -B --no-transfer-progress verify

What I tested

All runs used a small synthetic fixture (six utility classes, 60 two-test JUnit 5 classes, so 120 tests), Maven 3.9.12, JDK 25 and blastradius 0.4.0. Each job ran in a fresh clone with no shared files, mimicking actions/checkout on an empty runner. The PR changes one line in one utility class.

Caveats, so you read this correctly:

• The cache path was simulated by tarring and untarring .blastradius/, which is what actions/cache does. I did not run it on real GitHub-hosted runners.
• The S3 path ran against a local S3-compatible server (moto), not real AWS S3. I did not test IAM, OIDC role assumption or bucket policies. The endpoint, bucket and prefix behavior is the same code path as MinIO or any S3-compatible store, but verify your IAM setup separately.
• 83.3% reflects an evenly partitioned toy project. The realistic figure is the historical replay: 61.4% fewer test executions across 800 commits on four open-source projects, counted as executions rather than wall-clock time.

The cache-miss row is the safety property that matters for this architecture: losing the index costs you speed, never correctness. A missing, unreadable or mismatched index produces a full run, not a guess.

Gotchas I hit

1. actions/checkout on a PR has no local main, and the build fails. On a pull request, actions/checkout leaves you on a detached commit with only remote-tracking refs. With <baseRef>main</baseRef> the build failed outright:

base reference "main" does not resolve

Setting <baseRef>origin/main</baseRef> fixed it, and it still behaves as the base reference on the push-to-main job, where it is the current commit. Pair it with fetch-depth: 0 so the merge base is reachable. Note that a -DbaseRef=origin/main flag did not override a <baseRef> hard-coded in the POM in my test, so set it in the POM (or make the POM read a property).

2. Cache eviction turns into full runs. GitHub evicts caches that have not been accessed for a while and enforces a per-repository size limit. If main is quiet for long enough, the next PR misses and runs everything. That is safe but slow, so make sure something on main runs regularly (a scheduled workflow works) to keep a fresh snapshot.

3. Tracking can fail without failing the build. If the tracking agent is not attached to the test JVM, the build passes and logs tracking failed ... falling back to the full suite, and no index is saved. I found this with JaCoCo plus an explicit @{argLine} in Surefire; the fix is in the JaCoCo post in this series. Whatever your storage choice, grep the main job's log for [blastradius] TRACK and for the absence of tracking failed.

4. Forks. Fork PRs do not receive secrets or OIDC roles, so they cannot use S3, and they may not restore your cache. They fall back to the full suite, which is the correct behavior.

Cache or S3?

Start with the cache. Move to S3 when eviction, runner topology or access control gives you a reason.

What you give up versus a hosted service

Be honest about the trade. A dependency-based tool like this one is deterministic and self-contained, and it has no model that learns from failure history, no hosted dashboard and no cross-project analytics. You own index freshness, and you own the policy for when to fall back. If those hosted features are what you need, and your security and budget allow it, a hosted product may be the better fit. If your constraint is "no new vendor, no data leaving our accounts", this is a workable alternative.

Either way, keep a regular full-suite run (daily is the recommendation) as a backstop. Selection is sound by default, not a guarantee.

FAQ

Can I run predictive test selection in Maven without a SaaS product?

Yes. Dependency-based test impact analysis needs only a saved map of which classes each test loads. Blastradius builds that map on your base branch and stores it in a local directory, the GitHub Actions cache or an S3 bucket you own.

What data does the index contain?

Test class and method names, the production class names each test loaded, the commit SHA and a timestamp. It does not contain source code, test output or data. It is still internal metadata, so keep it out of public buckets and do not store secrets alongside it.

Do I need S3?

No. The default file store plus a GitHub Actions cache is enough on GitHub-hosted runners. S3 is for runners that cannot share a cache, or when you want IAM-level access control.

What happens if the cache or bucket is empty?

The PR build runs in FALLBACK mode and executes the full suite. Nothing is skipped on a guess.

Why does my PR build fail with "base reference does not resolve"?

actions/checkout on a pull request provides no local main branch. Use origin/main as the base reference and fetch-depth: 0.

Does it work with Gradle?

There is a Gradle plugin in the repository with the same cache and S3 settings, but at the time of writing it is not yet published to the Gradle Plugin Portal. The Maven plugin is on Maven Central.

Is skipping tests safe?

Selection is sound by default but not guaranteed. Reflection, generated code and string-dispatched APIs can escape dependency tracking. Keep a scheduled full-suite run as a safety net.

Disclosure: I maintain blastradius.

This post walks through how blastradius avoids that on pull requests, where the one setup that does collide on the base branch is, and the one-property fix. Everything here comes from a small fixture I built and ran, so the results are reproducible rather than asserted.

Where the collision comes from

JaCoCo's prepare-agent goal writes a -javaagent: option into a Maven property, argLine by default. Surefire then reads that property when it forks the test JVM. A TIA tool that records which classes each test loads needs its own agent in the same fork, so both want to write argLine.

Where they disagree is when each one needs it:

• Recording dependencies needs the agent once, on a trusted baseline build.
• Using the recording needs no agent at all. It only needs to decide which tests to run.

Blastradius is built around that split.

TRACK on the base branch, SELECT on pull requests

TRACK (base branch). The select goal forks a nested mvn clean test with the dependency-recording agent appended to argLine. During that run the agent records, for each test, the production classes it loaded. The result is saved as an index under .blastradius/. This is the only point where blastradius adds a Java agent.

SELECT (pull requests). Blastradius diffs the commit against the Git merge base, compares the changed classes with the saved index, and narrows Surefire with the standard test filter, the same property -Dtest= sets. If nothing is selected it sets skipTests. It does not touch argLine, so your JaCoCo agent, your custom flags, and any other tooling run exactly as they do in a normal build.

That is the part worth emphasising: on the builds developers wait for, the only change is which tests run, not how the test JVM is launched.

What I measured

The fixture is one Maven module with six small utility classes and 60 two-test JUnit 5 classes (120 tests). Each test class exercises exactly one utility class. Tooling: Maven 3.9.12, JDK 25, Surefire 3.5.2, JaCoCo 0.8.13, JUnit 5.10.2, blastradius 0.4.0. It is synthetic on purpose, so treat the numbers as an illustration of behavior, not a benchmark.

Does base-branch tracking survive JaCoCo?

I ran the TRACK build on main with four configurations:

Setup C is the one to watch, because it is also the most common way people write a JaCoCo-plus-custom-flags configuration. With it, blastradius logs:

[WARNING] [blastradius] tracking failed; discarding its incomplete dependency data and falling back to the full suite

The build still passes and the tests still run, so nothing breaks loudly. But no index is saved, which means every pull request build afterwards has nothing to select against and runs the full suite. You keep your coverage and lose the speedup, which is easy to miss.

Why setup C fails

In the forked tracking build, blastradius passes its agent on the command line as -DargLine=.... JaCoCo's prepare-agent reads and writes the project property argLine, which does not include that command-line value. It sets argLine to the JaCoCo agent alone, and an explicit <argLine>@{argLine} ...</argLine> in Surefire then resolves to exactly that. The tracking agent never reaches the test JVM, nothing is recorded, and blastradius correctly refuses to trust an empty recording.

(I confirmed this by running the same nested command by hand: the build logged argLine set to -javaagent:...jacoco... with no tracking agent in it, and no records were written.)

The fix: give JaCoCo its own property

Setup D separates the two concerns. JaCoCo writes to its own property, and your Surefire argLine lists it explicitly next to whatever -DargLine the build was given:

<properties>
  <!-- default so ${argLine} resolves in a normal build -->
  <argLine></argLine>
</properties>

<build>
  <plugins>
    <plugin>
      <groupId>org.jacoco</groupId>
      <artifactId>jacoco-maven-plugin</artifactId>
      <version>0.8.13</version>
      <executions>
        <execution>
          <goals><goal>prepare-agent</goal></goals>
          <configuration>
            <propertyName>jacocoArgLine</propertyName>
          </configuration>
        </execution>
        <execution>
          <id>report</id>
          <phase>verify</phase>
          <goals><goal>report</goal></goals>
        </execution>
      </executions>
    </plugin>

    <plugin>
      <artifactId>maven-surefire-plugin</artifactId>
      <version>3.5.2</version>
      <configuration>
        <argLine>@{jacocoArgLine} ${argLine} -Dcustom.flag=on</argLine>
      </configuration>
    </plugin>
  </plugins>
</build>

In a normal build ${argLine} is the empty default. In blastradius's tracking build it carries the tracking agent. JaCoCo's agent and the tracking agent both end up in the fork. If you do not set a custom argLine at all (setup B), the default behavior already works and you need no change.

What coverage looks like on a pull request

Here is the part to plan for. On a pull request, blastradius runs fewer tests, so JaCoCo reports coverage for fewer tests. The report is valid, not corrupted, but it only reflects what ran.

I measured it on the fixture with a one-line change to MathUtil, using setup D:

The fixture is the extreme case, because each test touches only one class. Real suites overlap far more. But the principle holds in general:

• Project-wide coverage percentages on a PR will drop if you run selection on that build. A jacoco:check rule with a project-wide threshold, evaluated on the PR build, can fail for a reason that has nothing to do with the change.
• Coverage of the changed code is still meaningful, because the tests that load a changed class are the ones that get selected. In the fixture, the changed class MathUtil is fully covered. If you use a diff-coverage tool, point it at the changed lines.
• Treat the base branch as the source of truth for overall coverage. Publish your coverage badge and project-wide gates from the full-suite build on main (the TRACK build is a full run), not from PRs.

If a team needs a project-wide coverage gate on every PR, that team should run the full suite for that gate and use selection only for the fast feedback build. That is a policy decision, not a limitation of the report.

A setup you can copy

1. Use setup B (no explicit argLine) or setup D above for JaCoCo.

2. Add the plugin:

   <plugin>
     <groupId>io.github.baokhang83.blastradius</groupId>
     <artifactId>blastradius-maven-plugin</artifactId>
     <version>0.4.0</version>
     <executions>
       <execution>
         <phase>process-test-classes</phase>
         <goals><goal>select</goal></goals>
       </execution>
     </executions>
     <configuration><baseRef>main</baseRef></configuration>
   </plugin>
   

3. On main, run your normal mvn verify. The first run tracks and saves .blastradius/.

4. Persist .blastradius/ with your CI cache between runs.

5. Check that tracking succeeded: look for [blastradius] TRACK and the absence of tracking failed in the log. A failure is silent as far as the build status goes.

6. On pull requests, run the same command. Blastradius selects tests from the saved index.

7. Keep a regular full-suite run (daily is the recommendation) as a safety net, since selection is sound by default but not guaranteed.

FAQ

Why do test impact analysis tools conflict with JaCoCo?

Both need a Java agent in the forked test JVM, and both usually set it through the argLine property. If either overwrites the other's value, you lose one of the two. The collision is in how the property is written, not in the agents themselves.

Does blastradius change argLine on pull requests?

No. In SELECT mode it sets Surefire's test filter (the same property as -Dtest=), or skipTests when nothing needs to run. It only adds a Java agent in the forked tracking build on the base branch.

My Maven build passes but PRs still run every test. Why?

Check the base-branch log for tracking failed. If your Surefire <argLine> uses @{argLine} with JaCoCo's default prepare-agent, the tracking agent can be overwritten and no index is saved. Switching JaCoCo to its own propertyName as shown above fixes it.

Will my JaCoCo coverage report be correct on pull requests?

It will be valid but partial: it reflects the tests that ran. Project-wide percentages drop when fewer tests run, while coverage of the changed code stays meaningful. Run overall coverage and any project-wide jacoco:check gate on the base-branch full run.

Does this work with Failsafe and integration tests?

Selection narrows both Surefire and Failsafe. I only measured Surefire for this post, so check your integration-test setup with the same TRACK log check before relying on it.

Is selection safe?

It is sound by default, not a guarantee. Reflection, generated code and string-dispatched APIs can escape dependency tracking, so keep a regular full-suite run as a backstop. The first post in this series has the replay numbers: 61.4% fewer test executions across 800 commits, with the exclusions and caveats listed.

Disclosure: I maintain blastradius.

That is not a bug in the cache. It is a granularity mismatch, and it is worth understanding precisely, because the two tools solve different halves of the problem and work well together.

What the build cache does

The extension fingerprints a module's inputs (sources, dependencies, plugin configuration) into a checksum. If it has seen that checksum before, it restores the module's outputs and skips the build. If it has not, it runs the module's whole lifecycle, Surefire included.

So the cache's smallest unit is the module. Anything inside it that changes, even a single character in one class, produces a new checksum for the module. There is no "mostly the same" in a checksum.

For a repository with many small modules that is a great trade. For a repository with one large module (a monolith, a core or common module, a service with its tests in the same artifact), it means a one-line edit costs the module's entire test suite.

Measuring it

I built a small fixture rather than assert this. It is deliberately simple and synthetic, so read the numbers as an illustration of the mechanism, not a benchmark:

• One Maven module, JDK 25 toolchain, Maven 3.9.12, maven-build-cache-extension 1.2.0, Surefire 3.5.2, JUnit 5.10.2.
• Six small utility classes in src/main.
• 60 JUnit 5 test classes, 2 tests each: 120 tests. Each test class exercises exactly one utility class, so every utility has 10 test classes (20 tests) that depend on it.

Steps and results:

The unchanged rebuild is the cache doing its job. Branch A is the problem: StringUtil is exercised by 20 of the 120 tests, and all 120 ran.

The other half: which tests does this change actually reach?

Narrowing inside the module needs information the cache does not have: which tests touch which production classes. blastradius gets it from the JVM instead of guessing from names or packages:

1. Track. On the base branch it runs the suite once with a java.lang.instrument agent attached. Each time a class is loaded while a test runs, it records the pair (test, class). That gives a ground-truth map from every test to the production classes it loaded.
2. Diff. On a change branch it diffs the commit against the Git merge base to find the changed classes.
3. Select. A test is selected if one of its recorded classes changed, if the test is new or modified, if an executed class reaches a changed class through one or two recorded static invocation edges, or if a non-source change forces the conservative full-suite fallback.
4. Gate. The selection becomes a standard Surefire/Failsafe -Dtest= filter.

The same fixture, with blastradius added alongside the cache extension:

The cache still misses, because the module's checksum changed. That is correct, and blastradius does not try to change it. What changes is what the missed build runs: only the 20 tests, from the 10 classes whose recorded dependencies include MathUtil, instead of all 120 (83.3% skipped in this fixture).

The two are complementary:

• The cache removes whole modules that did not change.
• blastradius removes tests inside the modules that did.

Is this just a toy result?

The fixture is evenly partitioned on purpose, so a clean 20/120 is what you would expect. Real suites are messier: shared fixtures, base classes and widely used utilities make some changes reach many tests. That is why I would not quote the fixture number, and why the historical replay matters more.

blastradius has been replayed over 800 consecutive commits on four open-source projects (apache/shenyu, apache/httpcomponents-client, jhy/jsoup, apache/commons-io). Across those replays, the tool selected 966,368 of 2,501,105 test executions: 61.4% fewer, with the per-project range running from 45.0% (jsoup) to 71.0% (shenyu). A module-level cache has to treat each of those commits that touches a module's sources as a full rerun of that module. The replay figures are counts of test executions, not wall-clock CI time, and the full methodology, exclusions and caveats are in the first post in this series and in HISTORICAL_REPLAY_ANALYSIS.md.

Setup notes for using both

Two things I hit while building the fixture, so you do not have to:

1. The base-branch (tracking) build must actually run the tests. Blastradius records dependencies by running the suite in a forked Maven build. If the cache extension restores that build from cache, no tests execute, nothing is recorded, and blastradius reports tracking failed; ... falling back to the full suite. That fallback is safe (you get a full run), but you get no index. The fix that worked for me was to disable the cache for the tracking build with MAVEN_OPTS="-Dmaven.build.cache.enabled=false". Passing the flag only on the outer command line did not reach the forked build.
2. Persist the .blastradius/ directory between runs, the same way you persist the cache. On a change branch, blastradius needs the saved index from the base branch; without one it runs the full suite.

Example plugin configuration, as used in the fixture:

<plugin>
  <groupId>io.github.baokhang83.blastradius</groupId>
  <artifactId>blastradius-maven-plugin</artifactId>
  <version>0.4.0</version>
  <executions>
    <execution>
      <phase>process-test-classes</phase>
      <goals><goal>select</goal></goals>
    </execution>
  </executions>
  <configuration><baseRef>main</baseRef></configuration>
</plugin>

Where this does not help

• Modules that are already small. If your tests are split across many modules and each module's tests are fast, the cache is already close to optimal.
• Changes that reach everything. A change to a class loaded by every test selects nearly every test, and a build-file or resource change triggers the full-suite fallback by design.
• Code reached through reflection or string-dispatched APIs. Selection is sound by default, not guaranteed. Selector engines, expression languages and query parsers are the known hard case (jsoup showed this in the replay and drove a fix), so keep a regular full-suite run, daily is the recommendation, as a backstop.

FAQ

Why does Maven Build Cache re-run all tests after a one-line change?

The extension caches at module granularity. Any change to a module's inputs produces a new checksum, so the module is rebuilt in full, including all of its Surefire tests. The cache has no record of which tests depend on which classes.

Can the Maven Build Cache Extension skip individual tests?

Not by itself: it restores or rebuilds whole modules. To run only the tests a change can affect, you need per-test dependency information, which is what test impact analysis tools such as blastradius record.

How does blastradius know which tests a class change affects?

On the base branch a Java agent records every class each test loads at runtime. On a change branch, tests whose recorded classes include a changed class are selected, along with new or modified tests and a bounded set of tests that reach the changed class through one or two static invocation edges.

Do I have to choose between the cache and blastradius?

No. The cache skips unchanged modules; blastradius narrows tests inside changed ones. One setup note: run the base-branch tracking build with the cache disabled, otherwise the restored build runs no tests and nothing is recorded.

Is it safe to skip tests?

Selection is sound by default but not a guarantee. Reflection, generated code and string-dispatched APIs can escape it. Run the full suite on a regular cadence as a safety net; the recommendation is daily.

Does 83% fewer tests mean 83% faster builds?

No. The fixture counts tests run, not time, and it is synthetic. Real savings depend on which tests are slow, fixed costs such as compilation and packaging, and how widely your changes reach.

What do I need to try it?

Maven, JDK 21 or later, a Git repository and JUnit 5. The plugin is on Maven Central as io.github.baokhang83.blastradius:blastradius-maven-plugin. The fixture in this post is easy to recreate: six utility classes, 60 two-test classes with one per utility, and the two plugins configured as shown.

Disclosure: I am the author of blastradius.

The short answer

• Cause: the kubelet will not apply a memory-limit decrease while the container's current usage is above the new limit. A JVM's usage stays high after garbage collection because it keeps the heap committed.
• Symptom: condition PodResizeInProgress=True with reason Error. containerStatuses[].allocatedResources shows the new value, but containerStatuses[].resources still shows the old one.
• It retries. The kubelet retried about every 11 to 15 seconds. The pod was not OOM-killed and did not restart. The check protected the JVM.
• Fix: make the JVM release memory, then let the resize finish. Releasing after the patch finished the resize on the next retry (about 20 s in my run). Releasing before the patch finished it 2 seconds after the patch, and the error never appeared.
• PodResizePending is a different problem. I saw it only when I asked for more memory than the node has (Infeasible).

What it looks like

I ran a JVM pod on a single-node kind cluster, Kubernetes v1.35.8. The pod has Guaranteed QoS with a 900Mi memory request and limit. The JVM (Temurin 21, ZGC, -Xmx600m) allocates 300 MB, holds it for 10 seconds, then drops all of it and idles. Inside the container, memory.current read 417 MB.

I shrank it to 300Mi through the resize subresource:

kubectl patch pod jvm --subresource resize -p \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"memory":"300Mi"},"limits":{"memory":"300Mi"}}}]}}'

A few minutes later, kubectl get pod jvm -o yaml (trimmed) showed:

status:
  conditions:
  - message: 'cannot decrease memory limits: [attempting to set pod memory limit (314572800)
      below current usage (435347456), attempting to set container "app" memory limit
      (314572800) below current usage (435347456)]'
    observedGeneration: 2
    reason: Error
    status: "True"
    type: PodResizeInProgress
  containerStatuses:
  - name: app
    allocatedResources:
      cpu: 500m
      memory: 300Mi        # what the kubelet accepted
    resources:
      limits:
        cpu: 500m
        memory: 900Mi      # what is actually applied
    restartCount: 0

At +10 s the condition was PodResizeInProgress=True with no reason. By +31 s the reason was Error. The cgroup never changed: memory.max stayed at 900 MB, and memory.current stayed at 417 MB for the whole minute I watched.

The events show a repeating error, and the kubelet log says why:

Normal   ResizeStarted   Pod resize started: {... "memory":"300Mi" ... "generation":2}
Warning  ResizeError     Pod resize error: {... "error":"cannot decrease memory limits: [attempting to set pod memory limit (314572800) below current usage (435347456) ...]"}
Warning  ResizeError     (repeats)

kubelet: "Allocated pod resize is not currently feasible" err="cannot decrease memory limits: ..."
kubelet: ... ResizePodInPlaceError ... (retries at 18:50:18, :33, :47, :58, 18:51:13, :25)

That matches what the Kubernetes docs say: before decreasing a container's memory limit, if memory usage exceeds the requested limit, the resize is skipped and the status remains in an "In Progress" state. They call this best-effort because usage can spike right after the check. What the docs don't show is the Error reason and the repeating ResizeError events, which look alarming but are the kubelet waiting.

Why a JVM does this

The kubelet compares the container's current memory usage with the new limit. A JVM's usage is dominated by committed heap, and the JVM does not return committed heap just because the objects in it are gone.

In my test the JVM sat idle for more than a minute with ZUncommitDelay=1 and gave nothing back. The likely reason is that no garbage-collection cycle ran after the data was dropped, so there was nothing to trigger an uncommit. I did not confirm that, and the behavior depends on the collector. Don't count on a JVM to release by itself.

PodResizeInProgress vs PodResizePending

If you see Infeasible or Deferred, the problem is node capacity. If you see InProgress plus "below current usage", the problem is the container's own memory, and the fix is below.

The fix: release memory from inside the JVM

On ZGC, lower the soft ceiling and force a collection:

kubectl exec jvm -- jcmd 1 VM.set_flag SoftMaxHeapSize 104857600
kubectl exec jvm -- jcmd 1 GC.run

Releasing while the resize was pending. I ran the two commands against the stuck pod. memory.current fell from 417 MB to 96 MB. The kubelet applied the resize on its next retry, 20 seconds after I started the release. memory.max became 300 MB, restartCount stayed 0, the condition cleared, and a ResizeCompleted event appeared.

Releasing before the patch. I recreated the pod, released first (424 MB → 101 MB), then patched. The resize applied 2 seconds after the patch. For this pod, the events were ResizeStarted and ResizeCompleted only. The Error condition never appeared.

Releasing first is cleaner: no error condition, no wait for a retry. But if you are already stuck, you don't need to re-patch. Free the memory and the kubelet finishes the job.

When it stays stuck

The resize stays "In Progress" if the JVM can't get below the new limit:

• Live data is bigger than the new limit. If the heap holds more live objects than the limit, I'd expect no collection to free enough, and the resize to stay pending. I didn't test this case. Pick a limit that fits.
• The ceiling is fixed. -Xmx can't change on a running JVM. The new limit must be above -Xmx plus metaspace, code cache, thread stacks and direct buffers. If it isn't, you need a restart, which is what resizePolicy: RestartContainer is for.
• Memory outside the heap. SoftMaxHeapSize only moves the heap. Native memory and direct buffers stay.

A debugging checklist

# 1. Which condition? InProgress or Pending, and the reason
kubectl get pod <pod> -o jsonpath='{range .status.conditions[?(@.type=="PodResizeInProgress")]}{.reason}: {.message}{"\n"}{end}'
kubectl get pod <pod> -o jsonpath='{range .status.conditions[?(@.type=="PodResizePending")]}{.reason}: {.message}{"\n"}{end}'

# 2. What is accepted vs actually applied
kubectl get pod <pod> -o jsonpath='allocated={.status.containerStatuses[0].allocatedResources.memory} applied={.status.containerStatuses[0].resources.limits.memory}{"\n"}'

# 3. What the container is really using
kubectl exec <pod> -- sh -c 'cat /sys/fs/cgroup/memory.current /sys/fs/cgroup/memory.max'

# 4. The events
kubectl get events --field-selector involvedObject.name=<pod>

If memory.current is above the new limit, the JVM is the thing to fix.

One gotcha from my own test: use the default strategic merge patch, as in the command above. With --type merge, the whole containers list is replaced, and the API server rejects it with "resource requests cannot be removed".

What I didn't test

• Collectors other than ZGC in a cluster. I tested ZGC, Shenandoah and G1 in a container separately, in another article, but not through a kubelet.
• A multi-container pod, or any version other than v1.35.8.
• Whether the retry interval changes under load, or what happens if usage spikes right after the kubelet's check.
• How long a release takes on a large heap under real traffic.

Reproduce it

You need Docker, kind and kubectl. Hold.java:

import java.util.*;
public class Hold {
  public static void main(String[] a) throws Exception {
    List<byte[]> l = new ArrayList<>();
    for (int i = 0; i < 300; i++) { byte[] b = new byte[1024*1024]; Arrays.fill(b,(byte)1); l.add(b); }
    System.out.println("allocated 300MB"); System.out.flush();
    Thread.sleep(10000);
    l.clear(); l = null;
    System.out.println("released refs"); System.out.flush();
    Thread.sleep(600000);
  }
}

pod.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: jvm
spec:
  containers:
  - name: app
    image: eclipse-temurin:21-jdk
    imagePullPolicy: IfNotPresent
    command: ["java","-XX:+UseZGC","-XX:ZUncommitDelay=1","-Xmx600m","/w/Hold.java"]
    volumeMounts: [{name: w, mountPath: /w}]
    resources:
      requests: {cpu: 500m, memory: 900Mi}
      limits:   {cpu: 500m, memory: 900Mi}
    resizePolicy:
    - {resourceName: memory, restartPolicy: NotRequired}
    - {resourceName: cpu, restartPolicy: NotRequired}
  volumes:
  - name: w
    configMap: {name: hold}

kind create cluster --name resize-test --image kindest/node:v1.35.8
kubectl create configmap hold --from-file=Hold.java
kubectl apply -f pod.yaml
kubectl wait --for=condition=Ready pod/jvm --timeout=240s
# wait ~15 s until the JVM logs "released refs", then:
kubectl patch pod jvm --subresource resize -p \
  '{"spec":{"containers":[{"name":"app","resources":{"requests":{"memory":"300Mi"},"limits":{"memory":"300Mi"}}}]}}'
kubectl get pod jvm -o yaml        # PodResizeInProgress, reason Error
kubectl exec jvm -- jcmd 1 VM.set_flag SoftMaxHeapSize 104857600
kubectl exec jvm -- jcmd 1 GC.run  # the kubelet finishes on its next retry
kind delete cluster --name resize-test

The first image pull took about 90 seconds on my machine. The megabyte numbers will differ a little on yours.

FAQ

Why is my Kubernetes pod resize stuck in progress?

The kubelet will not lower a memory limit below the container's current usage, and it keeps retrying. Check kubectl get pod <pod> -o yaml for PodResizeInProgress with a message saying "below current usage". Free memory inside the container and the resize finishes.

What is the difference between PodResizePending and PodResizeInProgress?

PodResizePending means the resize can't start yet. I saw it with reason=Infeasible when the request exceeded the node's capacity. PodResizeInProgress means the kubelet accepted the resize and is applying it. For a memory decrease it stays that way while usage is above the new limit.

Why won't the kubelet complete an in-place memory decrease for my Java pod?

A JVM keeps its committed heap after garbage collection, so the container's usage stays above the new limit. On ZGC or Shenandoah, lower SoftMaxHeapSize and run GC.run. On G1, use GC.run or G1PeriodicGCInterval.

Will the pod be OOM-killed while the resize is stuck?

Not in my test. The pod kept running at 900Mi, with 0 restarts, until I freed memory. The Kubernetes docs say the check is best-effort, with no guarantees.

Do I need to patch again after freeing memory?

No. The kubelet retries on its own. In my run it completed 20 seconds after I released memory, with no new patch.

How long does the kubelet take to apply the resize after the JVM releases memory?

20 seconds in the run where I released after the patch (the kubelet was retrying about every 11 to 15 seconds). If you release before the patch, it applied 2 seconds after.

Where Warden fits

I maintain mnemo-jvm-warden, an open-source Kubernetes controller and sidecar that does the release-then-resize order. The sidecar lowers the soft heap ceiling where the collector supports it, forces GC.run, and checks that the process's resident memory dropped. Only then does the resize reach the kubelet. That order is the second case in this article, where the error never appears. I haven't tested that exact path in the cluster in this article.

You don't need a tool for the fix: the two jcmd commands above work on their own.

The short answer

• ZGC and Shenandoah: SoftMaxHeapSize works at runtime. Lower it, run GC.run, and memory drops.
• G1: SoftMaxHeapSize is accepted by jcmd and has no effect. By default nothing returns memory. Two things did, in my test: an explicit GC.run, and G1PeriodicGCInterval set to a few seconds.
• Release speed: I measured no real difference. Each collector released a 300 MB working set within about 1 to 7 seconds. I did not measure large heaps or busy applications, which is where I'd expect differences, so treat "G1 is slower" as unproven.
• The real G1 difference is the missing ceiling. Nothing holds G1's heap below a target after you release, so you size -Xmx for the peak and accept that it can grow back.
• Lowering the limit first killed all three (OOMKilled, exit 137). Release first, then resize.

What each collector gives you

The test

The setup is the same as in my earlier test of RestartContainer: Eclipse Temurin 21.0.13 in a Linux container on Docker Desktop, cgroup v2, and a small program that allocates and touches 300 MB, holds it for 10 seconds, then drops all of it and idles. docker update --memory=300m lowers the cgroup limit on the running container in place. That is the kernel mechanism a pod resize uses, but there is no kubelet.

For each collector, I ran two cases. Naive: lower the limit straight away. Ordered: set SoftMaxHeapSize=100MB, run GC.run, wait until memory.current falls, then lower the limit.

G1 passes the ordered case, but I want to be exact about why. In that run I set SoftMaxHeapSize too, and it did nothing. The explicit GC.run did the work.

What actually released memory on G1

I tested each G1 lever on its own, starting from about 460 to 530 MB in use:

SoftMaxHeapSize is listed as a manageable flag on G1 in JDK 21, so jcmd VM.set_flag returns "Command executed successfully". That success message is misleading. Nothing in G1 reads the flag. If your automation sets the flag and then checks that the command succeeded, it will believe it lowered the ceiling on G1 when it did not.

How to make G1 release memory back to the OS

Pick one of these. I tested the first two.

1. Run GC.run from outside. jcmd <pid> GC.run triggers a full GC, after which G1 shrinks its heap. It is immediate, and it is a stop-the-world collection, so its pause grows with the live set.
2. Enable periodic GC. G1PeriodicGCInterval is a manageable flag, default 0 (off). Set it at startup, or live: jcmd <pid> VM.set_flag G1PeriodicGCInterval 3000 (milliseconds). In my test, memory fell within 7 seconds on an idle JVM. A related flag, G1PeriodicGCSystemLoadThreshold, is also manageable and, per JEP 346, can hold the periodic GC back while the system is busy. I did not test it.
3. Tune MinHeapFreeRatio and MaxHeapFreeRatio. These are manageable and control how much free heap G1 keeps after a collection. I did not test them here.

What this means for in-place resize

• Release before you resize, on every collector. The failure was identical across all three: lowering the limit while the JVM held committed memory got it killed.
• Don't trust "command succeeded" on G1. Verify that memory.current actually dropped, then resize. In Kubernetes, the kubelet's check also compares current usage to the new limit and holds the resize while usage is higher, although the docs say that check is best-effort.
• Plan the ceiling on G1. With ZGC and Shenandoah, SoftMaxHeapSize keeps the heap near your target after release. On G1 nothing does, so a G1 heap that was just released can grow back toward -Xmx before the limit drops. I did not test how fast. The practical rule is the one from my earlier article: the smallest limit must exceed -Xmx plus non-heap memory, and on G1 you cannot rely on a soft ceiling to enforce it.
• Mind the delay on ZGC. ZUncommitDelay defaults to 300 seconds. I shortened it to 1 second for the test. Without that, a ZGC release can take minutes unless you also lower the soft ceiling and wait for the uncommit.

What I didn't test

• Heap sizes above about 600 MB, live data that doesn't die, or JVMs under load. A full GC on G1 can cost far more at scale, and I haven't measured it.
• Whether G1 re-expands after release.
• G1PeriodicGCSystemLoadThreshold, MinHeapFreeRatio and MaxHeapFreeRatio.
• A real kubelet. I used docker update.
• G1 with the default region size. I set 4 MB regions because my test allocates 1 MB arrays, which are "humongous" objects in G1's default configuration, and the first G1 run ran out of heap.

Reproduce it

You need Docker. Save the files below in an empty folder.

Hold.java:

import java.util.*;
public class Hold {
  public static void main(String[] a) throws Exception {
    List<byte[]> l = new ArrayList<>();
    for (int i = 0; i < 300; i++) { byte[] b = new byte[1024*1024]; Arrays.fill(b,(byte)1); l.add(b); }
    System.out.println("allocated 300MB"); System.out.flush();
    Thread.sleep(10000);
    l.clear(); l = null;
    System.out.println("released refs"); System.out.flush();
    Thread.sleep(600000);
  }
}

run.sh runs the naive and ordered cases for any collector:

#!/bin/bash
# usage: ./run.sh <naive|ordered> <jvm flags...>      (run from the folder containing Hold.java)
# Example: LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1
set -u
mode=$1; shift; n=warden-test
docker rm -f $n >/dev/null 2>&1
docker run --rm -v "$PWD":/w -w /w eclipse-temurin:21-jdk javac Hold.java
docker run -d --name $n --memory=${LIM:-700}m --memory-swap=${LIM:-700}m -v "$PWD":/w -w /w \
  eclipse-temurin:21-jdk java "$@" -Xmx${XMX:-500}m Hold >/dev/null
for i in $(seq 1 60); do docker logs $n 2>&1 | grep -q 'released refs' && break; sleep 1; done; sleep 2
mem() { echo $(( $(docker exec $n cat /sys/fs/cgroup/memory.current) / 1048576 )); }
echo "memory.current after the app dropped its data: $(mem) MB"
if [ "$mode" = ordered ]; then
  docker exec $n jcmd 1 VM.set_flag SoftMaxHeapSize 104857600   # 100 MB soft ceiling
  docker exec $n jcmd 1 GC.run
  for i in $(seq 1 30); do [ "$(mem)" -lt 250 ] && break; sleep 1; done
  echo "memory.current after SoftMaxHeapSize + GC.run: $(mem) MB"
fi
docker update --memory=300m --memory-swap=300m $n        # in-place cgroup change, no restart
sleep 8
docker inspect -f 'running={{.State.Running}} OOMKilled={{.State.OOMKilled}} exit={{.State.ExitCode}}' $n
docker rm -f $n >/dev/null 2>&1

chmod +x run.sh
LIM=900 XMX=600 ./run.sh naive   -XX:+UseG1GC -XX:G1HeapRegionSize=4m
LIM=900 XMX=600 ./run.sh ordered -XX:+UseG1GC -XX:G1HeapRegionSize=4m
LIM=900 XMX=600 ./run.sh naive   -XX:+UseZGC -XX:ZUncommitDelay=1
LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1
LIM=700 XMX=500 ./run.sh naive   -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000
LIM=700 XMX=500 ./run.sh ordered -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000

g1-probe.sh tests each G1 lever on its own:

#!/bin/bash
# Which G1 lever actually returns memory? Run from the folder containing Hold.java.
set -u
docker run --rm -v "$PWD":/w -w /w eclipse-temurin:21-jdk javac Hold.java
start() { docker rm -f $1 >/dev/null 2>&1
  docker run -d --name $1 --memory=900m --memory-swap=900m -v "$PWD":/w -w /w eclipse-temurin:21-jdk \
    java -XX:+UseG1GC -XX:G1HeapRegionSize=4m -Xmx600m Hold >/dev/null
  for i in $(seq 1 60); do docker logs $1 2>&1 | grep -q 'released refs' && break; sleep 1; done; sleep 2; }
mem() { echo $(( $(docker exec $1 cat /sys/fs/cgroup/memory.current) / 1048576 )); }

echo "1) nothing (default), wait 25s";           start g1; b=$(mem g1); sleep 25; echo "   $b MB -> $(mem g1) MB"
echo "2) SoftMaxHeapSize=100MB only, wait 20s";  start g1; b=$(mem g1)
docker exec g1 jcmd 1 VM.set_flag SoftMaxHeapSize 104857600 >/dev/null; sleep 20; echo "   $b MB -> $(mem g1) MB"
echo "3) GC.run only";                           start g1; b=$(mem g1)
docker exec g1 jcmd 1 GC.run >/dev/null; sleep 4; echo "   $b MB -> $(mem g1) MB"
echo "4) G1PeriodicGCInterval=3000 set live";    start g1; b=$(mem g1)
docker exec g1 jcmd 1 VM.set_flag G1PeriodicGCInterval 3000 >/dev/null
for i in $(seq 1 40); do [ "$(mem g1)" -lt 250 ] && break; sleep 1; done; echo "   $b MB -> $(mem g1) MB (after ${i}s)"
docker rm -f g1 >/dev/null 2>&1

Expect OOMKilled=true exit=137 for naive runs and running=true for ordered ones. The megabyte numbers will differ a little on your machine.

FAQ

Which garbage collector is best for Kubernetes in-place pod resize?

In my test, ZGC and Shenandoah are the easiest, because SoftMaxHeapSize changes their behavior and keeps the heap near a target. G1 also works if something triggers a release, but it has no soft ceiling. Pick the collector your latency requirements need first, and then use the matching release method.

How do I make G1 release memory back to the OS?

Run jcmd <pid> GC.run, or set G1PeriodicGCInterval to a value in milliseconds (default 0, off). In my test both returned memory within a few seconds on a small idle heap.

Does SoftMaxHeapSize work with G1?

No. jcmd VM.set_flag SoftMaxHeapSize succeeds on G1 in JDK 21, but I saw no change in memory over 20 seconds, so I conclude G1 ignores it. It works on ZGC and Shenandoah.

Why was my pod OOMKilled after an in-place memory decrease?

The limit dropped while the JVM still held more committed memory than the new limit allows. In my test this killed the JVM on G1, ZGC and Shenandoah every time. Release memory first, check the resident memory, then lower the limit.

Is GC.run safe to use in production?

It triggers a full collection. On G1 that is stop-the-world, and its pause grows with the live set. I only tested a 600 MB heap on an idle JVM, so measure the pause on your own heap before relying on it.

Do I still need RestartContainer?

Only if the heap ceiling has to move. See my earlier article on RestartContainer for the case where it is and isn't required.

Where Warden fits

I maintain mnemo-jvm-warden, an open-source Kubernetes controller and sidecar that does the release-then-resize order for you. From its code and docs, it adjusts to the collector like this:

• Detection: the agent reads the target's GC beans and records, per collector, whether a soft max and uncommit are supported. ZGC and Shenandoah support both. G1 supports uncommit only. Serial, Parallel and Epsilon are treated as unsupported, and the agent stays read-only for them.
• Release: it sets SoftMaxHeapSize only where the collector has it, so on G1 that step is skipped instead of being sent and ignored. It then forces GC.run and polls the committed heap until it drops and holds steady. It checks the process's resident memory before it shrinks the pod.
• Boost on G1: a G1 workload that wants a Boost lease must already launch with -Xmx covering the largest heapTarget it will request, because there is no soft ceiling to raise later. The agent rejects a heap target above the launch-time -Xmx for every collector, and the Boost docs note that G1 has no live soft-max knob, so a G1 Boost only ever confirms that the allocation change landed.

The conclusions above do not depend on any tool. The jcmd commands work without it.

The short version

• Kubernetes 1.35 can change a running container's memory in place (the pods/resize subresource is GA).
• A JVM fixes its maximum heap at startup. Raising the limit in place does not raise the heap. Two popular articles are right about this.
• That does not mean every memory change needs a restart. Three different operations hide inside "resize memory", and only one of them requires it.
• On ZGC and Shenandoah you can lower and raise the heap's soft ceiling at runtime. In my test, releasing memory first let a container's limit drop from 900 MB (or 700 MB) to 300 MB with no restart. Lowering the limit first got the JVM OOM-killed every time.

What the advice says, and where it's right

Rack2Cloud's VPA in production article says: "-Xmx is a startup parameter. The JVM reads it once at initialization and sets the maximum heap ceiling for the lifetime of the process." It concludes that resizePolicy: RestartContainer for memory "is the correct setting for JVM services, not NotRequired". Its Kubernetes 1.35 article lists RestartContainer as appropriate "for memory decreases on workloads where the allocator won't release memory without a restart".

Shubham Chepe's post on the JVM and in-place resize gives the same prescription: "CPU NotRequired, memory RestartContainer". He explains why: "If you raise the memory limit in place, the heap ceiling doesn't grow." On shrinking he says it is "riskier still, since the JVM may already be using more than the new limit allows."

Both are correct that:

• the heap ceiling comes from -Xmx or -XX:MaxRAMPercentage at launch, and
• a bigger cgroup limit does not make a running JVM use it.

If you need the heap ceiling itself to move, you need a restart. This article doesn't dispute that.

Three operations, not one

The advice treats all three as the first one. That is the right call if you size -Xmx to today's limit. It is the wrong one if you set -Xmx for the peak and treat memory below that as adjustable.

ZGC and Shenandoah make this possible because of SoftMaxHeapSize, a manageable flag. You can change it on a running JVM:

jcmd <pid> VM.set_flag SoftMaxHeapSize <bytes>
jcmd <pid> GC.run

The collector tries to stay under the soft limit and uncommits what it frees. -Xmx remains the hard ceiling.

The test

I wanted to see what actually happens, so I ran it. The setup:

• Eclipse Temurin 21.0.13 in a Linux container on Docker Desktop, on cgroup v2 (/sys/fs/cgroup/memory.current was present).
• A small program that allocates and touches 300 MB, holds it for 10 seconds, then drops all references and idles.
• docker update --memory=300m --memory-swap=300m to lower the cgroup limit on the running container. This changes memory.max in place, which is the same kernel mechanism a pod resize uses, but it is not the kubelet.
• Two runs per collector. Naive: lower the limit straight away. Ordered: set SoftMaxHeapSize=100MB, run GC.run, wait for memory.current to fall, and only then lower the limit.
• Uncommit delays were shortened (ZUncommitDelay=1, ShenandoahUncommitDelay=1000) so the test finishes quickly. Defaults are longer.

Results:

With the ordered sequence, memory.max read back as 314572800 (300 MB) and the container never restarted.

What the kubelet adds

My test used Docker, so it had no kubelet. In a real cluster, the kubelet tries to avoid an OOM kill when a memory limit goes down: according to the Kubernetes docs, if current usage is above the requested limit, the resize is skipped and stays "In Progress". The docs call this best-effort, with no guarantee, and note a race where usage can spike right after the check.

For a JVM, that has a practical consequence. Committed heap counts as memory in use even when most of it is free, so the resize usually sits pending until the JVM gives memory back. Releasing first is what gets usage under the new limit, so the kubelet applies the resize. The ordered sequence works with the kubelet's check, not against it.

When RestartContainer is still the right answer

• You need the heap ceiling to change. -Xmx is fixed per process.
• You run G1 and haven't enabled periodic GC. G1 has no soft ceiling and returns heap mainly at a full GC or concurrent cycle, so release is slower and less predictable. I didn't test it here.
• You can accept the restart. A restart discards compiled code, loaded classes, pools and caches, and a service with few replicas loses capacity while it warms up. If that is fine for you, RestartContainer is simpler and needs no extra tooling.
• Your limit is already close to -Xmx plus non-heap memory. Shrinking safely needs room: the smallest limit must exceed -Xmx plus metaspace, code cache, thread stacks and direct buffers.

What this test does not show

• It is a synthetic program, not a production service, and I tested ZGC and Shenandoah on one JDK build.
• ZGC and Shenandoah used slightly different heap and limit sizes (600/900 MB and 500/700 MB). My first ZGC run at 500 MB ran out of heap while filling 300 MB, so I gave it more room.
• It was not run through a kubelet. The kubelet's check will change what you see on a real cluster.
• I haven't measured how long a release takes on a large heap under real load.

Reproduce it

You need Docker. Save the two files below in an empty folder.

Hold.java allocates and touches 300 MB, holds it for 10 seconds, then drops it:

import java.util.*;
public class Hold {
  public static void main(String[] a) throws Exception {
    List<byte[]> l = new ArrayList<>();
    for (int i = 0; i < 300; i++) { byte[] b = new byte[1024*1024]; Arrays.fill(b,(byte)1); l.add(b); }
    System.out.println("allocated 300MB"); System.out.flush();
    Thread.sleep(10000);
    l.clear(); l = null;
    System.out.println("released refs"); System.out.flush();
    Thread.sleep(600000);
  }
}

run.sh starts a container, optionally releases memory first, then lowers the cgroup limit in place:

#!/bin/bash
# usage: ./run.sh <naive|ordered> <jvm flags...>      (run from the folder containing Hold.java)
# Example: LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1
set -u
mode=$1; shift; n=warden-test
docker rm -f $n >/dev/null 2>&1
docker run --rm -v "$PWD":/w -w /w eclipse-temurin:21-jdk javac Hold.java
docker run -d --name $n --memory=${LIM:-700}m --memory-swap=${LIM:-700}m -v "$PWD":/w -w /w \
  eclipse-temurin:21-jdk java "$@" -Xmx${XMX:-500}m Hold >/dev/null
for i in $(seq 1 60); do docker logs $n 2>&1 | grep -q 'released refs' && break; sleep 1; done; sleep 2
mem() { echo $(( $(docker exec $n cat /sys/fs/cgroup/memory.current) / 1048576 )); }
echo "memory.current after the app dropped its data: $(mem) MB"
if [ "$mode" = ordered ]; then
  docker exec $n jcmd 1 VM.set_flag SoftMaxHeapSize 104857600   # 100 MB soft ceiling
  docker exec $n jcmd 1 GC.run
  for i in $(seq 1 30); do [ "$(mem)" -lt 250 ] && break; sleep 1; done
  echo "memory.current after SoftMaxHeapSize + GC.run: $(mem) MB"
fi
docker update --memory=300m --memory-swap=300m $n        # in-place cgroup change, no restart
sleep 8
docker inspect -f 'running={{.State.Running}} OOMKilled={{.State.OOMKilled}} exit={{.State.ExitCode}}' $n
docker rm -f $n >/dev/null 2>&1

Then run the four cases:

chmod +x run.sh
LIM=900 XMX=600 ./run.sh naive   -XX:+UseZGC -XX:ZUncommitDelay=1
LIM=900 XMX=600 ./run.sh ordered -XX:+UseZGC -XX:ZUncommitDelay=1
LIM=700 XMX=500 ./run.sh naive   -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000
LIM=700 XMX=500 ./run.sh ordered -XX:+UseShenandoahGC -XX:+UnlockExperimentalVMOptions -XX:ShenandoahUncommitDelay=1000

Expect OOMKilled=true exit=137 for the naive runs and running=true OOMKilled=false for the ordered ones. The exact megabyte numbers will differ a little on your machine.

FAQ

Do I need resizePolicy: RestartContainer for JVM memory?

Only if the heap ceiling has to move. A JVM fixes its maximum heap at startup, so growing beyond the -Xmx it launched with needs a restart. Shrinking, and growing back up to the launch ceiling, did not need one on ZGC or Shenandoah in my test, as long as the JVM released memory before the limit dropped.

Why doesn't raising the container's memory limit give the JVM more heap?

The JVM computes its maximum heap once, from -Xmx or from -XX:MaxRAMPercentage applied to the limit it saw at startup. A later in-place change to the cgroup limit does not change that number.

Can I change -Xmx on a running JVM?

No. What you can change on ZGC and Shenandoah is the soft ceiling, SoftMaxHeapSize, which sits at or below -Xmx.

What happens if I lower the memory limit without releasing memory first?

In my test the JVM was OOM-killed (exit code 137) every time, on both ZGC and Shenandoah. On a real cluster the kubelet makes a best-effort attempt to avoid this and may keep the resize pending while usage is above the new limit. The Kubernetes docs say this check gives no guarantees, because usage can spike right after it.

How do I make a JVM give memory back without restarting it?

On ZGC and Shenandoah, lower the soft ceiling and force a collection:

jcmd <pid> VM.set_flag SoftMaxHeapSize <bytes>
jcmd <pid> GC.run

Then check that the process's resident memory dropped before you lower the limit. The collector's uncommit delay applies: I shortened it for the test, and the defaults are longer.

Does this work with G1?

I didn't test it. G1 has no soft heap ceiling, so you cannot use the same flag. Plan for a slower, less predictable release, or use RestartContainer.

What is the default resizePolicy?

NotRequired for both CPU and memory, which means the change is applied without restarting the container. Setting RestartContainer for memory restarts the container, not the whole pod.

How small can the memory limit go?

The smallest limit must stay above -Xmx plus non-heap memory (metaspace, code cache, thread stacks, direct buffers) plus working-set headroom. For a typical Spring Boot service the non-heap part is commonly 100 to 150 MiB.

Where Warden fits

I maintain mnemo-jvm-warden, an open-source Kubernetes controller and sidecar that runs this order for you: lower the soft ceiling, force a GC, verify the process's resident memory dropped, and only then shrink the pod in place. To grow, it reverses the order. It does not change -Xmx, so the same sizing rule applies. There is a guide with the full reasoning, and a walkthrough on a kind cluster.

The point of this article doesn't depend on any tool. The jcmd commands above work without it.