Java Issues & Solutions, OpenShift, Openshift Issues & Solutions, Springboot Issues & Solutions

[Solved] OpenShift : MetaSpace Issue with SpringBoot based Micro-services

Leave a comment

The java.lang.OutOfMemoryError: Metaspace indicates that allocated native memory for Java class metadata is exausted. That’s why issue occured in standalone and cloud based applications.

In Java 8 and later versions, the maximum amount of memory allocated for Java classes (MaxMetaspaceSize) is by default unlimited, so in most cases there is no need to change this setting. On the other hand, if you want to fix the amount of memory allocated for Java classes, you can set it as follows:

java -XX:MaxMetaspaceSize=1024m

This JVM parameter -XX:MaxMetaspaceSize is just an set the upper limit of MetaSpace. The current Metaspace size (i.e. committed) will be smaller. In fact, there is a setting called MaxMetaspaceFreeRatio (default 70%) which means that the actual metaspace size will never exceed 230% of its occupancy.

And for it to grow it first would have to fill up, forcing a garbage collection in an attempt to free objects and only when it cannot meet its MinMetaspaceFreeRatio (default 40%) goal it would expand the current metaspace. That can however not be greater than 230% of the occupancy after the GC cycle.

Monitoring MetaSpace Size with Java Native Memory tracking

A good way to monitor the exact amount of Metadata is by using the NativeMemoryTracking, which can be added through the following settings:

-XX:+UnlockDiagnosticVMOptions -XX:NativeMemoryTracking=detail -XX:+PrintNMTStatistics

When Native Memory Tracking is enabled, you can request a report on the JVM memory usage using the following command:

$ jcmd <pid> VM.native_memory

OutOfMemoryError: Metaspace on OpenShift/Kubernetes

When using openjdk Image on OpenShift/Kubernetes, the default maxium value for the Metaspace is XX:MaxMetaspaceSize=100m. You might have noticed that setting this value through the JAVA_OPTIONS environment variable, doesn’t work as the default value is appended to the bottom:

VM Arguments: -Xms128m -Xmx1024m -XX:MetaspaceSize=128M -XX:MaxMetaspaceSize=256m    -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError

Once the MetaSpace get full in your application it will stop the service and through exception as below in your logs.

oc logs XYZ-service-7b856cc89-kpc6k  | grep -i metaspace
INFO exec  java -javaagent:/usr/share/java/jolokia-jvm-agent/jolokia-jvm.jar=config=/opt/jboss/container/jolokia/etc/jolokia.properties -XX:+UseParallelOldGC -XX:MinHeapFreeRatio=10 -XX:MaxHeapFreeRatio=20 -XX:GCTimeRatio=4 -XX:AdaptiveSizePolicyWeight=90 -XX:MaxMetaspaceSize=100m -XX:+ExitOnOutOfMemoryError -cp "." -jar /deployments/XYZ-service-0.0.1-SNAPSHOT.jar
Picked up JAVA_TOOL_OPTIONS:  -Dappdynamics.agent.accountAccessKey=600a90af-582a-4ae2-87b1-4599708b65dd -Dappdynamics.agent.reuse.nodeName=true -Dappdynamics.socket.collection.bci.enable=true -XX:MaxMetaspaceSize=1024m -javaagent:/opt/appdynamics-java/javaagent.jar
Terminating due to java.lang.OutOfMemoryError: Metaspace

Solutions

The correct way to set the MaxMetaspaceSize is through the GC_MAX_METASPACE_SIZE environment variable. Here are the different cases ti implement this solutions:

  • Jenkins Pipeline: For example, if you are using a jenkins pipeline to deploy your application or services then you can meke following changes in the json template to refelect these changes in deployment.yaml file to deploy your application with JKube, the following settings will override the default values for the MaxMetaspaceSize and MaxMetaspaceSize:
spec:
  template:
    spec:
      containers:
      - env:
        - name: JAVA_OPTIONS
          value: '-Xms256m -Xmx1024m'
        - name: GC_MAX_METASPACE_SIZE
          value: 1024
        - name: GC_METASPACE_SIZE
          value: 256
  • Manual Deployment through S2I: You can directly pass these MetaSpace parameters while deploying your service manually.
oc new-app xyz-service -e JAVA_OPTIONS="-Xms256m -Xmx1024m" -e GC_MAX_METASPACE_SIZE=1024 -e GC_METASPACE_SIZE=256

Note : After deploying your service over OpenShift/Kernates validate the deployment configuration file(deployment.yml) for these parameters. In case not not reflecting then delete your pods completly and reploy the application.

In case, you want make changes on other parameters also for improving the puformance of your applictaion then you can follow these documents to list of parameters for OpenShift.

Let me know your thought on this post. if this solution was helpful for you make comment on the post.

Happy Learning !!!