Showing posts with label JMX. Show all posts
Showing posts with label JMX. Show all posts

Saturday, April 30, 2016

Programmatic jcmd Access

Questions posed online demonstrate the occasional desire of developers to access via their Java applications the types of information normally gained from running JConsole, VisualVM, or JDK command-line tools externally against the application. Here are some examples of those types of questions:

I recently blogged on the usefulness of jcmd. This "Swiss-army knife" of command-line JDK tools provides much information about running Java processes and many of the types of information developers might want to acquire programatically are available via jcmd.

The Tools Enhancements in JDK 8 page states, "JDK 8 provides remote access to diagnostic commands which were previously accessible only locally via the jcmd tool. Remote access is provided using the Java Management Extensions (JMX), so diagnostic commands are exposed to a platform MBean registered to the platform MBean server. The MBean is the com.sun.management.DiagnosticCommandMBean interface." In the post Looking at DiagnosticCommandMBean in JConsole and VisualVM, I looked at using JMX and DiagnosticCommandMBean via JConsole and VisualVM to access jcmd type information on a running JVM process.

In this post I look at how to use com.sun.management.DiagnosticCommandMBean to programmatically access information provided by jcmd.

The DiagnosticCommandMBean is accessed via JMX using the ObjectName "com.sun.management:type=DiagnosticCommand". Most of the operations require no arguments and return a String. With this in mind, a general approach to accessing these operations that have String[] signatures, a String return type, and don't actually require a parameter is shown in the following code snippet.

Setting Up DiagnosticCommandMBean and General Method for Accessing Operations of Same Signature
   /** Object Name of DiagnosticCommandMBean. */
   public final static String DIAGNOSTIC_COMMAND_MBEAN_NAME =
      "com.sun.management:type=DiagnosticCommand";

   /** My MBean Server. */
   private final MBeanServer server = ManagementFactory.getPlatformMBeanServer();

   /** Platform MBean Server. */
   private final ObjectName objectName;

   //  . . .

   /**
    * Invoke operation on the DiagnosticCommandMBean that accepts
    *    String array argument but does not require any String
    *    argument and returns a String.
    *
    * @param operationName Name of operation on DiagnosticCommandMBean.
    * @param operationDescription Description of operation being invoked
    *    on the DiagnosticCommandMBean.
    * @return String returned by DiagnosticCommandMBean operation.
    */
   private String invokeNoStringArgumentsCommand(
      final String operationName, final String operationDescription)
   {
      String result;
      try
      {
         result = (String) server.invoke(objectName, operationName, new Object[] {null}, new String[]{String[].class.getName()});
      }
      catch (InstanceNotFoundException | ReflectionException | MBeanException exception)
      {
         result = "ERROR: Unable to access '" + operationDescription + "' - " + exception;
      }
      return result;
   }

With the code above in place, the method invokeNoStringArgumentsCommand(String, String) can be used to access several of the operations that DiagnosticCommandMBean provides. The help sub-command is useful when used with jcmd because it lists the available sub-commands associated with a specified Java process. Likewise, the "help" operation provided by the DiagnosticCommandMBean is similarly helpful in providing a list of commands that the MBean supports. This is easy to access with the code shown next that uses the code just shown.

/**
 * Provide list of supported operations (help).
 *
 * @return Single string containing names of supported operations.
 */
public String getAvailableOperations()
{
   return invokeNoStringArgumentsCommand("help", "Help (List Commands)");
}

Running this from a simple Java application leads to output similar to that shown next:

The following commands are available:
JFR.stop
JFR.start
JFR.dump
JFR.check
VM.native_memory
VM.check_commercial_features
VM.unlock_commercial_features
GC.rotate_log
Thread.print
GC.class_stats
GC.class_histogram
GC.run_finalization
GC.run
VM.uptime
VM.flags
VM.system_properties
VM.command_line
VM.version
help

For more information about a specific command use 'help '.

In my blog post Determining the Active HotSpot Garbage Collector, I wrote that jcmd can be used to identify the VM flags of a running Java process and that, from those flags, one can glean which garbage collector is in use. Because the DiagnosticCommandMBean supports the VM.flags operation as shown above, this operation can be used to view the VM flags and, from those flags, determine which collector is in use. The next code listing shows two example methods with the first accessing the VM flags of the process and the second method providing a simplistic example of how to use the first method to identify the garbage collector in use.

Obtain Virtual Machine Flags and Identify Active Garbage Collector
/**
 * Provide a String representing the Virtual Machine flags.
 *
 * @return String containing the virtual machine flags.
 */
public String getVirtualMachineFlags()
{
   return invokeNoStringArgumentsCommand("vmFlags", "Determine VM flags");
}

/**
 * Provide String representing active/current garbage collector.
 *
 * @return String representation of current garbage collector
 *    ("Parallel/Throughput", "Concurrent Mark Sweep (CMS)",
 *    "Garbage First", or "UNDETERMINED").
 */
public String determineGarbageCollector()
{
   String garbageCollector;
   final String vmFlags = getVirtualMachineFlags();
   if (vmFlags.contains("+UseParallelGC") || vmFlags.contains("+UseParallelOldGC"))
   {
      garbageCollector = "Parallel/Throughput";
   }
   else if (vmFlags.contains("+UseConcMarkSweepGC"))
   {
      garbageCollector = "Concurrent Mark Sweep (CMS)";
   }
   else if (vmFlags.contains("+UseG1GC"))
   {
      garbageCollector = "Garbage First";
   }
   else
   {
      garbageCollector = "UNDETERMINED";
   }
   return garbageCollector;
}

The String with the VM flags and the output of the identified garbage collector are shown next:

-XX:CICompilerCount=3 -XX:InitialHeapSize=134217728 -XX:MaxHeapSize=2128609280 -XX:MaxNewSize=709361664 -XX:MinHeapDeltaBytes=524288 -XX:NewSize=44564480 -XX:OldSize=89653248 -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:-UseLargePagesIndividualAllocation -XX:+UseParallelGC
Parallel/Throughput

The examples already shown (accessing help and VM flags) are demonstrative of the same general approach that can be used to programmatically access other operations on DiagnosticCommandMBean with the same signature. A series of code listings and corresponding output are shown next to illustrate some of these.

Accessing Virtual Machine Uptime
/**
 * Provide virtual machine uptime as single String.
 *
 * @return Single string containing virtual machine uptime.
 */
public String getVirtualMachineUptime()
{
   return invokeNoStringArgumentsCommand("vmUptime", "Virtual Machine Uptime");
}
0.272 s
Accessing Thread Dump
/**
 * Provide thread dump as single String.
 *
 * @return Single string containing formatted thread dump.
 */
public String getThreadDump()
{
   return invokeNoStringArgumentsCommand("threadPrint", "Thread Dump");
}
2016-04-30 20:21:22
Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.25-b02 mixed mode):

"Monitor Ctrl-Break" #10 daemon prio=5 os_prio=0 tid=0x00000000189ea800 nid=0x1590 runnable [0x000000001903e000]
   java.lang.Thread.State: RUNNABLE
 at java.net.DualStackPlainSocketImpl.accept0(Native Method)
 at java.net.DualStackPlainSocketImpl.socketAccept(DualStackPlainSocketImpl.java:131)
 at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
 at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:199)
 - locked <0x0000000081208810> (a java.net.SocksSocketImpl)
 at java.net.ServerSocket.implAccept(ServerSocket.java:545)
 at java.net.ServerSocket.accept(ServerSocket.java:513)
 at com.intellij.rt.execution.application.AppMain$1.run(AppMain.java:90)
 at java.lang.Thread.run(Thread.java:745)

"Service Thread" #9 daemon prio=9 os_prio=0 tid=0x0000000018915800 nid=0x2468 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C1 CompilerThread2" #8 daemon prio=9 os_prio=2 tid=0x0000000017087000 nid=0x17a0 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" #7 daemon prio=9 os_prio=2 tid=0x0000000017080000 nid=0x1560 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" #6 daemon prio=9 os_prio=2 tid=0x000000001707d000 nid=0x2004 waiting on condition [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Attach Listener" #5 daemon prio=5 os_prio=2 tid=0x000000001707b000 nid=0x2160 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" #4 daemon prio=9 os_prio=2 tid=0x000000001707a000 nid=0x458 runnable [0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"Finalizer" #3 daemon prio=8 os_prio=1 tid=0x00000000024f7000 nid=0x1964 in Object.wait() [0x00000000183ef000]
   java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x0000000081201570> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:142)
 - locked <0x0000000081201570> (a java.lang.ref.ReferenceQueue$Lock)
 at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:158)
 at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)

"Reference Handler" #2 daemon prio=10 os_prio=2 tid=0x00000000024ee000 nid=0x270c in Object.wait() [0x00000000182ef000]
   java.lang.Thread.State: WAITING (on object monitor)
 at java.lang.Object.wait(Native Method)
 - waiting on <0x0000000081208c28> (a java.lang.ref.Reference$Lock)
 at java.lang.Object.wait(Object.java:502)
 at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:157)
 - locked <0x0000000081208c28> (a java.lang.ref.Reference$Lock)

"main" #1 prio=5 os_prio=0 tid=0x0000000000c4e000 nid=0x24f8 waiting on condition [0x0000000000dee000]
   java.lang.Thread.State: RUNNABLE
 at sun.management.DiagnosticCommandImpl.executeDiagnosticCommand(Native Method)
 at sun.management.DiagnosticCommandImpl.access$000(DiagnosticCommandImpl.java:40)
 at sun.management.DiagnosticCommandImpl$Wrapper.execute(DiagnosticCommandImpl.java:128)
 at sun.management.DiagnosticCommandImpl.invoke(DiagnosticCommandImpl.java:230)
 at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
 at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
 at dustin.examples.diagnostics.VirtualMachineDiagnostics.invokeNoStringArgumentsCommand(VirtualMachineDiagnostics.java:167)
 at dustin.examples.diagnostics.VirtualMachineDiagnostics.getThreadDump(VirtualMachineDiagnostics.java:88)
 at dustin.examples.diagnostics.VirtualMachineDiagnostics.main(VirtualMachineDiagnostics.java:187)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:483)
 at com.intellij.rt.execution.application.AppMain.main(AppMain.java:144)

"VM Thread" os_prio=2 tid=0x0000000017046000 nid=0x22b0 runnable 

"GC task thread#0 (ParallelGC)" os_prio=0 tid=0x0000000002417800 nid=0x1580 runnable 

"GC task thread#1 (ParallelGC)" os_prio=0 tid=0x0000000002419000 nid=0x16d8 runnable 

"GC task thread#2 (ParallelGC)" os_prio=0 tid=0x000000000241a800 nid=0x177c runnable 

"GC task thread#3 (ParallelGC)" os_prio=0 tid=0x000000000241c800 nid=0x1974 runnable 

"VM Periodic Task Thread" os_prio=2 tid=0x0000000018918000 nid=0x10dc waiting on condition 

JNI global references: 15
Getting Class Histogram
/**
 * Provide class histogram as single String.
 *
 * @return Single string containing formatted class histogram.
 */
public String getHistogram()
{
   return invokeNoStringArgumentsCommand("gcClassHistogram", "GC Class Histogram");
}
 num     #instances         #bytes  class name
----------------------------------------------
   1:          3485         243064  [C
   2:           467         124976  [B
   3:           992         104200  java.lang.Class
   4:          3452          82848  java.lang.String
   5:          1006          46744  [Ljava.lang.Object;
   6:           525          46200  java.lang.reflect.Method
   7:           309          22248  java.lang.reflect.Field
   8:           519          16608  java.util.HashMap$Node
   9:           326          11760  [Ljava.lang.String;
  10:           211           8968  [I
  11:           434           8312  [Ljava.lang.Class;
  12:            77           7416  [Ljava.util.HashMap$Node;
  13:           179           7160  java.lang.ref.SoftReference
  14:           101           5656  java.lang.Class$ReflectionData
  15:            73           4672  java.net.URL
  16:            97           4656  java.util.HashMap
  17:            89           4560  [Ljava.lang.reflect.Method;
  18:           189           4536  javax.management.ImmutableDescriptor
  19:           279           4464  java.lang.Integer
  20:           139           4448  java.util.Hashtable$Entry
  21:           105           4200  java.util.WeakHashMap$Entry
  22:           129           4128  java.util.concurrent.ConcurrentHashMap$Node
  23:           128           4096  com.sun.jmx.mbeanserver.ConvertingMethod
  24:            90           3600  java.util.TreeMap$Entry
  25:            34           3360  [Ljava.util.WeakHashMap$Entry;
  26:             8           3008  java.lang.Thread
  27:            45           2880  javax.management.openmbean.OpenMBeanAttributeInfoSupport
  28:            87           2784  java.lang.ref.WeakReference
  29:           160           2560  java.lang.Object
  30:            49           2544  [Ljavax.management.MBeanAttributeInfo;
  31:           135           2536  [Lcom.sun.jmx.mbeanserver.MXBeanMapping;
  32:            30           2400  java.lang.reflect.Constructor
  33:            48           2304  javax.management.MBeanInfo
  34:            49           1960  java.io.ObjectStreamField
  35:            60           1920  javax.management.MBeanAttributeInfo
  36:            46           1840  java.util.LinkedHashMap$Entry
  37:            38           1824  sun.util.locale.LocaleObjectCache$CacheEntry
  38:            44           1760  java.lang.ref.Finalizer
  39:            30           1752  [Ljava.lang.reflect.Field;
  40:            36           1728  sun.misc.URLClassPath$JarLoader
  41:             1           1712  [[B
  42:            40           1600  sun.management.DiagnosticCommandArgumentInfo
  43:            31           1488  java.util.WeakHashMap
  44:            60           1440  java.util.ArrayList
  45:            11           1392  [Ljava.util.Hashtable$Entry;
  46:            42           1344  java.lang.ref.ReferenceQueue
  47:             2           1320  [J
  48:            10           1312  [Ljava.util.concurrent.ConcurrentHashMap$Node;
  49:            26           1248  java.util.logging.LogManager$LoggerWeakRef
  50:            47           1120  [Ljavax.management.ObjectName$Property;
  51:             1           1040  [Ljava.lang.Integer;
  52:            39           1008  [Ljavax.management.MBeanOperationInfo;
  53:            24            960  javax.management.ObjectName
  54:            38            912  java.io.ExpiringCache$Entry
  55:            19            912  sun.management.DiagnosticCommandInfo
  56:            18            864  java.util.TreeMap
  57:            12            864  java.util.logging.Logger
  58:            35            840  javax.management.ObjectName$Property
  59:            15            840  javax.management.openmbean.ArrayType
  60:            20            800  com.sun.jmx.mbeanserver.MXBeanSupport
  61:            20            800  javax.management.MBeanOperationInfo
  62:            14            784  sun.nio.cs.UTF_8$Encoder
  63:            32            768  com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$IdentityMapping
  64:            12            768  java.util.jar.JarFile
  65:            19            760  sun.util.locale.BaseLocale$Key
  66:            15            720  java.lang.management.PlatformComponent
  67:            44            704  java.lang.ref.ReferenceQueue$Lock
  68:            11            704  java.util.concurrent.ConcurrentHashMap
  69:            21            672  com.sun.jmx.mbeanserver.WeakIdentityHashMap$IdentityWeakReference
  70:             7            672  java.util.jar.JarFile$JarFileEntry
  71:            12            672  java.util.zip.ZipFile$ZipFileInputStream
  72:            19            608  java.util.Locale
  73:            19            608  sun.management.DiagnosticCommandImpl$Wrapper
  74:            19            608  sun.util.locale.BaseLocale
  75:            18            576  javax.management.MBeanParameterInfo
  76:            10            560  java.util.LinkedHashMap
  77:            14            560  javax.management.openmbean.SimpleType
  78:            23            552  com.sun.jmx.mbeanserver.PerInterface$MethodAndSig
  79:            26            536  [Ljava.lang.reflect.Constructor;
  80:            22            528  com.sun.jmx.mbeanserver.NamedObject
  81:            22            528  sun.reflect.generics.tree.SimpleClassTypeSignature
  82:            13            520  java.security.AccessControlContext
  83:            16            512  java.util.logging.LogManager$LogNode
  84:            17            504  [Ljava.io.ObjectStreamField;
  85:             9            504  javax.management.openmbean.CompositeType
  86:            19            488  [Lsun.management.DiagnosticCommandArgumentInfo;
  87:            12            480  com.sun.jmx.mbeanserver.PerInterface
  88:            20            480  java.lang.Class$AnnotationData
  89:            20            480  java.util.Arrays$ArrayList
  90:            20            480  java.util.jar.Attributes$Name
  91:            19            456  [Ljavax.management.MBeanParameterInfo;
  92:            19            456  java.util.Locale$LocaleKey
  93:            14            448  java.util.concurrent.locks.ReentrantLock$NonfairSync
  94:             8            448  javax.management.openmbean.OpenMBeanParameterInfoSupport
  95:            20            440  [Ljavax.management.MBeanNotificationInfo;
  96:            13            416  java.io.File
  97:            22            408  [Lsun.reflect.generics.tree.TypeArgument;
  98:            10            400  java.io.FileDescriptor
  99:            10            400  sun.reflect.UnsafeQualifiedStaticObjectFieldAccessorImpl
 100:             1            384  java.lang.ref.Finalizer$FinalizerThread
 101:             6            384  java.nio.DirectByteBuffer
 102:             8            384  java.util.Hashtable
 103:             1            384  java.util.logging.LogManager$Cleaner
 104:            12            384  java.util.zip.ZipCoder
 105:             1            376  java.lang.ref.Reference$ReferenceHandler
 106:            15            360  javax.management.StandardMBean
 107:            22            352  sun.reflect.generics.tree.ClassTypeSignature
 108:             4            336  [D
 109:             7            336  com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CompositeMapping
 110:             6            336  java.nio.DirectLongBufferU
 111:            14            336  java.util.concurrent.CopyOnWriteArrayList
 112:             6            336  sun.management.MemoryPoolImpl
 113:            14            336  sun.reflect.NativeConstructorAccessorImpl
 114:            16            328  [Ljava.lang.management.PlatformComponent;
 115:             4            320  [S
 116:            13            312  java.lang.management.ManagementPermission
 117:             9            288  java.lang.OutOfMemoryError
 118:            12            288  java.util.ArrayDeque
 119:            12            288  java.util.Collections$SingletonList
 120:             9            288  java.util.logging.Level
 121:            12            288  sun.misc.MetaIndex
 122:             5            280  sun.util.calendar.ZoneInfo
 123:             8            256  sun.misc.ProxyGenerator$PrimitiveTypeInfo
 124:             2            240  java.net.SocksSocketImpl
 125:             6            240  sun.management.MemoryPoolImpl$CollectionSensor
 126:             6            240  sun.management.MemoryPoolImpl$PoolSensor
 127:             6            240  sun.reflect.generics.repository.MethodRepository
 128:            14            224  java.util.concurrent.locks.ReentrantLock
 129:            14            224  sun.reflect.DelegatingConstructorAccessorImpl
 130:            13            216  [Ljavax.management.MBeanConstructorInfo;
 131:             9            216  java.util.logging.Level$KnownLevel
 132:             9            216  sun.util.logging.PlatformLogger$Level
 133:            10            208  [Ljava.lang.reflect.Type;
 134:             5            200  java.lang.ClassLoader$NativeLibrary
 135:             6            192  java.util.Vector
 136:             8            192  sun.reflect.generics.factory.CoreReflectionFactory
 137:             6            192  sun.reflect.generics.tree.MethodTypeSignature
 138:             2            176  java.net.DualStackPlainSocketImpl
 139:             9            168  [Lsun.reflect.generics.tree.FieldTypeSignature;
 140:             3            168  javax.management.openmbean.OpenMBeanOperationInfoSupport
 141:             9            160  [Lsun.reflect.generics.tree.FormalTypeParameter;
 142:             5            160  java.io.FileInputStream
 143:             4            160  java.security.ProtectionDomain
 144:             5            160  javax.management.MBeanNotificationInfo
 145:             5            160  javax.management.StandardEmitterMBean
 146:             6            144  java.util.LinkedList$Node
 147:             3            144  java.util.Properties
 148:             6            144  sun.misc.PerfCounter
 149:             6            144  sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl
 150:             6            144  sun.reflect.generics.scope.MethodScope
 151:             4            128  com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$ArrayMapping
 152:             2            128  java.io.ExpiringCache$1
 153:             4            128  java.security.CodeSource
 154:             8            128  java.util.HashSet
 155:             4            128  java.util.LinkedList
 156:             1            120  [[Ljava.lang.String;
 157:             5            120  sun.misc.FloatingDecimal$PreparedASCIIToBinaryBuffer
 158:             2            112  java.lang.Package
 159:             2            112  java.util.zip.ZipFile$ZipFileInflaterInputStream
 160:             1             96  [Ljava.lang.invoke.MethodType;
 161:             6             96  [Lsun.reflect.generics.tree.TypeSignature;
 162:             3             96  com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$EnumMapping
 163:             4             96  java.lang.RuntimePermission
 164:             2             96  java.lang.ThreadGroup
 165:             6             96  java.lang.ThreadLocal
 166:             2             96  java.nio.HeapByteBuffer
 167:             4             96  java.util.Collections$UnmodifiableRandomAccessList
 168:             3             96  java.util.Stack
 169:             2             96  java.util.zip.Inflater
 170:             2             96  javax.management.openmbean.TabularType
 171:             2             96  sun.management.GarbageCollectorImpl
 172:             2             96  sun.nio.cs.StreamEncoder
 173:             4             96  sun.reflect.annotation.AnnotationInvocationHandler
 174:             3             96  sun.reflect.generics.reflectiveObjects.TypeVariableImpl
 175:             3             96  sun.reflect.generics.repository.ClassRepository
 176:             1             88  sun.misc.Launcher$AppClassLoader
 177:             1             88  sun.misc.Launcher$ExtClassLoader
 178:             1             80  [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry;
 179:             1             80  [Ljava.lang.invoke.LambdaForm;
 180:             2             80  com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$TabularMapping
 181:             2             80  java.io.BufferedWriter
 182:             2             80  java.io.ExpiringCache
 183:             2             80  java.lang.invoke.MethodType
 184:             2             80  java.util.IdentityHashMap
 185:             2             80  sun.management.MemoryManagerImpl
 186:             2             80  sun.misc.FloatingDecimal$BinaryToASCIIBuffer
 187:             2             80  sun.misc.URLClassPath
 188:             1             72  [Ljavax.management.openmbean.SimpleType;
 189:             3             72  java.lang.annotation.RetentionPolicy
 190:             1             72  java.lang.invoke.MethodTypeForm
 191:             3             72  java.net.InetAddress$InetAddressHolder
 192:             3             72  java.util.Collections$SynchronizedSet
 193:             1             72  java.util.logging.LogManager$RootLogger
 194:             3             72  sun.misc.FloatingDecimal$ExceptionalBinaryToASCIIBuffer
 195:             3             72  sun.reflect.NativeMethodAccessorImpl
 196:             3             72  sun.reflect.generics.tree.ClassSignature
 197:             3             72  sun.reflect.generics.tree.FormalTypeParameter
 198:             1             64  [F
 199:             2             64  [Ljava.lang.Thread;
 200:             2             64  [Ljava.lang.annotation.RetentionPolicy;
 201:             4             64  [Ljava.security.Principal;
 202:             2             64  com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$CollectionMapping
 203:             4             64  com.sun.proxy.$Proxy1
 204:             2             64  java.io.FileOutputStream
 205:             2             64  java.io.FilePermission
 206:             2             64  java.io.PrintStream
 207:             2             64  java.lang.ThreadLocal$ThreadLocalMap$Entry
 208:             2             64  java.lang.VirtualMachineError
 209:             2             64  java.lang.invoke.MethodType$ConcurrentWeakInternSet$WeakEntry
 210:             2             64  java.lang.ref.ReferenceQueue$Null
 211:             2             64  java.lang.reflect.Proxy$Key1
 212:             2             64  java.lang.reflect.WeakCache$CacheValue
 213:             2             64  java.security.BasicPermissionCollection
 214:             2             64  java.security.Permissions
 215:             4             64  java.security.ProtectionDomain$Key
 216:             4             64  java.util.LinkedHashMap$LinkedValues
 217:             2             64  java.util.PropertyPermission
 218:             2             64  sun.reflect.annotation.AnnotationType
 219:             1             56  [Ljava.lang.Runnable;
 220:             3             56  [Lsun.reflect.generics.tree.ClassTypeSignature;
 221:             1             56  [Lsun.util.logging.PlatformLogger$Level;
 222:             1             56  java.lang.invoke.MemberName
 223:             1             56  java.util.logging.LogManager
 224:             3             48  [Ljava.lang.annotation.Annotation;
 225:             2             48  [Ljava.lang.reflect.TypeVariable;
 226:             1             48  [[Ljava.lang.Object;
 227:             2             48  com.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry
 228:             1             48  com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory$Mappings
 229:             1             48  com.sun.jmx.mbeanserver.MBeanIntrospector$MBeanInfoMap
 230:             1             48  com.sun.jmx.mbeanserver.MBeanIntrospector$PerInterfaceMap
 231:             2             48  com.sun.jmx.mbeanserver.WeakIdentityHashMap
 232:             2             48  java.io.BufferedOutputStream
 233:             2             48  java.io.File$PathStatus
 234:             2             48  java.io.FilePermissionCollection
 235:             2             48  java.io.OutputStreamWriter
 236:             2             48  java.lang.StringBuilder
 237:             2             48  java.net.InetAddress
 238:             2             48  java.net.InetAddress$Cache
 239:             2             48  java.net.InetAddress$Cache$Type
 240:             2             48  java.nio.charset.CoderResult
 241:             3             48  java.nio.charset.CodingErrorAction
 242:             1             48  java.util.concurrent.locks.ReentrantReadWriteLock$FairSync
 243:             2             48  java.util.logging.Logger$LoggerBundle
 244:             2             48  java.util.zip.ZStreamRef
 245:             2             48  sun.management.ManagementFactoryHelper$1
 246:             2             48  sun.misc.NativeSignalHandler
 247:             2             48  sun.misc.Signal
 248:             1             48  sun.nio.cs.SingleByte$Decoder
 249:             3             48  sun.reflect.DelegatingMethodAccessorImpl
 250:             2             48  sun.reflect.generics.scope.ClassScope
 251:             2             40  [Lcom.sun.jmx.mbeanserver.ClassLoaderRepositorySupport$LoaderEntry;
 252:             1             40  [Ljava.lang.management.MemoryPoolMXBean;
 253:             1             40  com.sun.jmx.interceptor.DefaultMBeanServerInterceptor
 254:             1             40  com.sun.jmx.mbeanserver.JmxMBeanServer
 255:             1             40  com.sun.jmx.mbeanserver.MBeanServerDelegateImpl
 256:             1             40  java.io.BufferedInputStream
 257:             1             40  sun.management.DiagnosticCommandImpl
 258:             1             40  sun.nio.cs.StandardCharsets$Aliases
 259:             1             40  sun.nio.cs.StandardCharsets$Cache
 260:             1             40  sun.nio.cs.StandardCharsets$Classes
 261:             1             40  sun.nio.cs.UTF_8$Decoder
 262:             1             32  [Ljava.lang.OutOfMemoryError;
 263:             2             32  [Ljava.lang.StackTraceElement;
 264:             1             32  [Ljava.lang.ThreadGroup;
 265:             1             32  [Ljava.lang.management.MemoryManagerMXBean;
 266:             1             32  com.sun.jmx.mbeanserver.Repository
 267:             1             32  java.io.WinNTFileSystem
 268:             1             32  java.lang.ArithmeticException
 269:             2             32  java.lang.Boolean
 270:             1             32  java.lang.NullPointerException
 271:             2             32  java.lang.Shutdown$Lock
 272:             1             32  java.lang.StringCoding$StringDecoder
 273:             1             32  java.lang.reflect.WeakCache
 274:             1             32  java.net.Socket
 275:             2             32  java.nio.ByteOrder
 276:             1             32  java.util.Collections$UnmodifiableMap
 277:             2             32  java.util.HashMap$EntrySet
 278:             1             32  java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
 279:             1             32  java.util.logging.LogManager$SystemLoggerContext
 280:             1             32  javax.management.MBeanConstructorInfo
 281:             1             32  sun.management.MemoryImpl
 282:             2             32  sun.net.www.protocol.jar.Handler
 283:             1             32  sun.nio.cs.StandardCharsets
 284:             1             24  [Ljava.io.File$PathStatus;
 285:             1             24  [Ljava.lang.invoke.MethodHandle;
 286:             1             24  [Ljava.net.InetAddress$Cache$Type;
 287:             1             24  [Ljava.net.InetSocketAddress;
 288:             1             24  [Ljava.security.ProtectionDomain;
 289:             1             24  [Lsun.launcher.LauncherHelper;
 290:             1             24  com.sun.jmx.mbeanserver.ClassLoaderRepositorySupport
 291:             1             24  com.sun.jmx.mbeanserver.MXBeanLookup
 292:             1             24  com.sun.jmx.remote.util.ClassLogger
 293:             1             24  dustin.examples.diagnostics.VirtualMachineDiagnostics
 294:             1             24  java.lang.ThreadLocal$ThreadLocalMap
 295:             1             24  java.lang.invoke.LambdaForm$NamedFunction
 296:             1             24  java.lang.invoke.MethodType$ConcurrentWeakInternSet
 297:             1             24  java.lang.reflect.ReflectPermission
 298:             1             24  java.net.Inet4Address
 299:             1             24  java.net.Inet6AddressImpl
 300:             1             24  java.net.ServerSocket
 301:             1             24  java.util.BitSet
 302:             1             24  java.util.Collections$EmptyMap
 303:             1             24  java.util.Collections$SetFromMap
 304:             1             24  java.util.Locale$Cache
 305:             1             24  java.util.concurrent.atomic.AtomicLong
 306:             1             24  java.util.concurrent.locks.ReentrantReadWriteLock
 307:             1             24  java.util.jar.Manifest
 308:             1             24  java.util.logging.LogManager$LoggerContext
 309:             1             24  java.util.logging.LoggingPermission
 310:             1             24  javax.management.NotificationBroadcasterSupport
 311:             1             24  sun.launcher.LauncherHelper
 312:             1             24  sun.management.CompilationImpl
 313:             1             24  sun.management.OperatingSystemImpl
 314:             1             24  sun.management.RuntimeImpl
 315:             1             24  sun.management.ThreadImpl
 316:             1             24  sun.management.VMManagementImpl
 317:             1             24  sun.misc.JarIndex
 318:             1             24  sun.misc.URLClassPath$FileLoader
 319:             1             24  sun.nio.cs.ISO_8859_1
 320:             1             24  sun.nio.cs.MS1252
 321:             1             24  sun.nio.cs.ThreadLocalCoders$1
 322:             1             24  sun.nio.cs.ThreadLocalCoders$2
 323:             1             24  sun.nio.cs.US_ASCII
 324:             1             24  sun.nio.cs.UTF_16
 325:             1             24  sun.nio.cs.UTF_16BE
 326:             1             24  sun.nio.cs.UTF_16LE
 327:             1             24  sun.nio.cs.UTF_8
 328:             1             24  sun.util.locale.BaseLocale$Cache
 329:             1             16  [Ljava.lang.Throwable;
 330:             1             16  [Ljava.nio.file.attribute.FileAttribute;
 331:             1             16  [Ljava.security.cert.Certificate;
 332:             1             16  [Ljava.util.logging.Handler;
 333:             1             16  com.intellij.rt.execution.application.AppMain$1
 334:             1             16  com.sun.jmx.interceptor.DefaultMBeanServerInterceptor$ResourceContext$1
 335:             1             16  com.sun.jmx.mbeanserver.DefaultMXBeanMappingFactory
 336:             1             16  com.sun.jmx.mbeanserver.DescriptorCache
 337:             1             16  com.sun.jmx.mbeanserver.MBeanAnalyzer$MethodOrder
 338:             1             16  com.sun.jmx.mbeanserver.MBeanInstantiator
 339:             1             16  com.sun.jmx.mbeanserver.MXBeanIntrospector
 340:             1             16  com.sun.jmx.mbeanserver.SecureClassLoaderRepository
 341:             1             16  java.io.FileDescriptor$1
 342:             1             16  java.lang.ApplicationShutdownHooks$1
 343:             1             16  java.lang.CharacterDataLatin1
 344:             1             16  java.lang.Runtime
 345:             1             16  java.lang.String$CaseInsensitiveComparator
 346:             1             16  java.lang.System$2
 347:             1             16  java.lang.Terminator$1
 348:             1             16  java.lang.invoke.MemberName$Factory
 349:             1             16  java.lang.management.PlatformComponent$1
 350:             1             16  java.lang.management.PlatformComponent$10
 351:             1             16  java.lang.management.PlatformComponent$11
 352:             1             16  java.lang.management.PlatformComponent$12
 353:             1             16  java.lang.management.PlatformComponent$13
 354:             1             16  java.lang.management.PlatformComponent$14
 355:             1             16  java.lang.management.PlatformComponent$15
 356:             1             16  java.lang.management.PlatformComponent$2
 357:             1             16  java.lang.management.PlatformComponent$3
 358:             1             16  java.lang.management.PlatformComponent$4
 359:             1             16  java.lang.management.PlatformComponent$5
 360:             1             16  java.lang.management.PlatformComponent$6
 361:             1             16  java.lang.management.PlatformComponent$7
 362:             1             16  java.lang.management.PlatformComponent$8
 363:             1             16  java.lang.management.PlatformComponent$9
 364:             1             16  java.lang.ref.Reference$Lock
 365:             1             16  java.lang.reflect.Proxy$KeyFactory
 366:             1             16  java.lang.reflect.Proxy$ProxyClassFactory
 367:             1             16  java.lang.reflect.ReflectAccess
 368:             1             16  java.net.InetAddress$2
 369:             1             16  java.net.URLClassLoader$7
 370:             1             16  java.nio.Bits$1
 371:             1             16  java.nio.Bits$1$1
 372:             1             16  java.nio.charset.CoderResult$1
 373:             1             16  java.nio.charset.CoderResult$2
 374:             1             16  java.security.ProtectionDomain$1
 375:             1             16  java.security.ProtectionDomain$3
 376:             1             16  java.util.Collections$EmptyIterator
 377:             1             16  java.util.Collections$EmptyList
 378:             1             16  java.util.Collections$EmptySet
 379:             1             16  java.util.Collections$SingletonSet
 380:             1             16  java.util.HashMap$Values
 381:             1             16  java.util.Hashtable$EntrySet
 382:             1             16  java.util.WeakHashMap$KeySet
 383:             1             16  java.util.concurrent.atomic.AtomicInteger
 384:             1             16  java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock
 385:             1             16  java.util.concurrent.locks.ReentrantReadWriteLock$Sync$ThreadLocalHoldCounter
 386:             1             16  java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock
 387:             1             16  java.util.jar.Attributes
 388:             1             16  java.util.jar.JavaUtilJarAccessImpl
 389:             1             16  java.util.logging.LoggingProxyImpl
 390:             1             16  java.util.zip.ZipFile$1
 391:             1             16  javax.management.JMX
 392:             1             16  javax.management.MBeanServerBuilder
 393:             1             16  javax.management.NotificationBroadcasterSupport$1
 394:             1             16  sun.management.ClassLoadingImpl
 395:             1             16  sun.management.HotSpotDiagnostic
 396:             1             16  sun.management.ManagementFactoryHelper$PlatformLoggingImpl
 397:             1             16  sun.misc.ASCIICaseInsensitiveComparator
 398:             1             16  sun.misc.FloatingDecimal$1
 399:             1             16  sun.misc.Launcher
 400:             1             16  sun.misc.Launcher$Factory
 401:             1             16  sun.misc.Perf
 402:             1             16  sun.misc.Unsafe
 403:             1             16  sun.net.www.protocol.file.Handler
 404:             1             16  sun.nio.ch.FileChannelImpl$1
 405:             1             16  sun.reflect.ReflectionFactory
 406:             1             16  sun.reflect.generics.tree.TypeVariableSignature
 407:             1             16  sun.util.calendar.Gregorian
Total         16267         885480

Some of the sub-commands available with jcmd are only available when diagnostic operations are unlocked with -XX:+UnlockDiagnosticVMOptions (specified when running the Java process). This is true of the DiagnosticCommandMBean counterparts as demonstrated in the next code listing and output listing.

Accessing Class Statistics
/**
 * Provide class statistics as single String.
 *
 * This is only supported when {@code -XX:+UnlockDiagnosticVMOptions} is enabled.
 *
 * @return Single string containing formatted class statistics.
 */
public String getClassStatistics()
{
   return invokeNoStringArgumentsCommand("gcClassStats", "GC Class Statistics");
}
GC.class_stats command requires -XX:+UnlockDiagnosticVMOptions

One jcmd subcommand I have not seen exposed as an operation on DiagnosticCommandMBean is that for generating a heap dump (jcmd's GC.heap_dump). There is no equivalent operation exposed by DiagnosticCommandMBean as far as I can tell from looking at the exposed operations in VisualVM and JConsole and from looking at the available commands listed by the "help" operation as demonstrated above. Although DiagnosticCommandMBean does not appear to provide an operation for invoking a heap dump, another MBean (com.sun.management.HotSpotDiagnostic with ObjectName of "com.sun.management:type=HotSpotDiagnostic") does with operation "dumpHeap" and this is demonstrated in A. Sundararajan's's blog post "Programmatically dumping heap from Java applications."

Conclusion

The DiagnosticCommandMBean provides most of jcmd's functionality and it therefore makes jcmd functionality available to Java applications to run that functionality against themselves.

The code listings shown in this post are available in the class VirtualMachineDiagnostics, which is available on GitHub. I may add or modify this class in the future, but the original version has the same code as shown in this post.

Monday, March 21, 2016

Looking at DiagnosticCommandMBean in JConsole and VisualVM

I've used JConsole for many years as a suitable generic JMX client. This tool is generally available with the Oracle JDK and is easy to use. In terms of JMX interaction, the most significant advantage of JConsole over VisualVM is that JConsole comes with a built-in MBeans tab while a plugin must be applied for this same functionality in VisualVM. However, as I explained in the blog post From JConsole to VisualVM, this plug-in is easy to install. In this post, I look at an area where VisualVM with the MBeans plugin is superior to JConsole with its built-in MBeans support: DiagnosticCommandMBean. I am using the versions of JConsole and VisualVM that are provided with the Oracle JDK (Java 8).

The next three screen snapshots demonstrate accessing operations of DiagnosticCommandMBean via JConsole. The first image displays about details regarding the DiagnosticCommandMBean such as its ObjectName (com.sun.management:type=DiagnosticCommand). The second image demonstrates that JConsole allows one to see the VM's system properties by clicking on the operation's "vmSystemProperties" button. The third image demonstrates that some operations of the DiagnosticCommandMBean ("help" in this case) cannot be invoked from JConsole.

As the last image shown demonstrates, some MBean operations are disabled. The reason for this is that, as described on StackOverflow, "they are enabled only for operations which take in simple types." Andreas Veithen has elaborated on this: "JConsole [only] allows to invoke methods that only have parameters with simple types. That includes primitive types, wrapper classes for primitive types and strings. Methods that have parameters with other types cannot be invoked because JConsole doesn't know how to construct instances of these types."

Fortunately, the MBeans plugin for VisualVM does provide support for operations against DiagnosticCommandMBean that deal with more complex data types. This support was explicitly added via VISUALVM-574 ("Add support for DiagnosticCommandMBean in MBeans plugin"). The next screen snapshot depicts basic background information regarding this plugin and is available in VisualVM by clicking on Tools -> Plugins -> Available Plugins -> VisualVM-MBeans.

After downloading and installing the VisualVM-MBeans plugin, VisualVM makes all operations on DiagnosticCommandMBean available as demonstrated in the next three screen snapshots. The first image shows that none of the operations are "grayed out." The second and third images show that both the "system properties" and the "help operations" are supported.

In the example of invoking "help" above, I did not pass it any arguments, so it behaved exactly as jcmd would behave when "help" is invoked without any other arguments and returns a list of jcmd command options available for the given Java process. The next screen snapshot depicts how this works similarly to jcmd when I provide one of those command names to the "help" operation as an argument (getting help on VM.uptime in this case).

The Tool Enhancements in JDK 8 page explains that DiagnosticCommandMBean makes jcmd functions available programatically and remotely: "JDK 8 provides remote access to diagnostic commands which were previously accessible only locally via the jcmd tool. Remote access is provided using the Java Management Extensions (JMX), so diagnostic commands are exposed to a platform MBean registered to the platform MBean server. The MBean is the com.sun.management.DiagnosticCommandMBean interface." The VisualVM plugin for JConsole/MBeans support makes use of the graphical VisualVM tool behave similarly to using jcmd from the command-line.

This post has demonstrated that DiagnosticCommandMBean provides access to the same data that jcmd provides for remote and programmatic access and that VisualVM provides a useful graphical interface for taking advantage of DiagnosticCommandMBean via the VisualVM-MBeans plugin.

Saturday, March 30, 2013

Detecting Java Threads in Deadlock with Groovy and JMX

Unfortunately, Java applications taking advantage of multiple threads can at times run into the dreaded deadlock condition. Fortunately, the Java Platform makes deadlock detection relatively easy. In fact, the built-in (since J2SE 5) ThreadMXBean (a PlatformManagedObject exposed via JMX) makes this information available to any client that "speaks JMX" via the findDeadlockedThreads() and findMonitorDeadlockThreads() methods. General "JMX clients" such as JConsole and VisualVM use this to provide information on detected deadlocks, but custom tools and scripts can be written to provide the same details. In this post, I look at using Groovy in conjunction with the Attach API to detect deadlocked threads for a locally running JVM process.

The Java Tutorials provides a simple and fairly interesting example of Java code that will typically result in deadlock on the "Deadlock" page in the "Concurrency" lesson of the "Essential Classes" trail. I provided that example here in only the slightest adapted form as the code I will run JMX clients against to detect deadlock. (As a side note, I keep seeing posts on reddit/Java and other online forums asking for the best free online introductory resources for learning Java; I cannot think of a better answer to this than the Java Tutorials.)

Deadlock.java (Adapted from Java Tutorials)
package dustin.examples.jmx.threading;

import static java.lang.System.out;

/**
 * Example of a class that often will lead to deadlock adapted from the 
 * Java Tutorials "Concurrency" section on "Deadlock":
 * http://docs.oracle.com/javase/tutorial/essential/concurrency/deadlock.html.
 */
public class Deadlock
{
   static class Friend
   {
      /** Friend's name. */
      private final String name;

      /**
       * Parameterized constructor accepting name for Friend instance.
       * @param newName Name of new Friend instance.
       */
      public Friend(final String newName)
      {
         this.name = newName;
      }
      
      /**
       * Provide this instance's name.
       * 
       * @return Friend's name.
       */
      public String getName()
      {
         return this.name;
      }

      /**
       * Bow from friend. Synchronized for thread-safe access.
       * 
       * @param bower Friend that is bowing.
       */
      public synchronized void bow(final Friend bower)
      {
         out.format("%s: %s has bowed to me!%n", 
            this.name, bower.getName());
         bower.bowBack(this);
      }

      /**
       * Bow back to friend who bowed to me. Synchronized for thread-safe access.
       * 
       * @param bower Friend who has bowed back to me.
       */
      public synchronized void bowBack(final Friend bower)
      {
         out.format("%s: %s  has bowed back to me!%n",
            this.name, bower.getName());
      }
   }

   /**
    * Simple executable function that demonstrates deadlock when two friends
    * are waiting on each other to bow to finish bowing.
    * 
    * @param arguments Command-line arguments: none expected.
    */
   public static void main(final String[] arguments)
   {
      final Friend alphonse = new Friend("Alphonse");
      final Friend gaston = new Friend("Gaston");
      new Thread(new Runnable()
      {
         public void run() { alphonse.bow(gaston); }
      }, "Gaston Bowing").start();
      new Thread(new Runnable()
      {
         public void run() { gaston.bow(alphonse); }
      }, "Alphonse Bowing").start();
   }
}

The next screen snapshot shows that this simple application becomes deadlocked between the two threads.

Even if we had not known this application was intended to demonstrate deadlock, the fact that it never ends or changes state is a good hint that it has run into deadlock. We can use JMX to determine that for certain. Whether we use a JMX client, jstack, or some other mechanism for determining deadlock, we'll likely need the Java process's pid (process ID), so the next screen snapshot indicates doing that (pid is 3792 in this case) with both jps and jcmd (the latter is only available since JDK 7).

Knowing that the pid is 3792, I can run "jconsole 3794" at the command line to have JConsole come up loaded with information about that deadlocked Java process. In this case, I'm interested in the threads and the next image shows the "Threads" tab in JConsole for this deadlock demonstration application.

The immediately previous screen snapshot has the two threads of interest circled. In my adapted code example, I named these "Gaston Bowing" and "Alphonse Bowing" to make them easier to find in JConsole and other tools. Had I not provided these explicit names in the example (which the original example in the Java Tutorials did not do), they would have been referenced by the more generic names "Thread-0" and "Thread-1." With JConsole loaded and the "Threads" tab available, I can click on the "Detect Deadlocks" button to see the information in the next two screen snapshots (one each for the two threads being selected).

By looking at the JConsole output for each of the identified deadlocked threads, we can see that each thread is BLOCKED and waiting on the other thread. The stack trace provided with each thread also provides pretty clear documentation of where this deadlock occurs in the code.

Java VisualVM can be started with the command jvisualvm and then the Java process in question can be selected as demonstrated in the next screen snapshot. Note that VisualVM automatically detects the deadlocks on the "Threads" tab of VisualVM and provides a message in red font indicating that.

When I click the "Thread Dump" button as indicated to see which threads are deadlocked, a thread dump is provided. Toward the bottom of that thread dump, as indicated in the next screen snapshot, are the details surrounding the deadlocked threads.

As JConsole did, VisualVM also indicates in the provided thread dump that the two bowing threads are waiting upon one another and "Java stack information" is provided for each of the threads identified as participating in the deadly embrace.

JConsole and Java VisualVM make it easy to identify the threads involved in deadlock. However, when a command line tool is preferred over a GUI-based tool, jstack is an obvious choice. Again using the pid identified earlier for this particular process (3792 in my case), one can simply type jstack 3792 to see a thread dump for that application. The next two screen snapshots show a piece of this with the first snapshot showing the command being run and the beginning of the output and the second snapshot showing the portion related to deadlock.

One doesn't have to look too closely to realize that jstack's output is the same as that provided in the "thread dump" in VisualVM.

JConsole and VisualVM provide a nice GUI approach for identifying deadlock and jstack does the same from the command-line. Given these tools, one might wonder why we'd care about being able to build our own tools and scripts to do the same. Often, these tools are enough, but there are times I may want to do specific things with the information once I have it or I want the answer to be even more direct than these tools provide. If I need to programmatically do something with the information or simply want the identification of threads involved in deadlock without the other threads' details, a custom tool might be in order. The remainder of this post focuses on that.

The following code listing contains Groovy code for printing threading information related to deadlocked threads as provided by the ThreadMXBean. This code makes use of a piece of Groovy code not shown here (JmxServer using Attach API) that can be found in my last blog post.

displayDetectedDeadlock.groovy
#!/usr/bin/env groovy
def pid = args[0]
def javaThreads = new javax.management.ObjectName("java.lang:type=Threading")
def server = JmxServer.retrieveServerConnection(pid)
long[] deadlockedThreadIds = server.invoke(javaThreads, "findDeadlockedThreads", null, null)
deadlockedThreadIds.each
{ threadId ->
   Object[] parameters = [threadId, 10]
   String[] signature = [Long.TYPE, Integer.TYPE]
   def threadInfo = server.invoke(javaThreads, "getThreadInfo", parameters, signature)
   print "\nThread '${threadInfo.threadName}' [${threadInfo.threadId}] is "
   print "${threadInfo.threadState} on thread '${threadInfo.lockOwnerName}' ["
   println "${threadInfo.lockOwnerId}]:\n"
   println "Java stack information for the threads listed above:"
   println "==================================================="
   println "'${threadInfo.threadName}':"
   threadInfo.stackTrace.each
   { compositeData ->
      print "\tat ${compositeData.className}.${compositeData.methodName}("
      println "${compositeData.fileName}:${compositeData.lineNumber})"
   }
   println "\n\n"
}

The Groovy script above specifies 10 as the maximum depth for the stack trace information to be provided as it provides details similar to those provided in JConsole, VisualVM, and jstack. Of course, the advantage of this approach is that any of this data that is printed to standard output here can also be used in runtime decisions and acted upon programatically from Java or Groovy code. The next screen snapshot shows the output from running this script against the deadlocked application used previously in this post.

Java makes it easy to detect threads locked in the deadly embrace known as deadlock. Although general tools like JConsole, VisualVM, and jstack are often sufficient to identify these cases, it is nice to be able to write custom scripts and tools for doing the same thing. This allows developers the flexibility to include deadlock detecting directly in their scripts and tools rather than needing to parse jstack output or resort to other data scraping approaches.

Saturday, March 23, 2013

Monitoring Key JVM Characteristics with Groovy, JMX, and RuntimeMXBean

Since J2SE 5, Platform MBeans have been available that allow some key characteristics regarding the JVM to be monitored and (even managed in some cases) via JMX. In addition, many JVM-based applications add their own JMX-enabled features for monitoring and management. In the blog post Groovy, JMX, and the Attach API, I looked at how to display many of the platform-provided MBeans using Groovy, JMX, and the Attach API. In this post, I look at more specific scripts that access a narrowly focused subset of these platform-exposed values available in the RuntimeMXBean.

My previous post demonstrated ability to view a wide spectrum of JVM details with Groovy and JMX. In most cases, I don't need all of these details all at once, but simply need one or two of the items. Although I could use a tool such as JConsole, VisualVM, or even a modern Java IDE to see these details, I sometimes wish to run a simple script to provide the exact results I'm looking for rather than using a general JMX client that I need to start up and navigate to that information. This is where simple scripts such as shown in this post are particularly handy.

All of my examples in this post assume the person running the Java process to be monitored/managed is the same person (same username) running the scripts and that they are being used to monitor Java processes running on the same local machine. This assumption allows the Attach API to be used. If a different user was running the script than ran the Java processes or if the scripts were to be run on remote Java processes, techniques for remote JMX would be used instead. Because the Attach API will be used in these examples, I have placed the code that each example uses in a Groovy script file that each example invokes. That file to be used by all of my scripts is called JmxServer.groovy and is shown next.

JmxServer.groovy
/*
 * JmxServer.groovy
 *
 * Functions meant to be called by other scripts or tools that need to use
 * the Attach API to access an MBeanServerConnection to use to manage and
 * monitor a particular JVM (and possibly the application hosted in that JVM)
 * identified by the provided Process ID (pid)
 */

import javax.management.MBeanServerConnection
import javax.management.ObjectName
import javax.management.remote.JMXConnector
import javax.management.remote.JMXConnectorFactory
import javax.management.remote.JMXServiceURL

import com.sun.tools.attach.VirtualMachine

/**
 * Provide an MBeanServerConnection based on the provided process ID (pid).
 *
 * @param pid Process ID of Java process for which MBeanServerConnection is
 *    desired.
 * @return MBeanServerConnection connecting to Java process identified by pid.
 */
def static MBeanServerConnection retrieveServerConnection(String pid)
{
   def connectorAddressStr = "com.sun.management.jmxremote.localConnectorAddress"
   def jmxUrl = retrieveUrlForPid(pid, connectorAddressStr)
   def jmxConnector = JMXConnectorFactory.connect(jmxUrl)
   return jmxConnector.getMBeanServerConnection()
}

/**
 * Provide JMX URL for attaching to the provided process ID (pid).
 *
 * @param @pid Process ID for which JMX URL is needed to connect.
 * @param @connectorAddressStr String for connecting.
 * @return JMX URL to communicating with Java process identified by pid.
 */
def static JMXServiceURL retrieveUrlForPid(String pid, String connectorAddressStr)
{
   // Attach to the target application's virtual machine
   def vm = VirtualMachine.attach(pid)

   // Obtain Connector Address
   def connectorAddress =
      vm.getAgentProperties().getProperty(connectorAddressStr)

   // Load Agent if no connector address is available
   if (connectorAddress == null)
   {
      def agent = vm.getSystemProperties().getProperty("java.home") +
          File.separator + "lib" + File.separator + "management-agent.jar"
      vm.loadAgent(agent)

      // agent is started, get the connector address
      connectorAddress =
         vm.getAgentProperties().getProperty(connectorAddressStr)
   }

   return new JMXServiceURL(connectorAddress);
}

The most important method defined in JmxServer.groovy, from a script client perspective, is the retrieveServerConnection(String) method. That methods excepts a pid as a process identifier and uses that and a call to the other method, retrieveUrlForPid(String,String) to attach to the Java process identified by that pid and provide an associated MBeanServerConnection instance. The easiest approaches for finding the appropriate Java pid are use of jps (pre-Java 7 and Java 7) or jcmd (Java 7+ only) as I briefly described in my recent post Booting AMX in GlassFish 3 with Groovy.

Having encapsulated the logic for acquiring a particular JVM (Java process's) MBeanServerConnection instance based on a provided pid, it is easy to start constructing simple Groovy scripts that use this provided MBeanServerConnection to provide specific desirable details about the JVM (Java process) in question. Note that all of these example scripts are in the same directory as the JmxServer.groovy file and so do not need to explicitly specify package or scoping details.

It is common to want to know what is on the classpath of a particular JVM for various reasons including detection of causes of ClassNotFoundException and NoClassDefFoundError. The classpath of a particular JVM is exposed by one of its platform MBean ("ClassPath" attribute of RuntimeMXBean) and can be easily discovered with the next Groovy script:

displayClasspath.groovy
#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def classpath = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "ClassPath")
println "\nClasspath for Java pid (${pid}):\n${classpath}\n"

That same RuntimeMXBean also provides the boot classpath and the Java library path. The next two code listings show Groovy scripts for accessing both of those.

displayBootClasspath.groovy
#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def bootClasspath = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "BootClassPath")
println "\nBoot Classpath for Java pid (${pid}):\n${bootClasspath}\n"
displayLibraryPath.groovy
#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def libraryPath = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "LibraryPath")
println "\nLibrary Path for Java pid (${pid}):\n${libraryPath}\n"

The Groovy scripts shown so far for displaying details related finding classes (the classpath, boot classpath, and library path) are all essentially the same script, but with a different attribute looked up on the RuntimeMXBean in each case. All three scripts are executed against a running instance of GlassFish in the following screen snapshot. The snapshot includes commands to jps and jcmd to determine the pid (5352) for the GlassFish instance.

The RuntimeMXBean has more to offer than just information on where classes were loaded from. It also includes attributes related to the JVM vendor such as name and version, but the other three methods that I like to have scripts to use are input arguments, system properties, and JVM uptime (and start time). The next three Groovy code listings show three scripts for displaying these details and each code snippet is followed by a screen snapshot demonstrating that script in action against the same GlassFish instance used in the last examples.

displayInputArguments.groovy
#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def inputArguments = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "InputArguments")
println "\nInput Arguments for Java pid (${pid}):\n${inputArguments}\n"
displaySystemProperties.groovy
#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def systemProperties = JmxServer.retrieveServerConnection(pid).getAttribute(javaRuntime, "SystemProperties")
println "\nSystem Properties for Java pid (${pid}):\n${systemProperties}\n"
displayJvmUptime.groovy
#!/usr/bin/env groovy
def pid = args[0]
def javaRuntime = new javax.management.ObjectName("java.lang:type=Runtime")
def server = JmxServer.retrieveServerConnection(pid)
def startTime = server.getAttribute(javaRuntime, "StartTime")
def uptime = server.getAttribute(javaRuntime, "Uptime")
println "\nJava process pid (${pid}) was started at ${new Date(startTime)} and has been up for ${uptime} ms.\n"

I have used this post to demonstrate simple Groovy scripts that acquire information from a JVM's Platform RuntimeMXBean such as the JVM's start time and uptime, the system properties and input arguments for the JVM process, and path information such as classpath, boot classpath, and library path. Although this information is available via tools such as JConsole, VisualVM, and Java IDEs (they all get it from the same source as these scripts!), the scripts can be advantageous at times (quicker to run and can be included within other scripts, for example).

Monday, March 18, 2013

"Booting AMX" in GlassFish 3 with Groovy

In my previous blog post, I looked at using JMX as one of multiple methods supported by GlassFish 3 for its administration, monitoring, and management. In this blog post, I look in more detail at monitoring and managing GlassFish 3 via JMX and Groovy. I focus on local connection to GlassFish using the Attach API in this post, but I have covered remote JMX access of GlassFish in a previous post (see also Remote Glassfish V3.1 and the mystical JMX settings). The MBeans used for administration are generally the same in either case.

Because I'm using the Attach API in this post's examples to connect to local Java processes that are started by me, I don't need to specify the JMX remote connection properties (java.rmi.server.hostname,com.sun.management.jmxremote.port, and com.sun.management.jmxremote.ssl=false, and com.sun.management.jmxremote.authenticate) for remote access. The easiest way to find Java processes meeting the standard of being on the local machine and being processes that I started is via use of jps (pre-JDK 7 and JDK 7) or jcmd (JDK 7 only), which I show in the next two screen snapshots.

As the above images indicate, GlassFish is running locally with PID (AKA Process ID or pid) 1584.

To begin, I'm going to use Groovy to access the relevant MBeanServerConnection via the Attach API. This is done as shown in the following Groovy code (next two code listings) in which the method retrieveServerConnection(String) accepts the PID and returns the MBeanServerConnection for that Java process. That method uses another method in the code listing, retrieveUrlForPid(String, String), which uses the Attach API to provide a JMXServiceURL for the Java process.

retrieveServerConnection(String)
/**
 * Provide an MBeanServerConnection based on the provided process ID (pid).
 *
 * @param pid Process ID of Java process for which MBeanServerConnection is
 *    desired.
 * @return MBeanServerConnection connecting to Java process identified by pid.
 */
def MBeanServerConnection retrieveServerConnection(String pid)
{
   def connectorAddressStr = "com.sun.management.jmxremote.localConnectorAddress"
   def jmxUrl = retrieveUrlForPid(pid, connectorAddressStr)
   def jmxConnector = JMXConnectorFactory.connect(jmxUrl)
   return jmxConnector.getMBeanServerConnection()
}
retrieveUrlForPid(String, String)
/**
 * Provide JMX URL for attaching to the provided process ID (pid).
 *
 * @param @pid Process ID for which JMX URL is needed to connect.
 * @param @connectorAddressStr String for connecting.
 * @return JMX URL to communicating with Java process identified by pid.
 */
def JMXServiceURL retrieveUrlForPid(String pid, String connectorAddressStr)
{
   // Attach to the target application's virtual machine
   def vm = VirtualMachine.attach(pid)

   // Obtain Connector Address
   def connectorAddress =
      vm.getAgentProperties().getProperty(connectorAddressStr)

   // Load Agent if no connector address is available
   if (connectorAddress == null)
   {
      def agent = vm.getSystemProperties().getProperty("java.home") +
          File.separator + "lib" + File.separator + "management-agent.jar"
      vm.loadAgent(agent)

      // agent is started, get the connector address
      connectorAddress =
         vm.getAgentProperties().getProperty(connectorAddressStr)
   }

   return new JMXServiceURL(connectorAddress);
}

With access to the MBeanServerConnection, I can start to do all types of useful things with Groovy and JMX to manage and monitor the a Java process. For example, the next code listing demonstrates how easy it is to now list the MBeans exposed by the Java process identified by a provided PID.

displayHostedMBeans(MBeanServerConnection)
/**
 * Display MBeans hosted on the provided MBeanServerConnection.
 *
 * @param mbeanServer MBeanServerConnection for which hosted MBeans are to be
 *    provided.
 */
def displayHostedMBeans(MBeanServerConnection mbeanServer)
{
   mbeanServer.queryNames(null, null).each
   {
      println it
   }
}

Running the above Groovy method against my running GlassFish instance with PID 1584 is demonstrated in the next screen snapshot.

The next code listing and associated screen snapshot demonstrate using Groovy and JMX to find not only the exposed MBeans, but to also provide the attributes and operations available on each of those exposed MBeans.

displayHostedMBeansAttributesAndOperations(MBeanServerConnection)
/**
 * Display MBeans hosted on provided MBean Server along with the attributes and
 * operations available on each MBean.
 */
def displayHostedMBeansAttributesAndOperations(MBeanServerConnection mbeanServer)
{
   mbeanServer.queryNames(null, null).each
   { mbeanObjectName ->
      def mbeanInfo = mbeanServer.getMBeanInfo(mbeanObjectName)
      println mbeanObjectName
      println "\tAttributes:"
      mbeanInfo.attributes.each
      { attribute ->
         println "\t\t${attribute.type} ${attribute.name}"
      }
      println "\tOperations:"
      mbeanInfo.operations.each
      { operation ->
         def operationStr = new StringBuilder();
         operationStr << "\t\t" << operation.name << "("
         operation.signature.each
         { parameter ->
            operationStr << parameter.type << " " << parameter.name << ", "
         }
         def operationString = operationStr.contains(",") ? operationStr.substring(0, operationStr.length()-2) : operationStr
         println "${operationString})"
      }
      println ""
   }
}

The previously displayed screen snapshot shows the bootAMX operation available on the amx-support:type=boot-amx MBean. This needs to be run to make other GlassFish-provided AMX MBean available. The next snippet of Groovy code, a method I've named bootAmx(MBeanServerConnection), invokes this bootAMX operation to instruct GlassFish to expose significantly more details via its JMX interface.

bootAmx(MBeanServerConnection)
/**
 * "Boot AMX" on GlassFish.
 *
 * @param mbeanServer MBeanServerConnection to be used to invoke bootAMX
 *    operation.
 */
def bootAmx(MBeanServerConnection mbeanServer)
{
   def amxObjectName = new ObjectName("amx-support:type=boot-amx")
   mbeanServer.invoke(amxObjectName, "bootAMX", null, null)
}

When the simple Groovy just shown is run, GlassFish exposes significant more functionality via JMX. Now that I've "booted AMX," I can rerun displayHostedMBeans(MBeanServerConnection) to see the new MBeans that are available. A portion of this is shown next in the screen snapshot and the small-font text below it contains the entire output.

amx:pp=/domain/configs/config[server-config]/network-config,type=transports
amx:pp=/domain/configs/config[default-config]/iiop-service,type=iiop-listener,name=orb-listener-1
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=encryption.key.alias
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=serverName
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-1]/http,type=file-cache
amx:pp=/domain/configs/config[server-config],type=mdb-container
amx:pp=/domain/configs/config[default-config],type=system-property,name=HTTP_LISTENER_PORT
amx:pp=/domain/configs/config[server-config]/web-container/session-config/session-manager,type=store-properties
amx:pp=/ext,type=config-tools
amx:pp=/domain/nodes,type=node,name=localhost-domain1
amx:pp=/domain/servers/server[server],type=application-ref,name=__admingui
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=sec-admin-listener
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=DownloadServlet,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[server-config]/network-config/transports,type=transport,name=tcp
amx:pp=/domain/configs/config[default-config],type=system-property,name=OSGI_SHELL_TELNET_PORT
amx:pp=/domain/configs/config[server-config],type=thread-pools
amx:pp=/domain/configs/config[server-config]/security-service,type=audit-module,name=default
amx:pp=/domain/secure-admin,type=secure-admin-principal,name="CN=localhost-instance,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=signature.key.alias
amx:pp=/,type=logging
java.lang:type=MemoryPool,name=Code Cache
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config]/web-container,type=session-config
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[file],type=property,name=file
amx:pp=/domain/configs/config[server-config]/iiop-service,type=iiop-listener,name=SSL
amx:pp=/domain/configs/config[server-config]/security-service,type=message-security-config,name=SOAP
amx:pp=/domain/configs/config[server-config]/iiop-service/iiop-listener[SSL_MUTUALAUTH],type=ssl
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-1],type=http
amx:pp=/domain/configs/config[default-config]/network-config,type=transports
amx:pp=/domain/configs/config[server-config]/http-service,type=virtual-server,name=server
amx:pp=/domain/configs/config[default-config]/web-container/session-config/session-manager,type=store-properties
amx:pp=/domain/configs/config[default-config],type=ejb-container
amx:pp=/ext,type=system-status
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[admin-listener],type=http
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=response-policy
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=property,name=loginPage
amx:pp=/domain/servers/server[server],type=resource-ref,name=jdbc/__TimerPool
java.lang:type=Memory
amx:pp=/mon/server-mon[server],type=compilation-system-mon,name=jvm/compilation-system
amx:pp=/domain/configs/config[server-config]/security-service,type=auth-realm,name=admin-realm
amx:pp=/domain/configs/config[default-config]/log-service,type=module-log-levels
amx:pp=/mon/server-mon[server],type=server-runtime-mon
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ClientProvider
amx:pp=/domain/configs/config[default-config]/web-container/session-config,type=session-manager
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ServerProvider
amx:pp=/domain/configs/config[server-config]/thread-pools,type=thread-pool,name=http-thread-pool
amx:pp=/domain/configs/config[default-config]/admin-service,type=property,name=adminConsoleDownloadLocation
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=connectionAttributes
amx:pp=/domain/resources/jdbc-connection-pool[__TimerPool],type=property,name=connectionAttributes
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=pu-protocol
amx:pp=/domain/configs/config[server-config]/network-config,type=protocols
amx:pp=/ext,type=connector-runtime-api-provider
amx:pp=/domain,type=resources
jmxremote:type=jmx-connector-server,protocol=rmi_jrmp,name=system
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-2],type=ssl
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-2],type=http
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=signature.key.alias
amx:pp=/,type=ext
amx:pp=/domain/configs/config[default-config]/monitoring-service,type=module-monitoring-levels
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=ClientProvider
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=property,name=restAuthURL
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-1],type=http
amx:pp=/domain/configs/config[server-config]/jms-service,type=jms-host,name=default_JMS_host
amx:pp=/domain/configs/config[server-config]/web-container/session-config/session-manager,type=manager-properties
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/network-config/network-listeners,type=network-listener,name=admin-listener
amx:pp=/domain/configs/config[default-config]/security-service/jacc-provider[default],type=property,name=repository
amx:pp=/mon/server-mon[server],type=operating-system-mon,name=jvm/operating-system
amx:pp=/domain/configs/config[default-config]/security-service,type=audit-module,name=default
java.lang:type=MemoryPool,name=Survivor Space
amx:pp=/domain/configs/config[default-config]/network-config,type=protocols
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-2],type=ssl
amx:pp=/domain/configs/config[default-config]/thread-pools,type=thread-pool,name=http-thread-pool
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=jsp,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[pu-protocol],type=port-unification
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=security.config
amx:pp=/domain/configs/config[default-config],type=system-property,name=HTTP_SSL_LISTENER_PORT
amx:pp=/domain/configs/config[default-config]/http-service,type=virtual-server,name=server
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=request-policy
amx:pp=/domain/configs/config[server-config],type=iiop-service
amx:pp=/domain/configs/config[default-config]/security-service,type=jacc-provider,name=default
amx:pp=/domain/configs/config[default-config]/web-container/session-config/session-manager,type=manager-properties
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=request-policy
amx:pp=/domain/configs/config[default-config]/security-service,type=auth-realm,name=certificate
java.nio:type=BufferPool,name=mapped
java.lang:type=Compilation
amx:pp=/,type=runtime
amx:pp=/domain/configs/config[server-config]/monitoring-service,type=module-monitoring-levels
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=dynamic.username.password
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=encryption.key.alias
amx-support:type=amx-loader,name=config
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=dynamic.username.password
amx:pp=/runtime,type=server-runtime,name=server
amx:pp=/domain/configs/config[server-config]/network-config/protocols,type=protocol,name=http-listener-1
amx:pp=/domain,type=system-applications
amx:pp=/domain/configs/config[server-config]/security-service,type=jacc-provider,name=default
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__default]/JDBCDataSource[jdbc/__default],type=JDBCDriver,name=jdbc/__default,j2eeType=JDBCDriver,J2EEServer=server,JDBCResource=jdbc/__default,JDBCDataSource=jdbc/__default
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[file],type=property,name=jaas-context
amx:pp=/domain/configs/config[server-config]/ejb-container,type=ejb-timer-service
amx:pp=/domain/configs/config[default-config]/security-service,type=auth-realm,name=file
java.lang:type=Runtime
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=default,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[default-config],type=web-container
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=debug
amx:pp=/domain/system-applications/application[__admingui]/module[__admingui],type=engine,name=security
amx:pp=/domain/configs/config[server-config]/admin-service,type=jmx-connector,name=system
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=response-policy
amx:pp=/ext,type=realms
amx:pp=/,type=query
amx:pp=/domain/configs/config[default-config],type=thread-pools
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=property,name=loginErrorPage
amx:pp=/domain/configs/config[server-config],type=connector-service
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config]/network-config,type=network-listeners
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__TimerPool]/JDBCDataSource[jdbc/__TimerPool],type=JDBCDriver,name=jdbc/__TimerPool,j2eeType=JDBCDriver,J2EEServer=server,JDBCResource=jdbc/__TimerPool,JDBCDataSource=jdbc/__TimerPool
amx:pp=/domain/configs/config[default-config],type=system-property,name=JMS_PROVIDER_PORT
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=request-policy
amx:pp=/domain/configs/config[default-config],type=availability-service
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[admin-http-redirect],type=http-redirect
amx:pp=/,type=J2EEDomain,j2eeType=J2EEDomain,name=amx
amx:pp=/domain/configs/config[default-config],type=system-property,name=IIOP_SSL_LISTENER_PORT
amx:pp=/domain/configs/config[default-config]/iiop-service,type=iiop-listener,name=SSL
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config]/network-config/transports,type=transport,name=tcp
amx:pp=/domain/configs/config[default-config],type=system-property,name=JMX_SYSTEM_CONNECTOR_PORT
amx:pp=/mon/server-mon[server],type=garbage-collector-mon,name=jvm/garbage-collectors/Copy
amx:pp=/domain/configs/config[default-config]/http-service,type=virtual-server,name=__asadmin
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=request-policy
amx:pp=/,type=system-info
amx:pp=/domain/configs/config[default-config],type=group-management-service
amx:pp=/domain/configs/config[default-config]/availability-service,type=web-container-availability
amx:pp=/domain/configs/config[server-config]/group-management-service,type=failure-detection
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=response-policy
java.lang:type=GarbageCollector,name=Copy
amx:pp=/domain/configs/config[server-config],type=java-config
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=ServerProvider
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[http-listener-2]/http,type=file-cache
amx:pp=/domain/configs/config[default-config],type=system-property,name=ASADMIN_LISTENER_PORT
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=request-policy
amx:pp=/domain/configs/config[server-config]/network-config/protocols,type=protocol,name=http-listener-2
amx:pp=/domain/configs/config[default-config]/jms-service,type=jms-host,name=default_JMS_host
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=ServerProvider
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=response-policy
amx:pp=/domain/configs/config[server-config],type=web-container
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[admin-realm],type=property,name=jaas-context
java.util.logging:type=Logging
amx:pp=/,type=tools
java.lang:type=MemoryPool,name=Eden Space
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=http-listener-2
com.sun.management:type=HotSpotDiagnostic
amx:pp=/domain/configs/config[default-config]/security-service,type=jacc-provider,name=simple
amx:pp=/domain/configs/config[default-config]/security-service,type=auth-realm,name=admin-realm
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=FacesServlet,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[server-config]/admin-service,type=property,name=adminConsoleContextRoot
amx:pp=/domain/configs/config[default-config],type=monitoring-service
amx:pp=/domain/configs/config[server-config]/security-service,type=auth-realm,name=file
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[admin-realm],type=property,name=jaas-context
amx:pp=/domain/configs/config[server-config]/security-service/audit-module[default],type=property,name=auditOn
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=response-policy
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[file],type=property,name=jaas-context
amx:pp=/domain/system-applications,type=application,name=__admingui
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=security.config
amx:pp=/domain/configs/config[default-config]/iiop-service/iiop-listener[SSL],type=ssl
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[default-config],type=connector-service
amx:pp=/domain/configs/config[server-config],type=ejb-container
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=request-policy
amx:pp=/mon/server-mon[server],type=garbage-collector-mon,name=jvm/garbage-collectors/MarkSweepCompact
amx:pp=/domain/configs/config[default-config]/security-service/audit-module[default],type=property,name=auditOn
amx:pp=/domain,type=secure-admin
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=http-listener-1
amx:pp=/,type=sample
amx:pp=/domain/configs/config[default-config]/group-management-service,type=failure-detection
amx:pp=/domain/configs/config[default-config]/network-config/network-listeners,type=network-listener,name=http-listener-1
amx:pp=/domain/system-applications/application[__admingui],type=module,name=__admingui
amx:pp=/J2EEDomain/J2EEServer[server]/WebModule[__admingui],type=Servlet,name=ThemeServlet,j2eeType=Servlet,J2EEServer=server,WebModule=__admingui,J2EEApplication=null
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=request-policy
amx:pp=/domain/configs,type=config,name=default-config
amx:pp=/domain/system-applications/application[__admingui]/module[__admingui],type=engine,name=web
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=dynamic.username.password
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=security.config
java.nio:type=BufferPool,name=direct
amx:pp=/domain/configs/config[server-config]/network-config/network-listeners,type=network-listener,name=http-listener-2
amx:pp=/domain/configs/config[server-config]/iiop-service,type=orb
amx:pp=/domain/configs/config[default-config],type=java-config
amx:pp=/domain,type=property,name=administrative.domain.name
amx:pp=/domain,type=load-balancers
amx:pp=/domain/resources/jdbc-connection-pool[__TimerPool],type=property,name=databaseName
amx:pp=/domain/resources,type=jdbc-connection-pool,name=DerbyPool
amx:pp=/domain/configs/config[default-config]/thread-pools,type=thread-pool,name=thread-pool-1
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config],type=diagnostic-service
amx:pp=/domain/configs/config[default-config],type=security-service
amx:pp=/domain/configs/config[default-config],type=system-property,name=IIOP_LISTENER_PORT
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__TimerPool],type=JDBCDataSource,name=jdbc/__TimerPool,j2eeType=JDBCDataSource,J2EEServer=server,JDBCResource=jdbc/__TimerPool
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=PortNumber
amx:pp=/domain/configs/config[server-config],type=admin-service
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=DatabaseName
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[admin-listener]/http,type=file-cache
amx:pp=/domain/configs/config[server-config]/web-container/session-config,type=session-manager
java.lang:type=MemoryPool,name=Tenured Gen
amx:pp=/domain/configs/config[server-config]/thread-pools,type=thread-pool,name=admin-thread-pool
amx:pp=/domain/configs/config[server-config]/security-service,type=jacc-provider,name=simple
amx:pp=/domain/configs/config[server-config],type=monitoring-service
amx:pp=/domain/configs/config[server-config]/security-service,type=property,name=default-digest-algorithm
amx:pp=/domain/configs/config[default-config],type=network-config
amx:pp=/,type=mon
amx:pp=/domain/configs/config[default-config],type=mdb-container
amx-support:type=boot-amx
amx:pp=/domain/configs/config[default-config],type=transaction-service
amx:pp=/J2EEDomain/J2EEServer[server],type=JDBCResource,name=jdbc/__default,j2eeType=JDBCResource,J2EEServer=server
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[admin-listener]/http,type=file-cache
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[admin-listener],type=http
amx:pp=/domain,type=servers
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=admin-http-redirect
amx:pp=/domain/servers,type=server,name=server
amx:pp=/domain/configs/config[default-config]/availability-service,type=ejb-container-availability
amx:pp=/domain/configs/config[server-config]/admin-service,type=property,name=ipsRoot
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=response-policy
amx:pp=/domain/configs/config[server-config]/iiop-service,type=iiop-listener,name=orb-listener-1
java.lang:type=GarbageCollector,name=MarkSweepCompact
amx:pp=/domain/configs/config[server-config]/admin-service,type=das-config
amx:pp=/mon/server-mon[server],type=thread-system-mon,name=jvm/thread-system
amx:pp=/domain/resources,type=jdbc-resource,name=jdbc/__TimerPool
amx:pp=/J2EEDomain/J2EEServer[server]/JDBCResource[jdbc/__default],type=JDBCDataSource,name=jdbc/__default,j2eeType=JDBCDataSource,J2EEServer=server,JDBCResource=jdbc/__default
amx:pp=/domain/configs/config[server-config]/security-service/jacc-provider[default],type=property,name=repository
amx:pp=/domain/configs/config[server-config]/http-service,type=access-log
java.lang:type=ClassLoading
amx:pp=/domain/configs/config[server-config]/network-config/protocols,type=protocol,name=admin-listener
java.lang:type=Threading
amx-support:type=amx-loader,name=j2ee
amx:pp=/domain/configs/config[server-config]/security-service/auth-realm[admin-realm],type=property,name=file
amx:pp=/mon/server-mon[server],type=memory-mon,name=jvm/memory
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/network-config/protocols,type=protocol,name=admin-listener
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=debug
amx:pp=/domain/configs/config[server-config]/security-service,type=message-security-config,name=HttpServlet
amx:pp=/domain/configs/config[server-config]/web-container,type=session-config
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=User
amx:pp=/domain/configs/config[server-config]/thread-pools,type=thread-pool,name=thread-pool-1
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[default-config]/http-service,type=access-log
amx:pp=/mon,type=server-mon,name=server
amx:pp=/domain/servers/server[server],type=resource-ref,name=jdbc/__default
amx:pp=/domain/configs/config[default-config],type=http-service
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=property,name=dynamic.username.password
amx:pp=/domain/resources/jdbc-connection-pool[DerbyPool],type=property,name=Password
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[pu-protocol]/port-unification,type=protocol-finder,name=admin-http-redirect
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[file],type=property,name=file
amx:pp=/domain/configs/config[default-config]/security-service/auth-realm[admin-realm],type=property,name=file
amx:pp=/domain/configs/config[default-config]/iiop-service,type=orb
amx:pp=/domain/configs/config[default-config]/admin-service,type=das-config
amx:pp=/domain/configs,type=config,name=server-config
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=request-policy
amx:pp=/domain/configs/config[default-config],type=log-service
amx:pp=/domain/configs/config[default-config]/iiop-service,type=iiop-listener,name=SSL_MUTUALAUTH
amx:pp=/domain/configs/config[default-config],type=iiop-service
amx:pp=/domain/configs/config[server-config]/iiop-service,type=iiop-listener,name=SSL_MUTUALAUTH
amx:pp=/domain/configs/config[server-config]/security-service,type=auth-realm,name=certificate
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=ClientProvider
amx:pp=/domain/configs/config[server-config],type=security-service
amx:pp=/domain/configs/config[default-config]/security-service,type=message-security-config,name=SOAP
amx:pp=/mon/server-mon[server],type=runtime-mon,name=jvm/runtime
amx:pp=/domain,type=lb-configs
amx:pp=/J2EEDomain/J2EEServer[server],type=WebModule,name=__admingui,j2eeType=WebModule,J2EEServer=server,J2EEApplication=null
java.lang:type=MemoryPool,name=Perm Gen
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet],type=provider-config,name=GFConsoleAuthModule
amx:pp=/mon/server-mon[server],type=class-loading-system-mon,name=jvm/class-loading-system
java.lang:type=MemoryManager,name=CodeCacheManager
amx:pp=/domain/resources,type=jdbc-resource,name=jdbc/__default
amx-support:type=mbean-tracker
amx:pp=/domain/configs/config[server-config],type=network-config
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-2],type=http
amx:pp=/domain,type=configs
amx:pp=/domain/configs/config[server-config]/http-service,type=virtual-server,name=__asadmin
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ClientProvider
amx:pp=/domain/configs/config[default-config]/web-container/session-config,type=session-properties
amx-support:type=amx-loader,name=startup
amx:pp=/domain/configs/config[default-config],type=admin-service
amx:pp=/domain/configs/config[default-config]/network-config/network-listeners,type=network-listener,name=http-listener-2
amx:pp=/domain/configs/config[server-config],type=jms-service
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[sec-admin-listener]/http,type=file-cache
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[sec-admin-listener],type=ssl
amx:pp=/domain,type=clusters
amx:pp=/domain,type=nodes
amx:pp=/domain/configs/config[default-config]/thread-pools,type=thread-pool,name=admin-thread-pool
amx-support:type=AMXConfigLoader
amx:pp=/domain/configs/config[default-config]/ejb-container,type=ejb-timer-service
amx:pp=/domain/configs/config[default-config]/iiop-service/iiop-listener[SSL_MUTUALAUTH],type=ssl
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-2]/http,type=file-cache
amx:pp=/domain/resources,type=jdbc-connection-pool,name=__TimerPool
amx:pp=/domain/configs/config[default-config],type=jms-service
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=request-policy
amx:pp=/domain/configs/config[server-config]/network-config/network-listeners,type=network-listener,name=admin-listener
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config],type=system-property,name=JAVA_DEBUGGER_PORT
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=encryption.key.alias
amx:pp=/domain/configs/config[server-config]/network-config,type=network-listeners
amx:pp=/domain/secure-admin,type=secure-admin-principal,name="CN=localhost,OU=GlassFish,O=Oracle Corporation,L=Santa Clara,ST=California,C=US"
amx:pp=/domain/configs/config[server-config]/network-config/network-listeners,type=network-listener,name=http-listener-1
amx:pp=/domain/configs/config[server-config]/web-container/session-config,type=session-properties
amx:pp=/,type=bulk-access
amx:pp=/domain/configs/config[server-config],type=group-management-service
amx:pp=/domain/configs/config[default-config],type=management-rules
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ServerProvider],type=property,name=signature.key.alias
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[XWS_ClientProvider],type=response-policy
amx:pp=/domain/configs/config[default-config]/availability-service,type=jms-availability
amx:pp=/domain/configs/config[default-config]/admin-service,type=jmx-connector,name=system
amx:pp=/domain/configs/config[default-config]/http-service/virtual-server[server],type=property,name=default-web-xml
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=response-policy
amx:pp=/domain/configs/config[server-config]/network-config/protocols/protocol[http-listener-1]/http,type=file-cache
amx:pp=/domain/configs/config[server-config]/admin-service,type=property,name=adminConsoleDownloadLocation
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[sec-admin-listener],type=http
amx:pp=/domain/configs/config[server-config],type=transaction-service
java.lang:type=OperatingSystem
amx:pp=/J2EEDomain/J2EEServer[server],type=JVM,j2eeType=JVM,J2EEServer=server
amx:pp=/,type=pathnames
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=security.config
amx-support:type=amx-loader,name=ext
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[SOAP],type=provider-config,name=XWS_ServerProvider
amx:pp=/domain/configs/config[server-config]/security-service/message-security-config[HttpServlet]/provider-config[GFConsoleAuthModule],type=response-policy
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ServerProvider],type=property,name=debug
amx:pp=/domain/configs/config[default-config]/network-config/protocols/protocol[pu-protocol]/port-unification,type=protocol-finder,name=http-finder
amx:pp=/J2EEDomain/J2EEServer[server],type=JDBCResource,name=jdbc/__TimerPool,j2eeType=JDBCResource,J2EEServer=server
amx:pp=/domain/configs/config[server-config],type=http-service
amx:pp=/,type=domain
amx:pp=/domain/configs/config[default-config]/security-service/message-security-config[SOAP]/provider-config[ClientProvider],type=property,name=encryption.key.alias
amx:pp=/domain,type=applications
amx:pp=/domain/configs/config[default-config],type=system-property,name=IIOP_SSL_MUTUALAUTH_PORT
amx:pp=/domain/configs/config[server-config]/iiop-service/iiop-listener[SSL],type=ssl
amx:pp=/J2EEDomain,type=J2EEServer,name=server,j2eeType=J2EEServer
amx:pp=,type=domain-root
JMImplementation:type=MBeanServerDelegate

As this image and the small-font text demonstrate, "booting AMX" has led to GlassFish exposing significant monitoring and management capabilities via JMX.

Conclusion

This post has demonstrated using Groovy to explore the standard JMX MBeans provided by the Java Platform as well as the AMX beans that GlassFish provides. The last Groovy code snippet invokes GlassFish's "bootAMX" operation to unleash significantly more GlassFish JMX-based monitoring and managing capabilities. Although all of this could be accomplished with a GUI-based JMX client such as JConsole or JVisualVM, there are advantages at times to running things in scripts. Using Groovy as demonstrated in this post allows scripts to easily interact with GlassFish for administration.