Contains the implementation of the external JMX APIs from org.apache.geode.admin.jmx.

JMX Support in GemFire

Our goal was to provide JMX administrative support in GemFire. The design was influenced by these important factors:

From a functional stand point, the JMX support was supposed to provide most of the same administrative and operational monitoring that the console already provides. In some cases we limited the functionality due to security concerns and in others because it was hard to express the features as JMX beans. The JMX Agent also provides some functionality (such as Health monitoring) that is not currently in the console.

The Agent communicates with the distributed system using the same distribution manager {@link org.apache.geode.distributed.internal.DistributionManager} as the console uses and thus has the same requirements that determine what distributed system it can manage. Although the Console currently supports managing multiple distributed systems, we decided that a given Agent should only be able to manage a single system. We have not tested the use of more than one Agent for the same system, however nothing currently prohibits this.

We decided to develop a simple public Admin API which in essence wraps the internal.admin API that the Console currently uses extensively. The Admin API also contains implementations of new functionality not in internal.admin. Our JMX support is an extension to this Admin API. In an overly simplified view, the GemFire JMX MBeans are ModelMBeans that manage instances of the Admin API objects housed in the Agent's MBeanServer.

The selected architecture consists of a Daemon Agent, which exists in a separate VM that GemFire does not depend on. This Agent hosts an MBeanServer, instances of any and all MBeans registered for managing a GemFire distributed system, and server connectors/adaptors that various types of clients can connect to.

The two server connectors we selected are the HttpAdaptor and the RMI Connector. The HttpAdaptor provides an HTML user interface of all MBeans in the MBeanServer. Although this generic UI is not as rich as an RMI client (or the GemFire Console) could be, it provides a functional and easy to use UI with no development required. The JMX Remote specification details the standard connectors. Although the HttpAdaptor is not required by this Sun spec. it is included in some form with all JMX implementations that I investigated. It should be noted that our JMX Agent currently starts up the HttpAdaptor, but not the RMI Connector. The latter is deferred as later work since some form of client must be developed for testing. Further research may also uncover a generic, configurable open-source RMI client for JMX. The GemFire Console could in theory be reworked as an RMI Connector client, but this is not currently planned.

Two open-source JMX implementations made it to our final review for consideration: XMOJO and MX4J. The decision to go with MX4J was based mainly on our perceptions of MX4J being more active and widespread in use. Additionally, XMOJO is associated with AdventNet which produces commercial products. This made MX4J seem more true to open-source and safer from corporate tampering.

ModelMBeans are very dynamic and capable of managing aggregate resources. Use of a ModelMBean entails specifying meta-data to an instance of javax.management.modelmbean.RequiredModelMBean. This meta-data identifies the manageble resource(s) which can consist of a single object, or many objects, including those in one VM or in any number of distributed VMs. We decided to subclass classes in the Admin API in order to massage them a little and make them easier to use as a managed resource by the ModelMBean. For example, org.apache.geode.admin.GemFireManager represents a type of member in a GemFire system which manages shared memory. When an MBean is registered for managing the GemFireManager, the JMX Agent instantiates a "JMX friendlier" subclass: org.apache.geode.admin.jmx.internal.GemFireMangerJmxImpl. Comparison of this class with the non-JMX super class org.apache.geode.admin.internal.GemFireManagerImpl will illustrate what "JMX friendly" means better than trying to explain it here...

One standard approach to defining a ModelMBean is to programmatically build the necessary meta-data. The folks on the Tomcat approach developed a better solution... an XML definition file which Jakarta Commons-Modeler parses to create the meta-data objects required to definie the ModelMBean. We currently have our XML descriptor file at org.apache.geode.admin.jmx.mbeans-descriptors.xml. Commons-Modeler can be found at http://jakarta.apache.org/commons/modeler/

Here's a quick run-down of the Admin and JMX pkgs in GemFire...

org.apache.geode.admin

org.apache.geode.admin.internal

org.apache.geode.admin.jmx

org.apache.geode.admin.jmx.internal

Some caveats, workarounds, and GemFire bugs to be aware of:

1) MX4J uses XSLT style sheets that are intended to work with the latest version of Xalan. JDK1.4.2 bundles an older version of Xalan which generates warnings.

2) MX4J's implementation of javax.management.modelmbean.RequiredModelMBean contains a bug such that any return type defined as an array of some object results in it attempting to Class.forName the class name with the "[]" still in the name of the class. Our current workaround is to return java.lang.Object where we'd like to return an array of some non-primitive type. I hope to look at this closer and work with MX4J to correct it (unless that latest code base at MX4J already has a fix).

3) Our MBeans currently have some return types of Admin interfaces and GemFire MBean types. This is not recommended. The correct approach is to return javax.management.ObjectName or an array of ObjectName so that remotability is not broken. We have a bug filed to correct this in GemFire.

4) Commons-Modeler provides a simple, incomplete implementation of ModelMBean called org.apache.commons.modeler.BaseModelMBean. We decided to use the standard RequiredModelMBean which all JMX implementations are required to supply. The JMX spec details several "managed resource types" that a ModelMBean can support. ObjectReference is the type that both MX4J's RequiredModelMBean and Modeler's BaseModelMBean support. However, MX4J is more strict in it's interpretation of the spec which spells it out as "ObjectReference". Modeler's BaseModelMBean performs no enforcement and simply assumes it is managing type ObjectReference. Modeler has a bug in org.apache.commons.modeler.ManagedBean because it specifies "objectReference" which is the incorrect case in comparison to the specification. I intend to work with the Jakarta-Commons folks to change this to comply with the spec. The Modeler will use BaseModelMBean by default even though it is actually depending on a JMX implementation such as MX4J or XMOJO. org.apache.geode.admin.jmx.internal.MBeanUtil tells Modeler to use RequiredModelMBean instead and also uses org.apache.geode.admin.jmx.internal.SpecManagedBean as a workaround to the whole "objectReference" issue. You could feasibly use org.apache.commons.modeler.BaseModelMBean or RequiredModelMBean.

Capabilities currently supported in GemFire via JMX:

TODO:

See also deferred backlog from Hardening/Operations sprint

It's very easy to use the JMX Monitor service to monitor an MBean's attribute and fire JMX Notifications. Statistics and Config Parameters are "MBean attributes".

Using Third Party Management Consoles/Tools

There is information on AdventNet's website detailing the use of their SNMP Adaptor to enable connectivity to HP OpenView, IBM Tivoli, CA Unicenter, and other SNMP managers.

Aside from using AdventNet's SNMP Adaptor, HPOpenView is the first I've seen of the big players offering what appears to be a way to plug our JMX support into their product for management of GemFire. I haven't had a chance to look at it yet. Most likely their SPI supports writing a small amount of glue code to have their console become an RMI client to our JMX VM (which is essentially an MBeanServer "server" that hosts our MBeans). I think it's unlikely that such vendors would want to support hosting of third party MBeans and the classpath/versioning headaches involved in that. The JMX Remote specification really is meant to cover how such console's would perform the remote connections to custom MBeans in external VMs/products. It is likely that the other management tool vendors such as Tivoli have a similar SPI available or in the works.

See HP Dev Resource Central's page on JMX for info and links on using JMX to integrate with HP OpenView. You can also sign up for HP Developer News

Deferred Backlog from Hardening/Operations Sprint

Note: some of these tasks may no longer be worded properly or even needed...