A REPL for the RHQ Plugin Container

Overview
RHQ plugins run inside of a plugin container that provides different services and manages the life cycles of  plugins. The plugin container in turn runs inside of the RHQ agent. If you are not familiar with the agent, it is deployed to each machine you want RHQ to manage. You can read more about it here and here. While the plugin container runs inside of the agent, it is not coupled to the agent. In fact is it used quite a bit outside of the agent. It is used in Embedded Jopr which was intended to be a replacement for the JMX web console in JBoss AS. The plugin container is also used a lot during development in automated tests.

My teammate, Heiko Rupp, has developed a cool wrapper application for the plugin container. It defines a handful of commands for working with the plugin container interactively. What is nice about this is that it can really speed up plugin development. Heiko has written several articles about the standalone container including Working on a standalone PluginContainer wrapper and I love the Standalone container in Jopr (updated). After reading some of his posts I got to thinking that a REPL for the plugin container would be really nice but not just any REPL though. I was thinking specifically about Clojure's REPL.

I have spent some time exploring the different ways Clojure could be effectively integrated with RHQ. There is little doubt in my mind that this is one of them. I recently started working on some Clojure functions to make working with the plugin container easier. I am utilizing Clojure's immutable and persistent data structures as well as some of the other great language features such as first class functions and multimethods. I am trying to make these functions easy enough to use so that someone who might not be a very experienced Clojure programmer might still find them useful during plugin development and testing.

Getting the Code
The project is available on github at https://github.com/jsanda/clj-rhq. It is built with leiningen so you want to get it installed. I typically run a swank server and connect from Emacs, but you can also start a REPL session directly if you are not an Emacs user. The project pulls in the necessary dependencies so that you can work with plugin container-related classes as you will see in the following sections.

Running the Code
These steps assume that you already have leiningen installed. First, clone the project:

git clone https://jsanda@github.com/jsanda/clj-rhq.git

Next, download project dependencies with:

lein deps

Some plugins reply  on native code provided by the Sigar library which you should find at clj-rhq/lib/sigar-dist-1.6.5.132.zip. Create a directory in lib named native and unzip sigar-dist-1.6.5.132.zip there. The project is configured to look for native libraries in lib/native.

Finally, if you are using Emacs run lein swank to start a swank server; otherwise, run lein repl to start a REPL session on the command line.

Starting/Stopping the Plugin Container

The first thing I do is call require to load the rhq.plugin-container namespace. Then I call the start function. The plugin container emits a line of output, and then the function returns nil. Next I verify that the plugin has started up by calling running?.  Then I call the stop function to shutdown the plugin container and finally call running? again to verify that the plugin container has indeed shutdown.

Executing Discovery Scans
So far we have looked at starting and stopping the PC. One of the nice things about working interactively in the REPL is that you are not limited to a pre-defined set of functions. If rhq.plugin-container did not offer any functions for executing a discovery scan, you could write something like the following:


The pc function simply returns the plugin container singleton which gives us access to the InventoryManager. We call InventoryManager's executeServerScanImmediately method and store the InventoryReport object that it returns in a variable named inventory-report. Alternatively you can use the discover function.


On the first call to discover we pass the keyword :SERVER as an argument. This results in a server scan being run. On the second call, we pass :SERVICE which results in a service scan being run. If you invoke discover with no arguments, a server scan is executed followed by a service scan. The two inventory reports from those scans are returned in a vector. The use of the count function to see how many resources were discovered is a good example that demonstrates how you can easily use functions defined outside of the rhq.plugin-container namespace to provide additional capabilities and functionality.

Searching the Local Inventory
Once you have the plugin container running and are able to execute discovery scans, you need a way query the inventory for resources with which you want to work. The inventory function does just that. It can be invoked in one of two ways. In its simpler form which takes no arguments, it returns the platform Resource object. In its more complex form, it takes a map of pre-defined filters and returns a lazy sequence of those resources that match the filters.


inventory is invoked on line 1 without any arguments, and then a string version of it is returned with a call to str. The type is Mac OS X indicating that the object is in fact the platform resource. On line 5 we invoke inventory with a single filter to include resources that are available.  That call shows that there are 62 resources in inventory that are up.  On line 7 we query for resources that are a service and see that there are 60 in inventory. On line 9 we specify multiple filters that will return down services. When multiple filters are specified, a resource must match each one in order to be included in the results. On line 10 we query for webapps from the JBossAS plugin. On line 13 we specify a custom filter in the form of an anonymous function with the :fn key. This filter finds resources that define at least two metrics.

Conclusion
We have looked at a number of functions to make working with the plugin container from the REPL a bit easier. Each function should also include a useful docstring as in the following example,


We have only scratched the surface with the functions in the rhq.plugin-container namesapce. In some future posts we will explore invoking resource operations, updating resource configurations, and deploying resources like EARs and WARs.

Writing an RHQ Plugin in Clojure

Clojure is a new, exciting language. My biggest problem with it is that I do not find enough time to work with it. One of the ways I am trying to increase my exposure to Clojure is by exploring ways of integrating it into RHQ. RHQ is well-suited for integrating non-Java, JVM languages because it was designed and built to be extended. In previous posts I have talked about various extension points including agent plugins, server plugins, and remote clients.

I decided to write an agent plugin in Clojure. If you are not familiar with RHQ plugins or what is involved with implementing one, check out this excellent tutorial from my colleague Heiko. Right now, I am just doing exploratory work. I have a few goals in mind though as I go down this path.

First, I have no desire to wind up writing Java in Clojure. By that I mean that I do not want to get bogged down dealing with mutable objects. One of the big draws to Clojure for me is that it is a purely functional language with immutable data structures; so, as I continue my exploration of integrating Clojure with RHQ, I want to write idiomatic Clojure code to the greatest extent possible.

Secondly, I want to preserve what I like to think of as the Clojure development experience. Clojure is a very dynamic language in which functions and name spaces can be loaded and reloaded on the fly. The REPL is an invaluable tool. It provides instant feedback. In my experience Test-Driven Development usually results in short, quick development iterations. TDD + REPL produces extremely fast development iterations.

Lastly, I want to build on the aforementioned goals in order to create a framework for writing RHQ plugins in Clojure. For instance, I want to be able to run and test my plugin in a running plugin container directly from the REPL. And then when I make a change to some function in my plugin, I want to be able to just reload that code without having to rebuild or redeploy the plugin.

Now that I have provided a little background on where I hope to go, let's take a look at where I am currently. Here is the first cut at my Clojure plugin.


I am using gen-class to generate the plugin component classes. As you can see, this is just a skeleton implementation. Here is the plugin descriptor.


I have run into some problems though when I deploy the plugin. When the plugin container attempts to instantiate the plugin component to perform a discovery scan, the following error is thrown:


I was not entirely surprised to see such an error because I have heard about some of the complexities involved with trying to run Clojure in an OSGi container, and the RHQ plugin container shares some similarities with OSGi. There is a lot of class loader magic that goes on with the plugin container. For instance, each plugin has its own class loader, plugins can be reloaded at runtime, and the container limits the visibility of certain classes and packages. I came across this Clojure Jira ticket, CLJ-260, which talks about setting the context class loader. Unfortunately this did not help my situation because the context class loader is already set to the plugin class loader.

After spinning my wheels a bit, I decided to try a different approach. I implemented my plugin component class in Java, and it delegates to a Clojure script. Here is the code for it.


And here is the Clojure script,


This version deploys without error. I have not fully grokked the class loading issues, but at least for now, I am going to stick with a thin Java layer that delegates to my Clojure code. Up until now, I have been using leiningen to build my code, but now that I am looking at a mixed code base, I may consider switching over to Maven. I use Emacs and Par Edit for Clojure, but I use IntelliJ for Java. The IDE support for Maven will come in handy when I am working on the Java side of things.

Find and Replace

I came across a post on the clojure users list the other day that discussed some code that read a file, removed a specified character from each line in the file, and the wrote the file back out to disk. It got me thinking about how I might implement some basic file editing functions. I started with a find function. (Please note for brevity and for clarity the code snippets that follow only include the ns function call when I introduce a function from a different name space for the first time.) Here is the initial implementation.

(ns clj-sandbox.io
  (:use [clojure.contrib.duck-streams :only [read-lines]]))   

(defn find [file s]
  (filter #(.contains % s) (read-lines file)))

This returns a sequence of all lines in the file that contain the string s. So far so good. I would like to get the line numbers as well though. I was ready to implement a solution using recur, but then I started thinking that there is probably a more functional way of achieving what I want. I started reading through the chapter on functional programming in Programming Clojure (chapter 5) to spark some ideas. That led me to some of the example code which in turn led me to clojure.contrib.seq/indexed. And now I have this function,

(ns clj-sandbox.io
  (:use [clojure.contrib.duck-streams :only [read-lines]])
  (:use [clojure.contrib.seq :only [indexed]]))

(defn find [file s]
  (map (fn [match] {:line-number (inc (first match)) :text (second match)})
       (filter #(.contains (second %) s) (indexed (read-lines file)))))

which returns a lazy sequence of lines and corresponding line numbers that contain at least one occurrence of s. The sequence is comprised of maps containing two keys, :line-number and :text. At this point I am pretty happy with my function and am ready to consider some additional enhancements. First, I want the capability to get back just the line numbers or just the text of each line. This is easily accomplished with a bit of refactoring.

(defn find
  ([file s opt] (map #(opt %) (find file s)))
  ([file s]
     (map (fn [match] {:line-number (inc (first match)) :text (second match)})
 (filter #(.contains (second %) s) (indexed (read-lines file))))))

Now I have a version that takes a third argument, opt, which should be one of the map keys, :line-number or :text. To get a sequence of just the lines numbers I can write,

user> (find "myfile" "mystring" :line-number)

At this point I am satisfied with find and ready to move onto a replace function. Here is a first cut at replace.

(ns clj-sandbox.io
  (:use [clojure.contrib.duck-streams :only [read-lines]])
  (:use [clojure.contrib.seq :only [indexed]])
  (:use [clojure.contrib.string :only [replace-str]]))

(defn replace [file s r] (map #(replace-str s r (:text %)) (find file s)))

This function replaces every occurrence of s with r, and the results are returned as a lazy sequence. I need to update the function to write the changes back to the file. Initially I consider,

(defn replace [f s r]
  (write-lines (str "." f) (map #(replace-str s r (:text %)) (find f s)))
  (copy (file (str "." f)) (file f)))

This implementation however is problematic. Changes are written to a copy of the file and when that is done, the original file is overwritten is replaced with the copy. Only those lines that match the search string are written back to the file; so, we wind up altogether losing lines that should be left intact. The function needs to write every line back to the file, including those that have not been modified. I decide to take out the call to find since it does not return all lines and replace it with a call to read-lines.

(ns clj-sandbox.io
  (:use [clojure.contrib.duck-streams :only [read-lines]])
  (:use [clojure.contrib.seq :only [indexed]])
  (:use [clojure.contrib.string :only [replace-str]])
  (:use [clojure.java.io :only [file]]))


(defn replace [f s r]
  (write-lines (str "." f) (map #(replace-str s r %) (read-lines f)))  
  (copy (file (str "." f)) (file f)))

This gives me the behavior that I want; however, I notice some duplication with creating the backup file name. That can easily be eliminated with a let binding.

(defn replace [f s r]
  (let [new-file (str "." f)]
    (write-lines new-file (map #(replace-str s r %) (read-lines f)))
    (copy (file new-file) (file f))))

Now I have a couple functions that can be used to perform a global find and replace. I spent some time working on functions that find/replace a specified number of matches. There is some additional effort needed for these however because I need to keep a running total of matches. Suppose I want the first 3 matches of some string. The first two occur on line 6, and the third match occurs on line 9. With the find function that has been discussed here, we can determine that we have matches on lines 6 and 9, but we cannot determine how many matches there are per line. I will try to revisit this in a future post. The source code for this can be found on github.