Introducing Distributed XA Transaction Support

We're as happy as a bear in a koi pond to announce support for Distributed (XA) Transactions in Immutant.

Messaging and Caching resources in Immutant are now automatically transactional and XA capable. And we make it easy for you to create DataSources for your XA compliant SQL databases so that you can then define transactions incorporating all three types of resources in your Clojure applications.

Some Background

X/Open XA is a standard specification for allowing multiple, independent resources to participate in a single, distributed transaction using a two-phase commit (2PC) protocol.

Say your application stores data in more than one place, perhaps an Oracle database and a Postgres database. When a function in your application writes data to those two databases, XA can ensure that it doesn't leave your data in an inconsistent state when the Oracle database fails. ;-)

To accomplish this, the commit and rollback methods are invoked not on any single resource like a JDBC or JMS connection but on a TransactionManager instead. Its job is to coordinate the commit or rollback of each resource involved in a particular transaction.

Defining an XA Transaction

Let's start with an example:

  (ns xa.example
    (:require [immutant.xa :as xa]
              [immutant.cache :as cache]
              [immutant.messaging :as msg]
              [ :as sql]))
  (defn do-three-things []
     (sql/with-connection {:datasource my-ds}
       (sql/insert-records :things {:name "foo"}))
     (cache/put my-cache :a 1)
     (msg/publish "/queue/test" "success!")))

The do-three-things function will insert a record into a SQL database, write an entry to a cache and publish a message to a queue, all within a single transaction. When it completes, either all of those things will have happened or none will, depending on whether an exception is tossed from the body passed to xa/transaction.

By the way, don't let that {:datasource my-ds} spec throw you just yet. We're going to discuss that in a minute.

So the xa/transaction macro starts a transaction, executes its body, and then commits the transaction unless an exception is caught, in which case the transaction is rolled back.

Transaction Scope

I lied a little. The xa/transaction macro is really just an alias for the required macro in the immutant.xa.transaction namespace, which is one of six macros matching the transaction attributes for JEE Container-Managed Persistence: required, requires-new, not-supported, supports, mandatory, and never. According to that spec, required is the default, so we alias it in the main immutant.xa namespace.

These macros allow you to control the scope of your transactions when your functions call each other. For example,

  (ns xa.example ...)
  (defn foo []

Here, we have one function, foo, defining a transaction that calls another function, do-three-things that, as you recall from above, seemingly defines another transaction. Or does it? In fact, the required macro won't start a new transaction if there's already one associated with the current thread. It'll simply include its body in the transaction started by the caller. If we really wanted a new transaction, we'd call requires-new inside do-three-things.

Here's another example:

  (ns xa.example
    (:require [immutant.xa.transaction :as tx]))
  (tx/required           ; begin tx #1
   (tx/not-supported     ; suspend tx #1
   (tx/requires-new      ; begin tx #2
    (three))             ; commit tx #2
   (throw (Exception.))) ; rollback tx #1

Here we have a function, one, running within a transaction that is suspended prior to calling the function two, that runs completely outside of any transaction, after which a second transaction is started before calling the function, three.

We then toss an exception (we could've also called tx/set-rollback-only) that causes everything we did in one to rollback. The exception does not affect what we did in two or three, however.

Incidentally, any exception tossed in two or three would also rollback the actions of one since all the macros re-throw whatever they catch.

Creating an XA DataSource

So now we'll discuss that {:datasource my-ds} spec from the first example.

To include your database in a distributed transaction, you need to create an XA DataSource for it. Do this using the immutant.xa/datasource function. It will expect the appropriate JDBC driver for your database to be available in the classpath, so you'll need to add one of the following to your Leiningen project.clj:

  (defproject foo "1.0.0-SNAPSHOT"
    :dependencies [[com.h2database/h2 "1.3.160"]              ; H2
                   [org.clojars.gukjoon/ojdbc "1.4"]          ; Oracle
                   [org.clojars.kjw/mysql-connector "5.1.11"] ; MySQL
                   [postgresql "9.0-801.jdbc4"]               ; Postgres
                   [net.sourceforge.jtds/jtds "1.2.4"]        ; MS SQL Server
                   [java.jdbc "0.2.2"]])

The comments on the right indicate the database types we currently support, and the versions above have been successfully tested on Immutant.

With the driver available, all that's left is to create the DataSource. Here are some examples from our integration tests:

  (defonce my-ds (xa/datasource "h2" {:adapter "h2" :database "mem:foo"}))
  (defonce my-ds (xa/datasource "oracle" 
                                {:adapter "oracle"
                                 :host ""
                                 :username "myuser"
                                 :password "mypassword"
                                 :database "mydb"}))
  (defonce my-ds (xa/datasource "mysql" 
                                {:adapter "mysql"
                                 :host ""
                                 :username "myuser"
                                 :password "mypassword"
                                 :database "mydb"}))
  (defonce my-ds (xa/datasource "postgres" 
                                {:adapter "postgresql"
                                 :username "myuser"
                                 :password "mypassword"
                                 :database "mydb"}))
  (defonce my-ds (xa/datasource "mssql" 
                                {:adapter "mssql"
                                 :host ""
                                 :username "myuser"
                                 :password "mypassword"
                                 :database "mydb"}))

To use one of those in a connection spec, you should associate it with the :datasource key, like so:

  (jdbc/with-connection {:datasource my-ds}
    (jdbc/create-table :things [:name "varchar(50)"]))

This should of course work with any Clojure SQL library built on, e.g. Korma, ClojureQL, Lobos, etc.

See the manual for more details.


XA is not for every application. It's useful when you have multiple JDBC backends or you need to synchronize your JDBC and JMS (HornetQ messaging) calls. Transactional data-grids (Infinispan caching) are also often handy, so we feel good about making all of these resources automatically transactional in Immutant, not to mention providing clean Clojure interfaces for them.

Distributed transactions are available now in our latest incremental builds and will of course be included in our upcoming 0.2.0 release of Immutant expected in the next week or so. As always, feel free to find us in the normal channels if you have any questions.

WTF is an enterprise-grade app server?

That's a great question. We get it a lot. It was asked on Hacker News in response to our birth announcement of Immutant yesterday.

So as someone who has worked in multiple enterprises, and now works for a company whose primary customers are enterprises, and since I routinely toss around the term like everyone knows what it means, and since it's one of those awful terms that means something different to everyone...

I'm compelled to answer the question, "WTF is an enterprise-grade application server?"

The answer requires answering another question first:

What's an Enterprise?

Here's my definition: it's an organization of mostly-independent teams building and maintaining applications used by other internal groups and external customers.

The key identifier of an enterprise is "a group of groups" in which more than one of them builds software.

There's usually a single "system operations" group. Their life is hell. They're ultimately responsibile for the security and integrity of the organization's data and the lifecycles of all the applications built for and used by all the other groups in the organization.

Did I mention their life is hell?

They can't afford to support all the myriad databases, message queues and web frameworks each team might decide to build their apps around. Not to mention supporting multiple languages! They prefer a single, "all in the tin" solution. Sadly, that usually means .Net or Java.

So enterprise-grade implies a capacity for supporting these types of environments. It usually involves, among other things, messaging so that the disparate apps may communicate, transactions to ensure the integrity of distributed data stores, and clustering, not only for scalability, but also to allow the lifecycles of the multiple versions of the apps to vary independently. And oh yeah, it also has to stay up, all the time.

What's an App Server?

An app server is a single product that provides all those services. It's a multi-threaded process that, once started, provides any app deployed to it with a web server, messaging, transactions, scheduling, security, caching, clustering, and more. JBoss AS7 is one such app server. It's open-source and it's fast.

Unfortunately, most popular commercial Java app servers provide those robust enterprise services at a very high price. Not only in dollars, but in the form of complex Java API's, overly-configured slow implementations, vendor-negotiated standards and general "acronym soup".

But it doesn't have to be that way. The Rails "convention over configuration" mantra inspired the creation of TorqueBox, encapsulating the enterprise-grade services provided by JBoss AS7 behind simple Ruby API's. The immediate goal for Immutant is to do the same with Clojure.

But by integrating any JVM-based language with JBoss AS7, the ultimate goal is to keep all the groups in an enterprise -- both operations and development -- happy. Or at least significantly less soul-sucky and hopefully more productive.

But why should you care?

Maybe you shouldn't. Maybe you only need a web server and a database, maybe just a JVM! Maybe you're not in a group of groups. Heck, maybe you're not even in a group!

Maybe you're perfectly content being both developer and admin for your apps and the various external processes on which they depend, or you're quite happy delegating some of those responsibilities to the fine folks at Heroku, EngineYard, or someone else.

If that's the case, you are probably a very happy person, and I'm very happy for you!

But if you feel you could benefit from a more integrated "all in the tin" solution, and especially if you're thinking along polyglot lines, I think TorqueBox (and Immutant, once it matures) is a compelling alternative, whether you work in an enterprise or not.