If a Test Case Falls in the Woods...

I've long been a fan of the Spring Framework and the base test classes they provide. A few years ago, a fellow consultant on a project introduced me to the concept of rollback testing.

Rollback testing implies that a transaction begins and ends with your test method. At the end of your test method, the transaction rolls back and nothing is persisted to the database. This certainly eliminates the problem of cleaning up state that is left hanging around at the end of an integration test. You simply have to do...nothing!

Spring also provides some base classes that you may extend to get this functionality with no additional work on your end. Their classes AbstractTestNGSpringContextTests and AbstractJUnit4SpringContextTests provide a convenient superclass that your classes can extend to benefit from this feature.

I'm a user of TestNG, so I used that flavor of Spring abstract test class as my starting point. From there, it is very simple to add a test that will validate that the create method works the way it is supposed to. I am using JPA with Hibernate as my JPA provider.

@ContextConfiguration(locations = {"classpath*:META-INF/spring.xml"})
public class TestDomain 
       extends AbstractTransactionalTestNGSpringContextTests {

    @Autowired(required = true)
    private DomainDao domainDao;

    @Test
    public void testCreate() {
        final Integer pk;
        {
            Domain domain = new Domain("md", "mccoy");
            pk = domainDao.insert(domain);
            Assert.assertNotNull(pk);
        }

        {
            final Domain domain = domainDao.select(pk);
            Assert.assertEquals(domain.getDomainId(), pk);
            Assert.assertEquals(domain.getDomainName(), "mccoy");
            Assert.assertEquals(domain.getTld(), "md");
            Assert.assertEquals(
                domain.getStatus(), Domain.Status.CANDIDATE);
        }
    }
}

You can imagine how pleased I was to see my integration test passed the first time I ran it. Damn, I'm good.

I've had the pleasure of working with a friend who has never met an ORM he trusts, and he has taught me many times to look at the log files and examine the SQL that is being executed. He does this to understand better how the ORM is approaching a problem. When I looked at the logs, I was surprised to see a lot of logging, but no SQL statements. I checked to make sure I had told Hibernate to log SQL and I had. In fact, I saw SQL statements in some other places, but none associated with this particular unit test.

Timber!

So why was there no SQL executed against my database. My DAO classes have a transactional boundary, so any call to them (like insert and select in my example), will create a transaction and close the transaction. When a transaction closes, it is a signal to Hibernate to flush the SQL statements and commit the transaction.

The reason I see no SQL is because my transaction begins and ends in relation to the test case, not because of my DAO. The transactional context surrounding my DAO calls recognize that a transaction is already in progress and simply participate in the existing transaction. When using Spring's base test classes, all transaction will rollback at the end of the test case. This means, unless something happened that forced Hibernate to flush SQL, there will be no SQL written to the database.

Is This A Problem?

Some people might argue whether this is a problem or not. After all, you are testing your integration code, not Hibernate's ability to persist your objects. In the past, I have bought that argument, however by closely watching my log files it is easy to see that some of my code can pass muster even though it is woefully wrong.

This rule sounds a bit obvious:

When changing the typical transactional boundaries of your application to surround the test case and not the service tier or DAO tier you may not be testing database impact on your application.

Even though the rule is obvious, the impact may be hard to tell when you are writing code. For example, in my testCreate() method above, this test will pass even if a annotate my domain class with a non-existent database table. I can specify @Table(name = "he_hate_me") and my test will pass. I can also supply the wrong schema  to this annotation or in my persistence.xml config and the test still will pass.

As you might expect, if you have a table with a unique index (other than the primary key) and you have a unit test that performs multiple inserts attempting to trip the constraint exception; these tests will also pass.

Workarounds

For the reasons I have shown, I would consider blind use of rollback testing to be a bad thing. Most JPA providers will function the same way as Hibernate and only flush statements when Hibernate feels it is necessary.

Of course, Hibernate can be explicitly told to flush, and there isn't a real downside to doing this in an integration test. But where and when should you flush? I find that a call to flush is warranted any time you make a call to a service method or DAO where a transactional context would normally be started if one wasn't already in progress.

Here is the previous example modified to show the call to flush().

@ContextConfiguration(locations = {"classpath*:META-INF/spring.xml"})
public class TestDomain 
       extends AbstractTransactionalTestNGSpringContextTests {

    @Autowired(required = true)
    private DomainDao domainDao;

    @Test
    public void testCreate() {
        final Integer pk;
        {
            Domain domain = new Domain("md", "mccoy");
            pk = domainDao.insert(domain);
            flush();
            Assert.assertNotNull(pk);
        }

        {
            final Domain domain = domainDao.select(pk);
            flush();
            Assert.assertEquals(domain.getDomainId(), pk);
            Assert.assertEquals(domain.getDomainName(), "mccoy");
            Assert.assertEquals(domain.getTld(), "md");
            Assert.assertEquals(
                domain.getStatus(), Domain.Status.CANDIDATE);
        }
    }
}

Flush Your Problems Away

Both Hibernate's Session and JPA's EntityManager interfaces have a flush() method. It can be a bit tricky to obtain the EntityManager interface in order to call flush in it, so I use a base class in my integration tests that includes a flush() method. The implementation looks like this:

@ContextConfiguration(locations = {"classpath*:META-INF/spring.xml"})
public class BaseTest 
       extends AbstractTransactionalTestNGSpringContextTests {

    protected EntityManager sharedEntityManager;

    protected void flush() {
        sharedEntityManager.flush();
    }

    @Autowired(required = true)
    public void setEntityManagerFactory(EntityManagerFactory emf) {
        sharedEntityManager = SharedEntityManagerCreator.
            createSharedEntityManager(emf);
    }
}

Some knowledgable Spring users will probably point out there is already an AbtractJpaTests class that exposes a sharedEntityManager. They would be right, and it takes care of load-time weaving cases that can make integration testing of JPA difficult to work with. Unfortunately for me, the good Spring folks tied this abstract class to JUnit (no TestNG version) and I'm using Hibernate which doesn't require load-time weaving.

Where Do You Generate Primary Keys?

Every once and a while I like to throw out any prior conceptions and biases I have amassed over the years and start with a fresh perspective. After all, how do you know what you take for granted on a regular basis is hard earned knowledge or simply just a case of "That's the way I've always done it."

A colleague and I have long debated the best approach to assigning primary keys on domain objects. He takes the stance that primary keys should be assigned prior to persistence (at instantiation), and mostly to be contrary, I take the opposite stance. Let the database assign the primary key, especially if it supports an IDENTITY or AUTOINCREMENT column type. To be truthful, I can find little fault in his approach. In fact, assigning a UUID as a primary key helps:

1. When you are adding transient entities to a collection prior to persistence.
2. To simplify equals and hash code implementations which can be quite a problem at times.
3. Keys are unique across all rows in the database which make it much easier for the DBA to perform a merge of tables as schemas evolve.

Perhaps I rely on the database mostly out of paranoia. His approach usually results in CHAR primary keys which have been shown to be less performant than INT/LONG keys. It is possible to reduce a UUID to a INT/LONG if you are willing to slightly increase your chances of a primary key collision.

The Hibernate guys, Gavin King and Christian Bauer have advocated the use of database-driven primary keys, and those guys seem to have a lot of practical experience and study in the field. Because of this, I decided to use an AUTOINCREMENT INT field for my primary keys in MySQL.

Test Driven Development

On my newest project, I threw out my personal biases and began with some unit tests. A typical unit test I write looks like this:

@Test
public void testCreate() {
 User user =
     new User("fred flintstone", "fred@bedrock.com");
 final Integer pk = userDao.insert(user);
 Assert.assertNotNull(pk);
}

It's fairly common for me to want the primary key of my newly inserted object. This can cause an issue for those of us who choose to let the database assign the primary key. At the time the insert method is called, the user's id value is null.

No problem, we just push responsibility for getting the primary key to the DAO tier.

In the DAO

I'm using JPA in my DAO tier. The JPA implementation of my DAO's insert method looks like this:

public void insert(final User user) {
 getEntityManager().persist(user);

/**
* Make an entity instance managed and persistent.
*/
public void persist(Object entity);

So here we can see that the persist() method, as mandated by the JPA specification, does not return a value; it is declared void. After issuing this call, our user object still has a value of null.

So what options do we have at this point? I mean, besides ditching JPA and going straight to JDBC which doesn't suffer from a handicapped abstraction layer?

One approach might be to make a call to retrieve the recently inserted object based on some unique properties of the object. In my particular case this is plausible, since the email address is a unique constraint in my database. Ironically however, if your object doesn't have any uniquely distinguishing set of properties, you might find yourself adding a dummy UUID to your class just to use as a unique identifier.

Re-fetch the Object

Even the benign sounding solution of fetching the recently inserted object from the database is not so clear cut. Even if there are unique properties to load the object, the object may not even be inserted in the database yet. I'm using Hibernate as my ORM, but I believe this issue would apply to most approaches that use a session cache as Hibernate does.

Often during a transaction, dozens of objects can be persisted or updated. By default, Hibernate defers the flush of SQL until the transaction ends, or until some other event forces a premature-flush. What this means is when calling getEntityManager().persist(user), no SQL is typically made to the database immediately.

The ORM has to be very smart about when it actually writes to the database, because when I follow the insert up with a getEntityManager.getUserByEmail(user.email), the ORM has to know that it is now time to execute the insert SQL call. After all, it's the only way the database is going to generate the primary key for the object.

Where to Fetch

So now the question becomes where does the fetch of the object belong. Do you fetch the object in the DAO in order to return the primary key, or do you force the client to fetch the object based on something other than the primary key?

// In UserDaoImpl
public Integer insert(final User user) {
 getEntityManager().persist(user);
 final User fetchUser =
         selectByEmail(user.getEmail());
 return fetchUser.getUserId();
}

// In Client
final User user =
 new User("fred flintstone", "fred@bedrock.com");
userDao.insert(user);
final User fetchUser =
 userDao.selectByEmail(user.getEmail());
Assert.assertNotNull(fetchUser);

Conclusion

In the end, the choice will obviously be yours. I'm interested to hear from others about your approach to this recurring problem. As for my colleague, his wisdom on this matter is proven by his lack of concern regarding any of these problems. He simply doesn't have to deal with it. I've chosen to heed to his example.

Who Needs Workflow?

Surprisingly enough, maybe you?

Workflow is one of those terms with which you have probably played buzzword bingo. If not the over-generalized category name, perhaps you have a bingo card covered in the acronyms that proliferate in this space, like the BP brothers -- BPM, BPEL, BPMN and BPDM. And, don't forget their webified cousins, WS-BPEL, WS-CDL and WSFL. Uh, uhh, BINGO!

For as many players in this space, it is surprising how few developers have ever laid their hands upon a workflow engine. Many companies tend to shy away from these offerings because they are part of a larger SOA initiative which takes a great amount of time and resources to roll out. There are also very few quality open source players in this area.

I've been meaning to dig in to see what makes a workflow engine tick, and a recent project afforded me the chance. It has a small workflow with twenty odd steps that has a few manual stages, credit card processing and email events. I could of written the process manually, but I wouldn't of learned anything new.

So now I can share with you some of my stumbles along the way and how workflow can function in a real world application. The first step is to choose your foil.

Keep in mind, I am not very knowledgeable of the players in the workflow arena. As I started this journey, I gained hands-on experience with only a few. Also note that I am developing personal software and price was a factor. :)

The Players

There are a boatload of players in the workflow market, and many have come and gone over the years. When I look at some of the players in this field, I sometimes get the feeling that programmers created a workflow for their in-house project and then open sourced it on the world. For many engines, there are probably only one or two applications that use it.

There are dozens of sites that contain a long list of commercial and open source workflow engines written in Java. Notably, OpenWFE was one of the more popular Java workflow engines and is now being developed as a Ruby application.

Open Source Workflow Engines Written in Java

www.manageability.org

Commercial Workflow BPM Vendors - Part I

Commercial Workflow/BPM Vendors - Part II

www.waria.com

While reviewing some of the different open source products, it dawned on me the world of the big standards belongs to the big commercial vendors as evidenced by the following chart from Gartner. That's fine with me because my needs are much simpler.

If anyone knows, I would be interested to see if any of these products are free or open source and if they are embeddable. Most that I inspected were commercial and standalone or integrated into an SOA or J2EE application server.

My Needs

I am modeling an order workflow, but I will probably create additional processes once I have handled my first priority. I need a workflow engine that I can embed in my application. I do not wish to run the workflow engine as a standalone server who's sole purpose in life is to process workflow lifecycles. These products have their place when the scale of an enterprise demands such needs, however my requirements are much more modest. It may be important for the engine I pick to have this capability or at least be able to cluster if scale becomes important.

I do want to be able to kick off tasks (execution processes in workflow-speak) and have the workflow engine shepherd the task along its path. At certain points along the workflow process I need the engine to delegate control to custom tasks that I have written in Java or Groovy. These tasks are very short and focus on a small part of the overall process like authorizing a credit card or checking the availability of a product.

I know there will be some situations when the workflow will have to pause for human intervention as might happen if a credit card was authorized successfully, but failed the capture stage. Or perhaps a human will manually review orders over a certain dollar amount to ensure the validity of the order. When these cases occur, the workflow is essentially paused while the user reacts to each situation on a case by case matter. This means that I will need to somehow track when these issues arise and bring them to the attention of the appropriate user. Once the employee handles the issue, control will return to the workflow engine to proceed based on the outcome of the manual intervention.

I also want an audit log that helps me visualize the path taken by the customer's order. This is helpful when dealing with customer service problems. I usually use the Spring Framework, and although I didn't need Spring to wrapper the engine for me, it would be nice if my java extensions to the workflow process could be configured within Spring.

My needs are simple. I don't need a graphical tool for manipulating the workflow, but neither will I turn one down if it is offered. I don't need identity support which allows the workflow engine to route tasks to users based on their roles in the system. I have one employee and that person is me.

Also note that I don't need any of the acronyms mentioned at the beginning of this post. I need a workflow engine, and I don't particularly care if is supports BPEL, BPMN or OSSAD. I don't have anything against standards, but they are still in a state of flux after nearly 10 years of formulation. Most workflow engines have a low level model similar to a finite state machine, and the business language du jour is just a layer above this. Many platforms that are designed this way will support a variety of business process modeling languages existing today. BPEL seems to be the current de facto standard in use amongst most of the market leaders right now.

The Search

Like most Java developers, I start my search for a library by searching the web for current articles and who the players are in the open source area. Unlike technologies like ORM where one or two libraries stand out from the rest, the open source Java workflow market is huge, which makes research difficult. Contrast this with the .NET market where there is one player. I wonder who?

The closest thing I found to an objective comparison of workflow engine's capabilities was this report on the Workflow Patterns website. The report confirmed that no one dominates this space, but it at least highlights three decent players who are JBoss jBPM, OpenWFE, and Enhydra Shark. I have not yet heard of Shark, but I have great respect for Enhydra from my very early days in Java when they competed with JBoss in the application server space.

These recent articles on The Server Side featured a new Finite State Machine from the Wazee Group called Physhun. It is completely configured using Spring, so it seemed like a potential fit.

Enhydra Shark had no documentation available on its website. The latest news said that version 2 was recently released, but checking in the logs it shows that happened in May 2007. As mentioned earlier, OpenWFE is now OpenWFEru where the "ru" stands for Ruby. I didn't look at it, but I would bet that the process definition language used can benefit from Ruby's DSL-like language capabilities. Physhun is very nice, and documented sparsely, but includes several good examples that are easy to follow along to. I find Physhun to work really well for what it is intended; Finite State Machines. It doesn't have the layers to handle tasks requiring human intervention. It seems like a decent base to build these capabilities on top of if you were so inclined. In the end, a FSM is not the same thing as a workflow engine.

JBoss jBPM

Like most things developed by JBoss, their workflow engine is meant to complement their application server, but can be used independently as well. jBPM version 3 has been out for three years and based on the forum posts, seems to have a decent user base. Support by the actual developers has waned over the past two years for whatever reasons, so this is a bit of a concern. Ronald van Kuijk is a member of the community and seems to be a one man army when it comes to helping people with jBPM. jBPM 4 is being developed, so perhaps the product will continue to be improved, and maybe that's what the core developers are spending their time on. jBPM has some relationship to the commercial JBoss product offering by Red Hat, so I would assume it is an important component to their product suite and is not going away anytime soon.

To help with my needs, jBPM provides me with a single jar library that is easily embeddable in my application. Although jBPM supports the BPEL standard, my needs are sufficiently handled by jPDL; an XML-based graph oriented programming model. (That's JBoss-speak for an XML configuration.)

You may read somewhere that jBPM does not require a database, however in order to do anything useful such as supporting tasks and being able to recover its state across server restarts, a database is required. Somewhat unfortunately, it uses Hibernate to persist its internal model. Not a deal breaker, but those people who weren't already using Hibernate will be now, and those using it already may have to take steps to get two different Hibernate session factories to play nice in the same application scope. This could involve JTA transaction management if you cannot share session factories. It would be nice if this could be extracted out of the engine in the future with a Hibernate implementation being the default.

On the plus side, jBPM is a workflow engine supported by the spring modules project. This project provides an object factory integration such that jBPM actions can be Spring Beans. Also, the creation of a Hibernate SessionFactory and sharing it with jBPM is made substantially easier.

What's Next

In the next installment, we will get into the code and some more concepts behind the workflow engine.

The Amish Will Hate These Buttons

I've always heard that the Amish have an aversion to buttons, although it could just be a rumour. A quick check of Wikipedia shows that indeed, some Amish dress codes prohibit the use of buttons on clothing because of their (the buttons, not the Amish) association with the military and their potential for vanity. Well, OK. For those of you Amish who have accidentally stumbled across the least read blog on the planet, please divert your attention by clicking on any of the Google Ads to the right. We are about to get downright vain.

Same Old Song and Dance

I was recently in need of adding some buttons to a web page I was designing. They weren't actually the typical command buttons like cancel or submit. They were acting as special toolbar buttons on my page and needed to be very graphical. They were also going to need to change color to indicate disabled and enabled states. The application is international, so the buttons would have to scale horizontally to accommodate differing text widths.

We've all seen this trick a thousand times, and its been documented to death. Here is a link to a popular article on the subject. I opened Photoshop and began to "draw" the button I had envisioned. It was a rounded rectangle with some Photoshop effects applied to achieve a green gradient background with the obligatory button shading.

Normally, I would copy this button and change the colors used in the gradient overlay to achieve a button with different colors for the different states of a button; rollover, disabled, and clicked. In the end, I would have four different button images and I would swap them out using a little JavaScript and CSS. (I could also opt to combine the images into a single large image and swap them using only CSS. There are a hundred ways to skin this cat.)

A New Perspective

I soon began to get bored with my options. I mean, how brain dead is it to keep swapping out these static images to respond to user interaction. Taking a cue from the Aqua Look and Feel and the work Kirill has done with the Substance Look and Feel, and the way every game console works, I realized that smooth transitions between button colors simulates the way natural light refracts on a normally translucent is simply cooler. A smooth transition from one color to the next can be achieved in HTML in two ways, AFAIK.

1. Create two elements, one with the starting color and one with the ending color. Make the ending color element have an opacity of 0 and align it on top of the starting element. In response to mouse events, animate the opacity of the ending color element to achieve the effect.
2. Using a single element, animate the background color from one color to another.

I think you might guess which approach I took. :-) Animating background colors from one value to another is a simple task in the RGB world.

Each unique color is comprised of a unique combination of red, green, and blue values ranging from 0 to 255. An example starting color may be bright green with a CSS color setting of rgb(0, 255, 0), and an ending color of bright blue with a color value of rgb(0, 0, 255). In order to animate along the path shown in the graphic above, we perform a linear transition of each individual color component. For each step in the animation, red will range from 0 to 0; green from 255 to 0, and blue from 0 to 255. You can see that half-way through the animation, our color is rgb(0, 128, 128).

Color Cubes - A Peek Under the Covers

It's worth pointing out that the small graphic above is a very simplified 2D view of color space. The path denoted by the arrow doesn't even exist in the RGB color space (although it does exist in the HSB space).

Think of the RGB space as a cube with one corner of the cube representing black (0, 0, 0) and the opposite corner is white (255, 255, 255). The three axis emanating from black are the red, blue and green axis. In the diagram below, I've filled in the left side of the cube with the representative color gradient.

Next is a 2D view of the same side of the color cube. You can see that the path denoted by the line shows the color range that is traversed by our linear transition.

 

Let's see an example. If you press the transition button, the box on the right will animate between green and blue.

Your opinion may differ, but when this transition occurs, I would prefer that the color hues change and the saturation and value not change. This is achievable by converting from RGB to the HSL color model. Then the hue can be transitioned from one color to another without an artificial darkening or lightening step. Of course, the HSL color values will have to be converted back to RGB in order to apply them to the element. In CSS3, HSL is supported as a valid color model, but only Firefox 2 and Safari 3 offer support.

Finishing it Off

Now that we are achieving a smooth transition from one color to another, it is a relatively simple matter to compose a translucent image that sits over a div with a changing background color. The image will allow the background color to shine through with varying opacities. In order to pull this off, the image will have to be a PNG, of course.

It's a bit trickier to create these images in Photoshop which is mostly due to the way effects are applied in the tool. You will find yourself creating a lot of the affects as grayscale gradients on separate layers and varying amounts of opacity.

Here are shots of a portion of the new button with its background color and one without the background. On the translucent image, you can barely make out the subtle shading. It is most helpful to work with a background color to bring out the detail. When you save the image, save it without the background and as a transparent PNG.

The transparent image appears with several layers of gradients and borders. They're subtle, but they are there.

The gradients show up nice against an opaque background.

Here are the Photoshop layers. Notice the Matte layer; this is the solid color in the corners. The matte is necessary for this particular example because the background color would show through otherwise. With a rectangular button, this would not be necessary.

Putting it all Together

Here are a couple examples. I am using the jQuery library to provide the color animation capabilities and event handling, but they can be performed with nearly any library or you can roll your own. Buttons that react to user events with smooth and elegant transitions are not just the realm of Flash and custom toolkits.

Who Needs Flash?

You can also add color transitions on mouse up and mouse down also. Try it out.

Sunday Funnies

I've been a big fan of Comic Life from Plasq since I first saw it a couple years ago. It almost made me reinstall Panther on my G4 PowerBook. I like all those kind of tools that let a person without true artistic ability fake it. You know, like Photoshop. I thought it would be cool to allow people to create their own comics, but allow them to embed the finished product on their blog or web page. This could be an image, or even better, a Google Map-type viewer. I also wanted to create the product as a web-based project. After all, that would be about the only way to differentiate the effort from Plasq's fine product. As I look at Plasq's web site today, I discovered that they now have a Windows version of their software. Even more incentive to try to build a web version. Such an undertaking would require some cutting edge Web 2.0 skills (I just can't bring myself to type skilz). The server architecture would have to be very interesting. I'm thinking of a JMS implementation in order to process events from the client side in an asynchronous manner. This would also allow me to leverage comet to improve the responsiveness and scalability of the application. This could be a very fun project. Of course it could get bogged down due to the browser's inability to perform certain tasks. I think I will try a few experiments to act as a proof of concept.

1. Allow drag and drop for image files external to the web browser. This is the first gotcha that will require an impure solution. Browsers can't do this yet. Probably use a very slim Java plugin or something in Flash. I hate to do it though. If I end up sticking Flash in there, the inclination is to do all the image manipulating in flash also. Before you know it, the whole app is just a Flash app. That's no fun.
2. Allow the user to right-click on an image and convert it to grayscale. Other image filters can come later.
3. Demonstrate how an image can be masked. This is important in comics because the image usually doesn't bleed beyond the confines of its cell.

That's probably a good enough start. If I can accomplish those tasks and they work very well and are cross-browser, that may represent 50% of the hard work.