Breaking All the Rules

Finally, after 20 years of being raised on the bread and butter of data persistence, I have a project where the client is open to using a non-SQL database. Of course that client is myself, but that alone wouldn't be fair justification for using a non-relational database.

So what are my primary motivations for using a document persistence mechanism on my project?

• A database that can evolve quickly
• Database that scales horizontally
• Accessible via JavaScript

A database that can evolve quickly

Most of the discussion about an evolving database relates to this concept of schema-less databases. While this is the same marketing blurb regurgitated by most of the so called "no-sql" vendors, most applications care that the data which is shoved into persistence conforms to an expected format. My application is no different, and all of the state representation supported by my application is validated against a JSON schema. After all, a schema is almost always necessary if the data you are storing needs to be queried by a field other than ID, or rendered to the screen.

Database that scales horizontally

Who wouldn't enjoy the ability of doubling their transaction count by only doubling the number of database servers? Only time will tell whether the scalability is linear or otherwise, but I am helping to develop a community site that could experience explosive growth if it is successful. So my ideal database would allow a simple mechanism for clustering instances, including a client-side connection that supports failover.

Accessible via JavaScript

This is a strange requirement for many developers right now. When I write it now, I suspect that most developers wonder whether I plan on making database calls directly from the client. While this is very feasible, it is not the case. Instead, I have chosen JavaScript as the language in which our server tier is developed.

Server Side JavaScript (SSJS)

While JavaScript on the server side sounds radical, I am finding it superior to many other approaches I have taken in the past. Perhaps because I have spent most of my career as a Java enterprise developer, I have a particular comfort and trust in the JVM. I have seen it perform faster and faster every year with more and more new languages developed to run upon it.

While I find Java a bit verbose, I like the syntax of Groovy much, much more. After all, Groovy is all Java plus many meta programming tricks and syntactic sugar to make development actually fun again. Over the past several years, I have also programmed in ActionScript, JavaScript, JRuby and read through some examples of Scala, Clojure and Jython.

Eventually, the time came to decide which JVM language to pick as my server side development language. At this point it should come as a surprise that I chose JavaScript. The main reasons for this choice are:

• Low barrier of entry. Most of our existing development staff knows how to read and write JavaScript. Rhino isn't your grandma's JavaScript either. Support for JavaScript 1.8 is in there and you might scratch your head the first time you see var [a,b] = carArray; as I did.
• Contractor accessibility. My company frequently augments our staff with contract developers when project load demands it. Instead of now looking for 5 Groovy resources, I am seeking 5 developers with JavaScript experience. It is much easier (and frankly, less expensive) for me to locate and onboard these resources.
• SSJS movement. My first exposure to server-side JavaScript was Jaxer from Aptana. It had some cool features, but in the end it seemed like ASP code using JavaScript as the server-side language. Since that time, a new movement has begun. The SSJS movement now has a JSGI specification (based on the oldest web server interface - CGI) which acts as a thin layer between a web server and JavaScript controller code. There aren't a lot of production ready MVC frameworks built on this spec yet, but rest assured they are coming.

After using Narwhal/Jack and Persevere and others, I chose RingoJS as our JSGI implementation. I am very happy with it thus far. I like the fact that RIngoJS supports a middleware stack and can be packaged in a standard Java WAR file for deployment.

The fact that RingoJS runs on Rhino was a primary consideration. This means that we have a direct bridge to the ecosystem of Java libraries and frameworks from within JavaScript. We are already using Mongo's Java database driver with a thin JavaScript layer on top to perform persistence. I'm very happy with our choices so far.

Skype Mobile on Verizon is Sad

I'm not sure who's idea of a sick joke this is, but I have been waiting for Skype Mobile for quite some time. I was very excited to get it on my Droid today. You see, I am traveling internationally, and my Droid is practically useless outside of the US. I do use Skype on my mac to make phone calls to other Skype users and I pay for Skype Out calls that let me call any phone number I need. So, I thought Skype Mobile would at least let me handle all my outbound calling needs as long as I have a Wifi connection.

Imagine my surprise when I learned that Skype Mobile doesn't use Wifi! So instead, I have to be on Verizon's phone network in order to use Skype Mobile. I'm dead in the water without Wifi support. This drawback alone makes it useless for me, and I eventually will have to drop Verizon to go with a GSM-based phone so I can use it when I travel.

So what happens when I get back to the US. It seems like I should be able to use Skype Mobile to make some of my international calls at a very low cost, correct? Well, not quite. In addition to the low rate I am charged by Skype Out, all the minutes I use are now impacting my Verizon plan.

Bottom line, without Wifi support, Skype Mobile is a toy. I haven't been able to try to use it yet even though it is on my phone. (It fails to start because I am in Saudi Arabia and not on the Verizon network at the moment.) I did watch the video and learned that there is no mention of being able to make Skype Mobile to Skype Mobile calls; only Skype Mobile to Skype Desktop. Can anyone confirm this?

Cleaning a Tree of .svn Directories

Subversion has an export command that will check a project out of a repository without creating the typical .svn directories used by the local Subversion command to keep track of state. Sometimes, however, it is necessary to strip these directories away. While dabbling in Linux and Mac OS X for several years, I am still a relative noob when it comes to the Unix command line commands.

This command will delete all the .svn directories in the current directory and below. It is a bit more complicated than I would of expected because it supports directories with spaces in their name.

find . -type d -name .svn -print0 | xargs -0 rm -fr

Using Astra Outside of Flash

Many of us Actionscript developers are self-described purists who eschew the trappings of Flash to the Designers who tread not in our world of text editors and command line compilers. OK, who are we kidding. You can’t do complex Flash development without loading up Flash every once and a while. Even if you spend most of your time in a text editor, the Flash development tool is a necessary part of the development process at times. 

Coming from a Java background where every library can be discovered on a class path and compiled into a single distributable archive, Actionscript programming still perplexes me a bit when it comes to using a button or a UIComponent. Of course these are component classes that are immediately accessible within the flash environment, but step outside into the cold world of the command line and you will find these classes are verboten.

So what gives? From the command line you have access to some pretty serious components from the Flash libraries. There are flash.net.URLLoader and flash.display.Sprite, but there isn’t access to fl.controls.Button or fl.core.UIComponent. In the case of the UIComponent class, it actually extends Sprite! I hope that someone can fill in the logic here because I came late to the game and it bewilders me that Adobe continues to make developers jump through Flash to get access to some of these classes and not to others. The cynic in me thinks this is Adobe’s way of selling more Flash licenses.

Yahoo’s Astra Library

I really like Yahoo’s developer website. There are years worth of relevant and practical knowledge there in videos, articles, and code. One of the areas of special interest to me is the Flash APIs that Yahoo has released under the name of ActionScript Toolkit for Rich Applications, or thankfully by its acronym, ASTRA.

ASTRA includes several high level components for Flash developers and a separate set of components for Flex development. There are classes for animation and badges. Yahoo also provides APIs for their own mapping, social network, weather, and other services.

What caught my eye was their layout containers.

The Case for Layout Management

I had come to Flash by way of Flex. Flex development has many similarities with Java client development, especially today with the marriage of XML and GUI found in JavaFX or the GUI builders in Groovy. Anyone who has developed a complex (or even simple) GUI knows how layout managers simplify a tremendous amount of tedious coding, especially if the application is resizable and/or dynamic.

At a previous Flex consulting job, I had noticed the customer was manually positioning components on the screen using X/Y coordinates. They weren’t even repositioning or resizing these components when the user stretched the window. In fact, they dealt with this shortcoming by making the window a fixed dimension. Never mind the average user had a large screen and the application ended up occupying a small corner and it had horizontal and vertical scrollbars so the content could be managed. It only took an hour to introduce Flex’s layout managers to the application, and the users were ecstatic.

I don’t know how common it is, but I often see developers crafting GUIs who are more adept on the server side of the coding equation. There are exceptions of course, but I think every developer who is creating a GUI has to put themselves in the shoes of a user. Not just every once and a while, but the entire time they are building the GUI.

Layouts in Flash

Flex has some great layout containers, very similar to Java’s. They can be nested to provide an infinite number of layout permutations. Never again should a developer code a window that cannot be smoothly resized with all the components reacting intuitively to a change in width or height. Seems like a pretty fundamental function for a GUI development environment, right?

Wrong!? Flash is probably the most visible GUI development platform in the world, and imagine my surprise the first time I went looking for a BorderLayout container. And there wasn’t one. I looked at a lot of code examples of some highly respected developers, and I was a bit surprised to find they mostly laid components out by hand. They manipulated the dimensions and position of components manually. This isn’t particularly difficult for a few components, but it becomes very troublesome when there are dozens of components and nested containers.

In fact, it’s precisely why layout containers exist in Flex, and more importantly to Flash developers, it is why Yahoo introduced the layout containers in Flash. The purpose of this blog post isn’t to show you how to use the Astra layout containers, but rather how to compile them in a manner that they can be used directly from Actionscript. For usage information, I highly recommend the examples included in the Astra download, especially the feed reader example.

Making ASTRA Layout Containers Accessible to Actionscript

Unfortunately (and inexplicably to me), Yahoo did not release the layout containers as a pure Actionscript library. This means you must use Flash to compile a swc library that can be linked with the command line compiler. Perhaps it is because the layout containers extend UIComponent, which I mentioned in the beginning was not immediately accessible outside the Flash IDE.

Download the ASTRA library from Yahoo’s Flash developer site. The installation of the ASTRA extensions are documented on Yahoo’s site. Since I have Adobe’s Extension Manager installed, simply double clicking the file installed it.

 

From here, we need to open Flash and create an new Flash file (Actionscript 3.0). Once the file is created go to WIndow –> Components (Ctrl-F7) if not already visible. In this dialog, you should see the Yahoo component categories have been added.

 

In the components dialog, expand the Yahoo! Layout node and drag each of the layout containers (BorderPane, FlowPane, HBoxPane, TilePane, and VBoxPane) to the Library dialog.

 

It’s not necessary to add any of the library items to the stage. At this point, we can simply publish our FLA, ensuring we check the ‘Export SWC’ option. When published, the resulting library should weigh in at nearly 50K. Not the lightest library you will come across, but still pretty reasonable for applications without a critical size limitation.

Now that you have a swc file, all of the ASTRA layout goodness is available to you in your Actionscript code. For instance, if you are using the MXMLC compiler, include the new SWC using the -compiler.include-libraries switch.

If a Test Case Falls in the Woods...

I've long been a fan of the Spring Framework and the base test classes they provide. A few years ago, a fellow consultant on a project introduced me to the concept of rollback testing.

Rollback testing implies that a transaction begins and ends with your test method. At the end of your test method, the transaction rolls back and nothing is persisted to the database. This certainly eliminates the problem of cleaning up state that is left hanging around at the end of an integration test. You simply have to do...nothing!

Spring also provides some base classes that you may extend to get this functionality with no additional work on your end. Their classes AbstractTestNGSpringContextTests and AbstractJUnit4SpringContextTests provide a convenient superclass that your classes can extend to benefit from this feature.

I'm a user of TestNG, so I used that flavor of Spring abstract test class as my starting point. From there, it is very simple to add a test that will validate that the create method works the way it is supposed to. I am using JPA with Hibernate as my JPA provider.

@ContextConfiguration(locations = {"classpath*:META-INF/spring.xml"})
public class TestDomain 
       extends AbstractTransactionalTestNGSpringContextTests {

    @Autowired(required = true)
    private DomainDao domainDao;

    @Test
    public void testCreate() {
        final Integer pk;
        {
            Domain domain = new Domain("md", "mccoy");
            pk = domainDao.insert(domain);
            Assert.assertNotNull(pk);
        }

        {
            final Domain domain = domainDao.select(pk);
            Assert.assertEquals(domain.getDomainId(), pk);
            Assert.assertEquals(domain.getDomainName(), "mccoy");
            Assert.assertEquals(domain.getTld(), "md");
            Assert.assertEquals(
                domain.getStatus(), Domain.Status.CANDIDATE);
        }
    }
}

You can imagine how pleased I was to see my integration test passed the first time I ran it. Damn, I'm good.

I've had the pleasure of working with a friend who has never met an ORM he trusts, and he has taught me many times to look at the log files and examine the SQL that is being executed. He does this to understand better how the ORM is approaching a problem. When I looked at the logs, I was surprised to see a lot of logging, but no SQL statements. I checked to make sure I had told Hibernate to log SQL and I had. In fact, I saw SQL statements in some other places, but none associated with this particular unit test.

Timber!

So why was there no SQL executed against my database. My DAO classes have a transactional boundary, so any call to them (like insert and select in my example), will create a transaction and close the transaction. When a transaction closes, it is a signal to Hibernate to flush the SQL statements and commit the transaction.

The reason I see no SQL is because my transaction begins and ends in relation to the test case, not because of my DAO. The transactional context surrounding my DAO calls recognize that a transaction is already in progress and simply participate in the existing transaction. When using Spring's base test classes, all transaction will rollback at the end of the test case. This means, unless something happened that forced Hibernate to flush SQL, there will be no SQL written to the database.

Is This A Problem?

Some people might argue whether this is a problem or not. After all, you are testing your integration code, not Hibernate's ability to persist your objects. In the past, I have bought that argument, however by closely watching my log files it is easy to see that some of my code can pass muster even though it is woefully wrong.

This rule sounds a bit obvious:

When changing the typical transactional boundaries of your application to surround the test case and not the service tier or DAO tier you may not be testing database impact on your application.

Even though the rule is obvious, the impact may be hard to tell when you are writing code. For example, in my testCreate() method above, this test will pass even if a annotate my domain class with a non-existent database table. I can specify @Table(name = "he_hate_me") and my test will pass. I can also supply the wrong schema  to this annotation or in my persistence.xml config and the test still will pass.

As you might expect, if you have a table with a unique index (other than the primary key) and you have a unit test that performs multiple inserts attempting to trip the constraint exception; these tests will also pass.

Workarounds

For the reasons I have shown, I would consider blind use of rollback testing to be a bad thing. Most JPA providers will function the same way as Hibernate and only flush statements when Hibernate feels it is necessary.

Of course, Hibernate can be explicitly told to flush, and there isn't a real downside to doing this in an integration test. But where and when should you flush? I find that a call to flush is warranted any time you make a call to a service method or DAO where a transactional context would normally be started if one wasn't already in progress.

Here is the previous example modified to show the call to flush().

@ContextConfiguration(locations = {"classpath*:META-INF/spring.xml"})
public class TestDomain 
       extends AbstractTransactionalTestNGSpringContextTests {

    @Autowired(required = true)
    private DomainDao domainDao;

    @Test
    public void testCreate() {
        final Integer pk;
        {
            Domain domain = new Domain("md", "mccoy");
            pk = domainDao.insert(domain);
            flush();
            Assert.assertNotNull(pk);
        }

        {
            final Domain domain = domainDao.select(pk);
            flush();
            Assert.assertEquals(domain.getDomainId(), pk);
            Assert.assertEquals(domain.getDomainName(), "mccoy");
            Assert.assertEquals(domain.getTld(), "md");
            Assert.assertEquals(
                domain.getStatus(), Domain.Status.CANDIDATE);
        }
    }
}

Flush Your Problems Away

Both Hibernate's Session and JPA's EntityManager interfaces have a flush() method. It can be a bit tricky to obtain the EntityManager interface in order to call flush in it, so I use a base class in my integration tests that includes a flush() method. The implementation looks like this:

@ContextConfiguration(locations = {"classpath*:META-INF/spring.xml"})
public class BaseTest 
       extends AbstractTransactionalTestNGSpringContextTests {

    protected EntityManager sharedEntityManager;

    protected void flush() {
        sharedEntityManager.flush();
    }

    @Autowired(required = true)
    public void setEntityManagerFactory(EntityManagerFactory emf) {
        sharedEntityManager = SharedEntityManagerCreator.
            createSharedEntityManager(emf);
    }
}

Some knowledgable Spring users will probably point out there is already an AbtractJpaTests class that exposes a sharedEntityManager. They would be right, and it takes care of load-time weaving cases that can make integration testing of JPA difficult to work with. Unfortunately for me, the good Spring folks tied this abstract class to JUnit (no TestNG version) and I'm using Hibernate which doesn't require load-time weaving.