Jason Chambers

Introduction to Gradle

2010-12-05T11:38:00.001-08:00

At my new gig, one of my first assignments was to re-engineer our build system. For this task I used Gradle. To ease the transition for the team, I put together a YouTube video

Tips on designing an upgrade process

2010-09-29T06:57:00.000-07:00

I’d like to share some of my lessons learned and pass these on.
Consider the current working version (x) sacred – the upgrade process should not touch it because you always need to provide an option to roll-back to (x) – perhaps even a couple of hours after the new version (y) has been deployed. Design hint: when upgrading, lay the new version (y) along-side and not on top of version (x). This implies a migration of configuration files at the application layer and data migration at the data layer from version (x) to version (y).

In a high-availability environment, consider is it possible to run both version (x) and (y) concurrently – this is to support the concept of a rolling upgrade where you upgrade each node in turn.

By far, the trickiest part of designing an upgrade process is considering evolutionary changes to the data schema. New tables, columns or constraints may have been added. You need to pay close attention to changes to the schema during development and always think about what this means for the migration of existing data. It’s usually a good idea to embed the version number in the name of the schema/database.

It’s essential that data created with the old version (x) is available with the new version (y) – but also, ensure that the operations the system provides on the data exhibit a consistent behavior where expected.

DevNexus presentation

2010-09-22T11:11:00.000-07:00

Back in March of 2010, I gave a presentation at the devnexus.com conference in Atlanta.

The presentation was entitled "From whiteboard to product launch". The intent being to share my teams experience in bringing a brand-new product to the marketplace. It covers a wide variety of areas including process, architecture and team organization.

The slides are available here and the audio is available here (I recommend you flip through the slides while listening to the audio).

How to scale the data layer

2008-12-29T08:54:00.000-08:00

Many a performance problem as a system struggles to support growing demand can be pointed to the data layer. The database can easily become a bottleneck and can be the hardest to scale after the fact because it manages state (in contrast, stateless components are easy to scale).

Of course, the problem may be solved by throwing more hardware at the problem – i.e. upgrading the database server by increasing memory and/or CPU horsepower. However, it is prudent to think about scalability before rather than after the fact.

First, let us consider some elementary physics:

pressure = force / area

For a given force, if we reduce the area to which the force is applied the pressure increases. Try for example, standing on tip-toes like a ballerina to get a sense of what I’m talking about. The force i.e. your body weight is constant, but we are reducing the surface area to which the force is applied resulting in increased pressure.

Likewise, if we increase the surface area the pressure is reduced. This is how a person can lay on a bed of nails without puncturing the skin – there needs to be of course enough nails.

Typically, we don’t have any real control over the forces applied to the products or solutions we are designing and building. We can speculate and design with certain limits in mind, however what if we come across a situation where those limits are shattered? In the design, we need to think of a way of how we can support this and yet reduce the pressure on the entire system.

One approach is to use horizontal and vertical portioning techniques in the design of the data layer. Both these approaches have the effect of increasing the surface area and thus reducing pressure points as the force (or load) on the system increases compared to a single monolithic data layer (i.e. single database server).

Horizontal partitioning is where the rows of a single logical table are spread over multiple physical databases. For example, customers whose last name begins with A-F may be stored in one database server and customers whose last name begins with G-L are stored in another database server and so on. It is a popular technique used by several large dot-coms as a way to spread the load.

Vertical portioning is a similar concept but involves storing different tables in different physical databases. For example, purchase orders may be stored in one database server whereas invoices may be stored in another.

To learn more about horizontal and vertical partitioning, take a look at http://en.wikipedia.org/wiki/Partition_(database)

Conceptually, the approaches are very straightforward however if you want to leverage them in your design there are several things to look out for.

For horizontal paritioning, you want to strive for even distribution across the partitions. For example, partitioning based on the first letter of the last name may not provide even distribution - V through Z for example may be quite light in the number of records. Hashing techniques may provide a good approach.

For vertical partitioning, references across data models may be tricky. In particular, forget about foreign key constraints. If the two data models have a high degree of coupling between them, then I would go back to the drawing board – perhaps you haven’t found the right boundary. It’s ok to go to one data model, find what you are looking for, then use the result from that query, to find what you ultimately need from the second data model. Avoid distributed transactions (XA/2PC) that span more than one data model if at all possible. Ask yourself, do you really need referential integrity across two data models at all times? If so, then again you should revisit your data design and carve out the right boundaries.

When designing data models at the highest level, strive for high-cohesion (if it changes together, then it stays together) and loose-coupling between the data models. In other words, apply some of the same desirable properties for code design to your data design.

You know you're a geek when

2008-12-08T11:49:00.001-08:00

Writing a shopping list the other day, I wrote down Guice instead of Juice. Time for a break from work I think.

Tomcat Controller update

2008-12-04T07:55:00.001-08:00

http://www.jasondchambers.com/2008/07/tomcat-controller.html

Using Grails to explore and develop the domain model

2008-11-12T15:31:00.000-08:00

Data modeling is typically a fairly important design activity. However, I have a hard time with EAR diagrams and data models as a starting point – particularly for a new system (you don’t really have a choice when dealing with legacy databases). Thinking in objects is much more natural to me. Also, I like the idea of using code to explore the domain model and try a few things out.

In the past, I have used the forward engineering features of Hibernate for this very purpose. Once the tables have been created in the database, I take advantage of a neat feature in JDeveloper where you can reverse engineer a data model diagram. So the process kind of goes like this Object model->Java code->Hibernate DDL->Database->Data model. It worked well in thrashing out the data model for my last project. Of course, it may not work for everyone but that’s ok – it works for me and that’s all that matters ;-)

For a new project I’m working on, I thought I’d try something different. I thought I’d give Grails a spin. This worked even better because Grails can generate such a lot of boiler-plate code for me enabling me to move a lot faster. It can also generate a scaffolding UI so I can interact with and test out the model. Here was the process:

Download and install Grails from grails.org

Follow the Quick Start guide to familiarize yourself with Grails http://grails.org/Quick+Start . During the Quick Start guide, you will learn how to create an application a domain class and a controller

Configure Grails to use Oracle instead of HSQL (I needed to externalize the database so I could browse it and reverse engineer the data model diagram using JDeveloper) – to do this modify grails-app/conf/DataSource.groovy – change the driver class name and the JDBC url – you may also have to copy the Oracle JDBC driver (ojdbc14.jar) to the lib directory

For each entity (where name is the name of the entity you want to model)
1. $ grails create-domain-class
2. Modify the generated class located in grails-app/domain/name.groovy – add the attributes
3. $ grails create-controller
4. Modify the generated controller class located in grails-app/controllers/nameController.groovy – change the body of the class to look like def scaffold = name
Run the application $ grails run-app

Point your browser to the application

Populate the model by interacting with the controllers through the generated scaffolding

Add some more entities by going to back to step 4

If you want to auto-populate the model with data on startup, add your code to grails-app/conf/BootStrap.groovy

I have to say, I found the out of the box experience with Grails pretty polished. It passed the 15 minute test with flying colors. (If I can’t get something working in under 15 minutes, I tend to dump it).

Performance testing tip - assumptions are dangerous

2008-10-08T14:40:00.000-07:00

I've been doing a lot of research over the past couple of weeks with a goal to understand how cryptography performs on different platforms, with different algorithms, with different data characteristics. In addition, I was also interested to learn how crypto accelerators perform compared to software based crypto.

I can't share all the findings with you, but I can tell you I was quite surprised by what I found. Once again, I was reminded that assumptions regarding performance are invariably wrong. The reason for this is there are so many subtle factors involved that can affect performance. Also, technology is constantly moving forward. Results and conclusions found today may not necessarily hold true tomorrow. As an example, does anybody remember how people used to chastise Java for being slow. That may have been the case in 1996, but in 2008 it most certainly is not.

A major assumption I had before I conducted my research was that I was convinced that the crypto accelerator would outperform s/w based crypto. This, it turned out was not the case. There were some cases where the s/w based crypto outperformed the crypto accelerator significantly. I also assumed that the SPARC chip would perform pretty well compared to AMD/Intel chips - it didn't really perform well at all in terms of raw speed/performance for the kinds of crypto operations I was performing.

The graph to the left shows the fastest and average performance for performing a SHA-1 hash on a credit card number on various platforms. The units are in nanoseconds. Look how fast Java performs on Intel on both Linux and Windows.

Quote of the day - Build one to throw it away

2008-09-13T09:03:00.000-07:00

This is a quote from Fred Brooks' "The Mythical Man Month".

“Where a new system concept or new technology is used, one has to build a system to throw away, for even the best planning is not so omniscient as to get it right the first time. Hence plan to throw one away; you will, anyhow.”

Great insight. What's also remarkable is this book is over thirty years old. In technology, we like to talk about how much and how fast things change - and they do, however there appear to be lots of things that remain timeless.

Using the Flickr Authentication API - part I - generating the api_sig

2008-09-12T12:51:00.001-07:00

All Flickr API calls using an authentication token must be signed. In addition, calls to the flickr.auth.* methods and redirections to the auth page on flickr must also be signed.

The api_sig is to be appended to the URL for all such API calls. The following code can be used to generate the api_sig (of course, you can use a higher-level library such as flickrj if you enjoy eating microwave ready to eat meals over growing your own tomatoes ;-).

package com.jasondchambers.flickr;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;
import java.util.Set;
import java.util.SortedMap;
import java.util.TreeMap;

/**
* Generates an api_sig from a list of parameter key value pairs
*
* The signing process goes like this. The parameters are sorted
* e.g. foo=1, bar=2, baz=3 sorts to bar=2, baz=3, foo=1
* The secret and the parameters are concatenated together as follows to provide
* the raw signature
* e.g. SECRETbar2baz3foo1
* An MD5 hash is created, converted to hex and returned - this is to be used as
* the api_sig parameter
*
* See section 8 of [1]
*
* [1] http://www.flickr.com/services/api/auth.spec.html
*
* @author Jason Chambers
*
*/
public class ApiSigGenerator {

 private String secret;

 public ApiSigGenerator(String secret) {
   this.secret = secret;
 }

 public String sign(String... paramKeyValuePairs) {
   try {
     // Sort the parameters first
     SortedMap<String, String> sortedParameterMap = sort(paramKeyValuePairs);
     // Generate the raw signature
     String rawApiSig = generateRawApiSig(sortedParameterMap);
     // Hash the raw signature and return
     return generateMd5(rawApiSig.toString());
   }
   catch (Exception e) {
     throw new FlickrClientException(e);
   }
 }

 private SortedMap<String, String> sort(String[] paramKeyValuePairs) {
   SortedMap<String, String> sortedParameterMap = new TreeMap<String, String>();
   final int KEY = 0;
   final int VALUE = 1;
   int i = KEY;
   String key = null;
   for (String o : paramKeyValuePairs) {
     if (i == KEY) {
       key = o;
     } else {
       sortedParameterMap.put(key, o);
     }
     i = ~i;
   }
   return sortedParameterMap;
 }

 private String generateRawApiSig(SortedMap<String, String> sortedParameterMap) {
   StringBuffer rawApiSig = new StringBuffer();
   rawApiSig.append(secret);
   Set<String> keySet = sortedParameterMap.keySet();
   for (String k1 : keySet) {
     rawApiSig.append(k1);
     rawApiSig.append(sortedParameterMap.get(k1));
   }
   return rawApiSig.toString();
 }

 private static String generateMd5(String input) throws NoSuchAlgorithmException {
   StringBuffer output = new StringBuffer();
   MessageDigest md;
     md = MessageDigest.getInstance("MD5");
   byte[] md5 = md.digest(input.getBytes());
   for (int i = 0; i < md5.length; i++) {
     String tmpStr = "0" + Integer.toHexString((0xff & md5[i]));
     output.append(tmpStr.substring(tmpStr.length() - 2));
   }
   return output.toString();
 }
}

Google's innovative approach to communicating Chrome

2008-09-10T14:51:00.000-07:00

A big part of my job is communicating. In particular, I communicate designs and vision such that our development team can go off and build product. I have tried several approaches to communicating designs to the (remote) development team over the past few years or so including (some with more success than others):

Powerpoints (2007 has awesome 3D capabilities – ideal for those m-architectures)
Demonstrations (I use iShowU)
Proof-of-concepts (i.e. working prototype code)
Visio’s (for when I feel constrained by UML)
Word documents (nice and formal)
Recorded Video presentations (I have found demonstration software works better)
UML models (using Enterprise Architect)

Why do I bring this up? - Because I have been studying Google Chrome of late. The thing that most impressed me most was not necessarily Google Chrome per se (although I applaud some of their innovations and their approach), but one of the innovative ways they communicated the project to the public when it was launched.

They developed a comic-book – go check it out.

Flickr API

2008-09-09T16:29:00.000-07:00

I have a ton of old prints I'd like to get published on-line at Flickr. I have a scanner at home but that sounds like a tedious process to me. It turns out that Walgreens (a US pharmacy) provides scanning services at a reasonable price. Sounds like a winner.

Typically whenever I publish photos to Flickr, I use iPhoto with FlickrExport from Connected Flow. This works fine for a manageable number of photos.

I'm thinking I want a better approach for bulk uploads so I signed up for an API key and intend to write some code to manage and automate the upload process for me.

I'll keep you posted with my findings and experiences using the API in due course.

iPhone - don't leave home without it

2008-09-05T14:31:00.000-07:00

It's been a couple of weeks since I picked up the new iPhone 3G. It had been on my shopping list for quite a while but being a bit of a cheapskate, I decided to wait and defer the purchase for as long as possible. The good thing about holding out on your technology purchases is that the gadgets only continue to get better (of course this is traded-off against the cost of going without).

Unfortunately, or fortunately whichever way you look at it, my RAZR finally packed up while on vacation in Kiawah, S.C. I happened to be passing the Apple store in nearby Charleston and decided to pop in and just see if by chance they had any iPhones in stock. To my delight and surprise they had and so I picked one up.

I immediately put it to use to help me find the car park - using it as a GPS locator if you will (~~although I don't think it doesn't actually uses GPS - I believe it locates via triangulation with cell towers~~ correction - it does actually use GPS and cell tower location technology).

Immediately, I found the user interface very intuitive. This is a major improvement over the cell phones I have used in the past. The RAZR phone had a terrible user interface. After using it for over two years, I could not identify the correct buttons to press to switch between silence/vibrate and loud ringtone. I ended up just pushing the buttons on the side until the display told me I'd finally reached my desired setting.

I have to hand it to Apple. They have created a very powerful capable device in the iPhone and have yet managed to make it's operation easier to use than less capable devices.

This past weekend, I ended up taking my Dad to an urgent care facility in Nashville to get a prescription for Diabetes meds he left back in Atlanta. After waiting 4 hours to see the doctor (we could have driven back to Atlanta in only 3.5 hours), we learned that the doctor had not heard of the medication. I couldn't quite believe what I was hearing. Out comes the iPhone and off I trot over to Google. It turns out the drug in question is not FDA approved (although it is approved in the UK). The doctor confirmed my findings after talking to a pharmacist.

Also, this past weekend we bought a new car. I did some preliminary research by tapping into edmunds.com when the salesman wasn't looking proved useful in figuring out if I had a good deal or not.

Finally, I have a terrible short term memory. I'm not very good at remembering things like where I parked the car or shopping lists. I use SpeakEasy which I use to record voice notes and reminders. Invaluable little tool.

I'm also experimenting with Evernote too as a way to organize some of the chaos - or at least make it accessible when I need it. I used it to compile and collect research for the recent car purchase. Good stuff.

The one down-side is that the battery, particularly when using 3G doesn't seem to last too long. Also, not quite sure why iTunes Store only works when connected over WiFi.

The touch-screen device seems to have just the right level of sensitivity.

All in all though, it's a very useful device.

Lucid definition of design

2008-09-02T12:59:00.000-07:00

I have just watched an interesting video of a presentation Richard P. Gabriel gave at QCon in 2007. The title of the presentation "Architectures of extraordinarily large. self-sustaining Systems" is available here for those of you who are interested.

During the video, Richard provided a wonderful definition of the term design:

"Design is the thinking one does before building"

I like this definition. What do you think?

Why GWT matters - part II

2008-08-27T14:54:00.000-07:00

Back in May, I wrote about why I thought GWT matters. As promised, I am finally getting around to completing my thoughts on the subject.

In the original post, I talked about how I felt we took a step back when we shifted our development focus more to the web platform and further away from the desktop. We lost some of the richness of the interface for our users and we retreated from advances in programming models in the areas of component and event based programming. It’s only been fairly recently where we finally can have it all (reach/zero deployment with productive programming models).

GWT does provide it all as far as I’m concerned. It has a nice programming model that should feel natural to anyone that has done any desktop based GUI programming in the past. It doesn’t lock you into a particular widget library which makes it a sound investment. It takes care of dealing with the nastiness of writing cross-browser code – because it generates all of this for you.

It makes it easy to write internationalized applications. It makes it easy to provide a working back-button for your users (a rarity for a lot of AJAX applications – hit the back button and you may be taken completely out of the application).

If you really do have to drop-down to JavaScript, there is a mechanism called JSNI which provides this.

However, most of all it is the amount of attention the GWT team has spent on tool support that really gets my vote. The integration with Eclipse makes for a very productive environment.

Of course, there are lots of alternatives in the RIA space. Flash/Flex, Silverlight, and Volta are just a couple of alternatives. Of course, if you are lucky enough to have exceptional JavaScript developers on your team then you may feel comfortable using JavaScript directly (although, I have to say although I like dynamically typed languages, one cannot argue with the ability of statically typed languages to assist you in finding problems earlier in the development cycle - i.e. compile-time vs run-time).

However, GWT seemed to be an ideal fit for my team (no new tools – continue to use Eclipse, no new languages to learn – continue to use Java, and no need to worry about the plethora of browser types and versions).

What are your experiences with GWT or RIA approaches in general? (the good, the bad, the ugly)?

Balancing “YAGNI” with “avoiding the cul-de-sac”

2008-08-20T10:48:00.001-07:00

This is a tough part of the architect’s job because it involves speculation. It would be easier if we had a crystal ball to peer into, but we don’t. All we have to go on is our instinct based on past experiences.

Let’s look at one end of the spectrum – “Avoiding the cul-de-sac”. You may have been burnt in the past due to significant amount of rework required as requirements changed or new requirements surfaced during the middle of a project. The common knee-jerk reaction is to make everything flexible, configurable and adaptable. However, like most engineering endeavors there are always trade-offs to be made. Be careful, you may have ended up with un-necessary complexity (which drives up the cost of the project).

Let’s look at the other end of the spectrum – You Ain’t Gonna Need It. With this approach, you eschew flexibility, configurability and adaptability until a requirement surfaces that demands it. This approach is a gamble, but the agile folks will tell you that this is ok because of continuous refactoring and unit tests.

So what is the architect to do? Well, it depends. You have to look at the sub-systems of the architecture and run through some what-if scenarios. To prioritize, I would focus on the sub-systems that have external dependencies. Wherever you have external dependencies, you have given up control. Anything could happen to the external dependency and you have no control over it. The vendor or organization that owns it could a) go out of business b) change their licensing model c) raise their prices d) break backwards compatibility e) discontinue support f) fail to meet performance SLAs. The architect is responsible for addressing these risks in such a way that if any of these events would happen, the impact is contained.

The single most effective way of containing impact is to effectively manage the degrees of coupling within the system. A Formula 1 team can change the wheel in under 10s because the wheel component is loosely (yet securely) coupled to the car. How long would it take to switch your system from say Oracle to SQL Server?

Where couplings between components are legitimate, there are several coding techniques and (albeit very similar) design patterns you can use to allow for independent evolution of the components yielding contained impact in the event of change. On the coding side, coupling to an interface rather than implementation is a safe bet – thus allowing for different implementations to be plugged in later. On the design front, the strategy, provider and adaptor patterns all help to introduce flexibility into the architecture. For example, the Java Cryptography Extensions (JCE) is a great example of the provider pattern. Initially, I may use a software based provider that performs cryptographic services – but I may upgrade to a hardware based provider (HSM) for when the time comes when a performance boost is required – all without having to change any code. A more common example is the JDBC architecture, where drivers can be plugged in to connect the application to a different database product for its persistence needs.

A goal of the architecture is therefore to provide flexibility without disruption where it makes sense.

Do Architects need coding skills?

2008-08-14T17:41:00.000-07:00

Do Architects need coding skills?

In the oft cited parallel world of construction and engineering (of buildings), it is true that to be an architect one does not need to necessarily possess brick-laying skills in order to be effective in their role. However, to be an architect one does need to understand the properties of the materials used for construction.

What about software? Most, in fact pretty much all of the (software) architects I know have some background coding experience. The question to be asked is how essential is it to have coding skills? If an architect can get by without having ever laid a brick, is it possible for a software architect to get by without ever written a line of code?

At least at the current state of our discipline, I would say no.

However, a more contentious question maybe – Do Architects need to maintain their coding skills once they’ve achieved the lofty status of architect. Well, I think it depends. Enterprise architects - possibly not (more on that later). However, for application, solution or product architects – I would say yes. There are many reasons why I think this is the case.

Firstly, the job of the architect is to take a definition of what needs to be built and figure out the best way for how to deliver the solution or product given the imposed time and financial constraints (in reality, the architect is also usually involved in figuring out what needs to be built although this should really be the role of the business analyst or product manager). He/she needs to figure out how to break the problem into manageable chunks (high-level design), decide what materials and tools to use (taking into account the current state of the art and the skills of the development organization) and forage for the non-functional requirements or quality attributes. Basically, the objective of the architect is to deliver better faster cheaper. It’s often a difficult job. The architect may also be stretched and be responsible for data design, dynamic modeling and even detailed design activities such as screen design. Essentially, the architect has to provide whatever it takes to ensure the development organization has enough information to be successful in the build out.

The architect may also be pulled into oversight of the development effort. Throwing designs over the wall and turning your back is likely to result in the development organization delivering something different to what is actually required.

Sometimes, models are not enough. They are by their nature abstract which means they are missing details. The details are where the devil lies. The details need to be figured out sooner or later and so the development organization may either choose to figure it out for themselves or involve the architect.

There may be some system requirements that are difficult to articulate in English specifications. Sometimes, code in the form of prototypes provides an ideal communication mechanism to the development team.
The architect is required to look around and observe and evaluate current trends in technology and processes. Maybe, there is some new toolkit or framework that has been developed that promises to help shorten the development cycle or provide some significant value. Of course, the architect has to try out the technology or process outside of the critical path of the project to ensure fit and readiness for the project at hand. Perhaps some prototypes need to be developed. The architect may need to dive in and learn the new technology before making a sound recommendation to embrace or avoid.

The architect shouldn’t be coding on a daily basis. If the architect is spending most of his/her time then I would argue they are not doing architecture. Having said that, coding is a small yet important component of the architect’s repertoire.

Caching tips

2008-07-25T08:35:00.000-07:00

Caching is all about keeping data closest to where it is needed. It is first and foremost a performance optimization, however you have to ask yourself a couple of questions about the data:

1. Is it read-only, read-mostly or write-mostly?
2. How often does the data change?
3. What is the tolerance in each case for data staleness?

Data that falls into the category of read-only/mostly, has a low rate of change and a reasonable tolerance for data staleness are ideal candidates for caching. A key performance metric is cache-hits versus cache-misses. Obviously, you want to look for cache-hits to get a good return on the investment. If you get few cache-hits and lots of cache-misses - be careful, you may need to rethink your caching strategy as you may have made performance worse rather than better.

For caching web-objects for a global web-site, edge caching providers such as Akamai are the way to go. This works well for static objects such as images, CSS, JS, HTML etc. It moves the content closer to the end user. Also, recall that every point across the network may have some degree of cache-ability (e.g. proxy servers, routers, browsers).

Ensure the HTTP cache-control headers are set appropriately to reduce trips back to the server when the version in the browser cache could be plenty fresh (enough)

For caching data on the app server that is sourced from some back-end service, database or EIS system, memcached or JBossCache work well. Clustering/cache replication may not always be required. Clustering is a good option if the data source is the end-user - if the data source is a database, you can recover the cache quite easily - if the data source is the end user (e.g. session information), you cannot very well ask them, "My cache seems to have gone down the toilet taking your session information with it - tell me again, how many shares of IBM did you want to buy again? (sorry)"

You can reduce a significant amount of load on back-end resources through caching - but only if the data is a good candidate for caching. Hibernate/eh-cache may work pretty well here in addition to memcached and JBossCache.

Oracle Coherence in my opinion has evolved way beyond it's caching roots. Using Coherence, you can build a huge data grid across commodity hardware. Each node in the grid has it's share of responsibilities and is backed up by n other nodes for redundancy. Operations against the data can move to where the data is rather than move the data to where the operation occurs (our traditional way of approaching data processing for the past gozillion years).

Tomcat Controller - updated

2008-07-23T17:53:00.000-07:00

Update 12/4/08
Thanks to everyone's comments, we took a deeper look and found what we were looking for in JMX. We plan on leveraging a single JVM/Tomcat instance with seperate contexts. The Tomcat Admin web application has the MBeans we were looking for enabling us to dynamically add/configure data sources.

Original post
I have a requirement that calls for breaking the product I am designing into multiple JVMs (Tomcat instances to be precise). The original plan was to have a single Tomcat instance with multiple contexts however that plan had to be shelved for various reasons. One of the Tomcat instances is to support the Management GUI. To cut a long story short, the Management GUI is required to manage the life-cycle (i.e. stop/start) of another Tomcat instance containing the core engine of the product.

I explored several options for approaching this problem. My first attempt was to look at OSGi. I have been eager to explore OSGi for quite a while now so this gave me a perfect opportunity. I managed to knock out a couple of simple bundles and deploy them to Equinox. Next step, try and build a bundle out of Tomcat. Unfortunately, this didn’t quite go to plan. Tomcat seems to have an issue with authority – it wouldn’t play nicely in an OSGi world leaving me with a stack trace of exception I didn’t have the time or energy to track down. I did think about switching to Jetty at this point, but this would be a very disruptive change and one that our schedule couldn’t allow.

Moving on, I recalled that Eclipse WTP does a pretty good job in managing server definitions and controlling their lifecycles. I proceeded to download the source for WTP and found a bunch of code called Server Tools. I tried to reverse engineer the structure from the code but was quite astonished by the size of the code base. I figured it would be too much to weed through to get to what I was looking for. The Server Tools is pretty well structured code – it’s just overkill for my needs.

Next step, there has to be something in Ant I can use. I remember using the Ant Catalina tools from years ago. These are basically Ant tasks that expose the controls of Tomcat through the Manager application to Ant. The tasks didn’t really give me what I was looking for. Sure, they could be used to control individual contexts but didn’t really give me anything to control the server instance itself.

I was really looking to avoid Runtime.exec() if I could. There had to be something that worked at a higher level of abstraction. I continued to dig around Ant and recalled the Java task. This sounded promising. Of course, I really wanted to leverage the class behind the Java task itself and not deal with build.xml, launching Ant and all that malarchy. The Java task provides me with what I wanted. Here’s the prototype code I came up with called the Tomcat Controller. I think the Java task fits my needs perfectly:

TomcatController.java


import java.io.File;

import org.apache.tools.ant.Project;
import org.apache.tools.ant.taskdefs.Java;
import org.apache.tools.ant.types.Path;

public class TomcatController {

  public enum State {
    STARTING,
    STARTED,
    STOPPING,
    STOPPED
  };
 
  State state = State.STOPPED;
 
  State getState() {
    return state;
  }
 
  private TomcatServerConfiguration config;
 
  public TomcatController(TomcatServerConfiguration config) {
    this.config = config;
  }
 
  // Asynchronously start this Tomcat
  public synchronized void start() {
    state = State.STARTING;
    Java javaTask = createTomcatJavaTask();
    javaTask.createArg().setValue("start");
    for (String s : config.getJvmArgs())
      javaTask.createJvmarg().setValue(s);
    javaTask.setFork(true);
    javaTask.setSpawn(true);

    System.out.println(javaTask.getCommandLine());
    javaTask.execute();   
    state = State.STARTED;// Not quite true - Tomcat may still be starting at this point
  }
 
  // Block while this Tomcat is stopping 
  public synchronized void stop() {
    state = State.STOPPING;
    Java javaTask = createTomcatJavaTask();
    javaTask.createArg().setValue("stop");
    javaTask.setFork(true);
    javaTask.setSpawn(false);

    System.out.println(javaTask.getCommandLine());
    javaTask.execute();
    state = State.STOPPED;   
  }

  private Java createTomcatJavaTask() {
    Project project = new Project();
    project.setBaseDir(new File(config.getTomcatHome()+File.separator+"bin"));
    Java javaTask = new Java();
    javaTask.setProject(project);
    javaTask.setDir(new java.io.File(config.getTomcatHome()+File.separator+"bin"));
    javaTask.setTaskName("tomcat");
    javaTask.setClassname("org.apache.catalina.startup.Bootstrap");
   
    Path classpath = new Path(project);
    classpath.createPathElement().setPath(config.getTomcatHome()+File.separator+"bin");
    classpath.createPathElement().setPath(config.getTomcatHome()+File.separator+"bin"+File.separator+"bootstrap.jar");
    classpath.createPathElement().setPath(config.getTomcatHome()+File.separator+"bin"+File.separator+"commons-logging-api.jar");
    javaTask.setClasspath(classpath);
    javaTask.createJvmarg().setValue("-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager");
    javaTask.createJvmarg().setValue("-Djava.util.logging.config.file="+config.getTomcatHome()+File.separator+"conf"+File.separator+"logging.properties");
    javaTask.createJvmarg().setValue("-Djava.endorsed.dirs="+config.getTomcatHome()+File.separator+"common"+File.separator+"endorsed");
    javaTask.createJvmarg().setValue("-Dcatalina.base="+config.getTomcatHome());
    javaTask.createJvmarg().setValue("-Dcatalina.home="+config.getTomcatHome());
    javaTask.createJvmarg().setValue("-Djava.io.tmpdir="+config.getTomcatHome()+File.separator+"temp");
    return javaTask;
  }
 
  public static void main(String[] args) throws InterruptedException {
   
    // Quick test - should really be in a JUnit test
    TomcatServerConfiguration config = new TomcatServerConfiguration();
    config.setName("Engine");
    config.setTomcatHome("c:\\apache-tomcat-5.5.26");
    config.addJvmArg("-Xmx512m");
    config.addJvmArg("-Dcom.sun.management.jmxremote");
    config.addJvmArg("-Dcom.sun.management.jmxremote.port=9128");
    config.addJvmArg("-Dcom.sun.management.jmxremote.ssl=false");
    config.addJvmArg("-Dcom.sun.management.jmxremote.authenticate=false");            
    TomcatController tc = new TomcatController(config);
    tc.start();
    Thread.currentThread().sleep(20000);
    tc.stop();
  }
}

TomcatServerConfiguration


import java.util.ArrayList;
import java.util.List;


public class TomcatServerConfiguration {

 private String name;
 private String tomcatHome;
 private List<String> jvmArgs = new ArrayList<String>();

 public TomcatServerConfiguration() {

 }
 public String getName() {
   return name;
 }
 public TomcatServerConfiguration setName(String name) {
   this.name = name;
   return this;
 }

 public String getTomcatHome() {
   return tomcatHome;
 }
 public TomcatServerConfiguration setTomcatHome(String tomcatHome) {
   this.tomcatHome = tomcatHome;
   return this;
 }

 public List<String> getJvmArgs() {
   return jvmArgs;
 }

 public void addJvmArg(String arg) {
   jvmArgs.add(arg);
 }

 public void clearJvmArgs() {
   jvmArgs.clear();
 }
}

Agile and mountain climbing

2008-06-18T06:34:00.000-07:00

I love agile. I cannot imagine getting software done in a traditional waterfront/BUFD way ever again.

Each sprint or iteration is kind of like making camp at the end of the day as you climb a mountain. However, the mountain does not go on forever (although it probably feels that way). You have to reach the summit one day and declare victory (then move on to another mountain - once you descend, recover and replenish supplies of course).

With agile you end up with a ton of releases accumulating - 1 for the end of each sprint. During the sprints, we pile features and fixes and we pray the documentation and QA can keep up with the pace. These releases are not really intended for external consumption. However, sometimes they may find their way into customers hands. This could happen from time to time.

However, let's consider the scenario where you had reached 8 sprints after a period of ~8 months (assuming 4 week sprints). You now have accumulated 8 releases in your wake. If each of those 8 releases have found their way into different customers hands, you are obliged to support that release. This means you have to consider 8 different upgrade scenarios. You also have to consider 8 x n different compatibility tests where n is the number of components in the product. This becomes un-manageable.

What is the solution? The answer is you have to bake into your release cycle a GA release. How many sprints maketh a GA release? It depends. How often should you do a GA release? It depends. The GA release is where you finally say - yes, we've reached the summit. It's time to put this effort to rest. Time to move on to the next mountain.

Are there any special considerations for the GA release? Yes. The emphasis for the GA release should be all about quality. Often, during the sprint the QA effort is most likely going to be focussed on functional testing. For a GA release, there is so much more testing that needs to be performed.

The GA release is about slowing down the pace of change and allowing the product to bake.

The parallels between Oil and Relational Database Management Systems

2008-06-16T19:42:00.000-07:00

We are addicted to Relational Database Management Systems. It's a safe, well-understood mature technology. Whenever we have
persistence requirements, our knee-jerk reaction is typically "let's stick it in a relational database". And why not indeed. It's not
a bad choice, and in some cases it may be your only choice.

Along the way, there have been some contenders to the crown. OODBMS - Object Oriented Database Management Systems spring to mind. But they never really took off, finding themselves pushed into a niche.

It kind of reminds me of our dependence on oil for energy. Instead of OPEC though, we have IBM, Oracle and Microsoft plus
I would include Sun (MySQL). I'd probably get slammed if I didn't include PostgreSQL in the mix too (although I would say this
is probably not high octane).

However, in 2008 things are a little different. There are some worthy alternatives you may want to check out. So what are the alternatives? Well, we have data grids. There are a bunch to chose from. Oracle Coherence is one that I first became familiar
with through my work at Delta (back when it was Tangosol). There are others available too now for both Java and .NET platforms.

Bigtable (http://labs.google.com/papers/bigtable.html) from Google and SimpleDB/S3 are also very interesting alternatives.

Next time you are working on designing a new system, you owe it to yourself to at least check out some alternatives.

Why GWT matters - part I

2008-05-29T07:20:00.000-07:00

I have been heads down taking a deep look at GWT of late. This is the first installment of my thoughts on why I believe GWT matters.

Before I got into web development, I was heavily involved in GUI development mostly for UNIX workstations. Back in the day, I used toolkits based on Xt - first OLIT and then later Motif (when OLIT was dropped). At about this point, C++ really started to become popular and later on, for another project I used Rogue-Wave Tools.h++.

The programming model was fairly natural and consisted largely of creating components (or widgets) such as buttons, text fields, sliders, scroll bars and the like; adding them to other components such as panels or frames; specifying properties such as colour, size, font; organizing into a layout; adding callbacks (or listeners) to respond to events (e.g. button pushed, slider dragged).

All components were created in the same way, although some components naturally exposed different properties you could set.

This standardized programming model laid the foundation for impressive tools such as Visual Basic and Borland Delphi. Back in the day, the buzzword of the day was RAD or Rapid Application Development. These tools, enabled a blank canvas representing the application under construction on which you could drag and drop components onto the canvas from a palette of available components. Behind the scenes, code was being generated to reflect the visual design.

And then, the web happened. Once it got past the initial "brochure-ware" stage, thanks initially to CGI it quickly evolved into an application platform. The web was great in that it truly solved the distribution problem. You no longer had to worry about getting your application installed. Instead, you would drop it on the server and the user would just point their browser and boom you're away. Upgrades - not a problem. Who cares what platform the browser runs on - you no longer had to worry about porting code natively to the platform. As long as the browser was available on the platform, your application was accessible.

However, this came at some cost. We had to learn new stuff like HTML, HTTP and unlearn some old stuff.

Read the next installment - part II (when I get around to writing it)