Introduction to Gradle

At my new gig, one of my first assignments was to re-engineer our build system. For this task I used Gradle. To ease the transition for the team, I put together a YouTube video

Tips on designing an upgrade process

I’d like to share some of my lessons learned and pass these on.
Consider the current working version (x) sacred – the upgrade process should not touch it because you always need to provide an option to roll-back to (x) – perhaps even a couple of hours after the new version (y) has been deployed. Design hint: when upgrading, lay the new version (y) along-side and not on top of version (x). This implies a migration of configuration files at the application layer and data migration at the data layer from version (x) to version (y).

In a high-availability environment, consider is it possible to run both version (x) and (y) concurrently – this is to support the concept of a rolling upgrade where you upgrade each node in turn.

By far, the trickiest part of designing an upgrade process is considering evolutionary changes to the data schema. New tables, columns or constraints may have been added. You need to pay close attention to changes to the schema during development and always think about what this means for the migration of existing data. It’s usually a good idea to embed the version number in the name of the schema/database.

It’s essential that data created with the old version (x) is available with the new version (y) – but also, ensure that the operations the system provides on the data exhibit a consistent behavior where expected.

DevNexus presentation

Back in March of 2010, I gave a presentation at the devnexus.com conference in Atlanta.

The presentation was entitled "From whiteboard to product launch". The intent being to share my teams experience in bringing a brand-new product to the marketplace. It covers a wide variety of areas including process, architecture and team organization.

The slides are available here and the audio is available here (I recommend you flip through the slides while listening to the audio).

How to scale the data layer

Many a performance problem as a system struggles to support growing demand can be pointed to the data layer. The database can easily become a bottleneck and can be the hardest to scale after the fact because it manages state (in contrast, stateless components are easy to scale).

Of course, the problem may be solved by throwing more hardware at the problem – i.e. upgrading the database server by increasing memory and/or CPU horsepower. However, it is prudent to think about scalability before rather than after the fact.

First, let us consider some elementary physics:

pressure = force / area

For a given force, if we reduce the area to which the force is applied the pressure increases. Try for example, standing on tip-toes like a ballerina to get a sense of what I’m talking about. The force i.e. your body weight is constant, but we are reducing the surface area to which the force is applied resulting in increased pressure.

Likewise, if we increase the surface area the pressure is reduced. This is how a person can lay on a bed of nails without puncturing the skin – there needs to be of course enough nails.

Typically, we don’t have any real control over the forces applied to the products or solutions we are designing and building. We can speculate and design with certain limits in mind, however what if we come across a situation where those limits are shattered? In the design, we need to think of a way of how we can support this and yet reduce the pressure on the entire system.

One approach is to use horizontal and vertical portioning techniques in the design of the data layer. Both these approaches have the effect of increasing the surface area and thus reducing pressure points as the force (or load) on the system increases compared to a single monolithic data layer (i.e. single database server).

Horizontal partitioning is where the rows of a single logical table are spread over multiple physical databases. For example, customers whose last name begins with A-F may be stored in one database server and customers whose last name begins with G-L are stored in another database server and so on. It is a popular technique used by several large dot-coms as a way to spread the load.

Vertical portioning is a similar concept but involves storing different tables in different physical databases. For example, purchase orders may be stored in one database server whereas invoices may be stored in another.

To learn more about horizontal and vertical partitioning, take a look at http://en.wikipedia.org/wiki/Partition_(database)

Conceptually, the approaches are very straightforward however if you want to leverage them in your design there are several things to look out for.

For horizontal paritioning, you want to strive for even distribution across the partitions. For example, partitioning based on the first letter of the last name may not provide even distribution - V through Z for example may be quite light in the number of records. Hashing techniques may provide a good approach.

For vertical partitioning, references across data models may be tricky. In particular, forget about foreign key constraints. If the two data models have a high degree of coupling between them, then I would go back to the drawing board – perhaps you haven’t found the right boundary. It’s ok to go to one data model, find what you are looking for, then use the result from that query, to find what you ultimately need from the second data model. Avoid distributed transactions (XA/2PC) that span more than one data model if at all possible. Ask yourself, do you really need referential integrity across two data models at all times? If so, then again you should revisit your data design and carve out the right boundaries.

When designing data models at the highest level, strive for high-cohesion (if it changes together, then it stays together) and loose-coupling between the data models. In other words, apply some of the same desirable properties for code design to your data design.

You know you're a geek when

Writing a shopping list the other day, I wrote down Guice instead of Juice. Time for a break from work I think.

Tomcat Controller update

http://www.jasondchambers.com/2008/07/tomcat-controller.html

Using Grails to explore and develop the domain model

Data modeling is typically a fairly important design activity. However, I have a hard time with EAR diagrams and data models as a starting point – particularly for a new system (you don’t really have a choice when dealing with legacy databases). Thinking in objects is much more natural to me. Also, I like the idea of using code to explore the domain model and try a few things out.

In the past, I have used the forward engineering features of Hibernate for this very purpose. Once the tables have been created in the database, I take advantage of a neat feature in JDeveloper where you can reverse engineer a data model diagram. So the process kind of goes like this Object model->Java code->Hibernate DDL->Database->Data model. It worked well in thrashing out the data model for my last project. Of course, it may not work for everyone but that’s ok – it works for me and that’s all that matters ;-)

For a new project I’m working on, I thought I’d try something different. I thought I’d give Grails a spin. This worked even better because Grails can generate such a lot of boiler-plate code for me enabling me to move a lot faster. It can also generate a scaffolding UI so I can interact with and test out the model. Here was the process:

1. Download and install Grails from grails.org


2. Follow the Quick Start guide to familiarize yourself with Grails http://grails.org/Quick+Start . During the Quick Start guide, you will learn how to create an application a domain class and a controller


3. Configure Grails to use Oracle instead of HSQL (I needed to externalize the database so I could browse it and reverse engineer the data model diagram using JDeveloper) – to do this modify grails-app/conf/DataSource.groovy – change the driver class name and the JDBC url – you may also have to copy the Oracle JDBC driver (ojdbc14.jar) to the lib directory


4. For each entity (where name is the name of the entity you want to model)
   
   
   
   1. $ grails create-domain-class
   
   
   2. Modify the generated class located in grails-app/domain/name.groovy – add the attributes
   
   
   3. $ grails create-controller
   
   
   4. Modify the generated controller class located in grails-app/controllers/nameController.groovy – change the body of the class to look like def scaffold = name
   
   
5. Run the application $ grails run-app


6. Point your browser to the application


7. Populate the model by interacting with the controllers through the generated scaffolding


8. Add some more entities by going to back to step 4


9. If you want to auto-populate the model with data on startup, add your code to grails-app/conf/BootStrap.groovy

I have to say, I found the out of the box experience with Grails pretty polished. It passed the 15 minute test with flying colors. (If I can’t get something working in under 15 minutes, I tend to dump it).