I retired from personal blogging in July 2008.
But you can find me over at http://blog.xero.com.
Shipping days are always a buzz at Xero. You can code away for as long as you like but it counts for nothing until we ship and go live.
The quote attributed to Steve Jobs ‘Real Artist Ship‘ has stuck with me for years.
Shipping is the final step in a number processes. Releases in a Software as a Service (SaaS) world are completely different from Enterprise Software.
At AfterMail, which was Enterprise Software - so installed on each customers site, we tried to upgrade the software no more than 3 monthly. That is because each client site needs to be upgraded. Shipping a new release creates all sorts of downstream work for customers and partners. Often we would do a number of releases before shipping the new version. The more customers you have the harder it becomes. You create more and more drag as you go along.
The consequences of a bad software release were huge as it often means that another version has to be shipped and will require repeating the upgrade process. This might mean a partner has to jump in the car for a few hours to get on site again. They don’t like that!
Sometimes we had to do mini-releases or patches.
It’s common for established software products, like the desktop incumbents that we are displacing, to only release new versions annually or even go 2 years or more between releases, such is the effort to upgrade.
One of the many big benefits of SaaS is that we only have a single version of the application in production. Every customer is on the same current version. So we can do lots of small releases with minimal impact to customers. At Xero we try to release every 2-3 weeks 1 or 2 major features and as many improvements as we can get in.
These bite sized chunks are easy for customers to digest as well.
It’s an evolutionary and responsive process. The cycle starts with the prioritization debate which I’ve mentioned before.
The development process starts with our BA’s and interaction designers building a specification for each new feature or improvement. We use Flash as a prototyping tool, starting with fairly low-fidelity screen walk-throughs which refine to more detailed working models.
These allow us to nail the design decisions before a line of code is written. These prototypes might take a few hours to a few weeks. During this process we talk to lots of customers and show the prototypes internally until we have agreement on what we want to do.
The design is handed over to development and the coding begins. Development time can be very fast. A big bit of work might take 2-3 people a month.
At any one time we have a number of features being built and tested on our internal servers.
2 weeks prior to a release we pull together the features that are ready into an integrated build where integration testing starts. We have functionality being built that may not ship for several release cycles.
Integration testing includes running a suite of automated tests.
Once the QA team is relatively happy, we then deploy to our secure live staging environment where we can test the migration scripts, complete regression testing and test the application for a couple of days.
Once QA have signed it off then we proceed to live deployment.
As we have evolved our processes and teaming models we have noticed that we can write code faster than we can test. In an application such as ours when we are dealing with peoples money it just has to be right. So we invest heavily in testing and try to give our testing team enough time to complete their testing work programs. They have the final call as to whether we can ship or not.
The actual process of deploying to production is fairly straight forward, and a dream compared to AfterMail.
- We generate the database change scripts
- We publish a release build of the Xero applications
- We upload those files to our release server
- If there are database changes we use ASP.Net app_offline feature. We try to release at 6am to ensure minimal disruption to our customers.
- We xcopy deploy the applications to the application servers and run the sqlserver change scripts. The process might be a fast as 5 minutes.
- We then bring the new version online and run through a bunch of standard regression tests just in case something strange happens.
- We then put up a blog and tell the world, like we did this morning.
As we are shipping every couple of weeks, as soon as we ship our testing team flips straight into the integration testing of the next release.
Releasing software is the end result of a big, but quick, set of processes. It’s a real buzz to see features that we’ve made be used and commented on by customers
So hopefully you can see why we get excited on Shipping Day.
blog
Excellent article. I was one of the primary designers of the clear.net ISP back in the ’90s. A couple of things that we learnt quickly when launching was:
*** It is easy to change things before you launch, once you have tens or hundreds of thousands of users on it, it is much much harder! In other words, get it right from the start.
*** You screw something up, you get instant feedback from your customers! (The development team was sitting right beside the helpdesk team then. The feedback loop was almost instantaneous.)
*** Online is ideal for incremental development and deployment. Convincing management of this is sometimes harder…
*** A mind shift was required from an IT perspective: going from 8×5 business hours availability to 7×24x365 was challenging in the beginning.
*** We had to design our platform for continuous operation that could handle OS and software upgrades without downtime. We could not exactly shut down the ISP for a couple of hours while we upgraded some systems.
*** We had to design our platforms for high availability, using clustering, and other techniques. (Load-balancers were not yet available then.) Having N+1 redundancy on everything saved us on more occasions than I can count.
*** We had to learn to get up early in the mornings for deployments to ensure we rolled out in the off-peak hours. Typically 4am-6am. Sometimes earlier. Arghhh.
As software companies grow, they generally go through the same learning curve of developing internal processes to manage incremental software deployment. The main driver is realising that the existing ad hoc process is taking too much time, and too error prone.
I’ve seen lots of companies go through the same learning curve, and generally they all end up with the same conclusions. While software development has had a lot of focus to reduce costs and improve efficency through common procedures, I hope the wider software development lifecycle will be looked into further - versioning, testing, deployment.
Although there could be roadblocks to this nirvana if some people have their way: http://www.freepatentsonline.com/7340520.html