Wednesday, 9 January 2008

Where did my database go?

Once upon a time, when I was coding a lot more for my day job, my SQL skills were pretty hot. Least, I thought so. I'd do pretty full on crunching with large data sets; I remember one project in particular, which was 75% SQL, 10% "Classic" VB, 10% Excel and 5% VB Script, that crunched itemised bills which totalled millions of pounds. Needed to be accurate and performant, even thought it was offline, I remember running one batch job that lasted about a week; mind you it only had a teeny NT4 server to run on, looks pathetic now; the apps output wasn't very time sensitive, but it did have to finish each job!

Pretty much every development that I've done in the last 15 years used an RDMBS of some kind. That meant understanding SQL.

But I see a sea-change. I see the emergence of hosted database platforms, except they're not general purpose RDMBS databases, they tend to be more special purpose.

I'm sure the first one that jumps to most people minds is Amazon's SimpleDB, after all the Web Services team at Amazon seems to do no wrong. It's not an RDMBS, apparently it's more like a directory service, with REST and SOAP interfaces. Yet to see what interesting apps get built on it, and how publicly we get told.

Next on the list is Force.Com. Seems this gives you a reduced SQL set called SOQL; notably UPDATE, INSERT and DELETE are missing, instead you SELECT a bunch of objects, and then act on them through a dot notation. So a bit of a mix and match.

There's quite a few others, and one could even argue that the Blogger platform I'm writing this on is a database service, inasmuch as it stores data, and links it to other things like the comment you may add.

I like the look of Astoria too; Pablo is one of the few folk who work at Microsoft who seem to grok the web. If this initiative does what I hope it will do, then I'll be able to develop my data tier locally, and deploy to the cloud with very little friction. They do make good and consistent, if huge and bloated, developer tools. I think the combo of the Asp.Net MVC framework and this piece is gonna actually make developing web apps on a Microsoft platform a possible choice again; though they do need something like Scriptaculous or Dojo to pull it all together at the front end, not sure their AJAX framework is quite there; and I'm concerned they may get distracted by future plans like Volta; it's too early to say whether that's the future of development.

Another trend, which has been around for a good long while now, is the abstraction away from the database, at least for developers, by ORM frameworks, sometimes built into a larger framework (eg Rails, Django). I'm pretty excited about the Hibernate Shards project for breaking the distributed database problem, without the average developer having to learn how that all complexity works, but it may be moot soon with all the cloud based data storage services springing up.

It seems to be a current wisdom that startups should reach "scale" before thinking about business models. Which may or may not be correct, and I'll let you draw your own conclusions. Does look though, that modern, high level, services on the web and programming frameworks are gonna start solving that "write cheap and cheerful code to scale up, deploy to scale out" problem.

I reckon In the next two years we'll all be writing extremely concurrent code with highly distributed data stores all over the web (let's just forget the emotional term database and let DBAs find a new career by themselves) and wonder why we ever did it the old way.

Note. There are safe harbor, copyright, privacy and DPA problems to be resolved, and, as ever, technology is ahead of regulation. But, for the purposes of this post, everything is permissible.


Robbie said...

It's an interesting trend that I hope continues, the more these utility services are hosted the easier it should become for developers and businesses to build new applications.

Data (S3, SDB), compute (EC2) and communication (Mojo), all on the web!

DE said...

For me, a database is always something that should be hosted Elsewhere. But putting your DB in the cloud is exchanging one problem set for another.

You rightly point out that Blogger is database, and that is certainly the future. Lots of useful apps and data driven services on the web until maybe I won't need my own database for any reason.

Tim Stevens

Tim Stevens
Be Silent