To SQL or not to SQL?

Tue, 21 Jul 2009 10:07:00 GMT

Eric Lai wrote an interesting article for Computerworld entitled No to SQL? Anti-database movement gains steam that highlighted the small but growing trend for not using a traditional relational database for managing data. Nati Shalom's No to SQL? Anti-database movement gains steam - My Take is a good follow-up.

I'm sure that most of us have good experiences of using a relational database in our projects and, despite the pain of sometimes mapping the data into a relational schema, relational databases provide an easy to use known quantity for managing data. With this in mind, you should certainly let your experience guide you but bear in mind that relational databases aren't the answer to every question.

The case study that we use as a basis for the exercises in our software architecture training course is relatively small yet most people decide to use a relational database to store the data without really giving the alternatives a second thought. Most of the time it comes down to experience in that most people are comfortable with using a relational database in their architecture. But there are many viable alternatives; from using flat files and object databases through to in-memory data structures, data grids and the cloud. So while relational databases might be a solution for your particular problem, it's always worth spending a couple of minutes assessing whether they are the *best* solution. Here are some things to think about before deciding on whether to go down the SQL or non-SQL route.

Do you have special non-functional requirements that would be hard to satisfy with a relational database? (e.g. high performance/low latency, massive scalability, etc)
What is the available skillset of the team?
Do you have existing licenses for an RDBMS?
Could an open source RDBMS be appropriate?
Do you need to access legacy systems where the data is already in an RDBMS?
Do you have the hardware available to run an RDBMS?
Do you already have existing backup and archival processes and procedures for relational databases?
What are your management information and reporting requirements (scheduled and ad hoc)? Is it possible to satisfy these with a non-SQL solution?
Do other systems need access to your data via a SQL interface? (a service gateway might be a better approach, but that's another issue)
Do you have data migration requirements from an existing relational database?
Can the data management problem be split up into transactional and non-transactional partitions, maybe using a relational database for only one of them?
Do you really *need* persistence?

Relational databases are established mainstream solutions that are applicable in many cases. Just don't forget that there are other alternatives too.

The joy of sets

Wed, 13 Feb 2008 17:30:00 GMT

I've recently seen impressive performance gains in a data-centric process, which is a generic enough concept to be of general interest.

Imagine a system which consolidates the trades done in 10 different branches of a supermarket chain. We receive these trades on a batch basis: a file per branch every night. This file contains de-normalised rows of data containing information about a customer (e.g. identified through their loyalty card number), a product, and how much was purchased. We need to convert this data into a normalised schema where product and customer have their own set of relational tables. Our database doesn't store an exhaustive set of every product sold by the supermarket chain, so we only bother to store information about products when they appear the input files. Let's also assume we're doing this by hand through stored procedures -- no fancy ETL tools here.

The iterative-programmer's style of dealing with this problem is likely to be looping through the input file, and having conditional checks on each row, e.g.:

For each row in the input file:

if the product doesn't exist in our database, insert the product
if the customer doesn't exist, insert the customer
finally, insert the price and quantity which was purchased

We apply the above algorithm to each file we receive from each supermarket branch we are importing data for, and, because major relational databases pride themselves on their ability to deal with concurrent access, we have a thread (or process) per branch, where each thread imports all its trades in its own transaction. We kick them off all at once, hence finishing in approximately as much time as it takes to import the branch with the most data. Right?

Wrong. When certain conditions arise, this process serialises so it takes 10 times longer than expected, as if we'd done each branch one at a time. (In fact, with naive error handling, this approach may never work due to race conditions).

If one product appears for the first time in 2 different branches, let's say the West London branch and the South Manchester branch, then whichever branch encounters that product first in the feed file will insert it. When the other branch then tries to insert this product, it needs to wait for the first branch's transaction to commit or rollback before knowing if it can create the same row (in our database we naturally have primary key constraints to ensure each product is represented by 1 unique row). So the second branch waits for the whole of the first branch's transaction to commit!

When you're writing PL/SQL and you're thinking at the level of cursors and if-then-else statements, the problems outlined above may not seem obvious. If you're purely a Java/C#/C++ programmer you may start thinking along the lines of "well, perhaps we should reduce the size of our transaction", or "perhaps we can ignore this row and come back to it". These are also illustrations of thinking about the problem in the wrong way.

The correct approach is not to consider the input data as a succession of rows, but rather as a set of sets: identify the set of products you are importing, the set of customers, and the set of prices and quantities. Logic can then be applied to different sets: identify the union of new customers, and the union of all new products, and insert these before inserting the price and volume information in a concurrent manner.

The overhead of the pre-processing required is quickly recouped by the excellent performance achievable once the problem has been reduced in such a way that the lion's share of the work is embarrassingly parallel.

The original problem here arose again due to a paradigm clash: procedural programming, such as the constructs at your fingertips in languages like PL/SQL, encourage you to think of solutions to problems which are probably better solved using relational algebra, which ANSI SQL captures pretty well. Of course, PL/SQL, T-SQL and the like will always have their place. But if you consider yourself an iterative or object oriented programmer, and you find yourself writing stored procedures, the most elegant solutions will often be achieved if you approach the problem thinking in sets, not loops.

Coding the Architecture - database tag

To SQL or not to SQL?

The joy of sets