Saturday, January 16, 2010

Database Scalability in a High-Volume Environment

Traditionally, the term "database" as used by technical professionals implied "relational, row-based, transactional." These implied extra characteristics do us a disservice when speaking of databases in a high-volume environment where there must be several types of databases deployed, some of which aren't relational, some of which aren't row-based, and most of which aren't transactional.

What is a Database?

What is a database? A database is merely a system for storing data such that retrieval is efficient. Therefore, a database administrator is responsible for picking the best database for the job. If you find yourself requiring words like "transactional" or "normalised data" when speaking of the fundamental capabilities of a database, you are missing the subtleties. Such capabilities are useful in some situations, but are often not required. Having capabilities where they are not required is a waste of resources which becomes painfully apparent in high-volume environments.

One capability databases need to provide in a high-volume environment is write load distribution across nodes in a cluster. This is a capability not traditionally provided by the DBMS, and engineers in high-volume environments have rolled their own systems to do it hundreds of times. DBMSs with this capability have only recently started to appear in the Open Source world.