LumoSQL

LumoSQL is a precise modification of SQLite to add performance, security and privacy features. The result is a new category of software compatible with the original SQLite. LumoSQL is currently a working prototype, with the next goal being production testing. The NLnet Foundation funded Phase I of LumoSQL to running proof of concept, and is continuing to fund Phase II.

Surprising Background

The following facts often surprise people:

• the most-deployed software, by a factor of at least four zeros, is a compact but full-featured database
• this database is called SQLite, and a typical mobile phone stores all non-streaming data in several hundred SQLite databases.
• SQLite is also used in web browsers, operating systems, vehicles of all kinds and so on.
• SQLite is open source, exceptionally well-maintained (with many contributions) by fewer than 8 people loosely connected in a small, unambitious company.

The corollaries are significant:

• This is uncharted territory for Computer Science: is SQLite's ultra-conservative compatibility commitment to its hundreds of billions of installations the right choice? Is SQLite's fast-moving support of formal database standards the best way forward?
• Why are there so many forks of SQLite, none with more than trivial (say, a few handsfuls of millions) of deployments? The nature of the project seems to guarantee its success and also constrain its future.
• Why does nobody appear to have noticed there is a strategic problem in that the most-used software is incompatible with security, privacy and other requirements of the 21st century, some of them mandated by law in many countries?

LumoSQL Phase I Completed

This is a technical paragraph. For even more technical detail see the code development page.

LumoSQL is a modification (not a fork) of the SQLite embedded data storage library. LumoSQL offers multiple key-value backend storage systems selectable by the user. It offers features not found in any other mainstream database:

• ability to checksum every row on write and verify on read
• ability to trigger arbitrary functions on per-row read and write
• a general test suite for benchmarking precisely how LumoSQL (or SQLite) is performing and the full context of that benchmark run. For some reason database benchmark is very poorly done, including by the TCP-C consortium founded for solely that purpose.
• a general build system able to mix and match multiple versions of the database with multiple versions of multiple backends. Never before has it been possible to compare the different strategies of various Key-Value stores with the same database frontend.

If you are an SQLite user familiar with C development wanting an easier way to benchmark and measure SQLite, or if you are wanting features only available in other key-value storage engines, then you will find that LumoSQL offers new features even in its prototype stage.

LumoSQL Phase II Has Started

In Phase II LumoSQL is implementing at-rest encryption and privacy using the features developed in Phase I, and readying LumoSQL for more general testing.