LumoSQL

LumoSQL modifies the SQLite Open Source database software to add performance, security and privacy features. LumoSQL matters because there are many copies of SQLite on every mobile phone in the world. LumoSQL is a technical product for software developers to use.

LumoSQL is a new category of software compatible with the original SQLite. LumoSQL is currently a working prototype. The NLnet Foundation funded Phase I of LumoSQL to running proof of concept, and is continuing to fund Phase II, however, LumoSQL mostly exists due to the volunteer effort contributed by coders, cryptographers, statisticians, system administrators and more.

Surprising Background

The following facts often surprise people:

• SQLite is probably the most-deployed software, by a factor of at least four zeros.
• SQLite is a full-featured database, supporting the standard SQL language despite being tiny compared to all the other mainstream SQL databases.
• a typical mobile phone stores all non-streaming data in several hundred SQLite databases.
• SQLite is also used in web browsers, operating systems, vehicles of all kinds and so on.
• SQLite is open source, exceptionally well-maintained (with many contributions) by around 8 people loosely connected in a small, unambitious company. (There are many more than 8 people who contribute occasionally to SQLite.)

The corollaries are significant:

• This is uncharted territory for Computer Science: is SQLite's ultra-conservative compatibility commitment to its hundreds of billions of installations the right choice? Is SQLite's fast-moving support of formal database standards the best way forward?
• Why are there so many forks of SQLite, none with more than trivial (say, a few handsfuls of millions) of deployments? The nature of the project seems to guarantee its success and also constrain its future.
• Why are the obvious strategic problems with SQLite not discussed more widely? The most-used software in the world is incompatible with security, privacy and other requirements of the 21st century, so why isn't this a hot topic?

LumoSQL Phase I Completed

This is a technical paragraph. For even more technical detail see the code development page.

LumoSQL is a modification (not a fork) of the SQLite embedded data storage library. LumoSQL offers multiple key-value backend storage systems selectable by the user. It offers features not found in any other mainstream database:

• ability to checksum every row on write and verify on read
• ability to trigger arbitrary functions on per-row read and write
• a general test suite for benchmarking precisely how LumoSQL (or SQLite) is performing and the full context of that benchmark run. For some reason database benchmark is very poorly done, including by the TCP-C consortium founded for solely that purpose.
• a general build system able to mix and match multiple versions of the database with multiple versions of multiple backends. Never before has it been possible to compare the different strategies of various Key-Value stores with the same database frontend.

If you are an SQLite user familiar with C development wanting an easier way to benchmark and measure SQLite, or if you are wanting features only available in other key-value storage engines, then you will find that LumoSQL offers new features even in its prototype stage.

LumoSQL Phase II Has Started

In Phase II LumoSQL is implementing at-rest encryption and privacy using the features developed in Phase I, and readying LumoSQL for more general testing.

Notable outcomes from LumoSQL already include:

✅ The only mainstream database without a Write-ahead Log

✅ The only mainstream database with swappable Key-Value stores, where all stores are peers rather than one having special knowledge that gives it technical advantages

✅ preliminary Benchmarking results

✅ The Not Forking tool, which avoids forks in both simple and complicated source code

Exciting things in progress:

⛅ Lumions are RFC for universal encrypted blobs with authentication

⛅ Documented API for arbitary key-value stores

⛅ Documented API for accessing the key-value stores via the SQLite library, instantly making the SQLite key-value store the most widely-distributed key-value store. Nothing calls the SQLite key-value store today except SQLite