LumoSQL

Technically: LumoSQL modifies the SQLite Open Source database software to add essential performance, security and privacy features. LumoSQL is a technical product for software developers to use. LumoSQL is currently a working prototype.

Socially and Commercially: LumoSQL matters because virtually all data for all apps on mobile devices is stored on SQLite, making it by far the most-used database anywhere. Yet this database is vulnerable to well-informed snoopers and does not even come close to the gold-standard privacy requirements of the EU. LumoSQL is a new category of software compatible with the original SQLite.

Practically: LumoSQL exists due to the volunteer effort contributed by many skilled people. The Vrije Universiteit Brussel continues to fund valuable contributions in cryptography and mathematical analysis. The NLnet Foundation funded Phase I of LumoSQL, and is continuing to fund Phase II.

Surprising Background

The following facts often surprise people:

• SQLite is very likely the most-deployed software, by a factor of at least four zeros. It is likely that SQLite is the only trillion-scale software in existence, although that statement needs some validation work to be sure.
• SQLite is a full-featured database, supporting the standard SQL language despite being tiny compared to all the other mainstream SQL databases.
• a typical mobile phone stores all non-streaming data in several hundred SQLite databases.
• SQLite is also used in web browsers, operating systems, vehicles of all kinds and so on.
• SQLite is open source, exceptionally well-maintained (with many contributions) by around 8 people loosely connected in a small, unambitious company. (There are many more than 8 people who contribute occasionally to SQLite.)

The corollaries are significant:

• This is uncharted territory for Computer Science: is SQLite's ultra-conservative compatibility commitment to its (at least) hundreds of billions of installations the right choice? Is SQLite's fast-moving support of formal database standards the best way forward?
• Why are there so many forks of SQLite, none with more than trivial (likely at very most a few handfuls of millions) of deployments? The nature of the SQLite project seems to guarantee its success and also constrain its future.
• Why are the obvious strategic problems with SQLite not discussed more widely? The most-used software in the world is incompatible with security, privacy and other requirements of the 21st century, so why isn't this a hot topic? Perhaps because it works so well that people don't wish to rock the boat. But with new features required, and it being unacceptable to store data without any form of encryption, there needs to be some compatible new option. LumoSQL wants to be that option.

LumoSQL Phase I Completed

This is a technical section non-technical readers can skip. For even more technical detail see the code development page.

LumoSQL is a modification (not a fork) of the SQLite embedded data storage library. LumoSQL offers multiple key-value backend storage systems selectable by the user. It offers features not found in any other mainstream database:

• ability to checksum every row on write and verify on read
• ability to trigger arbitrary functions on per-row read and write
• a general test suite for benchmarking precisely how LumoSQL (or SQLite) is performing and the full context of that benchmark run. For some reason database benchmark is very poorly done, including by the TCP-C consortium founded for solely that purpose.
• a general build system able to mix and match multiple versions of the database with multiple versions of multiple backends. Never before has it been possible to compare the different strategies of various Key-Value stores with the same database frontend.

If you are an SQLite user familiar with C development wanting an easier way to benchmark and measure SQLite, or if you are wanting features only available in other key-value storage engines, then you will find that LumoSQL offers new features even in its prototype stage.

LumoSQL Phase II Has Started

In Phase II LumoSQL is implementing at-rest encryption and privacy using the features developed in Phase I, and readying LumoSQL for more general testing.

Notable outcomes from LumoSQL already include:

✅ The only mainstream database optionally without a Write-ahead Log

✅ The only mainstream database with swappable Key-Value stores, where all stores are peers rather than one having special knowledge that gives it technical advantages

✅ preliminary Benchmarking results

✅ The Not Forking tool, which avoids forks in both simple and complicated source code

Exciting things in progress:

⛅ Lumions are described in a draft RFC for universal encrypted blobs with authentication

⛅ Documented API for arbitary key-value stores

⛅ Documented API for accessing the key-value stores via the SQLite library, instantly making the SQLite key-value store the most widely-distributed key-value store. Nothing calls the SQLite key-value store today except SQLite