LumoSQL

LumoSQL protects data on mobile phones using a new data storage technology compatible with what is already used on every device. The device owner has ultimate right to decide who can read or change their data... and this decision continues to be enforced even after it has been copied off the phone to (for example) a bank or insurance company for processing with their in-house database software.

If a criminal or government officer takes a phone away from its owner, LumoSQL data cannot be read without the consent of either the phone owner or someone(s) to whom the phone owner has granted access.

The detailed strategy document LumoSQLMotivation-1.0.pdf explains the social, business and technical pressures that mean LumoSQL brings useful and radical change to data storage.

In summary:

Technically: LumoSQL modifies the SQLite Open Source database software to add essential performance, security and privacy features. LumoSQL is a technical product for software developers to use. LumoSQL is currently a working prototype, and uses Predicate Based Encryption to manipulate Lumions.

Socially and Commercially: LumoSQL matters because virtually all data for all apps on mobile devices is stored using SQLite, making it by far the most-used database anywhere. Yet this database is vulnerable to snoopers and does not even come close to the gold-standard privacy requirements of the EU and various non-EU countries. LumoSQL is a new category of software, highly compatible with the original SQLite and protecting our most intimate data.

Practically: LumoSQL exists due to the volunteer effort contributed by many skilled people. The Vrije Universiteit Brussel continues to fund valuable contributions in cryptography and mathematical analysis. The NLnet Foundation funded Phase I of LumoSQL, and is continuing to fund Phase II.

Surprising Background

The following facts often surprise people:

• SQLite is very likely the most-deployed software, by a factor of at least four zeros. It is likely that SQLite is the only trillion-scale software in existence, although that statement needs some validation work to be sure.
• SQLite is a full-featured database, supporting the standard SQL language despite being tiny compared to all the other mainstream SQL databases.
• a typical mobile phone stores all non-streaming data in several hundred SQLite databases.
• SQLite is also used in web browsers, operating systems, vehicles of all kinds and so on.
• SQLite is open source, exceptionally well-maintained (with many contributions) by around 8 people loosely connected in a small, unambitious company. (There are many more than 8 people who contribute occasionally to SQLite.)

The corollaries to these unusual facts are significant:

• This is uncharted territory for Computer Science: is SQLite's ultra-conservative compatibility commitment to its (at least) hundreds of billions of installations the right choice? Is SQLite's fast-moving support of formal database standards the best way forward?
• Of the many forks of SQLite, none have more than trivial (likely at very most a few handfuls of millions) of deployments. This includes useful, stable and well-maintained forks. It seems the nature of the SQLite project ensures its success but also constrains its future into the very long term.
• The obvious strategic problems with SQLite are not widely discussed. Evidently, the most-used software in the world is incompatible with security, privacy and other requirements of the 21st century. Perhaps this is not a hotly-discussed topic because SQLite works so well and is so ubiquitous that people can't imagine an alternative. But with new features required, and it being unacceptable to store personal data unencrypted, there needs to be some compatible new option. LumoSQL wants to be that option.

LumoSQL Phase I Completed

This is a technical section non-technical readers can skip. For even more technical detail see the code development page.

LumoSQL is a modification (not a fork) of the SQLite embedded data storage library. LumoSQL offers multiple key-value backend storage systems selectable by the user. It offers features not found in any other mainstream database:

• ability to checksum every row on write and verify on read
• ability to trigger arbitrary functions on per-row read and write
• a general test suite for benchmarking precisely how LumoSQL (or SQLite) is performing and the full context of that benchmark run. For some reason database benchmark is very poorly done, including by the TCP-C consortium founded for solely that purpose.
• a general build system able to mix and match multiple versions of the database with multiple versions of multiple backends. Never before has it been possible to compare the different strategies of various Key-Value stores with the same database frontend.

If you are an SQLite user familiar with C development wanting an easier way to benchmark and measure SQLite, or if you are wanting features only available in other key-value storage engines, then you will find that LumoSQL offers new features even in its prototype stage.

LumoSQL Phase II Has Started

In Phase II LumoSQL is implementing at-rest encryption and privacy using the features developed in Phase I, and readying LumoSQL for more general testing.

Notable outcomes from LumoSQL already include:

✅ The only mainstream database with swappable Key-Value stores, where all stores are peers rather than one store having special knowledge that gives it technical advantages. The two stores we have concentrated on so far are (1) the existing SQLite store with optional, binary-compatible modifications for encrypted rows and tables and the associated metadata and (2) the LMDB memory-based store which may have advantages when uses with the most modern high-performance RAM-based storage hardware. We look forward to integrating other stores, and to prove the point we also supply the ancient Oracle BDB backend as an example third store.

✅ The only mainstream database optionally without a Write-ahead Log, when using the LMDB storage backend.

✅ preliminary Benchmarking results

✅ The Not Forking tool, which avoids forks in both simple and complicated source code

Exciting things in progress:

⛅ Lumions are described in a draft RFC for universal encrypted blobs with authentication. For the first time, a piece of data can have all the security rights of a full database, even it is called "mydata.txt" and attached to an email. LumoSQL uses Lumions as rows and tables in SQLite but is just one use case. As soon as the cryptographic design has settled we will update this RFC and consult even more widely

⛅ Documented API for arbitrary key-value stores

⛅ Documented API for accessing the key-value stores via the SQLite library, instantly making the SQLite key-value store the most widely-distributed key-value store. Nothing calls the SQLite key-value store today except SQLite