LumoSQL

LumoSQL protects data on mobile phones using a new data storage technology called Lumions. The phone owner has ultimate right of deciding who can read or change Lumion data... even after it has been copied off the phone to (for example) a bank or insurance company for processing with very different database software. The LumoSQL design means that if a criminal or government officer copies LumoSQL data off a phone, it cannot be read without the consent of either the phone owner or someone(s) to whom the phone owner has granted access. In addition, write access is a different permission to read access, and administrative access is different again. It made sense for us to make our first product a modification to the database found on every phone because that can help billions of people without significant change in their daily software usage.

Technically: LumoSQL modifies the SQLite Open Source database software to add essential performance, security and privacy features. LumoSQL is a technical product for software developers to use. LumoSQL is currently a working prototype, and uses Predicate Based Encryption to manipulate Lumions.

Socially and Commercially: LumoSQL matters because virtually all data for all apps on mobile devices is stored using SQLite, making it by far the most-used database anywhere. Yet this database is vulnerable to snoopers and does not even come close to the gold-standard privacy requirements of the EU and other countries. LumoSQL is a new category of software, highly compatible with the original SQLite and protecting our most intimate data.

Practically: LumoSQL exists due to the volunteer effort contributed by many skilled people. The Vrije Universiteit Brussel continues to fund valuable contributions in cryptography and mathematical analysis. The NLnet Foundation funded Phase I of LumoSQL, and is continuing to fund Phase II.

Surprising Background

The following facts often surprise people:

• SQLite is very likely the most-deployed software, by a factor of at least four zeros. It is likely that SQLite is the only trillion-scale software in existence, although that statement needs some validation work to be sure.
• SQLite is a full-featured database, supporting the standard SQL language despite being tiny compared to all the other mainstream SQL databases.
• a typical mobile phone stores all non-streaming data in several hundred SQLite databases.
• SQLite is also used in web browsers, operating systems, vehicles of all kinds and so on.
• SQLite is open source, exceptionally well-maintained (with many contributions) by around 8 people loosely connected in a small, unambitious company. (There are many more than 8 people who contribute occasionally to SQLite.)

The corollaries are significant:

• This is uncharted territory for Computer Science: is SQLite's ultra-conservative compatibility commitment to its (at least) hundreds of billions of installations the right choice? Is SQLite's fast-moving support of formal database standards the best way forward?
• Why are there so many forks of SQLite, none with more than trivial (likely at very most a few handfuls of millions) of deployments? The nature of the SQLite project seems to guarantee its success and also constrain its future.
• Why are the obvious strategic problems with SQLite not discussed more widely? The most-used software in the world is incompatible with security, privacy and other requirements of the 21st century, so why isn't this a hot topic? Perhaps because it works so well that people don't wish to rock the boat. But with new features required, and it being unacceptable to store data without any form of encryption, there needs to be some compatible new option. LumoSQL wants to be that option.

LumoSQL Phase I Completed

This is a technical section non-technical readers can skip. For even more technical detail see the code development page.

LumoSQL is a modification (not a fork) of the SQLite embedded data storage library. LumoSQL offers multiple key-value backend storage systems selectable by the user. It offers features not found in any other mainstream database:

• ability to checksum every row on write and verify on read
• ability to trigger arbitrary functions on per-row read and write
• a general test suite for benchmarking precisely how LumoSQL (or SQLite) is performing and the full context of that benchmark run. For some reason database benchmark is very poorly done, including by the TCP-C consortium founded for solely that purpose.
• a general build system able to mix and match multiple versions of the database with multiple versions of multiple backends. Never before has it been possible to compare the different strategies of various Key-Value stores with the same database frontend.

If you are an SQLite user familiar with C development wanting an easier way to benchmark and measure SQLite, or if you are wanting features only available in other key-value storage engines, then you will find that LumoSQL offers new features even in its prototype stage.

LumoSQL Phase II Has Started

In Phase II LumoSQL is implementing at-rest encryption and privacy using the features developed in Phase I, and readying LumoSQL for more general testing.

Notable outcomes from LumoSQL already include:

✅ The only mainstream database with swappable Key-Value stores, where all stores are peers rather than one store having special knowledge that gives it technical advantages. The two stores we have concentrated on so far are (1) the existing SQLite store with optional, binary-compatible modifications for encrypted rows and tables and the associated metadata and (2) the LMDB memory-based store which may have advantages when uses with the most modern high-performance RAM-based storage hardware. We look forward to integrating other stores, and to prove the point we also supply the ancient Oracle BDB backend as an example third store.

✅ The only mainstream database optionally without a Write-ahead Log, when using the LMDB storage backend.

✅ preliminary Benchmarking results

✅ The Not Forking tool, which avoids forks in both simple and complicated source code

Exciting things in progress:

⛅ Lumions are described in a draft RFC for universal encrypted blobs with authentication. For the first time, a piece of data can have all the security rights of a full database, even it is called "mydata.txt" and attached to an email. LumoSQL uses Lumions as rows and tables in SQLite but is just one use case. As soon as the cryptographic design has settled we will update this RFC and consult even more widely

⛅ Documented API for arbitrary key-value stores

⛅ Documented API for accessing the key-value stores via the SQLite library, instantly making the SQLite key-value store the most widely-distributed key-value store. Nothing calls the SQLite key-value store today except SQLite