Building a better-sqlite3 compatible JavaScript package with Rust

The libSQL project is a fork of SQLite that adds capabilities that you want for production use of SQLite, such as server mode, replication and backups to S3. Early on we decided to implement all of the extensions in Rust instead of C for a variety of reasons, including safety but also development speed.

However, the decision to use Rust also meant that we could not just rely on the existing ecosystem, but had to provide language bindings ourselves. For example, in the JavaScript ecosystem, one popular package that provides SQLite3 bindings is the better-sqlite3 package.

We aimed to develop a package designed to seamlessly integrate as a substitute for better-sqlite3, allowing users to effortlessly transition without altering their existing codebases, while also providing the option to leverage libSQL's advanced features and extensions.

Of course, we were not the first ones to want to bridge Rust code into the JavaScript ecosystem. There are two Rust crates, Neon Bindings and NAPI-RS, that allow you to write code in Rust and export them into JavaScript. We ended up going with Neon Bindings, but later found out that Prisma, for example, uses NAPI-RS, so that's likely a good option as well.

#So what does the better-sqlite3 API look like?

A simple example of opening a local database file, querying a table, and printing out the results looks like this:

import Database from 'better-sqlite3';
const opts = {};
const db = new Database(‘local.db', opts);
const stmt = db.prepare('SELECT * FROM users WHERE id = ?');
const userId = 1;
const row = stmt.get(userId);
console.log(row.firstName, row.lastName, row.email);

The import statement pulls the Database type into the application, which is the main entry point to better-sqlite3. To open a connection to a local database, you pass a path and options to the Database constructor. In the example above, we keep the options empty, but we'll get back to that later.

The db.prepare() function prepares a SQL statement for execution. In our example, we query a users table if the id column matches some value marked with ?, which is a place-holder for a value we bind to the statement when we execute it. That value is the userId variable passed to the get() function, which binds userId to the place-holder value and returns a single row from the table.

Finally, we print out the resulting row with console.log() by accessing properties of the row object, which are automatically added to the JavaScript object based on the SQLite schema.

#How did we implement the API using Neon Bindings?

Following the Neon example project, we had to build a source tree that contains both a package.json and Cargo.toml files as it's a dual-language package. We then added the neon crate as a dependency, but also the libsql crate, which implements the libSQL extensions, as well.

We then added an index.js file with a bit of Neon magic that imports Rust functions and a Database class:

const { currentTarget } = require('@neon-rs/load');
const target = currentTarget();
const { databaseOpen } = require(`@libsql/${target}`);
class Database {
  constructor(path, opts) {
    this.db = databaseOpen(path);
  }
}

The @neon-rs/load magic imports functions by identifying the operating system and machine architecture you are using, such as Linux on arm64, and subsequently loads a package tailored to your specific target, namely @libsql/<target>.

For the Database constructor we call into a native function named databaseOpen, which is implemented in the target specific package.

We now require a touch of Neon magic to export functions that are used to build native code.For openDatabase, we have the following (in slightly simplified form):

struct Database {
    db: Arc<Mutex<libsql::Database>>,
    conn: Arc<Mutex<libsql::Connection>>,
}

impl Database {
    fn js_open(mut cx: FunctionContext) -> JsResult<JsBox<Database>> {
        let rt = runtime(&mut cx)?;
        let db_path = cx.argument::<JsString>(0)?.value(&mut cx);
        let db = {
            let mut builder = libsql::Builder::new_local(&db_path);
            rt.block_on(builder.build())
        }.or_else(|err| throw_libsql_error(&mut cx, err))?;
        let conn = db
            .connect()
            .or_else(|err| throw_libsql_error(&mut cx, err))?;
        let db = Database::new(db, conn);
        Ok(cx.boxed(db))
    }

    fn new(db: libsql::Database, conn: libsql::Connection) -> Self {
        Database { }
    }
}

That is, there is a related function js_open, which extracts the parameters we passed from JavaScript and then invokes the libsql crate API. One additional complication is that the libsql crate has async methods, but since the better-sqlite3 API is synchronous, we need to block_on() for some of the calls. We also have a async variant of the better-sqlite3 API to avoid the blocking, but we also wanted to keep the synchronous variant for compatibility.

Finally, to export js_open as a JavaScript databaseOpen function, we have some more Neon magic:

#[neon::main]
fn main(mut cx: ModuleContext) -> NeonResult<()> {
    cx.export_function("databaseOpen", Database::js_open)?;
}

We've successfully implemented a JavaScript function with Rust. This process involves developing a JavaScript API that serves to wrap a Rust object inside a JavaScript object.

Given that this object is opaque in JavaScript, it's forwarded as a parameter to any functions that necessitate its use. For instance, when examining prepare() in JavaScript:

prepare(sql) {
  try {
    const stmt = databasePrepareSync.call(this.db, sql);
    return new Statement(stmt);
  } catch (err) {
    throw new SqliteError(err.message, err.code, err.rawCode);
  }
}

The prepare() method calls databasePrepareSync(), which is a Neon-exported native function, passing the database object stored in this.db to it. On the Rust side, we extract the object from parameters and just use the Rust type:

pub fn js_prepare_sync<'a>(mut cx: FunctionContext) -> JsResult<JsBox<Statement>> {
    let db: Handle<'_, JsBox<Database>> = cx.this()?;
    let sql = cx.argument::<JsString>(0)?.value(&mut cx);
    let conn = db.get_conn();
    let rt = runtime(&mut cx)?;
    let result = rt.block_on(async { conn.lock().await.prepare(&sql).await });
    let stmt = result.or_else(|err| throw_libsql_error(&mut cx, err))?;
    let stmt = Arc::new(Mutex::new(stmt));
    let stmt = Statement {
        conn: conn.clone(),
        stmt,
    };
    Ok(cx.boxed(stmt))
}

Notice how js_prepare_sync() returns a Statement from Rust, a symmetrical approach in the API's implementation: corresponding to each JavaScript class within the better-sqlite3 API, there's a Rust struct that's passed into functions as needed.

To recap, in this post we learned about the components required to build a better-sqlite3 compatible JavaScript package with Rust. The components we covered are:

• libsql npm package: This is a Node.js package which provides the JavaScript API layer. This layer is what JavaScript developers interact with when they want to perform database operations.
• @libsql/(target): This is a target-specific package that acts as a bridge between the JavaScript API and the Rust crate.
• libsql Rust crate: This contains the database specific code to execute operations.

That's more or less all there is to implementing a JavaScript API in Rust. Of course, there are some complications around transforming Rust values into JavaScript values, especially for more complex cases like when better-sqlite3 rows have properties reflecting table column names. But pretty much just boilerplate code, which you either perform on Rust side or on JavaScript side, depending on what makes sense.