Building a better-sqlite3 compatible JavaScript package with Rust
Building a better-sqlite3 compatible JavaScript package with Rust
The libSQL project is a fork of SQLite that adds capabilities that you want for production use of SQLite, such as server mode, replication and backups to S3. Early on we decided to implement all of the extensions in Rust instead of C for a variety of reasons, including safety but also development speed.
However, the decision to use Rust also meant that we could not just rely on the existing ecosystem, but had to provide language bindings ourselves. For example, in the JavaScript ecosystem, one popular package that provides SQLite3 bindings is the better-sqlite3
package.
We aimed to develop a package designed to seamlessly integrate as a substitute for better-sqlite3
, allowing users to effortlessly transition without altering their existing codebases, while also providing the option to leverage libSQL's advanced features and extensions.
Of course, we were not the first ones to want to bridge Rust code into the JavaScript ecosystem. There are two Rust crates, Neon Bindings and NAPI-RS, that allow you to write code in Rust and export them into JavaScript. We ended up going with Neon Bindings, but later found out that Prisma, for example, uses NAPI-RS, so that's likely a good option as well.
# So what does the better-sqlite3
API look like?
A simple example of opening a local database file, querying a table, and printing out the results looks like this:
import Database from 'better-sqlite3';
const opts = {};
const db = new Database(‘local.db', opts);
const stmt = db.prepare('SELECT * FROM users WHERE id = ?');
const userId = 1;
const row = stmt.get(userId);
console.log(row.firstName, row.lastName, row.email);
The import
statement pulls the Database
type into the application, which is the main entry point to better-sqlite3
. To open a connection to a local database, you pass a path and options to the Database
constructor. In the example above, we keep the options empty, but we'll get back to that later.
The db.prepare()
function prepares a SQL statement for execution. In our example, we query a users
table if the id
column matches some value marked with ?
, which is a place-holder for a value we bind to the statement when we execute it. That value is the userId
variable passed to the get()
function, which binds userId
to the place-holder value and returns a single row from the table.
Finally, we print out the resulting row with console.log()
by accessing properties of the row object, which are automatically added to the JavaScript object based on the SQLite schema.
# How did we implement the API using Neon Bindings?
Following the Neon example project, we had to build a source tree that contains both a package.json
and Cargo.toml
files as it's a dual-language package. We then added the neon
crate as a dependency, but also the libsql
crate, which implements the libSQL extensions, as well.
We then added an index.js
file with a bit of Neon magic that imports Rust functions and a Database
class:
const { currentTarget } = require('@neon-rs/load');
const target = currentTarget();
const { databaseOpen } = require(`@libsql/${target}`);
class Database {
constructor(path, opts) {
this.db = databaseOpen(path);
}
}
The @neon-rs/load
magic imports functions by identifying the operating system and machine architecture you are using, such as Linux on arm64, and subsequently loads a package tailored to your specific target, namely @libsql/<target>
.
For the Database
constructor we call into a native function named databaseOpen
, which is implemented in the target specific package.
We now require a touch of Neon magic to export functions that are used to build native code.For openDatabase
, we have the following (in slightly simplified form):
struct Database {
db: Arc<Mutex<libsql::Database>>,
conn: Arc<Mutex<libsql::Connection>>,
}
impl Database {
fn js_open(mut cx: FunctionContext) -> JsResult<JsBox<Database>> {
let rt = runtime(&mut cx)?;
let db_path = cx.argument::<JsString>(0)?.value(&mut cx);
let db = {
let mut builder = libsql::Builder::new_local(&db_path);
rt.block_on(builder.build())
}.or_else(|err| throw_libsql_error(&mut cx, err))?;
let conn = db
.connect()
.or_else(|err| throw_libsql_error(&mut cx, err))?;
let db = Database::new(db, conn);
Ok(cx.boxed(db))
}
fn new(db: libsql::Database, conn: libsql::Connection) -> Self {
Database { }
}
}
That is, there is a related function js_open
, which extracts the parameters we passed from JavaScript and then invokes the libsql
crate API. One additional complication is that the libsql
crate has async
methods, but since the better-sqlite3
API is synchronous, we need to block_on()
for some of the calls. We also have a async
variant of the better-sqlite3
API to avoid the blocking, but we also wanted to keep the synchronous variant for compatibility.
Finally, to export js_open
as a JavaScript databaseOpen
function, we have some more Neon magic:
#[neon::main]
fn main(mut cx: ModuleContext) -> NeonResult<()> {
cx.export_function("databaseOpen", Database::js_open)?;
}
We've successfully implemented a JavaScript function with Rust. This process involves developing a JavaScript API that serves to wrap a Rust object inside a JavaScript object.
Given that this object is opaque in JavaScript, it's forwarded as a parameter to any functions that necessitate its use. For instance, when examining prepare()
in JavaScript:
prepare(sql) {
try {
const stmt = databasePrepareSync.call(this.db, sql);
return new Statement(stmt);
} catch (err) {
throw new SqliteError(err.message, err.code, err.rawCode);
}
}
The prepare()
method calls databasePrepareSync()
, which is a Neon-exported native function, passing the database object stored in this.db
to it. On the Rust side, we extract the object from parameters and just use the Rust type:
pub fn js_prepare_sync<'a>(mut cx: FunctionContext) -> JsResult<JsBox<Statement>> {
let db: Handle<'_, JsBox<Database>> = cx.this()?;
let sql = cx.argument::<JsString>(0)?.value(&mut cx);
let conn = db.get_conn();
let rt = runtime(&mut cx)?;
let result = rt.block_on(async { conn.lock().await.prepare(&sql).await });
let stmt = result.or_else(|err| throw_libsql_error(&mut cx, err))?;
let stmt = Arc::new(Mutex::new(stmt));
let stmt = Statement {
conn: conn.clone(),
stmt,
};
Ok(cx.boxed(stmt))
}
Notice how js_prepare_sync()
returns a Statement
from Rust, a symmetrical approach in the API's implementation: corresponding to each JavaScript class within the better-sqlite3
API, there's a Rust struct that's passed into functions as needed.
To recap, in this post we learned about the components required to build a better-sqlite3
compatible JavaScript package with Rust. The components we covered are:
- libsql npm package: This is a Node.js package which provides the JavaScript API layer. This layer is what JavaScript developers interact with when they want to perform database operations.
- @libsql/(target): This is a target-specific package that acts as a bridge between the JavaScript API and the Rust crate.
- libsql Rust crate: This contains the database specific code to execute operations.
That's more or less all there is to implementing a JavaScript API in Rust. Of course, there are some complications around transforming Rust values into JavaScript values, especially for more complex cases like when better-sqlite3
rows have properties reflecting table column names. But pretty much just boilerplate code, which you either perform on Rust side or on JavaScript side, depending on what makes sense.