Chiseling in the Open

Around 6 months ago, we embarked on a journey to come up with a simpler way for application developers to handle their data layers. By viewing everything in the backend as code — from the database to the business logic — we believed we could capture enough intent to automate the data layer enough so that developers could feel like they are just interacting with a big pool of global shared memory instead of databases, connectors, ORMs, etc. And better yet, tie all of that to the git workflow application developers already know.

We now believe our hosted platform is reaching enough maturity to be moved to public beta. That means it's time to do what I've been very excited about: the core of ChiselStrike is going Open Source! Our repo is available on Github under Apache 2.0, allowing you develop your ChiselStrike projects locally and influence the future of our API. Then, when you're ready to take it to the public, deploy your projects to our hosted service.

Now is a great occasion to spend some cycles explaining how ChiselStrike works, and where we're going from here.

#The ChiselStrike story

Modern full-stack projects suffer from an explosion of complexity, as we described in an earlier article. Below your frontend framework, where the application presentation layer lives, there is “the backend”. From a high level, many recent developments make the backend feel smoothly integrated with the frontend. Through things like Node.js and Deno, one can code the backend with the same languages and stack used in the frontend.

But looking closely, there are way too many pieces to this puzzle: even if business logic is in JavaScript/TypeScript, you still need to deal with ORMs, APIs, databases, etc. And that's for a simple project. As complexity grows, you then need to plug in authentication, authorization, security policies, and the list goes on.

Our vision for ChiselStrike is that there should really only be two pieces to this puzzle: the frontend, and the backend. And everything that goes below the line should be generated from declarative code in your language of choice: no ops, no commands, only the good stuff!

#What's wrong with databases?

Databases are at the core of any storage infrastructure. But usually they offer a lot more complexity than application developers would like.

Developers want to take advantage of particular properties of databases. Some of them are related to ordering and consistency guarantees under load (think strong consistency vs eventual consistency), and some are related to how you shape your data and access (like document versus relational).

But because databases want to cater to a wide variety of developers, with a wide variety of needs, they expose their capabilities through database specific languages, such as SQL.

SQL is great, a true survivor, as we wrote about before. After decades of NoSQL, it became clear that SQL really is almost impossible to displace, and a really complete way to express the kind of general functionality databases typically offer.

But being popular doesn't change the fact that, for application developers, it is one more thing to learn. Even if we don't take into account all the operations usually required to keep them up and running, which serverless databases minimize, there is still the issue of handling schemas, migrations, tables, queries, etc.

SQL in particular is not actually that hard to learn. But it is very hard to master. Add the fact developers have plenty of other stuff to master and database-speak is clearly a departure from the mental model of other parts of the application, and here's the first big hurdle.

#What's wrong with ORMs?

ORMs provide a very useful abstraction around the database, and in a sense fix many of the issues that we discussed above.

But they don't go all the way in fixing the mismatch between app-land and data-land. Let's look, for example, at TypeORM's (a popular ORM for Typescript) as a reference. Here's how, according to their documentation, one would expose a particular table to TypeORM:

import { Entity, Column, PrimaryGeneratedColumn } from 'typeorm';

@Entity()
export class Photo {
  @PrimaryGeneratedColumn()
  id: number;

  @Column({
    length: 100,
  })
  name: string;

  @Column('text')
  description: string;

  @Column()
  filename: string;

  @Column('double')
  views: number;

  @Column()
  isPublished: boolean;
}

Developers suddenly have to be aware of database types — that can be different or incompatible across different databases, of concepts like primary keys, the ways in which databases increment fields, etc.

Prisma, another very popular ORM has similar concepts during the model definition phase. And to construct a query, one would write:

const usersWithPosts = await prisma.user.findMany({
  orderBy: [
    {
      role: 'desc',
    },
    {
      name: 'desc',
    },
  ],
  include: {
    posts: {
      orderBy: {
        title: 'desc',
      },
      select: {
        title: true,
      },
    },
  },
});

Yes, you are now writing code in TypeScript, and that is great. But the code above can hardly be called TypeScript. It is more like SQL and JSON had a baby. For more complex queries, you end up essentially crafting a SQL query in JSON, which takes you out of the mental model of your application, back into the mental model of databases.

Yes — you can work with native types, integrate with your backend code, but you still have to take your mind into SQL and back. And to be clear, that's not to dis on either Prisma or TypeORM. They are fantastic ORMs. But the question is: can we have a better abstraction?

ChiselStrike is fundamentally different. Let's look for example at how we would represent a Person:

import { ChiselEntity } from '@chiselstrike/api';

export class Person extends ChiselEntity {
  name: string;
  age: number;
}

There'sno need to specify IDs, indexes, column mappings, internal database types, or worry about database implementation details.

Migrations are also extremely simplified. We take advantage of the fact that in TypeScript, properties can be optional and have defaults to simplify the most common cases around schema evolution. Forexample, you can just add a field that has a default value (and change the default value at will too). There are no ALTER statements and no database meddling; just change your type as you would if you were doing this all in memory.

import { ChiselEntity } from '@chiselstrike/api';

export class Person extends ChiselEntity {
  name: string;
  age: number;
  isHuman: boolean = false;
}

And how about queries? You can just issue lambdas and express your business logic naturally, like this:

await Person.findMany((p) => p.age > 40);

We will go into more details in a later article about how this is done, but the key realization of ChiselStrike is that this mapping should start from TypeScript down, not from the database up. Your code is king, and determines what needs to be done.

I'll share a personal anecdote: in the early days when ChiselStrike was nothing more than an idea in our heads, one of the concerns that popped in some conversations was “how will you expose all of the complexity of SQL into TypeScript?” Asking that, is akin to asking how will you expose all the complexity of a very rich processor instruction set into a programming language: the key is to realize that you don't need to. What you want is to map your programming language to the processor's instruction set, and if large parts of the instruction set are unused, does it really matter?

#Tying it all together

ChiselStrike is built with a mixture of TypeScript and Rust. We use Hyper to provide a fast and reliable http layer. Because requests are independent of each other, we use a thread-per-core architecture to distribute them across all cores of the machine. Each thread instantiates a JavaScript interpreter: Deno.

But tied to that is the ChiselStrike infrastructure & API, in TypeScript. That defines how requests are received, prepared, and routed. The API allows you to access persistent objects.

Lots of the API, especially the higher level sugary ones, are really just plain TypeScript. But at key points, it calls into ChiselStrike's Rust component. An example is the code below to access secrets:

export function getSecret(key: string): JSONValue | undefined {
  const secret = Deno.core.opSync('op_chisel_get_secret', key);
  if (secret === undefined || secret === null) {
    return undefined;
  }
  return secret;
}

The Rust component is the one responsible for brokering all access to the database, mapping the entities to database objects, etc.

For most deployments, the database is itself embedded in the platform. Meaning that for many deployments, in particular local ones for development, you have everything you need right there at your fingertips, without any extra or external components.

#The hosted Platform

If you're interested in what we're doing we'd love to have you in our community! You can join our Discord community, and when you're ready to get serious, deploy on our platform (which is in beta, but hold tight!).

If you already deploy your frontend to any of the popular platforms like Vercel, Netlify, or Gatsby Cloud, you can just add a subdirectory with ChiselStrike code and point the same repository to our platform (or create a separate repository, up to you!). Any pushes to your main branch are automatically deployed and reflected in your data layer.

Happy Chiseling!