While testing HTTP requests with bodies in the RealWorld server, I ran into a problem in an unexpected place. The Worker could read the body from its TCP connection, but passing those Bytes to the Actor that ran the application exposed a bug in the Protos runtime. Fixing it became part of building the server. This was exactly the kind of problem I wanted to find by taking the language beyond small examples.

Protos is an experimental prototype-based language I am developing on GraalVM and Truffle. I chose RealWorld, also known as Conduit, to put it to the test. RealWorld is an article-publishing application with a common API that allows different frontends and backends to work together. My implementation covers all 19 API routes, including registration, login, profiles, articles, the feed, comments, follows and favourites. RealWorld project · Protos backend source

What I had to implement in Protos

I started by bringing up a TCP service, serving the first public routes and checking the responses from an HTTP client. From there, each part of the application exercised a different piece of the language and its library.

• Services and networking. I added a service mode and an explicit network grant to the launcher. The server needed to keep listening while two Workers accepted connections and handled requests, then close its resources on shutdown.
• Transfer between Actors. Moving from public queries to requests with bodies exposed a bug I had to fix in the transfer of Bytes between Actors. The application also needed to define which request data could cross that boundary and who remained responsible for the connection.
• Entropy and the clock. I added explicit capabilities for obtaining secure randomness and reading the wall clock. The application uses the former to generate identifiers and the latter to calculate and check JWT timestamps.
• Cryptography in the standard library. I added Argon2id for passwords and the pieces needed for HS256 JWTs, including HMAC-SHA256 and Base64url. The backend uses those APIs and defines its own policies for identity, authorisation and token revocation. Protos repository

I also implemented routing, the domain model and JSON validation before state changes. In a partial update, for example, an omitted field and a field explicitly set to null mean different things. The server also rejects duplicate JSON keys and validates listing parameters such as limit.

An HTTP request through Protos, step by step

In these snippets, name: value declares a local variable or an object slot, depending on context. name = value updates a variable. The form (arguments) => { ... } creates a function whose result is its final expression. The excerpts come from the repository, with some comments removed for readability.

1. Start a Worker with the authority it needs

worker: Actor.spawn(
    "./HttpWorker.protos",
    "start",
    listener.acceptCapability(),
    application,
    index
)

HttpWorkers.protos

Actor.spawn loads HttpWorker.protos in a new Actor and calls its start function with the next three arguments. listener.acceptCapability() lets the Worker accept connections from the existing listener. application is a communication reference of type GroupRef, which lets it request replies from the RootActor. index identifies the Worker. The full listener, the mutable store and the signing key stay with the RootActor.

2. Read the body and cross the Actor boundary

handleRequest: (request) => {
    content: Encoding.Latin1.encode("")
    chunk: request.body.read(8192).value()
    while(chunk !== null) {
        for(octet in chunk) {
            content.add(octet)
        }
        chunk = request.body.read(8192).value()
    }

    application.request("handle", {
        method: request.method
        target: request.target
        version: request.version
        headers: request.headers
        body: content
    }).value()
}

HttpWorker.protos

request.body is a stream tied to the HTTP session. The Worker reads chunks of up to 8 KiB until it receives null, which marks the end of the body. The HTTP library limits the decoded body to 1 MiB. The object passed to request("handle", ...) contains the data the RootActor needs. The request target retains its original bytes for the router to interpret. The headers retain their order and any duplicates for security checks.

Reading the stream and requesting a reply from the other Actor both return a Future. .value() obtains the result and suspends the current execution if it is still pending. The Worker can then process control messages such as drain and abort. When the RootActor replies, the Worker writes the response to its connection.

3. Validate and create an article

{ method: "POST"
    path: "/api/articles"
    handle: route(true, (request, match, principal) => {
    input: RequestJson.createArticle(request.body)
    response: null
    if(!input.valid) { response = Responses.validation(input) } else {
        fields: input.fields
        time: auth.timestamp()
        slug: Slug.allocate(store, fields["title"].value)
        result: store.createArticle(principal.id, slug, fields["title"].value,
            fields["description"].value, fields["body"].value,
            RequestJson.tagsOr(fields, []), time)
        if(result.ok) {
            response = Responses.created(Articles.singleFor(result.value, principal.id, store))
        } else { response = Responses.domainError(result) }
    }
    response
}) },

Api.protos

route(true, ...) requires an authenticated user before running the route handler. Inside the handler, RequestJson.createArticle validates the body. If validation passes, the server obtains the time, allocates a slug and calls the store with principal.id as the author. That ID comes from authentication, so fields in the JSON cannot choose who authors the article. Responses.created prepares the HTTP 201 response.

Separating validation from writes also gives partial updates a precise meaning. For an optional field such as bio, the following code preserves the existing value when the field is omitted, clears it when the field is explicitly null and replaces it when the field contains text.

textOr: (fields, name, previous) => {
    value: previous
    if(fields.containsKey(name)) {
        node: fields[name]
        value = null
        if(node.kind == "string") { value = node.value }
    }
    value
}

RequestJson.protos

Shutdown is also part of the server protocol. With drain, the Workers finish work they have already admitted. With abort, they cancel outstanding work. The quit console command requests a graceful shutdown.

The password change that made Hurl fail

The backend uses Argon2id for passwords and issues HS256-signed JWTs. The token subject is a stable account identifier. When a user changes their password, the application increments the credential generation and returns a new token. On subsequent requests, it compares the generation in the token with the generation stored for the account. This invalidates an older token even if its signature is valid and it has yet to expire.

What can change while the hash is being computed

A password change exposes a detail of the Protos concurrency model. Argon2id hashing returns a Future, so the current RootActor turn can suspend while waiting. During that interval, another turn can update the credentials for the same account. That is why updateAccount checks the generation on entry and reads the record again after waiting.

if(checked.valid && (record !== null) && (record.generation == expectedGeneration)) {
    verifier: null
    if(fields.containsKey("password")) {
        verifier = Password.hash(fields["password"].value, entropy).value()
    }
    record = byId(id)
    if(record.generation == expectedGeneration) {
        current: identityStore.identityById(id)
        username: Input.textOr(fields, "username", current.username)
        email: Input.textOr(fields, "email", current.email)
        result = conflict(id, username, email)
        if(result === null) {
            result = identityStore.replaceIdentity(id, username, email,
                Input.textOr(fields, "bio", current.bio),
                Input.textOr(fields, "image", current.image))
            if(result.ok && (verifier !== null)) { replace(record, verifier) }
        }
    }
}

AuthStore.protos

The first check compares the authenticated user's credential generation with the current one. The second prevents a request that started with old credentials from overwriting a change made while it waited. If the update succeeds, replacing the verifier advances the generation. On the next request, JWT.verify checks the token against the signature and validity policy, and the application compares the token's generation with the account's generation. Both checks must pass before the application treats the user as authenticated. Auth.protos

One scenario in the RealWorld Hurl suite kept sending the old token after changing the password. It failed. I included a patch that captures the replacement token from the response and uses it in subsequent requests. The make test-hurl target applies the patch to a temporary copy of the suite. Token refresh patch

The compatibility result for the official suite is 12 of 13. The second run checks the full sequence with this backend's revocation policy. The exact revision of the external suite used for these runs was not recorded with the results.

Try it locally

With a compatible build of Protos installed, start the server from the repository.

git clone https://github.com/guillermomolina/protos-realworld-example-app.git
cd protos-realworld-example-app
make serve

make serve starts the service at 127.0.0.1:3000 and explicitly grants the network, entropy and clock capabilities through --allow-network, --allow-entropy and --allow-clock. From another terminal, try the public endpoints.

curl -i http://127.0.0.1:3000/api/tags
curl -i http://127.0.0.1:3000/api/articles

Persistence is the next step

Accounts, credentials, articles, comments, follows and favourites live in MemoryStore. Restarting the process clears that state. Retaining the signing key alone cannot restore a session, because the account and its credential generation disappear with the store.

The next problem I want to tackle is durable storage. SQLite is the first candidate for developing a reusable data access layer in Protos. RealWorld already provides concrete cases to test it against, from unique usernames to the relationships between articles, authors and favourites.