Beacon: Design Docs

As promised in my last post, I've put together designs for various parts of of the Beacon.

First, a disclaimer: As I started to actually read the details of the NIST Randomness Beacon Spec, I decided it was just way more complicated than was necessary for what this project actually aims to achieve. Instead, I've decided to take inspiration from that project and 1. improve on the API design and 2. mostly ignore the cryptophy part. I'll build an app that emits a UUID every 60 seconds. As a follow up, it may be interesting to replace the UUID with a cryptographic random number and somehow involve an AWS CMK, just to do some exploration, but it's not relevant to the main goals of the project.

As a reminder, the main goal of the project is to build something relatively simple to understand, but just complex enough to demonstrate a bunch of the things I learned building Check Run Reporter.

REST API

Even though Apiary is dying a slow death at Oracle, I still find it and API Blueprint to be the best way to get started with API design. Writing an API in Swagger/OpenAPI is just tedious and unpleasant; writing one with API Blueprint is almost as easy as writing Markdown. More importantly, apiary.io will then serve as a free host of the resultant API documention.

I'm including the current API design here, but it's also available on Apiary. I assume it'll evolve a bit before the final product ships.

FORMAT: 1A
HOST: https://beacon.code-like-a-carpent.com/beacon/2.0

# Beacon

This is toy project inspired by the [NIST Randomness Beacon](https://beacon.nist.gov/home) for demonstrating underlying technologies. It is extraordinarily non-conformant with NIST's spec.

The NIST spec is not particularly RESTful and, further, the actual cryptographic features would require a lot of effort that's nort particularly interesting for demonstrating the underlying technologies.

Instead, we'll build a service that emits a random UUID once per minute. It will be possible to receive those UUIDs in real time via RSS or a WebSocket and authenticated users may retrieve historical data.

We want to be able to find any pulse by its
- index
- timestamp
- value
- outputValue

And we should be able to do range queries on same. Since sorting by `value` or `outputValue` doesn't really make sense, range queries using either of those attributes will use the specified value as a starting point, but order by pulseIndex. 

# Group REST API

## Pulse [/pulse]

### Retrieve the latest pulse [GET /pulse]

+ Attributes (Pulse)

+ Response 200 (application/json)

### Retrieve a histical pulse [GET /pulse/{id}]

+ Attributes (Pulse)

+ Response 200 (application/json)

### Retrieve historical pulses [GET /pulses{?before}{?after}{?first}{?last}{?attribute}]

+ Parameters

    + before (string, optional) - When specified, returns Pulses before this value of `attribute` (not including this Pulse)
    + after (string, optional) - When specified, returns Pulses after this value of `attribute` (not including this Pulse)
    + first (number, optional) - Number of pulses to return starting from `after` (non-inclusive)
        + Default: 100
    + last (number, optional) - Number of pulses to return starting from `before` (non-inclusive)
        + Default: 100
    + attribute (enum[string], optional) - indicates which attribute to query against.
        + Default: pulseIndex
        + Members
            + pulseIndex
            + timeStamp
            + value
            + outputValue

+ Attributes (array[Pulse])

+ Response 200 (application/json)

### Retrieve the first pulse at or after a specific time [GET /pulse{?after}]

+ Attributes (Pulse)

+ Response 200 (application/json)

### Retrieve the last pulse before a specific time [GET /pulse{?before}]

+ Attributes (Pulse)

+ Response 200 (application/json)

### RSS feed of pulses [GET /pulse.rss]

+ Response 200 (application/xml)

### WebSocket stream of pulses [GET /pulse.ws]

+ Response 101

# Group Web App

## Load the landing page [GET /]

+ Response 200 (text/html)

# Group Internal Operations

### Ping [GET /ping]

In a serverless world, ping and healthcheck endpoints aren't particularly useful in terms of confirming a service hasn't crashed, however, a simple ping endpoint can be super helpful in development and test environments for confirming that builds are configured correctly.

+ Response 200 (application/json)

    + Body 

            {
                "ok": true
            }

# Data Structures

## Pulse (object)

+ uri (string) a uniform resource identifier (URI) that uniquely identifies the pulse
+ version (string) the version of the beacon format being used
+ period (number) the number (denoted by π) of milliseconds between the timestamps of this pulse and the expected subsequent pulse
+ pulseIndex (number) the pulse index (integer identifier, starting at 1), conveying the order of generation of this pulse within its chain
+ external (PulseExternal)
+ timeStamp (string) the time (UTC) of pulse release by the Beacon Engine (the actual release time MAY be slightly larger, but SHALL NOT be smaller)
+ previous (string) the outputValue of the previous pulse;
+ hour (string) the outputValue of the first pulse in the (UTC) hour of the previous pulse
+ day (string) the outputValue of the first pulse in the (UTC) day of the previous pulse
+ month (string) the outputValue of the first pulse in the (UTC) month of the previous pulse
+ year (string) the outputValue of the first pulse in the (UTC) year of the previous pulse
+ outputValue (string) the hash() of all the above fields

## PulseExternal (object)

+ sourceId (string) the hash() of a text description of the source of the external value, or an all-zeros byte string (with exactly hLB = d|hash()| /8e bytes) if there is no external value
+ statusCode (number) the status of the external value
+ value (string) the hash() of an external value, drawn from a verifiable external source from time to time, or an all-zeros string if there is no external value

Data Modeling

For a while now, I've been thinking about using GraphQL to model DynamoDB tables. This is a first pass at what that might look like, using the Beacon as an example project. I don't know if I'll actually build the code gen to make this work for the beacon, but it was still a useful exercises for thinking about the data model.

scalar DateTime
scalar JSONObject

"""
Indicates that this type is backed by DynamoDB.
"""
directive @model on OBJECT 

"""
Tells the model which table its in.
"""
directive @table(
  """
  Runtime environment variable that will store the table's name.
  """
  env: String!
) on OBJECT 

directive @pk(
  partitionKey: [String!]!
  pkPrefix: String
  sortKey: [String]
  skPrefix: String
) on OBJECT 

directive @index(
  name: [String!]!
  partitionKey: String!
  pkPrefix: String
  sortKey: [String] 
  skPrefix: String
) on OBJECT 

"""
Indicates that this field should be used with DynamoDB's ttl feature. 
"""
directive @ttl on FIELD

@model
@table(env: "TABLE_PULSES")
@pk(
  partitionKey: [timeStamp]
  pkPrefix: "PULSE"
)
@index(
  name: "gsi1"
  partitionKey: [pulseIndexMillenium]
  pkPrefix: "PULSE_MILLENIUM"
  sortKey: [pulseIndex]
  skPrefix: "INDEX"
)
@index(
  name: "gsi2"
  partitionKey: [externalValue]
  pkPrefix: "PULSE_VALUE"
)
@index(
  name: "gsi3"
  partitionKey: [outputValue]
  pkPrefix: "PULSE_OUTPUT_VALUE"
)
type Pulse {
  uri: String
  version: String
  period: Number
  pulseIndex: Number
  """
  This is a hidden field that just serves to produce a vaguely useful partition
  key for gsi2. 
  """
  pulseIndexMillenium: Number @hidden @computed(inline: "Math.floor(pulseIndex/1000)")
  externalSourceId: String
  externalStatusCode: String
  externalValue: String
  timeStamp: DateTime
  previous: String
  hour: String
  day: String
  month: String
  year: String
  outputValue: String
}

@model
@table(env: "TABLE_SESSIONS")
@pk(
  partitionKey: [id]
  pkPrefix: "USER_SESSION"
)
type UserSession {
  id: String
  expires: DateTime @ttl
  session: JSONObject
}

So, obivously there are a bunch of directives here that aren't part of GraphQL. Long-run, I want to put together some combination of schema transforms and graphql codegen that uses those directives to generate not only a type defintion, but also a CloudFormation template TypeScript client that can read and write those models.

I can already see some potential issues with the directive design (for example, @ttl could be used on two differently named fields of two different models that live in the same table, which would be invalid), so this is probably not a final design for the directives, but the data models themselves appear to agree with our REST API design.

GraphQL API

In addition to using GraphQL to define the internal data models (and, potentially, for using queries internally), I also want to expose the domain models using GraphQL. This'll be a wholly-different GraphQL schema and runtime context that the data model described above.

scalar DateTime

interface Node {
  id: ID!
}

type PageInfo {
  hasNextPage: Boolean!
  hasPreviousPage: Boolean!
  startCursor: String!
  endCursor: String!
}

interface Connection {
  pageInfo: PageInfo!
  edges: [Edge]
}

interface Edge {
  cursor: String!
  node: Node
}

type Query {
  node(id: ID!): Node
  nodes(ids: [ID!]!): [Node]!
}

type Pulse implements Node {
  # Based on outputValue
  id: ID!
  uri: String
  version: String
  period: Number
  pulseIndex: Number
  external: PulseExternal
  timeStamp: DateTime
  previous: String
  hour: String
  day: String
  month: String
  year: String
  outputValue: String
}

type PulseConnection implements Connection {
  pageInfo: PageInfo!
  edges: [PulseEdge]
}

type PulseEdge implements Edge {
  cursor: String!
  node: Pulse
}

type PulseExternal {
  sourceId: String
  statusCode: String
  value: String
}

extend type Query {
  pulses(first: Int, after: String, last: Int, before: String): PulseConnection!
  pulse(outputValue: String, ): Pulse
}

type Subscription {
  pulses: [Pulses!]
}

Permissions

I've been eyeing Authzed for a while now. It's a SaaS based on the Google Zanzibar paper fo performantly defining and querying fine-grained access control.

I think the schema for this project is pretty simple (though, addmittedly, I've never used Authzed before. I imagine this'll evolve as the project gets built.

definition anon {}

definition user {}

definition pulse {
    relation viewer: anon:* | user:*

    permission view = viewer
}

These are the test relationships I added in the playground, but since we're not actually dealing with ownership beyond checking if the user is logged in, this should be the complete set of relationships. I don't need to worry about sending a new relationship every time there's a new touple. Which is good, since, while Authzed's pricing is incredibly reasonable, at one new record per minute, it would actually become really pricy really quickly.

// Some example relationships
pulse:last#viewer@anon:*
pulse:last#viewer@user:*
pulse:*#viewer@user:*

And finally some test assertions:

assertTrue:
  - pulse:last#view@user:1
  - pulse:last#view@user:2
  - pulse:last#view@anon:any
  - pulse:1#view@user:1
  - pulse:1#view@user:2
  - pulse:2#view@user:1
  - pulse:2#view@user:2
assertFalse:
  - pulse:1#view@anon:any
  - pulse:2#view@anon:any

What's Next?

This covers the majority of the upfront design I intend to do. If this were a real project, I'd put together some with bounce diagrams and/or flow charts talking about how all the pieces will fit together. Dince I'll be blogging along as I do the implementation, I'll instead do a bit of design work at the start of most posts, then talk about how to implement it.