Schema-First API Development with committee × OpenAPI × Rails

When we build a web application, we usually develop the frontend and backend separately. The frontend is almost always Nuxt.js, but the API side varies from project to project depending on who's leading it and how fast things need to move. On projects I lead where speed is important, I mainly develop with Ruby on Rails. (The rest of this post assumes Rails is being used for development.)

Premise

The problem

Whether the frontend and backend are separate projects isn't really the issue. The bigger problem arises when the person implementing the API and the person integrating it on the frontend are different people—getting everyone aligned on the API spec.

Even if the person doing the integration can read the API's code, they probably don't want to dig through it every time they integrate something. Likewise, the person implementing the API probably doesn't want to spend time explaining things every time a new endpoint is added or a new member joins.

A common solution is to document the API spec, but simply deciding to "write documentation" rarely works out well on its own.

Using OpenAPI

At our company, we use OpenAPI to document API specs.

• OpenAPI Specification

There are libraries that generate OpenAPI (or Swagger) definition files from a DSL, but since the supported version tends to depend on the library, I personally write YAML directly.

As soon as a YAML file describing the API spec is pushed to the repository, we deploy it as a Swagger UI via CircleCI Artifacts so project members can view it. (No matter where it's deployed, you still need to make people aware they should look at it 😅)

Even so, problems can still occur—forgetting to update the documentation, or updating it in a way that no longer matches the implementation.

Schema-first development

No matter when you write the API spec documentation, "writing documentation" is a tedious task, and it's hard to stay motivated if it doesn't feel connected to the implementation.

While looking for a better approach, I came across an article introducing a gem called committee, which lets you write API response tests based on an OpenAPI schema. I tried it out, and it seemed pretty promising, so here's a brief introduction to how to use it and how it might be applied.

• github.com/interagent/committee
• github.com/willnet/committee-rails

Example test using committee

I'll skip the detailed setup here, but suppose you define an API spec like this:

openapi: 3.0.2

info:
  title: example
  version: '1'

servers:
  - url: http://localhost:3000/{api_version}
    description: local server
    variables:
      api_version:
        default: 'v1'
        enum:
          - 'v1'

components:
  schemas:
    User:
      type: object
      description: user
      required: [id, email, first_name, last_name]
      properties:
        id:
          type: integer
          description: user id
        email:
          type: string
          format: email
          description: email address
        first_name:
          type: string
          description: first name
        last_name:
          type: string
          description: last name

paths:
  /users/{user_id}:
    get:
      tags: [user]
      parameters:
        - in: path
          name: user_id
          description: user id
          required: true
          schema:
            type: integer
      responses:
        200:
          description: ok
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/User'

In this example, the path is split to allow versioning, which you can support by adding a prefix in the configuration as shown below. The committee-rails gem makes the setup easy.

  config.include Committee::Rails::Test::Methods
  config.add_setting :committee_options
  config.committee_options = {
    schema_path: Rails.root.join('doc', 'openapi.yml').to_s,
    prefix: '/v1'
  }

Assuming GET /users/{user_id} is an API that retrieves user information, the code to verify this with RSpec and committee looks like this:

RSpec.describe 'user api', type: :request do
  let(:user) { create(:user) }

  describe 'GET /users/:id' do
    it 'success' do
      get user_path(user.id)
      expect(response).to have_http_status(:ok)
      assert_response_schema_confirm
    end
  end
end

With this test, we can now verify whether the actual response matches the spec described in OpenAPI. (The committee-rails README shows an example using assert_schema_conform, but since it produced a deprecation warning, I'm using assert_response_schema_confirm instead.)

Starting from writing the schema

Now that we can write tests based on the spec described in OpenAPI, let's think about how best to apply this in an actual development flow:

1. Discuss the API spec
2. Once the spec is decided, write documentation following OpenAPI
3. Write tests based on the documentation
4. Implement the API
5. Hand off for integration

This seemed like the right approach.

You could call this approach TDD, but the key point is that at steps 1 and 2, the backend and frontend align on the API spec and produce base documentation together.

If the schema is established as shared understanding from the very first step, and the API implementation is guaranteed to satisfy it, this should reduce rework caused by misaligned understanding.

Since I was thinking this through on my own, I still want to try applying it as a team going forward.

→ I've written about a method for generating and integrating a client from the schema here.

Aside: things I noticed about committee

Property validation

In the User schema definition, I set required: [id, email, first_name, last_name], but committee can't validate the presence or absence of properties that aren't marked required. (It can, however, catch things like a clearly wrong type being returned.) This isn't really odd as a spec, but it does seem like something you could easily forget to update when adding a new property.

Also, if the response includes created_at or updated_at that aren't described in the spec, the test still passes.

It would be nice to be able to easily verify whether the properties described in the spec are neither missing nor excessive, so I plan to look into the documentation and think it over further.

File references aren't resolved

A common pain point when writing OpenAPI or Swagger files is that the file size tends to balloon relative to the content. (Sometimes there aren't that many endpoints, yet the file runs to hundreds of lines 😇)

A common workaround is to run a task that splits and merges YAML or JSON files using custom rules, but OpenAPI version 3 actually supports file references natively.

• Using $ref

paths:
  /users:
    $ref: '../resources/users.yaml'
  /users/{userId}:
    $ref: '../resources/users-by-id.yaml'

Since references to files split this way are supported, there's no need to run any additional merge process.

However, it seems committee doesn't correctly resolve file references, so it can't be used to validate schemas split this way.

There may be a workaround, but in the end it looks like you'd need to output everything into a single file and have committee reference that instead.

→ I wrote about a method for managing a split schema here.