Setting Up a GAE/Go Development Environment

Last time, I wrote about the code for implementing an authenticated API with GAE/Go and Firebase. This time, building on that code, I'll summarize how to set up an environment for developing with GAE/Go and Firebase.

Local development environment

Code for GAE can be run locally using a command called dev_appserver.py. Authentication is required to deploy, but it's not needed just to run things locally.

Also, to match the GAE/Go environment, I'm using Go version 1.11.

• Using the local development server
• Google App Engine Go 1.11 Standard Environment documentation

Setting up this kind of development environment is fine to leave to each individual, but when multiple people are developing together, everyone's environment ends up with some differences, and it's a hassle to go back and forth whenever something changes. So I generally use Docker to keep the development environment consistent.

On package management

For Go package management, I initially assumed I'd use dep, but there was information about running into issues when deploying to GAE (and I did in fact run into them), and since Modules — while still in a transitional period — was becoming usable, I decided to use Modules instead.

• dep
• Modules

Docker environment for GAE/Go

What's needed is an environment where the minimum required commands for Go 1.11 can run, so I prepared a Dockerfile based on golang:1.11.6 that just installs gcloud and golangci-lint.

• Quickstart for Debian/Ubuntu
• golangci-lint

# Dockerfile
FROM golang:1.11.6

RUN apt-get update -y && \
    apt-get install lsb-release -y

RUN export CLOUD_SDK_REPO="cloud-sdk-$(lsb_release -c -s)" && \
    echo "deb http://packages.cloud.google.com/apt $CLOUD_SDK_REPO main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list && \
    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - && \
    apt-get update -y && \
    apt-get install google-cloud-sdk google-cloud-sdk-app-engine-python google-cloud-sdk-app-engine-go google-cloud-sdk-datastore-emulator -y

RUN curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.15.0

WORKDIR /go/src/YOUR_VCS/YOUR_NAME/YOUR_REPO

Replace /YOUR_VCS/YOUR_NAME/YOUR_REPO with your own environment's values.

Write the necessary environment variables into .env, and set up docker-compose.yml as follows.

GO111MODULE is set to enable the use of Modules mentioned above, and GOOGLE_APPLICATION_CREDENTIALS is set so that Firebase credentials can be loaded from JSON.

# .env
GO111MODULE=on
GOOGLE_APPLICATION_CREDENTIALS=/go/src/YOUR_VCS/YOUR_NAME/YOUR_REPO/app/serviceAccountKey.json

# docker-compose.yml
version: '3'

services:
  go:
    build: dockerfiles/go
    env_file:
      - .env
    volumes:
      - .:/go/src/YOUR_VCS/YOUR_NAME/YOUR_REPO
    ports:
      - 8080:8080
      - 8000:8000
    restart: 'no'
    command: make serve

• How to Install and Activate Module Support
• Getting started with authentication

The docker-compose.yml command is set to make serve, which is defined in the Makefile as follows.

# Makefile
serve:
	dev_appserver.py app.yaml --host=0.0.0.0 --admin_host=0.0.0.0 --support_datastore_emulator=False

At this point, the directory structure looks like this. serviceAccountKey.json has been downloaded from the Firebase console (and already added to .gitignore).

.
├── .env
├── .gitignore
├── Makefile
├── README.md
├── app
│   ├── main.go
│   └── serviceAccountKey.json
├── app.yaml
├── docker-compose.yml
├── dockerfiles
│   └── go
│       └── Dockerfile
├── go.mod
└── go.sum

Here's app.yaml at its bare minimum for now.

# app.yaml
runtime: go111

main: app
handlers:
- url: /.*
  script: auto
  secure: always

With that, we're ready to spin up the development environment with docker-compose.

$ docker-compose up
# Now accessible at localhost:8080, localhost:8000

dev_appserver.py detects code changes and builds automatically, and when a build runs it pulls in any required Modules on its own — so from here you can just write code in your editor of choice. (That said, getting editor support tools like formatters working is still something each person needs to sort out for themselves.)

Deploying to GAE

If you've already completed the authentication setup with the gcloud command as described above, deploying is as simple as a single command.

$ gcloud app deploy app.yaml

Almost too easy.

Using CircleCI

Since I wanted CircleCI to handle various things whenever code was pushed, I put together the following config file for now.

# .circleci/config.yml
version: 2
jobs:
  build:
    docker:
      - image: golang:1.11.6
        environment:
          GO111MODULE: 'on'
    working_directory: /go/src/YOUR_VCS/YOUR_NAME/YOUR_REPO
    steps:
      - checkout
      - restore_cache:
          key: modules-{{ checksum "go.mod" }}
      - run:
          name: install modules
          command: go mod tidy
      - save_cache:
          key: modules-{{ checksum "go.mod" }}
          paths:
            - /go/pkg/mod
      - run:
          name: lint
          command: |
            curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin v1.15.0
            golangci-lint run --verbose
      - run:
          name: build
          command: go build app/main.go

This just checks that the code passes the default golangci-lint linter settings and that it builds, so from here I'm planning to add testing and deployment into the mix.

About caching Modules

CircleCI is set up to cache Modules, saving the downloaded Modules under /go/pkg/mod keyed on the checksum of go.mod.

Regarding this: since using Modules generates both go.mod and go.sum, I somehow assumed — based on the feel of other package managers — that go.sum was a so-called lock file, but it turns out that's not the case.

• Is 'go.sum' a lock file? Why does 'go.sum' include information for module versions I am no longer using?

According to the wiki, go.sum is not a lock file, and reproducible builds are possible with go.mod alone, so I set key: modules-{{ checksum "go.mod" }}.

With this, the environment from local development, through deployment, to CI has been set up.

Both GAE and Go seem to be evolving quickly, so I plan to keep catching up on new information and improving things little by little.