cache

[This post was originally written in 2020.]

This post is dedicated primarily on making a cicd pipeline, a pretty generic boilerplate pipeline that uses Cloud Build, which is google's CICD serverless platform. Modularly building upwards so that my next project can use this. We're using Cloud Source Repos to host repos, but the pattern is all the same. All these CI pipelines follow the same structure:

• There's a steps file to define steps (e.g. steps to test a pipeline)
• There's a trigger to invoke some steps (e.g. pushing to a branch)

My trigger is defined as any changes to my master branch, and my cloudbuild.yml file has pretty self-explanatory steps: run a test, build docker image, push docker image (and optionally deploy docker image).

This tutorial, which is what I'm generally following, has a separate branch that has a separate cloudbuild.yaml file to manage deployment. This way, the CI step which is generally managed by the developer is decoupled from the CD step, which is generally managed by automated systems. The CD step uses the kubectl builder to automatically link a cluster to deploy, which is a nice abstraction. I made some custom modifications to conform to the way I want feature-branch tested and the final result deployed.

Feature branch runs test and builds docker image.

Master branch runs test, builds, pushes image, and deploys

Cool! Everything works like a charm. You can also do more fancy things. I think if you have more complex docker images, you can simply layer images on top of the GCP builder images. These A-records are mapped to LoadBalancer IPs:

adding a service

I'm using postgresql as the backend hosted on Cloud SQL; it's a managed database service that does scaling, security, and backups for you. For postgres tooling, everything I'm using is pretty standard: psycopg2/sqlalchemy/alembic via the flask ecosystem.

Anyways, everything works great! I can also see it exposed on a lightweight front end I made.

I am using cloudsqlproxy to run everything local against a production database, but alembic makes it really easy to sync your data models up.

Note that sqlalchemy expires on commit; this means that when the data is transferred from local memory to be committed onto the database, the object can't be used anymore. It doesn’t matter too much since you do business logic before you commit anyway.

Cool! I’m ready to hire some engineers now for my project (joking).

Other random stuff:

• I recommend this to follow to do what I did here.
• easy-to-follow alembic set up and reasoning + commit/rollback code

This is just me trying out Nomad and Waypoint. This post was originally wrote in 2020.

nomad

Nomad acts as an online server/system to manage applications on that infra. (Alternative: k8s-terraform) Also, according to the intro talk, it's a really good platform for serverless compute.

What nomad does better than kubernetes:

• true serverless: maximizes utilization, minimizes resources needed
• can use windows/legacy systems, nomad isn't limited to kubernetes
• light: single binary

We can run a simple local server/client with nomad agent -dev -config config.hcl.

Clients are basically worker nodes, and servers makes sure there's high availability (everything is running across regions); our servers and clients create a cluster of virtual machines that have instances of consul and nomad, etc. We can port-forward (tunnel-through-iap) virtual machine ports to local ports (similar to kubectl port-forward).

I'm connected to my nomad cluster locally and I can just schedule a docker image simply with nomad comands: run/plan/stop, etc. It's like kubectl apply -f but with a .nomad file.

I scheduled a hello-world image. I see that it's running in hashistack-client-1. I see that consul automatically picked this guy up in services; the service mesh is all mapped out, etc.

Kubernetes is probably still better in most cases. In terms of other alternatives, there's also rancher, docker swarm, mesos , etc.

waypoint

Hashicorp also announced waypoint; it’s a tool to build and deploy.

  deploy {
    use "exec" {
      command = ["kubectl", "apply", "-f", "<TPL>"]
      template {
        path = "./example-nodejs-exec.yml"
      }
    }
  }

It’s like a makefile that also connects to k8s, AWS, GCP, and also nomad.

Other things:

• people need to use google cloud shell or other cloud terminal equivalents more when writing tutorials like they do here
• api if you just want nomad to be some compute
• Nomad Links: Good list of tooling, example with binary instead of docker, a nice templating walkthrough, a library of reference nomad files