SaltyCrane Blog — Notes on JavaScript and web development

Simple codemod example with jscodeshift

jscodeshift codemods allow refactoring JavaScript or TypeScript code by manipulating the abstract syntax tree.

This is an example showing how to rename variables named foo to bar.

Install jscodeshift

npm install -g jscodeshift

Create an example file to modify

  • create a folder

    mkdir my-project
    cd my-project
    
  • create an example file, my-file-to-modify.js

    const foo = 1;
    console.log(foo);
    

Create a transform

create a file my-transform.js

module.exports = function transformer(fileInfo, api) {
  return api
    .jscodeshift(fileInfo.source)
    .find(api.jscodeshift.Identifier)
    .forEach(function (path) {
      if (path.value.name === "foo") {
        api.jscodeshift(path).replaceWith(api.jscodeshift.identifier("bar"));
      }
    })
    .toSource();
};

Run it

jscodeshift -t my-transform.js ./my-file-to-modify.js

The file my-file-to-modify.js now contains:

const bar = 1;
console.log(bar);

Another example

This example removes the React JSX element <MyHeader /> and removes the MyHeader import. I'm not sure why, but it added some extra parentheses. Prettier cleaned this up for me, but if you have an improvement, let me know.

// removeMyHeader.js
module.exports = function transformer(file, api) {
  const jscodeshift = api.jscodeshift;

  const withoutElement = jscodeshift(file.source)
    .find(jscodeshift.JSXElement)
    .forEach(function (path) {
      if (path.value.openingElement.name.name === "MyHeader") {
        path.prune();
      }
    })
    .toSource();

  const withoutImport = jscodeshift(withoutElement)
    .find(jscodeshift.ImportDefaultSpecifier)
    .forEach(function (path) {
      if (path.value.local.name === "MyHeader") {
        path.parentPath.parentPath.prune();
      }
    })
    .toSource();

  return withoutImport;
};

Here is a command to run it for a React TypeScript codebase:

jscodeshift --parser=tsx --extensions=tsx -t ./removeMyHeader.js ./src

AST Explorer

AST Explorer is a very helpful tool to experiment and learn the API with code completion. Go to https://astexplorer.net/ and select "jscodeshift" under the "Transform" menu.

lodash error

Error: Cannot find module 'lodash'

When running jscodeshift, I got the above error so I ran npm install -g lodash and this got rid of the error for me.

Buildtime vs runtime environment variables with Next.js and Docker

For a Next.js app, buildtime environment variables are variables that are used when the next build command runs. Runtime variables are variables used when the next start command runs.

Below are ways to set buildtime and rutime environment variables with Docker and ways to use buildtime and runtime environment variables with Next.js. Note the Dockerfile is written for simplicity to illustrate the examples. For a more optimized Next.js Docker build see my Docker multi-stage CI example.

Methods for setting environment variables with Docker

MethodAvailable at buildtimeAvailable at runtimeValue passed to docker buildValue passed to docker run
ARG
ENV
ARG + docker build --build-arg
ARG + ENV + docker build --build-arg
docker run --env

Methods for using environment variables in Next.js

MethodSet atAvailable in Next.js client side rendered code (browser)Available in Next.js server side rendered codeAvailable in Node.jsNotes
.env files?both?process.env cannot be destructured or accessed with dynamic properties
NEXT_PUBLIC_ prefixed vars in .env filesbuildtimeprocess.env cannot be destructured or accessed with dynamic properties
env in next.config.jsbuildtimeprocess.env cannot be destructured or accessed with dynamic properties
publicRuntimeConfigruntimeRequires page uses SSR
serverRuntimeConfigruntime
process.envruntime

Assume this package.json for the examples below

{
  "scripts": {
    "build": "next build",
    "dev": "next",
    "start": "next start"
  },
  "dependencies": {
    "next": "^10.0.9",
    "react": "^17.0.2",
    "react-dom": "^17.0.2"
  }
}

Setting static environment variables for buildtime and runtime

Environment variables can be specified with the ENV instruction in a Dockerfile. Below MY_VAR will be available to both next build and next start. For more information see https://docs.docker.com/engine/reference/builder/#env

Dockerfile

FROM node:14-alpine

ENV MY_VAR=cake

WORKDIR /app
COPY . ./
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]

Docker build

docker build -t mytag .

Docker run

docker run mytag

Setting dynamic buildtime environment variables

Dynamic environment variables can be passed to the docker build command using --build-arg and used in the Dockerfile with the ARG statement. Below MY_VAR is an environment variable available to next build.

Note that MY_VAR is not available to next start. ARG statements act like ENV statements in that they are treated like environment variables during docker build, but they are not persisted in the image. To make them available during docker run (and next start) set the value using ENV (see the next example).

For more information see https://docs.docker.com/engine/reference/builder/#arg

Dockerfile

FROM node:14-alpine

ARG MY_VAR

WORKDIR /app
COPY . ./
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]

Docker build

docker build --build-arg MY_VAR=cake -t mytag .

Docker run

docker run mytag

Setting dynamic buildtime environment variables that are available at runtime also

The variable in the previous example, set using ARG, is not persisted in the Docker image so it is not available at runtime. To make it available at runtime, copy the value from ARG to ENV.

Dockerfile

FROM node:14-alpine

ARG MY_VAR
ENV MY_VAR=$MYVAR

WORKDIR /app
COPY . ./
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]

Docker build

docker build --build-arg MY_VAR=cake -t mytag .

Docker run

docker run mytag

Setting dynamic runtime environment variables

Dynamic environment variables can be passed to docker run using the --env flag. These will not be available to next build but they will be available to next start. For more information see https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file

Dockerfile

FROM node:14-alpine
WORKDIR /app
COPY . ./
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]

Docker build

docker build -t mytag .

Docker run

docker run --env MY_VAR=cake mytag

Using buildtime environment variables

To use buildtime environment variables in Next.js code, set them using env in next.config.js. Then access them via process.env in your app code. NOTE: process.env cannot be destructured or used with dynamic property access. Next.js does a string substituion at build time using the webpack DefinePlugin. For more information see https://nextjs.org/docs/api-reference/next.config.js/environment-variables

next.config.js

module.exports = {
  env: {
    MY_VAR: process.env.MY_VAR
  }
}

my-app-file.js

console.log(process.env.MY_VAR)

Using runtime environment variables (client-side or server-side)

To use runtime environment variables (client-side or server-side), set them using publicRuntimeConfig in next.config.js. Then access them using getConfig from next/config. NOTE: this only works for Next.js pages where server-side rendering (SSR) is used. i.e. the page must use getServerSideProps or getInitialProps. For more information see https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration

next.config.js

module.exports = {
  publicRuntimeConfig: {
    MY_VAR: process.env.MY_VAR
  }
}

my-app-file.js

import getConfig from "next/config";
const { publicRuntimeConfig } = getConfig();
console.log(publicRuntimeConfig.MY_VAR)

Using runtime environment variables (server-side only)

To use runtime environment variables (server-side only), set them using serverRuntimeConfig in next.config.js. Then access them using getConfig from next/config. For more information see https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration

NOTE: this applies to to files Next.js "builds". Server run files not processed by Next.js can use process.env to access environment variables. See below.

next.config.js

module.exports = {
  serverRuntimeConfig: {
    MY_VAR: process.env.MY_VAR
  }
}

my-app-file.js

import getConfig from "next/config";
const { serverRuntimeConfig } = getConfig();
console.log(serverRuntimeConfig.MY_VAR)

Using runtime environment variables server-side (not processed by Next.js)

For files not processed by Next.js (next build) (e.g. a server.js file run by node), runtime environment variables can be accessed on the server via process.env. NOTE: "runtime" variables here means variables used when the Node.js process runs. For more information see https://nodejs.org/docs/latest-v14.x/api/process.html#process_process_env

server.js

console.log(process.env.MY_VAR)

Next.js assetPrefix

If the Next.js assetPrefix is set in next.config.js using an environment variable, the environment variable should be set at buildtime for Next.js static pages but set at runtime for server rendered pages.

next.config.js

module.exports = {
  assetPrefix: process.env.MY_ASSET_PREFIX
}

Next.js GitLab CI/CD Docker multi-stage example

This describes an example Next.js project with a GitLab CI/CD pipeline that does the following:

  • installs npm packages and builds static assets
  • runs ESLint, TypeScript, and Cypress
  • builds a Docker image for deployment
  • pushes the Docker image to the GitLab Container Registry

This example prepares a Docker image for deployment but doesn't actually deploy it. See an example CI/CD pipeline that deploys to Amazon ECS.

To increase speed and reduce image size, it uses Docker multi-stage builds.

Dockerfile

The Dockerfile defines 3 stages:

  • the "builder" stage installs npm packages and builds static assets. It produces artifacts (/app and /root/.cache) that are used by the cypress and deploy stages. It is also used to build an image used to run ESLint and TypeScript.
  • the "cypress" stage uses a different base image from the "builder" stage and is used to run cypress tests
  • the final deploy stage copies the /app directory from the "builder" stage and sets NODE_ENV to "production" and exposes port 3000
ARG BASE_IMAGE=node:14.16-alpine

# ================================================================
# builder stage
# ================================================================
FROM $BASE_IMAGE as builder
ENV NODE_ENV=test
ENV NEXT_TELEMETRY_DISABLED=1
RUN apk add --no-cache bash git
WORKDIR /app
COPY ./package.json ./
COPY ./package-lock.json ./
RUN CI=true npm ci
COPY . ./
RUN NODE_ENV=production npm run build

# ================================================================
# cypress stage
# ================================================================
FROM cypress/base:14.16.0 as cypress
WORKDIR /app
# copy cypress from the builder image
COPY --from=builder /root/.cache /root/.cache/
COPY --from=builder /app ./
ENV NODE_ENV=test
ENV NEXT_TELEMETRY_DISABLED=1

# ================================================================
# final deploy stage
# ================================================================
FROM $BASE_IMAGE
WORKDIR /app
COPY --from=builder /app ./
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
EXPOSE 3000
CMD ["npm", "start"]

.gitlab-ci.yml

  • 3 images are built: test, cypress, and deploy. The test image is used for running ESLint and TypeScript and is needed for cypress and deploy. The cypress image is used for running Cypress.
  • it uses Docker BuildKit to make caching easier. (With BuildKit, cached layers will be automatically pulled when needed. Without BuildKit, images used for caching need to be explicitly pulled.) For comparison, see this diff adding BuildKit. Note DOCKER_BUILDKIT is set to 1 to enable BuildKit.
variables:
  # enable docker buildkit. Used with `BUILDKIT_INLINE_CACHE=1` below
  DOCKER_BUILDKIT: 1
  DOCKER_TLS_CERTDIR: "/certs"
  IMAGE_TEST: $CI_REGISTRY_IMAGE/test:latest
  IMAGE_CYPRESS: $CI_REGISTRY_IMAGE/cypress:latest
  IMAGE_DEPLOY: $CI_REGISTRY_IMAGE/deploy:latest

stages:
  - build
  - misc
  - deploy

.base:
  image: docker:latest
  services:
    - docker:dind
  before_script:
    - docker --version
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"

build:builder:
  extends: .base
  stage: build
  script:
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "$IMAGE_TEST" --target builder -t "$IMAGE_TEST" .
    - docker push "$IMAGE_TEST"

build:deployimage:
  extends: .base
  stage: misc
  needs: ["build:builder"]
  script:
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "$IMAGE_DEPLOY" --cache-from "$IMAGE_TEST" --cache-from "$IMAGE_CYPRESS" -t "$IMAGE_DEPLOY" .
    - docker push "$IMAGE_DEPLOY"

test:cypress:
  extends: .base
  stage: misc
  needs: ["build:builder"]
  script:
    - docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "$IMAGE_CYPRESS" --cache-from "$IMAGE_TEST" --target cypress -t "$IMAGE_CYPRESS" .
    - docker push "$IMAGE_CYPRESS"
    - docker run "$IMAGE_CYPRESS" npm run cy:citest

test:eslint:
  extends: .base
  stage: misc
  needs: ["build:builder"]
  script:
    - docker run "$IMAGE_TEST" npm run eslint

test:typescript:
  extends: .base
  stage: misc
  needs: ["build:builder"]
  script:
    - docker run "$IMAGE_TEST" npm run tsc

deploy:
  stage: deploy
  needs: ["build:deployimage", "test:cypress", "test:eslint", "test:typescript"]
  script:
    - echo "deploy here"

.dockerignore

Adding the .git directory to .dockerignore prevented cache invalidation for the COPY . ./ command in the Dockerfile.

.git

References

How to run Docker in Docker on Mac

Docker in Docker can be used in GitLab CI/CD to build Docker images. This is how to run Docker in Docker on Mac.

  • create directory

    mkdir /tmp/my-project
    cd /tmp/my-project
    
  • create docker-compose.yml file:

    version: "3"
    services:
      docker-daemon:
        container_name: "my-docker-daemon"
        environment:
          DOCKER_TLS_CERTDIR: ""
        image: "docker:dind"
        networks:
          "my-network":
            aliases:
              - "docker"
        privileged: true
      docker-client:
        command: sh -c 'while [ 1 ]; do sleep 1000; done'
        container_name: "my-docker-client"
        depends_on:
          - "docker-daemon"
        environment:
          DOCKER_HOST: "tcp://docker:2375"
        image: "docker:latest"
        networks:
          "my-network": {}
    
    networks:
      "my-network":
        name: "my-network"
    
  • run the docker daemon and client containers

    docker-compose up -d
    
  • run a shell in the client container

    docker exec -it my-docker-client sh
    
  • run a docker command in the docker client container

    / # docker ps
    CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
    

References

Next.js Cypress GitLab CI example

This is an example Next.js project that runs a Cypress test in Docker using a GitLab CI pipeline. It also uses the GitLab Container Registry for caching purposes.

.gitlab-ci.yml

variables:
  DOCKER_TLS_CERTDIR: "/certs"

stages:
  - test

test-cypress:
  stage: test
  image: docker:latest
  services:
    - docker:dind
  variables:
    IMAGE_TAG: $CI_REGISTRY_IMAGE:latest
  script:
    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    - docker pull $IMAGE_TAG || true
    - docker build --cache-from $IMAGE_TAG -t $IMAGE_TAG .
    - docker push $IMAGE_TAG
    - docker run $IMAGE_TAG npm run cy:citest

Dockerfile

This uses the official Cypress Docker image (Dockerfile).

FROM cypress/base:14.16.0

WORKDIR /app
# run npm install before adding app code for better Docker caching
# https://semaphoreci.com/docs/docker/docker-layer-caching.html
COPY ./package.json /app
COPY ./package-lock.json /app
# CI=true suppresses Cypress progress log spam
RUN CI=true npm ci
COPY . /app
RUN npm run build

package.json

{
  "scripts": {
    "build": "next build",
    "cy:citest": "start-server-and-test start http://localhost:3000 cy:run",
    "cy:run": "cypress run",
    "dev": "next",
    "start": "next start"
  },
  "dependencies": {
    "next": "^10.0.9",
    "react": "^17.0.2",
    "react-dom": "^17.0.2"
  },
  "devDependencies": {
    "cypress": "^6.8.0",
    "start-server-and-test": "^1.12.1"
  }
}

cypress/integration/index_spec.js

describe("index page", () => {
  it("loads successfully", () => {
    cy.visit("http://localhost:3000");
    cy.contains("Index");
  });
});

References

Example Next.js GitLab CI/CD Amazon ECR and ECS deploy pipeline

I've created an example Next.js project with a GitLab CI/CD pipeline that builds a Docker image, pushes it to Amazon ECR, deploys it to an Amazon ECS Fargate cluster, and uploads static assets (JS, CSS, etc.) to Amazon S3. The example GitLab repo is here: https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example

Interesting files

Here are the interesting parts of some of the files. See the full source code in the GitLab repo.

  • .gitlab-ci.yml (view at gitlab)

    • the variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and ECR_HOST are set in the GitLab UI under "Settings" > "CI/CD" > "Variables"
    • this uses the saltycrane/aws-cli-and-docker Docker image which provides the aws v2 command line tools and docker in a single image. It is based on amazon/aws-cli and installs bc, curl, docker, jq, and tar. This idea is from Valentin's tutorial.
    variables:
      DOCKER_HOST: tcp://docker:2375
      DOCKER_TLS_CERTDIR: ""
      AWS_DEFAULT_REGION: "us-east-1"
      CI_APPLICATION_REPOSITORY: "$ECR_HOST/next-aws-ecr-ecs-gitlab-ci-cd-example"
      CI_APPLICATION_TAG: "$CI_PIPELINE_IID"
      CI_AWS_S3_BUCKET: "next-aws-ecr-ecs-gitlab-ci-cd-example"
      CI_AWS_ECS_CLUSTER: "next-aws-ecr-ecs-gitlab-ci-cd-example"
      CI_AWS_ECS_SERVICE: "next-aws-ecr-ecs-gitlab-ci-cd-example"
      CI_AWS_ECS_TASK_DEFINITION: "next-aws-ecr-ecs-gitlab-ci-cd-example"
      NEXT_JS_ASSET_URL: "https://$CI_AWS_S3_BUCKET.s3.amazonaws.com"
    
    stages:
      - build
      - deploy
    
    build:
      stage: build
      image: saltycrane/aws-cli-and-docker
      services:
        - docker:dind
      script:
        - ./bin/build-and-push-image-to-ecr
        - ./bin/upload-assets-to-s3
    
    deploy:
      stage: deploy
      image: saltycrane/aws-cli-and-docker
      services:
        - docker:dind
      script:
        - ./bin/ecs update-task-definition
    
  • Dockerfile (view at gitlab)

    The value of NEXT_JS_ASSET_URL is passed in using the --build-arg option of the docker build command run in bin/build-and-push-image-to-ecr. It is used like an environment variable in the RUN npm run build command below. In this project it is assigned to assetPrefix in next.config.js.

    FROM node:14.16-alpine
    ARG NEXT_JS_ASSET_URL
    ENV NODE_ENV=production
    WORKDIR /app
    COPY ./package.json ./
    COPY ./package-lock.json ./
    RUN npm ci
    COPY . ./
    RUN npm run build
    EXPOSE 3000
    CMD ["npm", "start"]
    
  • bin/build-and-push-image-to-ecr (view at gitlab)

    # log in to the amazon ecr docker registry
    aws ecr get-login-password | docker login --username AWS --password-stdin "$ECR_HOST"
    
    # build docker image
    docker pull "$CI_APPLICATION_REPOSITORY:latest" || true
    docker build --build-arg "NEXT_JS_ASSET_URL=$NEXT_JS_ASSET_URL" --cache-from "$CI_APPLICATION_REPOSITORY:latest" -t "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"     -t "$CI_APPLICATION_REPOSITORY:latest" .
    
    # push image to amazon ecr
    docker push "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"
    docker push "$CI_APPLICATION_REPOSITORY:latest"
    
  • bin/upload-assets-to-s3 (view at gitlab)

    LOCAL_ASSET_PATH=/tmp/upload-assets
    
    mkdir $LOCAL_ASSET_PATH
    
    # copy the generated assets out of the docker image
    docker run --rm --entrypoint tar "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG" cf - .next | tar xf - -C $LOCAL_ASSET_PATH
    
    # rename .next to _next
    mv "$LOCAL_ASSET_PATH/.next" "$LOCAL_ASSET_PATH/_next"
    
    # remove directories that should not be uploaded to S3
    rm -rf "$LOCAL_ASSET_PATH/_next/cache"
    rm -rf "$LOCAL_ASSET_PATH/_next/server"
    
    # gzip files
    find $LOCAL_ASSET_PATH -regex ".*\.\(css\|svg\|js\)$" -exec gzip {} \;
    
    # strip .gz extension off of gzipped files
    find $LOCAL_ASSET_PATH -name "*.gz" -exec sh -c 'mv $1 `echo $1 | sed "s/.gz$//"`' - {} \;
    
    # upload gzipped js, css, and svg assets
    aws s3 sync --no-progress $LOCAL_ASSET_PATH "s3://$CI_AWS_S3_BUCKET" --cache-control max-age=31536000 --content-encoding gzip --exclude "*" --include "*.js"     --include "*.css" --include "*.svg"
    
    # upload non-gzipped assets
    aws s3 sync --no-progress $LOCAL_ASSET_PATH "s3://$CI_AWS_S3_BUCKET" --cache-control max-age=31536000 --exclude "*.js" --exclude "*.css" --exclude "*.svg" --exclude "*.map"
    
  • bin/ecs (view full file) (This file was copied from the gitlab-org repo)

    #!/bin/bash -e
    
    update_task_definition() {
      local -A register_task_def_args=( \
        ['task-role-arn']='taskRoleArn' \
        ['execution-role-arn']='executionRoleArn' \
        ['network-mode']='networkMode' \
        ['cpu']='cpu' \
        ['memory']='memory' \
        ['pid-mode']='pidMode' \
        ['ipc-mode']='ipcMode' \
        ['proxy-configuration']='proxyConfiguration' \
        ['volumes']='volumes' \
        ['placement-constraints']='placementConstraints' \
        ['requires-compatibilities']='requiresCompatibilities' \
        ['inference-accelerators']='inferenceAccelerators' \
      )
    
      image_repository=$CI_APPLICATION_REPOSITORY
      image_tag=$CI_APPLICATION_TAG
      new_image_name="${image_repository}:${image_tag}"
    
      register_task_definition_from_remote
    
      new_task_definition=$(aws ecs register-task-definition "${args[@]}")
      new_task_revision=$(read_task "$new_task_definition" 'revision')
      new_task_definition_family=$(read_task "$new_task_definition" 'family')
    
      # Making sure that we at least have one running task (even if desiredCount gets updated again with new task definition below)
      service_task_count=$(aws ecs describe-services --cluster "$CI_AWS_ECS_CLUSTER" --services "$CI_AWS_ECS_SERVICE" --query "services[0].desiredCount")
    
      if [[ $service_task_count == 0 ]]; then
        aws ecs update-service --cluster "$CI_AWS_ECS_CLUSTER" --service "$CI_AWS_ECS_SERVICE" --desired-count 1
      fi
    
      # Update ECS service with newly created task defintion revision.
      aws ecs update-service \
                --cluster "$CI_AWS_ECS_CLUSTER" \
                --service "$CI_AWS_ECS_SERVICE" \
                --task-definition "$new_task_definition_family":"$new_task_revision"
    
      return 0
    }
    
    read_task() {
      val=$(echo "$1" | jq -r ".taskDefinition.$2")
      if [ "$val" == "null" ];then
        val=$(echo "$1" | jq -r ".$2")
      fi
      if [ "$val" != "null" ];then
        echo -n "${val}"
      fi
    }
    
    register_task_definition_from_remote() {
      task=$(aws ecs describe-task-definition --task-definition "$CI_AWS_ECS_TASK_DEFINITION")
      current_container_definitions=$(read_task "$task" 'containerDefinitions')
      new_container_definitions=$(echo "$current_container_definitions" | jq --arg val "$new_image_name" '.[0].image = $val')
      args+=("--family" "${CI_AWS_ECS_TASK_DEFINITION}")
      args+=("--container-definitions" "${new_container_definitions}")
      for option in "${!register_task_def_args[@]}"; do
        value=$(read_task "$task" "${register_task_def_args[$option]}")
        if [ -n "$value" ];then
          args+=("--${option}" "${value}")
        fi
      done
    }
    
    update_task_definition
    

Usage - set up AWS resources

Below are the minimum steps I needed to create the required AWS services for my example. I use the AWS region "us-east-1". For info about creating some of these services via the command line, see my Amazon ECS notes.

Create an ECR repository

Create an ECS Fargate cluster

Create an ECS task definition

  • click "Create new Task Definition" here: https://console.aws.amazon.com/ecs/home?region=us-east-1#/taskDefinitions
  • select "FARGATE" and click "Next step"
  • configure task
    • for "Task Definition Name" enter "next-aws-ecr-ecs-gitlab-ci-cd-example"
    • for "Task Role" select "None"
    • for "Task execution role" select "Create new role"
    • for "Task memory" select "0.5GB"
    • for "Task CPU" select "0.25 vCPU"
    • click "Add container"
      • for "Container Name" enter "next-aws-ecr-ecs-gitlab-ci-cd-example"
      • for "Image" enter "asdf" (this will be updated by the gitlab ci/cd job)
      • leave "Private repository authentication" unchecked
      • for "Port mappings" enter "3000"
      • click "Add"
    • click "Create"

Create an ECS service

  • click "Create" here: https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/next-aws-ecr-ecs-gitlab-ci-cd-example/services
  • configure service
    • for "Launch type" select "FARGATE"
    • for "Task Definition" enter "next-aws-ecr-ecs-gitlab-ci-cd-example"
    • for "Cluster" select "next-aws-ecr-ecs-gitlab-ci-cd-example"
    • for "Service name" enter "next-aws-ecr-ecs-gitlab-ci-cd-example"
    • for "Number of tasks" enter 1
    • for "Deployment type" select "Rolling update"
    • click "Next step"
  • configure network
    • select the appropriate "Cluster VPC" and two "Subnets"
    • click "Next step"
  • set Auto Scaling
    • click "Next step"
  • review
    • click "Create Service"

Open port 3000

Create a S3 bucket

  • click "Create bucket" here: https://s3.console.aws.amazon.com/s3/home?region=us-east-1
  • for "Bucket name" enter "next-aws-ecr-ecs-gitlab-ci-cd-example"
  • uncheck "Block all public access"
  • check the "I acknowledge that the current settings might result in this bucket and the objects within becoming public" checkbox
  • click "Create bucket"

Update permissions for S3 bucket

Create an IAM user

  • create an IAM user. The user must have at least ECR, ECS, and S3 permissions.
  • take note of the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY

Usage - run the CI/CD pipeline

Fork the example gitlab repo and configure CI/CD variables

  • fork https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example
  • go to "Settings" > "CI/CD" > "Variables" and add the following variables. You can choose to "protect" and "mask" all of them.
    • AWS_ACCESS_KEY_ID
    • AWS_SECRET_ACCESS_KEY
    • ECR_HOST (This is the part of the ECR repository URI before the /. It looks something like XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com)

Edit variables in .gitlab-ci.yml

If you used names other than "next-aws-ecr-ecs-gitlab-ci-cd-example", edit the variables in .gitlab-ci.yml.

Test it

References (build/push)

References (deploy)

GitLab CI/CD hello world examples

These are my notes for creating "hello world" and "docker hello world" GitLab CI/CD pipelines.

GitLab CI/CD hello world (gitlab repo)

  • create git repo, push it to gitlab, and set origin. Replace saltycrane with your username.

    mkdir gitlab-ci-cd-hello-world
    cd gitlab-ci-cd-hello-world
    git init
    touch .gitignore
    git add .
    git commit -m 'first commit'
    git push --set-upstream [email protected]:saltycrane/gitlab-ci-cd-hello-world.git --all
    git remote add origin [email protected]:saltycrane/gitlab-ci-cd-hello-world.git
    
  • add a .gitlab-ci.yml file

    build-hello:
      script:
        - echo "hello world"
    
  • commit and push

    git add .
    git commit -m 'add ci/cd config'
    git push origin
    
  • see the pipeline run (replace saltycrane with your username): https://gitlab.com/saltycrane/gitlab-ci-cd-hello-world/-/pipelines

GitLab CI/CD Docker hello world (gitlab repo)

  • create git repo, push it to gitlab, and set origin. Replace saltycrane with your username.

    mkdir gitlab-ci-cd-docker-hello-world
    cd gitlab-ci-cd-docker-hello-world
    git init
    touch .gitignore
    git add .
    git commit -m 'first commit'
    git push --set-upstream [email protected]:saltycrane/gitlab-ci-cd-docker-hello-world.git --all
    git remote add origin [email protected]:saltycrane/gitlab-ci-cd-docker-hello-world.git
    
  • add a Dockerfile file

    FROM alpine
    RUN echo "hello"
    
  • add a .gitlab-ci.yml file

    variables:
      DOCKER_TLS_CERTDIR: "/certs"
    
    build-docker:
      image: docker:latest
      services:
        - docker:dind
      script:
        - docker build -t hello .
    
  • commit and push

    git add .
    git commit -m 'add Dockerfile and ci/cd config'
    git push origin
    
  • see the pipeline run (replace saltycrane with your username): https://gitlab.com/saltycrane/gitlab-ci-cd-docker-hello-world/-/pipelines

Amazon ECS notes

These are my notes for creating a Docker image, pushing it to Amazon ECR (Elastic Container Registry), and deploying it to Amazon ECS (Elastic Container Service) using AWS Fargate (serverless for containers) using command line tools.

Create docker image on local machine

  • install docker (macOS)

    brew install homebrew/cask/docker
    
  • create directory

    mkdir /tmp/my-project
    cd /tmp/my-project
    
  • create /tmp/my-project/Dockerfile:

    FROM python:3.9-alpine3.13
    WORKDIR /app
    RUN echo 'Hello' > ./index.html
    EXPOSE 80
    CMD ["python", "-m", "http.server", "80"]
    
  • create Docker image

    docker build -t my-image .
    
  • test running the Docker image locally

    docker run -p 80:80 my-image
    
  • go to http://localhost in the browser and see the text "Hello"

Install and configure AWS command line tools

  • install AWS command line tools

    brew install awscli
    
  • create an IAM user

  • run aws configure and enter:

    • AWS Access Key ID
    • AWS Secret Access Key

    This creates the file ~/.aws/credentials

Create ECR repository and push image to it

From https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html#use-ecr

  • create an Amazon ECR repository using aws ecr create-repository

    aws ecr create-repository --repository-name my-repository --region us-east-1
    

    output:

    {
        "repository": {
            "repositoryArn": "arn:aws:ecr:us-east-1:AAAAAAAAAAAA:repository/my-repository",
            "registryId": "AAAAAAAAAAAA",
            "repositoryName": "my-repository",
            "repositoryUri": "AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com/my-repository",
            "createdAt": "2021-03-17T10:48:18-07:00",
            "imageTagMutability": "MUTABLE",
            "imageScanningConfiguration": {
                "scanOnPush": false
            },
            "encryptionConfiguration": {
                "encryptionType": "AES256"
            }
        }
    }
    

    Take note of the "registryId" and use it in place of "AAAAAAAAAAAA" below.

  • tag the docker image with the repositoryUri

    docker tag my-image AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com/my-repository
    
  • log in to the Amazon ECR registry using aws ecr get-login-password

    aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com
    
  • push the docker image to the Amazon ECR repository

    docker push AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com/my-repository
    
  • see the image in AWS console https://console.aws.amazon.com/ecr/repositories?region=us-east-1

Install ECS command line tools

  • install ecs-cli. Note there is ecs-cli in addition to aws ecs tools. The reason is probably similar to why some services are named "Amazon Service" and some are named "AWS Service". (It seems like ecs-cli provides higher level commands.)
    brew install amazon-ecs-cli
    

Create Amazon ECS Fargate cluster

From https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html

  • create a cluster using ecs-cli up
    ecs-cli up --cluster my-cluster --launch-type FARGATE --region us-east-1
    
    output:
    INFO[0001] Created cluster                               cluster=my-cluster region=us-east-1
    INFO[0002] Waiting for your cluster resources to be created...
    INFO[0002] Cloudformation stack status                   stackStatus=CREATE_IN_PROGRESS
    VPC created: vpc-BBBBBBBBBBBBBBBBB
    Subnet created: subnet-CCCCCCCCCCCCCCCCC
    Subnet created: subnet-DDDDDDDDDDDDDDDDD
    Cluster creation succeeded.
    
    Take note of the VPC (virtual private cloud), and two subnet IDs to use later. See the cluster in the AWS console UI: https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters

Gather parameters required to deploy to ECS cluster

From https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html

Create task execution IAM role
  • create a file /tmp/my-project/task-execution-assume-role.json

    {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Sid": "",
          "Effect": "Allow",
          "Principal": {
            "Service": "ecs-tasks.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    }
    
  • create the task execution role using aws iam create-role

    aws iam create-role --role-name my-task-execution-role --assume-role-policy-document file:///tmp/my-project/task-execution-assume-role.json --region us-east-1
    
  • attach the task execution role policy using aws iam attach-role-policy

    aws iam attach-role-policy --role-name my-task-execution-role --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy --region us-east-1
    
Get security group ID
  • get the default security group ID for the virtual private cloud (VPC) created when creating the ECS cluster using aws ec2 describe-security-groups. Replace "vpc-BBBBBBBBBBBBBBBBB" with your VPC ID

    aws ec2 describe-security-groups --filters Name=vpc-id,Values=vpc-BBBBBBBBBBBBBBBBB --region us-east-1
    

    output:

    {
        "SecurityGroups": [
            {
                "Description": "default VPC security group",
                "GroupName": "default",
                "IpPermissions": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": [
                            {
                                "GroupId": "sg-EEEEEEEEEEEEEEEEE",
                                "UserId": "AAAAAAAAAAAA"
                            }
                        ]
                    }
                ],
                "OwnerId": "AAAAAAAAAAAA",
                "GroupId": "sg-EEEEEEEEEEEEEEEEE",
                "IpPermissionsEgress": [
                    {
                        "IpProtocol": "-1",
                        "IpRanges": [
                            {
                                "CidrIp": "0.0.0.0/0"
                            }
                        ],
                        "Ipv6Ranges": [],
                        "PrefixListIds": [],
                        "UserIdGroupPairs": []
                    }
                ],
                "VpcId": "vpc-BBBBBBBBBBBBBBBBB"
            }
        ]
    }
    

    Take note of the "GroupId" to be used later

Deploy to Amazon ECS cluster

From https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html

  • create /tmp/my-project/ecs-params.yml replacing "subnet-CCCCCCCCCCCCCCCCC", "subnet-DDDDDDDDDDDDDDDDD", and "sg-EEEEEEEEEEEEEEEEE" with appropriate IDs from above. ECS Parameters docs

    version: 1
    task_definition:
      task_execution_role: my-task-execution-role
      ecs_network_mode: awsvpc
      task_size:
        mem_limit: 0.5GB
        cpu_limit: 256
    run_params:
      network_configuration:
        awsvpc_configuration:
          subnets:
            - "subnet-CCCCCCCCCCCCCCCCC"
            - "subnet-DDDDDDDDDDDDDDDDD"
          security_groups:
            - "sg-EEEEEEEEEEEEEEEEE"
          assign_public_ip: ENABLED
    
  • create /tmp/my-project/docker-compose.yml replacing AAAAAAAAAAAA with the registryId:

    version: '3'
    services:
      web:
        image: 'AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com/my-repository'
        ports:
          - '80:80'
    
  • deploy to the ECS cluster using ecs-cli compose service up. This creates a task definition and service. This uses the docker-compose.yml file in the current directory.

    ecs-cli compose --cluster my-cluster --project-name my-project --ecs-params ecs-params.yml --region us-east-1 service up --launch-type FARGATE
    

    see the service in the web UI: https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/my-cluster/services

Hit the server in the browser

  • configure security group to allow inbound access on port 80 using aws ec2 authorize-security-group-ingress

    aws ec2 authorize-security-group-ingress --group-id sg-EEEEEEEEEEEEEEEEE --protocol tcp --port 80 --cidr 0.0.0.0/0 --region us-east-1
    
  • get the IP address using ecs-cli compose service ps

    ecs-cli compose --cluster my-cluster --project-name my-project --region us-east-1 service ps
    

    output:

    Name                                             State    Ports                    TaskDefinition  Health
    my-cluster/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/web  RUNNING  FF.FF.FF.FF:80->80/tcp  my-project:1    UNKNOWN
    

    Take note of the IP address under "Ports"

  • visit in the browser: http://FF.FF.FF.FF replacing "FF.FF.FF.FF" with your IP address

Destroy

  • delete the ECS service using ecs-cli compose service down

    ecs-cli compose --cluster my-cluster --project-name my-project --region us-east-1 service down
    
  • delete the ECS cluster using ecs-cli down

    ecs-cli down --force --cluster my-cluster --region us-east-1
    
  • delete the ECR repository using aws ecr delete-repository

    aws ecr delete-repository --repository-name my-repository --region us-east-1 --force
    

References

How to download pull request metadata using the GitHub GraphQL API

This is a TypeScript Node.js script to download GitHub pull request information (title, body, comments, etc.) using the GitHub GraphQL API. The data is saved in a JSON file.

The GitHub repo is here: download-github-prs.

Create a GitHub personal access token

Create a GitHub personal access token as described here (no checkboxes need to be selected for public repos): https://docs.github.com/en/[email protected]/github/authenticating-to-github/creating-a-personal-access-token This is used to access the GitHub GraphQL API.

Intall libraries

  • Create a directory and cd into it
    $ mkdir -p /tmp/my-prs
    $ cd /tmp/my-prs 
    
  • Create package.json file:
    {
      "scripts": {
        "download": "ts-node index.ts"
      },
      "dependencies": {
        "node-fetch": "^2.6.1"
      },
      "devDependencies": {
        "@types/node-fetch": "^2.5.7",
        "ts-node": "^9.0.0",
        "typescript": "^4.1.2"
      }
    }
    
  • Install
    $ npm install 
    
  • Create script

    Creat a file, /tmp/my-prs/index.ts, replacing XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX with the GitHub personal access token described above.

    import * as fs from "fs";
    import fetch from "node-fetch";
    
    const GITHUB_TOKEN = "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX";
    const GITHUB_GRAPHQL_API_URL = "https://api.github.com/graphql";
    const OUTPUT_DIR = "/tmp/my-prs";
    const REPO_OWNER = "facebookexperimental";
    const REPO_NAME = "Recoil";
    
    fetchPullRequest(1);
    
    /**
     * fetchPullRequest
     */
    async function fetchPullRequest(prNumber: number) {
      const reactionFragment = `
        content
        user {
          login
        }
      `;
      const userContentEditFragment = `
        createdAt
        deletedAt
        deletedBy {
          login
        }
        diff
        editedAt
        editor {
          login
        }
        updatedAt
      `;
      const commentFragment = `
        author {
          login
        }
        body
        createdAt
        reactions(first: 100) {
          nodes {
            ${reactionFragment}
          }
        }
        userContentEdits(first: 100) {
          nodes {
            ${userContentEditFragment}
          }
        }
      `;
      const query = `
      query {
        repository(owner: "${REPO_OWNER}", name: "${REPO_NAME}") {
          nameWithOwner
          pullRequest(number: ${prNumber}) {
            author { login }
            baseRefName
            baseRefOid
            body
            closedAt
            comments(first: 100) {
              nodes {
                ${commentFragment}
              }
            }
            commits(first: 250) {
              nodes {
                commit {
                  oid
                }
              }
            }
            createdAt
            files(first: 100) {
              nodes { path }
            }
            headRefName
            headRefOid
            mergeCommit { oid }
            merged
            mergedAt
            mergedBy { login }
            number
            publishedAt
            reactions(first: 10) {
              nodes {
                ${reactionFragment}
              }
            }
            reviews(first: 10) {
              nodes {
                author { login }
                body
                comments(first: 10) {
                  nodes {
                    ${commentFragment}
                  }
                }
                commit {
                  oid
                }
                createdAt
                editor { login }
                publishedAt
                reactions(first: 10) {
                  nodes {
                    ${reactionFragment}
                  }
                }
                resourcePath
                submittedAt
                updatedAt
                userContentEdits(first: 10) {
                  nodes {
                    ${userContentEditFragment}
                  }
                }
              }
            }
            state
            title
            updatedAt
            userContentEdits(first: 10) {
              nodes {
                ${userContentEditFragment}
              }
            }
          }
        }
      }
      `;
    
      // make graphql query and strigify the response
      const resp = await fetchQuery(query);
      const respStr = JSON.stringify(resp, null, 2);
    
      // save json file
      const filepath = [
        `${OUTPUT_DIR}/`,
        `${REPO_NAME}-pr-${String(prNumber).padStart(4, "0")}.json`,
      ].join("");
      console.log(`Saving ${filepath}...`);
      fs.writeFileSync(filepath, respStr);
    }
    
    /**
     * fetchQuery
     */
    function fetchQuery(query: string, variables: Record<string, any> = {}) {
      return fetch(GITHUB_GRAPHQL_API_URL, {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          Authorization: `bearer ${GITHUB_TOKEN}`,
        },
        body: JSON.stringify({
          query,
          variables,
        }),
      }).then((response) => {
        return response.json();
      });
    }
    

    Run it

    $ npm run download 
    

    Output

    This produces the following JSON file

    $ cat /tmp/my-prs/Recoil-pr-0001.json
    {
      "data": {
        "repository": {
          "nameWithOwner": "facebookexperimental/Recoil",
          "pullRequest": {
            "author": {
              "login": "facebook-github-bot"
            },
            "baseRefName": "master",
            "baseRefOid": "40e870caadc159a87e81be291ff641410ab32e8f",
            "body": "This is pull request was created automatically because we noticed your project was missing a Contributing file.\n\nCONTRIBUTING files explain how a developer can contribute to the project - which you should actively encourage.\n\nThis PR was crafted with love by Facebook's Open Source Team.",
            "closedAt": "2020-05-13T04:12:15Z",
            "comments": {
              "nodes": [
                {
                  "author": {
                    "login": "davidmccabe"
                  },
                  "body": "Already added this manually.",
                  "createdAt": "2020-05-13T04:12:15Z",
                  "reactions": {
                    "nodes": []
                  },
                  "userContentEdits": {
                    "nodes": []
                  }
                }
              ]
            },
            "commits": {
              "nodes": [
                {
                  "commit": {
                    "oid": "96f91679540362fa96a6c92611a8ef5621447b42"
                  }
                }
              ]
            },
            "createdAt": "2020-05-06T22:31:01Z",
            "files": {
              "nodes": [
                {
                  "path": "CONTRIBUTING.md"
                }
              ]
            },
            "headRefName": "automated_fixup_contributing_file_exists",
            "headRefOid": "96f91679540362fa96a6c92611a8ef5621447b42",
            "mergeCommit": null,
            "merged": false,
            "mergedAt": null,
            "mergedBy": null,
            "number": 1,
            "publishedAt": "2020-05-06T22:31:01Z",
            "reactions": {
              "nodes": []
            },
            "reviews": {
              "nodes": []
            },
            "state": "CLOSED",
            "title": "Adding Contributing file",
            "updatedAt": "2020-10-07T20:23:05Z",
            "userContentEdits": {
              "nodes": []
            }
          }
        }
      }
    }
    

Notes on Fabric 2 and Python 3

Fabric 2 is a Python package used for running commands on remote machines via SSH. Fabric 2 supports Python 3 and is a rewrite of the Fabric I used years ago. Here are my notes on using Fabric 2 and Python 3.

Set up SSH config and SSH agent

  • Create or edit your ~/.ssh/config file to contain your remote host parameters
    Host myhost
        User myusername
        HostName myhost.com
        IdentityFile ~/.ssh/id_rsa
    
  • Add your private key to your SSH agent
    $ ssh-add ~/.ssh/id_rsa
    

Create a project, create a virtualenv, and install fabric2

$ mkdir -p /tmp/my-project
$ cd /tmp/my-project
$ python3 -m venv venv
$ source venv/bin/activate
$ pip install fabric2

Create a fabfile.py script

Create a file /tmp/my-project/fabfile.py with the following contents. Note: "myhost" is the same name used in ~/.ssh/config described above.

from fabric2 import task

hosts = ["myhost"]

@task(hosts=hosts)
def mytask(c):
    print("Starting mytask...")
    with c.cd("/var"):
        c.run("ls -l")
    print("Done.")

Run the fabric script

In /tmp/my-project, with the virtualenv activated, run the fabric task to list the contents of /var on the remote host.

$ fab2 mytask 

Output:

Starting mytask...

total 48
drwxr-xr-x  2 root root   4096 backups
drwxr-xr-x  9 root root   4096 cache
drwxrwxrwt  2 root root   4096 crash
drwxr-xr-x 38 root root   4096 lib
drwxrwsr-x  2 root root   4096 local
drwxrwxrwt  2 root root   4096 lock
drwxrwxr-x 14 root root   4096 log
drwxrwsr-x  2 root root   4096 mail
drwxr-xr-x  2 root root   4096 opt
drwxr-xr-x  5 root root   4096 spool
drwxrwxrwt  2 root root   4096 tmp
drwxr-xr-x  3 root root   4096 www

Done.

See also / References