SaltyCrane: dockerhttps://www.saltycrane.com/blog/2021-04-13T13:06:02-07:00Buildtime vs runtime environment variables with Next.js and Docker
2021-04-13T13:06:02-07:00https://www.saltycrane.com/blog/2021/04/buildtime-vs-runtime-environment-variables-nextjs-docker/<p>
For a Next.js app, <strong>buildtime</strong> environment variables are
variables that are used when the <code>next build</code> command runs.
<strong>Runtime</strong> variables are variables used when the
<code>next start</code> command runs.
</p>
<p>
Below are ways to set buildtime and rutime environment variables with Docker
and ways to use buildtime and runtime environment variables with Next.js. Note
the <code>Dockerfile</code> is written for simplicity to illustrate the
examples. For a more optimized Next.js Docker build see my
<a href="/blog/2021/04/nextjs-gitlab-cicd-docker-multi-stage-example/"
>Docker multi-stage CI example</a
>.
</p>
<h4 id="methods-for-setting-environment-variables-with-docker">
Methods for setting environment variables with Docker
</h4>
<table>
<thead>
<tr>
<th align="left">Method</th>
<th align="center">Available at buildtime</th>
<th align="center">Available at runtime</th>
<th align="center">Value passed to <code>docker build</code></th>
<th align="center">Value passed to <code>docker run</code></th>
</tr>
</thead>
<tbody>
<tr>
<td align="left"><code>ARG</code></td>
<td align="center">✔</td>
<td align="center"></td>
<td align="center"></td>
<td align="center"></td>
</tr>
<tr>
<td align="left">
<a
href="#setting-static-environment-variables-for-buildtime-and-runtime"
><code>ENV</code></a
>
</td>
<td align="center">✔</td>
<td align="center">✔</td>
<td align="center"></td>
<td align="center"></td>
</tr>
<tr>
<td align="left">
<a href="#setting-dynamic-buildtime-environment-variables"
><code>ARG</code> + <code>docker build --build-arg</code></a
>
</td>
<td align="center">✔</td>
<td align="center"></td>
<td align="center">✔</td>
<td align="center"></td>
</tr>
<tr>
<td align="left">
<a
href="#setting-dynamic-buildtime-environment-variables-that-are-available-at-runtime-also"
><code>ARG</code> + <code>ENV</code> +
<code>docker build --build-arg</code></a
>
</td>
<td align="center">✔</td>
<td align="center">✔</td>
<td align="center">✔</td>
<td align="center"></td>
</tr>
<tr>
<td align="left">
<a href="#setting-dynamic-runtime-environment-variables"
><code>docker run --env</code></a
>
</td>
<td align="center"></td>
<td align="center">✔</td>
<td align="center"></td>
<td align="center">✔</td>
</tr>
</tbody>
</table>
<h4 id="methods-for-using-environment-variables-in-nextjs">
Methods for using environment variables in Next.js
</h4>
<table>
<thead>
<tr>
<th align="left">Method</th>
<th align="left">Set at</th>
<th align="center">
Available in Next.js client side rendered code (browser)
</th>
<th align="center">Available in Next.js server side rendered code</th>
<th align="center">Available in Node.js</th>
<th>Notes</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">
<a href="https://nextjs.org/docs/basic-features/environment-variables"
><code>.env</code> files</a
>
</td>
<td align="left">?both?</td>
<td align="center"></td>
<td align="center">✔</td>
<td align="center"></td>
<td>
<code>process.env</code> cannot be destructured or accessed with dynamic
properties
</td>
</tr>
<tr>
<td align="left">
<a
href="https://nextjs.org/docs/basic-features/environment-variables#exposing-environment-variables-to-the-browser"
><code>NEXT_PUBLIC_</code> prefixed vars in <code>.env</code> files</a
>
</td>
<td align="left">buildtime</td>
<td align="center">✔</td>
<td align="center">✔</td>
<td align="center"></td>
<td>
<code>process.env</code> cannot be destructured or accessed with dynamic
properties
</td>
</tr>
<tr>
<td align="left">
<a href="#using-buildtime-environment-variables"
><code>env</code> in <code>next.config.js</code></a
>
</td>
<td align="left">buildtime</td>
<td align="center">✔</td>
<td align="center">✔</td>
<td align="center"></td>
<td>
<code>process.env</code> cannot be destructured or accessed with dynamic
properties
</td>
</tr>
<tr>
<td align="left">
<a
href="#using-runtime-environment-variables-client-side-or-server-side"
><code>publicRuntimeConfig</code></a
>
</td>
<td align="left">runtime</td>
<td align="center">✔</td>
<td align="center">✔</td>
<td align="center"></td>
<td>Requires page uses SSR</td>
</tr>
<tr>
<td align="left">
<a href="#using-runtime-environment-variables-server-side-only"
><code>serverRuntimeConfig</code></a
>
</td>
<td align="left">runtime</td>
<td align="center"></td>
<td align="center">✔</td>
<td align="center"></td>
<td></td>
</tr>
<tr>
<td align="left">
<a
href="#using-runtime-environment-variables-server-side-not-processed-by-nextjs"
><code>process.env</code></a
>
</td>
<td align="left">runtime</td>
<td align="center"></td>
<td align="center"></td>
<td align="center">✔</td>
<td></td>
</tr>
</tbody>
</table>
<h4 id="assume-this-packagejson-for-the-examples-below">
Assume this <code>package.json</code> for the examples below
</h4>
<pre><code class="language-json">{
"scripts": {
"build": "next build",
"dev": "next",
"start": "next start"
},
"dependencies": {
"next": "^10.0.9",
"react": "^17.0.2",
"react-dom": "^17.0.2"
}
}
</code></pre>
<h4 id="setting-static-environment-variables-for-buildtime-and-runtime">
Setting static environment variables for buildtime and runtime
</h4>
<p>
Environment variables can be specified with the <code>ENV</code> instruction
in a <code>Dockerfile</code>. Below <code>MY_VAR</code> will be available to
both <code>next build</code> and <code>next start</code>. For more information
see
<a href="https://docs.docker.com/engine/reference/builder/#env"
>https://docs.docker.com/engine/reference/builder/#env</a
>
</p>
<p>
<strong><code>Dockerfile</code></strong>
</p>
<pre><code class="language-Dockerfile">FROM node:14-alpine
ENV MY_VAR=cake
WORKDIR /app
COPY . ./
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]
</code></pre>
<p><strong>Docker build</strong></p>
<pre><code class="language-sh">docker build -t mytag .
</code></pre>
<p><strong>Docker run</strong></p>
<pre><code class="language-sh">docker run mytag
</code></pre>
<h4 id="setting-dynamic-buildtime-environment-variables">
Setting dynamic buildtime environment variables
</h4>
<p>
Dynamic environment variables can be passed to the
<code>docker build</code> command using <code>--build-arg</code> and used in
the <code>Dockerfile</code> with the <code>ARG</code> statement. Below
<code>MY_VAR</code> is an environment variable available to
<code>next build</code>.
</p>
<p>
Note that <code>MY_VAR</code> is not available to <code>next start</code>.
<code>ARG</code> statements act like <code>ENV</code> statements in that they
are treated like environment variables during <code>docker build</code>, but
they are not persisted in the image. To make them available during
<code>docker run</code> (and <code>next start</code>) set the value using
<code>ENV</code> (see the next example).
</p>
<p>
For more information see
<a href="https://docs.docker.com/engine/reference/builder/#arg"
>https://docs.docker.com/engine/reference/builder/#arg</a
>
</p>
<p>
<strong><code>Dockerfile</code></strong>
</p>
<pre><code>FROM node:14-alpine
ARG MY_VAR
WORKDIR /app
COPY . ./
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]
</code></pre>
<p><strong>Docker build</strong></p>
<pre><code>docker build --build-arg MY_VAR=cake -t mytag .
</code></pre>
<p><strong>Docker run</strong></p>
<pre><code>docker run mytag
</code></pre>
<h4
id="setting-dynamic-buildtime-environment-variables-that-are-available-at-runtime-also"
>
Setting dynamic buildtime environment variables that are available at runtime
also
</h4>
<p>
The variable in the previous example, set using <code>ARG</code>, is not
persisted in the Docker image so it is not available at runtime. To make it
available at runtime, copy the value from <code>ARG</code> to
<code>ENV</code>.
</p>
<p>
<strong><code>Dockerfile</code></strong>
</p>
<pre><code>FROM node:14-alpine
ARG MY_VAR
ENV MY_VAR=$MYVAR
WORKDIR /app
COPY . ./
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]
</code></pre>
<p><strong>Docker build</strong></p>
<pre><code>docker build --build-arg MY_VAR=cake -t mytag .
</code></pre>
<p><strong>Docker run</strong></p>
<pre><code>docker run mytag
</code></pre>
<h4 id="setting-dynamic-runtime-environment-variables">
Setting dynamic runtime environment variables
</h4>
<p>
Dynamic environment variables can be passed to <code>docker run</code> using
the <code>--env</code> flag. These will not be available to
<code>next build</code> but they will be available to <code>next start</code>.
For more information see
<a
href="https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file"
>https://docs.docker.com/engine/reference/commandline/run/#set-environment-variables--e---env---env-file</a
>
</p>
<p>
<strong><code>Dockerfile</code></strong>
</p>
<pre><code>FROM node:14-alpine
WORKDIR /app
COPY . ./
RUN npm install
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]
</code></pre>
<p><strong>Docker build</strong></p>
<pre><code>docker build -t mytag .
</code></pre>
<p><strong>Docker run</strong></p>
<pre><code>docker run --env MY_VAR=cake mytag
</code></pre>
<h4 id="using-buildtime-environment-variables">
Using buildtime environment variables
</h4>
<p>
To use buildtime environment variables in Next.js code, set them using
<code>env</code> in <code>next.config.js</code>. Then access them via
<code>process.env</code> in your app code. NOTE:
<code>process.env</code> cannot be destructured or used with dynamic property
access. Next.js does a string substituion at build time using the webpack
<a href="https://webpack.js.org/plugins/define-plugin/">DefinePlugin</a>. For
more information see
<a
href="https://nextjs.org/docs/api-reference/next.config.js/environment-variables"
>https://nextjs.org/docs/api-reference/next.config.js/environment-variables</a
>
</p>
<p>
<strong><code>next.config.js</code></strong>
</p>
<pre><code>module.exports = {
env: {
MY_VAR: process.env.MY_VAR
}
}
</code></pre>
<p>
<strong><code>my-app-file.js</code></strong>
</p>
<pre><code>console.log(process.env.MY_VAR)
</code></pre>
<h4 id="using-runtime-environment-variables-client-side-or-server-side">
Using runtime environment variables (client-side or server-side)
</h4>
<p>
To use runtime environment variables (client-side or server-side), set them
using <code>publicRuntimeConfig</code> in <code>next.config.js</code>. Then
access them using <code>getConfig</code> from <code>next/config</code>. NOTE:
this only works for Next.js pages where server-side rendering (SSR) is used.
i.e. the page must use
<a
href="https://nextjs.org/docs/basic-features/data-fetching#getserversideprops-server-side-rendering"
><code>getServerSideProps</code></a
>
or
<a href="https://nextjs.org/docs/api-reference/data-fetching/getInitialProps"
><code>getInitialProps</code></a
>. For more information see
<a
href="https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration"
>https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration</a
>
</p>
<p>
<strong><code>next.config.js</code></strong>
</p>
<pre><code>module.exports = {
publicRuntimeConfig: {
MY_VAR: process.env.MY_VAR
}
}
</code></pre>
<p>
<strong><code>my-app-file.js</code></strong>
</p>
<pre><code>import getConfig from "next/config";
const { publicRuntimeConfig } = getConfig();
console.log(publicRuntimeConfig.MY_VAR)
</code></pre>
<h4 id="using-runtime-environment-variables-server-side-only">
Using runtime environment variables (server-side only)
</h4>
<p>
To use runtime environment variables (server-side only), set them using
<code>serverRuntimeConfig</code> in <code>next.config.js</code>. Then access
them using <code>getConfig</code> from <code>next/config</code>. For more
information see
<a
href="https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration"
>https://nextjs.org/docs/api-reference/next.config.js/runtime-configuration</a
>
</p>
<p>
NOTE: this applies to to files Next.js "builds". Server run files
not processed by Next.js can use <code>process.env</code> to access
environment variables. See below.
</p>
<p>
<strong><code>next.config.js</code></strong>
</p>
<pre><code>module.exports = {
serverRuntimeConfig: {
MY_VAR: process.env.MY_VAR
}
}
</code></pre>
<p>
<strong><code>my-app-file.js</code></strong>
</p>
<pre><code>import getConfig from "next/config";
const { serverRuntimeConfig } = getConfig();
console.log(serverRuntimeConfig.MY_VAR)
</code></pre>
<h4
id="using-runtime-environment-variables-server-side-not-processed-by-nextjs"
>
Using runtime environment variables server-side (not processed by Next.js)
</h4>
<p>
For files not processed by Next.js (<code>next build</code>) (e.g. a
<code>server.js</code> file run by <code>node</code>), runtime environment
variables can be accessed on the server via <code>process.env</code>. NOTE:
"runtime" variables here means variables used when the Node.js
process runs. For more information see
<a
href="https://nodejs.org/docs/latest-v14.x/api/process.html#process_process_env"
>https://nodejs.org/docs/latest-v14.x/api/process.html#process_process_env</a
>
</p>
<p>
<strong><code>server.js</code></strong>
</p>
<pre><code>console.log(process.env.MY_VAR)
</code></pre>
<h4 id="nextjs-assetprefix">Next.js <code>assetPrefix</code></h4>
<p>
If the
<a
href="https://nextjs.org/docs/api-reference/next.config.js/cdn-support-with-asset-prefix"
>Next.js <code>assetPrefix</code></a
>
is set in <code>next.config.js</code> using an environment variable, the
environment variable should be set at buildtime for
<a
href="https://nextjs.org/docs/advanced-features/automatic-static-optimization"
>Next.js static pages</a
>
but set at runtime for
<a href="https://nextjs.org/docs/basic-features/pages#server-side-rendering"
>server rendered pages</a
>.
</p>
<p>
<strong><code>next.config.js</code></strong>
</p>
<pre><code>module.exports = {
assetPrefix: process.env.MY_ASSET_PREFIX
}
</code></pre>
Next.js GitLab CI/CD Docker multi-stage example
2021-04-06T09:59:46-07:00https://www.saltycrane.com/blog/2021/04/nextjs-gitlab-cicd-docker-multi-stage-example/<p>
This describes an example <a href="https://nextjs.org/">Next.js</a> project
with a <a href="https://docs.gitlab.com/ee/ci/">GitLab CI/CD</a> pipeline that
does the following:
</p>
<ul>
<li>installs npm packages and builds static assets</li>
<li>runs ESLint, TypeScript, and Cypress</li>
<li>builds a Docker image for deployment</li>
<li>
pushes the Docker image to the
<a href="https://docs.gitlab.com/ee/user/packages/container_registry/"
>GitLab Container Registry</a
>
</li>
</ul>
<p>
This example prepares a Docker image for deployment but doesn't actually
deploy it. See
<a
href="/blog/2021/03/example-nextjs-gitlab-cicd-amazon-ecr-and-ecs-deploy-pipeline/"
>an example CI/CD pipeline that deploys to Amazon ECS</a
>.
</p>
<p>
To increase speed and reduce image size, it uses
<a href="https://docs.docker.com/develop/develop-images/multistage-build/"
>Docker multi-stage builds</a
>.
</p>
<ul>
<li>
GitLab repo:
<a
href="https://gitlab.com/saltycrane/next-docker-multi-stage-gitlab-ci-cd-example"
>https://gitlab.com/saltycrane/next-docker-multi-stage-gitlab-ci-cd-example</a
>
</li>
<li>
CI/CD Pipelines:
<a
href="https://gitlab.com/saltycrane/next-docker-multi-stage-gitlab-ci-cd-example/-/pipelines"
>https://gitlab.com/saltycrane/next-docker-multi-stage-gitlab-ci-cd-example/-/pipelines</a
>
</li>
</ul>
<h4 id="dockerfile"><code>Dockerfile</code></h4>
<p>The Dockerfile defines 3 stages:</p>
<ul>
<li>
the "builder" stage installs npm packages and builds static
assets. It produces artifacts (<code>/app</code> and
<code>/root/.cache</code>) that are used by the cypress and deploy stages.
It is also used to build an image used to run ESLint and TypeScript.
</li>
<li>
the "cypress" stage uses a different base image from the
"builder" stage and is used to run cypress tests
</li>
<li>
the final deploy stage copies the <code>/app</code> directory from the
"builder" stage and sets <code>NODE_ENV</code> to
"production" and exposes port 3000
</li>
</ul>
<pre><code>ARG BASE_IMAGE=node:14.16-alpine
# ================================================================
# builder stage
# ================================================================
FROM $BASE_IMAGE as builder
ENV NODE_ENV=test
ENV NEXT_TELEMETRY_DISABLED=1
RUN apk add --no-cache bash git
WORKDIR /app
COPY ./package.json ./
COPY ./package-lock.json ./
RUN CI=true npm ci
COPY . ./
RUN NODE_ENV=production npm run build
# ================================================================
# cypress stage
# ================================================================
FROM cypress/base:14.16.0 as cypress
WORKDIR /app
# copy cypress from the builder image
COPY --from=builder /root/.cache /root/.cache/
COPY --from=builder /app ./
ENV NODE_ENV=test
ENV NEXT_TELEMETRY_DISABLED=1
# ================================================================
# final deploy stage
# ================================================================
FROM $BASE_IMAGE
WORKDIR /app
COPY --from=builder /app ./
ENV NODE_ENV=production
ENV NEXT_TELEMETRY_DISABLED=1
EXPOSE 3000
CMD ["npm", "start"]
</code></pre>
<h4 id="gitlab-ciyml"><code>.gitlab-ci.yml</code></h4>
<ul>
<li>
3 images are built: <code>test</code>, <code>cypress</code>, and
<code>deploy</code>. The <code>test</code> image is used for running ESLint
and TypeScript and is needed for <code>cypress</code> and
<code>deploy</code>. The <code>cypress</code> image is used for running
Cypress.
</li>
<li>
it uses
<a href="https://docs.docker.com/develop/develop-images/build_enhancements/"
>Docker BuildKit</a
>
to make caching easier. (With BuildKit, cached layers will be automatically
pulled when needed. Without BuildKit, images used for caching need to be
explicitly pulled.) For comparison, see
<a
href="https://gitlab.com/saltycrane/next-docker-multi-stage-gitlab-ci-cd-example/-/commit/6b238a34a8bdeb310d17fcbaa83227ba2eb78d92"
>this diff adding BuildKit</a
>. Note <code>DOCKER_BUILDKIT</code> is set to <code>1</code> to enable
BuildKit.
</li>
</ul>
<pre><code>variables:
# enable docker buildkit. Used with `BUILDKIT_INLINE_CACHE=1` below
DOCKER_BUILDKIT: 1
DOCKER_TLS_CERTDIR: "/certs"
IMAGE_TEST: $CI_REGISTRY_IMAGE/test:latest
IMAGE_CYPRESS: $CI_REGISTRY_IMAGE/cypress:latest
IMAGE_DEPLOY: $CI_REGISTRY_IMAGE/deploy:latest
stages:
- build
- misc
- deploy
.base:
image: docker:latest
services:
- docker:dind
before_script:
- docker --version
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
build:builder:
extends: .base
stage: build
script:
- docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "$IMAGE_TEST" --target builder -t "$IMAGE_TEST" .
- docker push "$IMAGE_TEST"
build:deployimage:
extends: .base
stage: misc
needs: ["build:builder"]
script:
- docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "$IMAGE_DEPLOY" --cache-from "$IMAGE_TEST" --cache-from "$IMAGE_CYPRESS" -t "$IMAGE_DEPLOY" .
- docker push "$IMAGE_DEPLOY"
test:cypress:
extends: .base
stage: misc
needs: ["build:builder"]
script:
- docker build --build-arg BUILDKIT_INLINE_CACHE=1 --cache-from "$IMAGE_CYPRESS" --cache-from "$IMAGE_TEST" --target cypress -t "$IMAGE_CYPRESS" .
- docker push "$IMAGE_CYPRESS"
- docker run "$IMAGE_CYPRESS" npm run cy:citest
test:eslint:
extends: .base
stage: misc
needs: ["build:builder"]
script:
- docker run "$IMAGE_TEST" npm run eslint
test:typescript:
extends: .base
stage: misc
needs: ["build:builder"]
script:
- docker run "$IMAGE_TEST" npm run tsc
deploy:
stage: deploy
needs: ["build:deployimage", "test:cypress", "test:eslint", "test:typescript"]
script:
- echo "deploy here"
</code></pre>
<h4 id="dockerignore"><code>.dockerignore</code></h4>
<p>
Adding the <code>.git</code> directory to <code>.dockerignore</code> prevented
cache invalidation for the <code>COPY . ./</code> command in the
<code>Dockerfile</code>.
</p>
<pre><code>.git
</code></pre>
<h4 id="references">References</h4>
<ul>
<li>
<a href="https://docs.docker.com/develop/develop-images/multistage-build/"
>https://docs.docker.com/develop/develop-images/multistage-build/</a
>
</li>
<li>
<a
href="https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#making-docker-in-docker-builds-faster-with-docker-layer-caching"
>https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#making-docker-in-docker-builds-faster-with-docker-layer-caching</a
>
</li>
<li>
<a href="https://docs.gitlab.com/ee/ci/directed_acyclic_graph/index.html"
>https://docs.gitlab.com/ee/ci/directed_acyclic_graph/index.html</a
>
</li>
<li>
<a href="https://docs.gitlab.com/ee/ci/yaml/README.html#needs"
>https://docs.gitlab.com/ee/ci/yaml/README.html#needs</a
>
</li>
<li>
<a href="https://testdriven.io/blog/faster-ci-builds-with-docker-cache/"
>https://testdriven.io/blog/faster-ci-builds-with-docker-cache/</a
>
</li>
<li>
<a
href="https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources"
>https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources</a
>
</li>
<li>
<a href="https://docs.docker.com/develop/develop-images/build_enhancements/"
>https://docs.docker.com/develop/develop-images/build_enhancements/</a
>
</li>
</ul>
How to run Docker in Docker on Mac
2021-04-06T09:48:14-07:00https://www.saltycrane.com/blog/2021/04/how-run-docker-docker-mac/<p>
<a href="https://hub.docker.com/_/docker">Docker in Docker</a> can be used in
<a href="https://docs.gitlab.com/ee/ci/docker/using_docker_build.html"
>GitLab CI/CD to build Docker images</a
>. This is how to run Docker in Docker on Mac.
</p>
<ul>
<li>
<p>create directory</p>
<pre><code>mkdir /tmp/my-project
cd /tmp/my-project
</code></pre>
</li>
<li>
<p>create <code>docker-compose.yml</code> file:</p>
<pre><code>version: "3"
services:
docker-daemon:
container_name: "my-docker-daemon"
environment:
DOCKER_TLS_CERTDIR: ""
image: "docker:dind"
networks:
"my-network":
aliases:
- "docker"
privileged: true
docker-client:
command: sh -c 'while [ 1 ]; do sleep 1000; done'
container_name: "my-docker-client"
depends_on:
- "docker-daemon"
environment:
DOCKER_HOST: "tcp://docker:2375"
image: "docker:latest"
networks:
"my-network": {}
networks:
"my-network":
name: "my-network"
</code></pre>
</li>
<li>
<p>run the docker daemon and client containers</p>
<pre><code>docker-compose up -d
</code></pre>
</li>
<li>
<p>run a shell in the client container</p>
<pre><code>docker exec -it my-docker-client sh
</code></pre>
</li>
<li>
<p>run a docker command in the docker client container</p>
<pre><code>/ # docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
</code></pre>
</li>
</ul>
<h4 id="references">References</h4>
<ul>
<li>
<a href="https://hub.docker.com/_/docker"
>https://hub.docker.com/_/docker</a
>
</li>
<li>
<a href="https://www.caktusgroup.com/blog/2020/02/25/docker-image/"
>https://www.caktusgroup.com/blog/2020/02/25/docker-image/</a
>
</li>
</ul>
Next.js Cypress GitLab CI example
2021-03-25T16:12:37-07:00https://www.saltycrane.com/blog/2021/03/nextjs-cypress-gitlab-ci-example/<p>
This is an example <a href="https://nextjs.org/">Next.js</a> project that runs
a <a href="https://www.cypress.io/">Cypress</a> test in Docker using a
<a href="https://docs.gitlab.com/ee/ci/">GitLab CI</a> pipeline. It also uses
the
<a href="https://docs.gitlab.com/ee/user/packages/container_registry/"
>GitLab Container Registry</a
>
for caching purposes.
</p>
<ul>
<li>
GitLab repo:
<a href="https://gitlab.com/saltycrane/next-cypress-gitlab-ci-example"
>https://gitlab.com/saltycrane/next-cypress-gitlab-ci-example</a
>
</li>
<li>
CI Pipelines:
<a
href="https://gitlab.com/saltycrane/next-cypress-gitlab-ci-example/-/pipelines"
>https://gitlab.com/saltycrane/next-cypress-gitlab-ci-example/-/pipelines</a
>
</li>
</ul>
<h4 id="gitlab-ciyml"><code>.gitlab-ci.yml</code></h4>
<pre><code>variables:
DOCKER_TLS_CERTDIR: "/certs"
stages:
- test
test-cypress:
stage: test
image: docker:latest
services:
- docker:dind
variables:
IMAGE_TAG: $CI_REGISTRY_IMAGE:latest
script:
- docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
- docker pull $IMAGE_TAG || true
- docker build --cache-from $IMAGE_TAG -t $IMAGE_TAG .
- docker push $IMAGE_TAG
- docker run $IMAGE_TAG npm run cy:citest
</code></pre>
<h4 id="dockerfile"><code>Dockerfile</code></h4>
<p>
This uses the official Cypress Docker image (<a
href="https://github.com/cypress-io/cypress-docker-images/blob/master/base/14.16.0/Dockerfile"
><code>Dockerfile</code></a
>).
</p>
<pre><code>FROM cypress/base:14.16.0
WORKDIR /app
# run npm install before adding app code for better Docker caching
# https://semaphoreci.com/docs/docker/docker-layer-caching.html
COPY ./package.json /app
COPY ./package-lock.json /app
# CI=true suppresses Cypress progress log spam
RUN CI=true npm ci
COPY . /app
RUN npm run build
</code></pre>
<h4 id="packagejson"><code>package.json</code></h4>
<pre><code>{
"scripts": {
"build": "next build",
"cy:citest": "start-server-and-test start http://localhost:3000 cy:run",
"cy:run": "cypress run",
"dev": "next",
"start": "next start"
},
"dependencies": {
"next": "^10.0.9",
"react": "^17.0.2",
"react-dom": "^17.0.2"
},
"devDependencies": {
"cypress": "^6.8.0",
"start-server-and-test": "^1.12.1"
}
}
</code></pre>
<h4 id="cypressintegrationindex_specjs">
<code>cypress/integration/index_spec.js</code>
</h4>
<pre><code>describe("index page", () => {
it("loads successfully", () => {
cy.visit("http://localhost:3000");
cy.contains("Index");
});
});
</code></pre>
<h4 id="references">References</h4>
<ul>
<li>
<a
href="https://docs.cypress.io/guides/continuous-integration/introduction#Setting-up-CI"
>https://docs.cypress.io/guides/continuous-integration/introduction#Setting-up-CI</a
>
</li>
<li>
<a
href="https://docs.gitlab.com/ee/user/packages/container_registry/#build-and-push-by-using-gitlab-cicd"
>https://docs.gitlab.com/ee/user/packages/container_registry/#build-and-push-by-using-gitlab-cicd</a
>
</li>
</ul>
Example Next.js GitLab CI/CD Amazon ECR and ECS deploy pipeline
2021-03-25T10:29:45-07:00https://www.saltycrane.com/blog/2021/03/example-nextjs-gitlab-cicd-amazon-ecr-and-ecs-deploy-pipeline/<p>
I've created an example <a href="https://nextjs.org/">Next.js</a> project
with a <a href="https://docs.gitlab.com/ee/ci/">GitLab CI/CD</a> pipeline that
builds a Docker image, pushes it to
<a href="https://aws.amazon.com/ecr/">Amazon ECR</a>, deploys it to an
<a href="https://aws.amazon.com/ecs/">Amazon ECS</a>
<a href="https://aws.amazon.com/fargate/">Fargate</a> cluster, and uploads
static assets (JS, CSS, etc.) to
<a href="https://aws.amazon.com/s3/">Amazon S3</a>. The example GitLab repo is
here:
<a href="https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example"
>https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example</a
>
</p>
<h4 id="interesting-files">Interesting files</h4>
<p>
Here are the interesting parts of some of the files. See the full source code
in the
<a href="https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example"
>GitLab repo</a
>.
</p>
<ul>
<li>
<p>
<code>.gitlab-ci.yml</code> (<a
href="https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example/-/blob/main/.gitlab-ci.yml"
>view at gitlab</a
>)
</p>
<ul>
<li>
the variables <code>AWS_ACCESS_KEY_ID</code>,
<code>AWS_SECRET_ACCESS_KEY</code>, and <code>ECR_HOST</code> are set in
the GitLab UI under "Settings" > "CI/CD" >
"Variables"
</li>
<li>
this uses the
<a href="https://hub.docker.com/r/saltycrane/aws-cli-and-docker"
>saltycrane/aws-cli-and-docker</a
>
Docker image which provides the <code>aws</code> v2 command line tools
and <code>docker</code> in a single image. It is based on
<a href="https://hub.docker.com/r/amazon/aws-cli">amazon/aws-cli</a> and
installs bc, curl, docker, jq, and tar. This idea is from
<a href="https://www.youtube.com/watch?v=jg9sUceyGaQ"
>Valentin's tutorial</a
>.
</li>
</ul>
<pre><code>variables:
DOCKER_HOST: tcp://docker:2375
DOCKER_TLS_CERTDIR: ""
AWS_DEFAULT_REGION: "us-east-1"
CI_APPLICATION_REPOSITORY: "$ECR_HOST/next-aws-ecr-ecs-gitlab-ci-cd-example"
CI_APPLICATION_TAG: "$CI_PIPELINE_IID"
CI_AWS_S3_BUCKET: "next-aws-ecr-ecs-gitlab-ci-cd-example"
CI_AWS_ECS_CLUSTER: "next-aws-ecr-ecs-gitlab-ci-cd-example"
CI_AWS_ECS_SERVICE: "next-aws-ecr-ecs-gitlab-ci-cd-example"
CI_AWS_ECS_TASK_DEFINITION: "next-aws-ecr-ecs-gitlab-ci-cd-example"
NEXT_JS_ASSET_URL: "https://$CI_AWS_S3_BUCKET.s3.amazonaws.com"
stages:
- build
- deploy
build:
stage: build
image: saltycrane/aws-cli-and-docker
services:
- docker:dind
script:
- ./bin/build-and-push-image-to-ecr
- ./bin/upload-assets-to-s3
deploy:
stage: deploy
image: saltycrane/aws-cli-and-docker
services:
- docker:dind
script:
- ./bin/ecs update-task-definition
</code></pre>
</li>
<li>
<p>
<code>Dockerfile</code> (<a
href="https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example/-/blob/main/Dockerfile"
>view at gitlab</a
>)
</p>
<p>
The value of <code>NEXT_JS_ASSET_URL</code> is passed in using the
<code>--build-arg</code> option of the <code>docker build</code> command
run in <code>bin/build-and-push-image-to-ecr</code>. It is used like an
environment variable in the <code>RUN npm run build</code> command below.
In this project it is assigned to <code>assetPrefix</code> in
<code>next.config.js</code>.
</p>
<pre><code>FROM node:14.16-alpine
ARG NEXT_JS_ASSET_URL
ENV NODE_ENV=production
WORKDIR /app
COPY ./package.json ./
COPY ./package-lock.json ./
RUN npm ci
COPY . ./
RUN npm run build
EXPOSE 3000
CMD ["npm", "start"]
</code></pre>
</li>
<li>
<p>
<code>bin/build-and-push-image-to-ecr</code> (<a
href="https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example/-/blob/main/bin/build-and-push-image-to-ecr"
>view at gitlab</a
>)
</p>
<pre><code># log in to the amazon ecr docker registry
aws ecr get-login-password | docker login --username AWS --password-stdin "$ECR_HOST"
# build docker image
docker pull "$CI_APPLICATION_REPOSITORY:latest" || true
docker build --build-arg "NEXT_JS_ASSET_URL=$NEXT_JS_ASSET_URL" --cache-from "$CI_APPLICATION_REPOSITORY:latest" -t "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG" -t "$CI_APPLICATION_REPOSITORY:latest" .
# push image to amazon ecr
docker push "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG"
docker push "$CI_APPLICATION_REPOSITORY:latest"
</code></pre>
</li>
<li>
<p>
<code>bin/upload-assets-to-s3</code> (<a
href="https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example/-/blob/main/bin/upload-assets-to-s3"
>view at gitlab</a
>)
</p>
<pre><code>LOCAL_ASSET_PATH=/tmp/upload-assets
mkdir $LOCAL_ASSET_PATH
# copy the generated assets out of the docker image
docker run --rm --entrypoint tar "$CI_APPLICATION_REPOSITORY:$CI_APPLICATION_TAG" cf - .next | tar xf - -C $LOCAL_ASSET_PATH
# rename .next to _next
mv "$LOCAL_ASSET_PATH/.next" "$LOCAL_ASSET_PATH/_next"
# remove directories that should not be uploaded to S3
rm -rf "$LOCAL_ASSET_PATH/_next/cache"
rm -rf "$LOCAL_ASSET_PATH/_next/server"
# gzip files
find $LOCAL_ASSET_PATH -regex ".*\.\(css\|svg\|js\)$" -exec gzip {} \;
# strip .gz extension off of gzipped files
find $LOCAL_ASSET_PATH -name "*.gz" -exec sh -c 'mv $1 `echo $1 | sed "s/.gz$//"`' - {} \;
# upload gzipped js, css, and svg assets
aws s3 sync --no-progress $LOCAL_ASSET_PATH "s3://$CI_AWS_S3_BUCKET" --cache-control max-age=31536000 --content-encoding gzip --exclude "*" --include "*.js" --include "*.css" --include "*.svg"
# upload non-gzipped assets
aws s3 sync --no-progress $LOCAL_ASSET_PATH "s3://$CI_AWS_S3_BUCKET" --cache-control max-age=31536000 --exclude "*.js" --exclude "*.css" --exclude "*.svg" --exclude "*.map"
</code></pre>
</li>
<li>
<p>
<code>bin/ecs</code> (<a
href="https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example/-/blob/main/bin/ecs"
>view full file</a
>) (This file was copied from the
<a href=""><code>gitlab-org</code> repo</a>)
</p>
<pre><code>#!/bin/bash -e
update_task_definition() {
local -A register_task_def_args=( \
['task-role-arn']='taskRoleArn' \
['execution-role-arn']='executionRoleArn' \
['network-mode']='networkMode' \
['cpu']='cpu' \
['memory']='memory' \
['pid-mode']='pidMode' \
['ipc-mode']='ipcMode' \
['proxy-configuration']='proxyConfiguration' \
['volumes']='volumes' \
['placement-constraints']='placementConstraints' \
['requires-compatibilities']='requiresCompatibilities' \
['inference-accelerators']='inferenceAccelerators' \
)
image_repository=$CI_APPLICATION_REPOSITORY
image_tag=$CI_APPLICATION_TAG
new_image_name="${image_repository}:${image_tag}"
register_task_definition_from_remote
new_task_definition=$(aws ecs register-task-definition "${args[@]}")
new_task_revision=$(read_task "$new_task_definition" 'revision')
new_task_definition_family=$(read_task "$new_task_definition" 'family')
# Making sure that we at least have one running task (even if desiredCount gets updated again with new task definition below)
service_task_count=$(aws ecs describe-services --cluster "$CI_AWS_ECS_CLUSTER" --services "$CI_AWS_ECS_SERVICE" --query "services[0].desiredCount")
if [[ $service_task_count == 0 ]]; then
aws ecs update-service --cluster "$CI_AWS_ECS_CLUSTER" --service "$CI_AWS_ECS_SERVICE" --desired-count 1
fi
# Update ECS service with newly created task defintion revision.
aws ecs update-service \
--cluster "$CI_AWS_ECS_CLUSTER" \
--service "$CI_AWS_ECS_SERVICE" \
--task-definition "$new_task_definition_family":"$new_task_revision"
return 0
}
read_task() {
val=$(echo "$1" | jq -r ".taskDefinition.$2")
if [ "$val" == "null" ];then
val=$(echo "$1" | jq -r ".$2")
fi
if [ "$val" != "null" ];then
echo -n "${val}"
fi
}
register_task_definition_from_remote() {
task=$(aws ecs describe-task-definition --task-definition "$CI_AWS_ECS_TASK_DEFINITION")
current_container_definitions=$(read_task "$task" 'containerDefinitions')
new_container_definitions=$(echo "$current_container_definitions" | jq --arg val "$new_image_name" '.[0].image = $val')
args+=("--family" "${CI_AWS_ECS_TASK_DEFINITION}")
args+=("--container-definitions" "${new_container_definitions}")
for option in "${!register_task_def_args[@]}"; do
value=$(read_task "$task" "${register_task_def_args[$option]}")
if [ -n "$value" ];then
args+=("--${option}" "${value}")
fi
done
}
update_task_definition
</code></pre>
</li>
</ul>
<h4 id="usage---set-up-aws-resources">Usage - set up AWS resources</h4>
<p>
Below are the minimum steps I needed to create the required AWS services for
my example. I use the AWS region <code>"us-east-1"</code>. For info
about creating some of these services via the command line, see my
<a href="/blog/2021/03/amazon-ecs-notes/">Amazon ECS notes</a>.
</p>
<p><strong>Create an ECR repository</strong></p>
<ul>
<li>
create a private ECR repository here:
<a href="https://console.aws.amazon.com/ecr/repositories?region=us-east-1"
>https://console.aws.amazon.com/ecr/repositories?region=us-east-1</a
>
</li>
<li>name the repository "next-aws-ecr-ecs-gitlab-ci-cd-example"</li>
<li>
leave "Tag immutability" disabled to allow the "latest"
tag to be overwritten
</li>
</ul>
<p><strong>Create an ECS Fargate cluster</strong></p>
<ul>
<li>
click "Create Cluster" here:
<a
href="https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters"
>https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters</a
>
</li>
<li>choose "Networking only" (Fargate)</li>
<li>name the cluster "next-aws-ecr-ecs-gitlab-ci-cd-example"</li>
<li>check the "Create VPC" checkbox</li>
<li>click "Create"</li>
</ul>
<p><strong>Create an ECS task definition</strong></p>
<ul>
<li>
click "Create new Task Definition" here:
<a
href="https://console.aws.amazon.com/ecs/home?region=us-east-1#/taskDefinitions"
>https://console.aws.amazon.com/ecs/home?region=us-east-1#/taskDefinitions</a
>
</li>
<li>select "FARGATE" and click "Next step"</li>
<li>
configure task
<ul>
<li>
for "Task Definition Name" enter
"next-aws-ecr-ecs-gitlab-ci-cd-example"
</li>
<li>for "Task Role" select "None"</li>
<li>
for "Task execution role" select "Create new role"
</li>
<li>for "Task memory" select "0.5GB"</li>
<li>for "Task CPU" select "0.25 vCPU"</li>
<li>
click "Add container"
<ul>
<li>
for "Container Name" enter
"next-aws-ecr-ecs-gitlab-ci-cd-example"
</li>
<li>
for "Image" enter "asdf" (this will be updated
by the gitlab ci/cd job)
</li>
<li>leave "Private repository authentication" unchecked</li>
<li>for "Port mappings" enter "3000"</li>
<li>click "Add"</li>
</ul>
</li>
<li>click "Create"</li>
</ul>
</li>
</ul>
<p><strong>Create an ECS service</strong></p>
<ul>
<li>
click "Create" here:
<a
href="https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/next-aws-ecr-ecs-gitlab-ci-cd-example/services"
>https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/next-aws-ecr-ecs-gitlab-ci-cd-example/services</a
>
</li>
<li>
configure service
<ul>
<li>for "Launch type" select "FARGATE"</li>
<li>
for "Task Definition" enter
"next-aws-ecr-ecs-gitlab-ci-cd-example"
</li>
<li>
for "Cluster" select
"next-aws-ecr-ecs-gitlab-ci-cd-example"
</li>
<li>
for "Service name" enter
"next-aws-ecr-ecs-gitlab-ci-cd-example"
</li>
<li>for "Number of tasks" enter 1</li>
<li>for "Deployment type" select "Rolling update"</li>
<li>click "Next step"</li>
</ul>
</li>
<li>
configure network
<ul>
<li>
select the appropriate "Cluster VPC" and two
"Subnets"
</li>
<li>click "Next step"</li>
</ul>
</li>
<li>
set Auto Scaling
<ul>
<li>click "Next step"</li>
</ul>
</li>
<li>
review
<ul>
<li>click "Create Service"</li>
</ul>
</li>
</ul>
<p><strong>Open port 3000</strong></p>
<ul>
<li>
on the ECS service page
<a
href="https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/next-aws-ecr-ecs-gitlab-ci-cd-example/services/next-aws-ecr-ecs-gitlab-ci-cd-example/details"
>https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/next-aws-ecr-ecs-gitlab-ci-cd-example/services/next-aws-ecr-ecs-gitlab-ci-cd-example/details</a
>
under "Network Access", next to "Security groups", click
the link to the security group
</li>
<li>click "Actions" then click "Edit inbound rules"</li>
<li>click "Add rule"</li>
<li>for "Port range" enter "3000"</li>
<li>for "Source" select "0.0.0.0/0"</li>
<li>click "Save rules"</li>
</ul>
<p><strong>Create a S3 bucket</strong></p>
<ul>
<li>
click "Create bucket" here:
<a href="https://s3.console.aws.amazon.com/s3/home?region=us-east-1"
>https://s3.console.aws.amazon.com/s3/home?region=us-east-1</a
>
</li>
<li>
for "Bucket name" enter
"next-aws-ecr-ecs-gitlab-ci-cd-example"
</li>
<li>uncheck "Block all public access"</li>
<li>
check the "I acknowledge that the current settings might result in this
bucket and the objects within becoming public" checkbox
</li>
<li>click "Create bucket"</li>
</ul>
<p><strong>Update permissions for S3 bucket</strong></p>
<ul>
<li>
go to Permissions (<a
href="https://s3.console.aws.amazon.com/s3/buckets/next-aws-ecr-ecs-gitlab-ci-cd-example?region=us-east-1&tab=permissions"
>https://s3.console.aws.amazon.com/s3/buckets/next-aws-ecr-ecs-gitlab-ci-cd-example?region=us-east-1&tab=permissions</a
>) and under "Bucket policy", click "Edit"
</li>
<li>
enter:
<pre><code>{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::next-aws-ecr-ecs-gitlab-ci-cd-example/*"
}
]
}
</code></pre>
</li>
<li>click "Save changes"</li>
</ul>
<p><strong>Create an IAM user</strong></p>
<ul>
<li>
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html#create-an-iam-user"
>create an IAM user</a
>. The user must have at least ECR, ECS, and S3 permissions.
</li>
<li>
take note of the <code>AWS_ACCESS_KEY_ID</code> and
<code>AWS_SECRET_ACCESS_KEY</code>
</li>
</ul>
<h4 id="usage---run-the-cicd-pipeline">Usage - run the CI/CD pipeline</h4>
<p>
<strong>Fork the example gitlab repo and configure CI/CD variables</strong>
</p>
<ul>
<li>
fork
<a
href="https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example"
>https://gitlab.com/saltycrane/next-aws-ecr-ecs-gitlab-ci-cd-example</a
>
</li>
<li>
go to "Settings" > "CI/CD" > "Variables"
and add the following variables. You can choose to "protect" and
"mask" all of them.
<ul>
<li><code>AWS_ACCESS_KEY_ID</code></li>
<li><code>AWS_SECRET_ACCESS_KEY</code></li>
<li>
<code>ECR_HOST</code> (This is the part of the ECR repository URI before
the <code>/</code>. It looks something like
<code>XXXXXXXXXXXX.dkr.ecr.us-east-1.amazonaws.com</code>)
</li>
</ul>
</li>
</ul>
<p>
<strong>Edit variables in <code>.gitlab-ci.yml</code></strong>
</p>
<p>
If you used names other than
"next-aws-ecr-ecs-gitlab-ci-cd-example", edit the variables in
<code>.gitlab-ci.yml</code>.
</p>
<p><strong>Test it</strong></p>
<ul>
<li>clone the repo and push a commit</li>
<li>
see the pipeline running under "CI/CD" > "Pipelines"
</li>
<li>
go to the cluster tasks page:
<a
href="https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/next-aws-ecr-ecs-gitlab-ci-cd-example/tasks"
>https://us-east-1.console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/next-aws-ecr-ecs-gitlab-ci-cd-example/tasks</a
>
</li>
<li>click on the task and copy the "Public IP"</li>
<li>
enter the public IP followed by <code>:3000</code> in the browser (Note: the
IP address changes for every <code>git push</code>. A
<a href="https://aws.amazon.com/elasticloadbalancing/">load balancer</a>
should probably be used, but I didn't do that.)
</li>
</ul>
<h4 id="references-buildpush">References (build/push)</h4>
<ul>
<li>
<a href="https://www.youtube.com/watch?v=jg9sUceyGaQ"
>https://www.youtube.com/watch?v=jg9sUceyGaQ</a
>
</li>
<li>
<a
href="https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker"
>https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#docker</a
>
</li>
<li>
<a
href="https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#using-docker-caching"
>https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#using-docker-caching</a
>
</li>
</ul>
<h4 id="references-deploy">References (deploy)</h4>
<ul>
<li>
<a
href="https://docs.gitlab.com/ee/ci/cloud_deployment/#deploy-your-application-to-the-aws-elastic-container-service-ecs"
>https://docs.gitlab.com/ee/ci/cloud_deployment/#deploy-your-application-to-the-aws-elastic-container-service-ecs</a
>
</li>
<li>
<a
href="https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/AWS/Deploy-ECS.gitlab-ci.yml"
>https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/AWS/Deploy-ECS.gitlab-ci.yml</a
>
</li>
<li>
<a
href="https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy/ECS.gitlab-ci.yml"
>https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/templates/Jobs/Deploy/ECS.gitlab-ci.yml</a
>
</li>
<li>
<a
href="https://gitlab.com/gitlab-org/cloud-deploy/-/blob/master/aws/ecs/Dockerfile"
>https://gitlab.com/gitlab-org/cloud-deploy/-/blob/master/aws/ecs/Dockerfile</a
>
</li>
<li>
<a
href="https://gitlab.com/gitlab-org/cloud-deploy/-/blob/master/aws/src/bin/ecs"
>https://gitlab.com/gitlab-org/cloud-deploy/-/blob/master/aws/src/bin/ecs</a
>
</li>
</ul>
GitLab CI/CD hello world examples
2021-03-25T09:06:02-07:00https://www.saltycrane.com/blog/2021/03/gitlab-cicd-hello-world-examples/<p>
These are my notes for creating "hello world" and "docker hello
world"
<a href="https://docs.gitlab.com/ee/ci/README.html">GitLab CI/CD</a>
pipelines.
</p>
<h4 id="gitlab-cicd-hello-world-gitlab-repo">
GitLab CI/CD hello world
<small
>(<a href="https://gitlab.com/saltycrane/gitlab-ci-cd-hello-world"
>gitlab repo</a
>)</small
>
</h4>
<ul>
<li>
<p>
create git repo, push it to gitlab, and set origin. Replace
<code>saltycrane</code> with your username.
</p>
<pre><code>mkdir gitlab-ci-cd-hello-world
cd gitlab-ci-cd-hello-world
git init
touch .gitignore
git add .
git commit -m 'first commit'
git push --set-upstream git@gitlab.com:saltycrane/gitlab-ci-cd-hello-world.git --all
git remote add origin git@gitlab.com:saltycrane/gitlab-ci-cd-hello-world.git
</code></pre>
</li>
<li>
<p>add a <code>.gitlab-ci.yml</code> file</p>
<pre><code>build-hello:
script:
- echo "hello world"
</code></pre>
</li>
<li>
<p>commit and push</p>
<pre><code>git add .
git commit -m 'add ci/cd config'
git push origin
</code></pre>
</li>
<li>
<p>
see the pipeline run (replace <code>saltycrane</code> with your username):
<a
href="https://gitlab.com/saltycrane/gitlab-ci-cd-hello-world/-/pipelines"
>https://gitlab.com/saltycrane/gitlab-ci-cd-hello-world/-/pipelines</a
>
</p>
</li>
</ul>
<h4 id="gitlab-cicd-docker-hello-world-gitlab-repo">
GitLab CI/CD Docker hello world
<small
>(<a href="https://gitlab.com/saltycrane/gitlab-ci-cd-docker-hello-world"
>gitlab repo</a
>)</small
>
</h4>
<ul>
<li>
<p>
create git repo, push it to gitlab, and set origin. Replace
<code>saltycrane</code> with your username.
</p>
<pre><code>mkdir gitlab-ci-cd-docker-hello-world
cd gitlab-ci-cd-docker-hello-world
git init
touch .gitignore
git add .
git commit -m 'first commit'
git push --set-upstream git@gitlab.com:saltycrane/gitlab-ci-cd-docker-hello-world.git --all
git remote add origin git@gitlab.com:saltycrane/gitlab-ci-cd-docker-hello-world.git
</code></pre>
</li>
<li>
<p>add a <code>Dockerfile</code> file</p>
<pre><code>FROM alpine
RUN echo "hello"
</code></pre>
</li>
<li>
<p>add a <code>.gitlab-ci.yml</code> file</p>
<pre><code>variables:
DOCKER_TLS_CERTDIR: "/certs"
build-docker:
image: docker:latest
services:
- docker:dind
script:
- docker build -t hello .
</code></pre>
</li>
<li>
<p>commit and push</p>
<pre><code>git add .
git commit -m 'add Dockerfile and ci/cd config'
git push origin
</code></pre>
</li>
<li>
<p>
see the pipeline run (replace <code>saltycrane</code> with your username):
<a
href="https://gitlab.com/saltycrane/gitlab-ci-cd-docker-hello-world/-/pipelines"
>https://gitlab.com/saltycrane/gitlab-ci-cd-docker-hello-world/-/pipelines</a
>
</p>
</li>
</ul>
Amazon ECS notes
2021-03-19T10:04:07-07:00https://www.saltycrane.com/blog/2021/03/amazon-ecs-notes/<p>
These are my notes for creating a Docker image, pushing it to
<a href="https://aws.amazon.com/ecr/">Amazon ECR</a> (Elastic Container
Registry), and deploying it to
<a href="https://aws.amazon.com/ecs/">Amazon ECS</a> (Elastic Container
Service) using
<a href="https://aws.amazon.com/fargate/">AWS Fargate</a> (serverless for
containers) using command line tools.
</p>
<h4 id="create-docker-image-on-local-machine">
Create docker image on local machine
</h4>
<ul>
<li>
<p>install docker (macOS)</p>
<pre><code>brew install homebrew/cask/docker
</code></pre>
</li>
<li>
<p>create directory</p>
<pre><code>mkdir /tmp/my-project
cd /tmp/my-project
</code></pre>
</li>
<li>
<p>create <code>/tmp/my-project/Dockerfile</code>:</p>
<pre><code>FROM python:3.9-alpine3.13
WORKDIR /app
RUN echo 'Hello' > ./index.html
EXPOSE 80
CMD ["python", "-m", "http.server", "80"]
</code></pre>
</li>
<li>
<p>create Docker image</p>
<pre><code>docker build -t my-image .
</code></pre>
</li>
<li>
<p>test running the Docker image locally</p>
<pre><code>docker run -p 80:80 my-image
</code></pre>
</li>
<li>
<p>
go to <a href="http://localhost">http://localhost</a> in the browser and
see the text "Hello"
</p>
</li>
</ul>
<h4 id="install-and-configure-aws-command-line-tools">
Install and configure AWS command line tools
</h4>
<ul>
<li>
<p>install AWS command line tools</p>
<pre><code>brew install awscli
</code></pre>
</li>
<li>
<p>
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/get-set-up-for-amazon-ecs.html#create-an-iam-user"
>create an IAM user</a
>
</p>
</li>
<li>
<p>
run
<a
href="https://docs.aws.amazon.com/cli/latest/reference/configure/index.html"
><code>aws configure</code></a
>
and enter:
</p>
<ul>
<li>AWS Access Key ID</li>
<li>AWS Secret Access Key</li>
</ul>
<p>This creates the file <code>~/.aws/credentials</code></p>
</li>
</ul>
<h4 id="create-ecr-repository-and-push-image-to-it">
Create ECR repository and push image to it
</h4>
<p>
From
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html#use-ecr"
>https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html#use-ecr</a
>
</p>
<ul>
<li>
<p>
create an Amazon ECR repository using
<a
href="https://docs.aws.amazon.com/cli/latest/reference/ecr/create-repository.html"
><code>aws ecr create-repository</code></a
>
</p>
<pre><code>aws ecr create-repository --repository-name my-repository --region us-east-1
</code></pre>
<p>output:</p>
<pre><code>{
"repository": {
"repositoryArn": "arn:aws:ecr:us-east-1:AAAAAAAAAAAA:repository/my-repository",
"registryId": "AAAAAAAAAAAA",
"repositoryName": "my-repository",
"repositoryUri": "AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com/my-repository",
"createdAt": "2021-03-17T10:48:18-07:00",
"imageTagMutability": "MUTABLE",
"imageScanningConfiguration": {
"scanOnPush": false
},
"encryptionConfiguration": {
"encryptionType": "AES256"
}
}
}
</code></pre>
<p>
Take note of the "registryId" and use it in place of
"AAAAAAAAAAAA" below.
</p>
</li>
</ul>
<ul>
<li>
<p>tag the docker image with the <code>repositoryUri</code></p>
<pre><code>docker tag my-image AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com/my-repository
</code></pre>
</li>
<li>
<p>
log in to the Amazon ECR registry using
<a
href="https://docs.aws.amazon.com/cli/latest/reference/ecr/get-login-password.html"
><code>aws ecr get-login-password</code></a
>
</p>
<pre><code>aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com
</code></pre>
</li>
<li>
<p>push the docker image to the Amazon ECR repository</p>
<pre><code>docker push AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com/my-repository
</code></pre>
</li>
<li>
<p>
see the image in AWS console
<a href="https://console.aws.amazon.com/ecr/repositories?region=us-east-1"
>https://console.aws.amazon.com/ecr/repositories?region=us-east-1</a
>
</p>
</li>
</ul>
<h4 id="install-ecs-command-line-tools">Install ECS command line tools</h4>
<ul>
<li>
install <code>ecs-cli</code>. Note there is <code>ecs-cli</code> in addition
to <code>aws ecs</code> tools. The reason is probably similar to why some
services are named
<a href="https://docs.aws.amazon.com/"
>"Amazon Service" and some are named "AWS Service"</a
>. (It seems like <code>ecs-cli</code> provides higher level commands.)
<pre><code>brew install amazon-ecs-cli
</code></pre>
</li>
</ul>
<h4 id="create-amazon-ecs-fargate-cluster">
Create Amazon ECS Fargate cluster
</h4>
<p>
From
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html"
>https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html</a
>
</p>
<ul>
<li>
create a cluster using
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-up.html"
><code>ecs-cli up</code></a
>
<pre><code>ecs-cli up --cluster my-cluster --launch-type FARGATE --region us-east-1
</code></pre>
output:
<pre><code>INFO[0001] Created cluster cluster=my-cluster region=us-east-1
INFO[0002] Waiting for your cluster resources to be created...
INFO[0002] Cloudformation stack status stackStatus=CREATE_IN_PROGRESS
VPC created: vpc-BBBBBBBBBBBBBBBBB
Subnet created: subnet-CCCCCCCCCCCCCCCCC
Subnet created: subnet-DDDDDDDDDDDDDDDDD
Cluster creation succeeded.
</code></pre>
Take note of the VPC (virtual private cloud), and two subnet IDs to use
later. See the cluster in the AWS console UI:
<a href="https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters"
>https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters</a
>
</li>
</ul>
<h4 id="gather-parameters-required-to-deploy-to-ecs-cluster">
Gather parameters required to deploy to ECS cluster
</h4>
<p>
From
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html"
>https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html</a
>
</p>
<h5 id="create-task-execution-iam-role">Create task execution IAM role</h5>
<ul>
<li>
<p>
create a file <code>/tmp/my-project/task-execution-assume-role.json</code>
</p>
<pre><code>{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"Service": "ecs-tasks.amazonaws.com"
},
"Action": "sts:AssumeRole"
}
]
}
</code></pre>
</li>
<li>
<p>
create the task execution role using
<a
href="https://docs.aws.amazon.com/cli/latest/reference/iam/create-role.html"
><code>aws iam create-role</code></a
>
</p>
<pre><code>aws iam create-role --role-name my-task-execution-role --assume-role-policy-document file:///tmp/my-project/task-execution-assume-role.json --region us-east-1
</code></pre>
</li>
<li>
<p>
attach the task execution role policy using
<a
href="https://docs.aws.amazon.com/cli/latest/reference/iam/attach-role-policy.html"
><code>aws iam attach-role-policy</code></a
>
</p>
<pre><code>aws iam attach-role-policy --role-name my-task-execution-role --policy-arn arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy --region us-east-1
</code></pre>
</li>
</ul>
<h5 id="get-security-group-id">Get security group ID</h5>
<ul>
<li>
<p>
get the default security group ID for the virtual private cloud (VPC)
created when creating the ECS cluster using
<a
href="https://docs.aws.amazon.com/cli/latest/reference/ec2/describe-security-groups.html"
><code>aws ec2 describe-security-groups</code></a
>. Replace "vpc-BBBBBBBBBBBBBBBBB" with your VPC ID
</p>
<pre><code>aws ec2 describe-security-groups --filters Name=vpc-id,Values=vpc-BBBBBBBBBBBBBBBBB --region us-east-1
</code></pre>
<p>output:</p>
<pre><code>{
"SecurityGroups": [
{
"Description": "default VPC security group",
"GroupName": "default",
"IpPermissions": [
{
"IpProtocol": "-1",
"IpRanges": [],
"Ipv6Ranges": [],
"PrefixListIds": [],
"UserIdGroupPairs": [
{
"GroupId": "sg-EEEEEEEEEEEEEEEEE",
"UserId": "AAAAAAAAAAAA"
}
]
}
],
"OwnerId": "AAAAAAAAAAAA",
"GroupId": "sg-EEEEEEEEEEEEEEEEE",
"IpPermissionsEgress": [
{
"IpProtocol": "-1",
"IpRanges": [
{
"CidrIp": "0.0.0.0/0"
}
],
"Ipv6Ranges": [],
"PrefixListIds": [],
"UserIdGroupPairs": []
}
],
"VpcId": "vpc-BBBBBBBBBBBBBBBBB"
}
]
}
</code></pre>
<p>Take note of the "GroupId" to be used later</p>
</li>
</ul>
<h4 id="deploy-to-amazon-ecs-cluster">Deploy to Amazon ECS cluster</h4>
<p>
From
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html"
>https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html</a
>
</p>
<ul>
<li>
<p>
create <code>/tmp/my-project/ecs-params.yml</code> replacing
"subnet-CCCCCCCCCCCCCCCCC",
"subnet-DDDDDDDDDDDDDDDDD", and "sg-EEEEEEEEEEEEEEEEE"
with appropriate IDs from above.
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-ecsparams.html"
>ECS Parameters docs</a
>
</p>
<pre><code>version: 1
task_definition:
task_execution_role: my-task-execution-role
ecs_network_mode: awsvpc
task_size:
mem_limit: 0.5GB
cpu_limit: 256
run_params:
network_configuration:
awsvpc_configuration:
subnets:
- "subnet-CCCCCCCCCCCCCCCCC"
- "subnet-DDDDDDDDDDDDDDDDD"
security_groups:
- "sg-EEEEEEEEEEEEEEEEE"
assign_public_ip: ENABLED
</code></pre>
</li>
<li>
<p>
create <code>/tmp/my-project/docker-compose.yml</code> replacing
AAAAAAAAAAAA with the registryId:
</p>
<pre><code>version: '3'
services:
web:
image: 'AAAAAAAAAAAA.dkr.ecr.us-east-1.amazonaws.com/my-repository'
ports:
- '80:80'
</code></pre>
</li>
<li>
<p>
deploy to the ECS cluster using
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-service-up.html"
><code>ecs-cli compose service up</code></a
>. This creates a task definition and service. This uses the
<code>docker-compose.yml</code> file in the current directory.
</p>
<pre><code>ecs-cli compose --cluster my-cluster --project-name my-project --ecs-params ecs-params.yml --region us-east-1 service up --launch-type FARGATE
</code></pre>
<p>
see the service in the web UI:
<a
href="https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/my-cluster/services"
>https://console.aws.amazon.com/ecs/home?region=us-east-1#/clusters/my-cluster/services</a
>
</p>
</li>
</ul>
<h4 id="hit-the-server-in-the-browser">Hit the server in the browser</h4>
<ul>
<li>
<p>
configure security group to allow inbound access on port 80 using
<a
href="https://docs.aws.amazon.com/cli/latest/reference/ec2/authorize-security-group-ingress.html"
><code>aws ec2 authorize-security-group-ingress</code></a
>
</p>
<pre><code>aws ec2 authorize-security-group-ingress --group-id sg-EEEEEEEEEEEEEEEEE --protocol tcp --port 80 --cidr 0.0.0.0/0 --region us-east-1
</code></pre>
</li>
<li>
<p>
get the IP address using
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-service-ps.html"
><code>ecs-cli compose service ps</code></a
>
</p>
<pre><code>ecs-cli compose --cluster my-cluster --project-name my-project --region us-east-1 service ps
</code></pre>
<p>output:</p>
<pre><code>Name State Ports TaskDefinition Health
my-cluster/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx/web RUNNING FF.FF.FF.FF:80->80/tcp my-project:1 UNKNOWN
</code></pre>
<p>Take note of the IP address under "Ports"</p>
</li>
<li>
<p>
visit in the browser:
<a href="http://FF.FF.FF.FF">http://FF.FF.FF.FF</a> replacing
"FF.FF.FF.FF" with your IP address
</p>
</li>
</ul>
<h4 id="destroy">Destroy</h4>
<ul>
<li>
<p>
delete the ECS service using
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-compose-service-rm.html"
><code>ecs-cli compose service down</code></a
>
</p>
<pre><code>ecs-cli compose --cluster my-cluster --project-name my-project --region us-east-1 service down
</code></pre>
</li>
<li>
<p>
delete the ECS cluster using
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/cmd-ecs-cli-down.html"
><code>ecs-cli down</code></a
>
</p>
<pre><code>ecs-cli down --force --cluster my-cluster --region us-east-1
</code></pre>
</li>
<li>
<p>
delete the ECR repository using
<a
href="https://docs.aws.amazon.com/cli/latest/reference/ecr/delete-repository.html"
><code>aws ecr delete-repository</code></a
>
</p>
<pre><code>aws ecr delete-repository --repository-name my-repository --region us-east-1 --force
</code></pre>
</li>
</ul>
<h4 id="references">References</h4>
<ul>
<li>
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html"
>https://docs.aws.amazon.com/AmazonECS/latest/developerguide/docker-basics.html</a
>
</li>
<li>
<a
href="https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html"
>https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-cli-tutorial-fargate.html</a
>
</li>
</ul>
Creating a GraphQL API with Python, Graphene, and Postgres
2019-02-20T21:44:39-08:00https://www.saltycrane.com/blog/2019/02/creating-graphql-api-python-graphene-and-postgres/<p>
Here are my notes on creating a <a href="https://graphql.org/">GraphQL</a> API
with Python, Django, <a href="https://graphene-python.org/">Graphene</a>, and
Postgres. I learned almost everything from the
<a href="https://www.howtographql.com/graphql-python/0-introduction/">
excellent GraphQL Python tutorial at howtographql.com</a
>. Optimistically, I'll write a series of posts about how to make a React
Native app that no one will use. Realistically, I won't.
</p>
<h4 id="contents">Contents</h4>
<ul>
<li><a href="#install-python">Install Python</a></li>
<li>
<a href="#create-directory">Create project directory and virtualenv</a>
</li>
<li><a href="#django">Install Django and create a Django project</a></li>
<li><a href="#postgres-docker">Run Postgres in Docker</a></li>
<li><a href="#create-database">Create a database</a></li>
<li><a href="#django-postgres">Configure Django to use Postgres</a></li>
<li><a href="#graphene">Install and configure Graphene</a></li>
<li><a href="#django-app">Create a new Django app and add a model</a></li>
<li><a href="#graphql">GraphQL all the things</a></li>
<li><a href="#try">Try it using the GraphiQL explorer</a></li>
<li><a href="#references">References / See also</a></li>
</ul>
<h4 id="install-python">Install Python 3.7.2</h4>
<pre class="console">
$ brew install python
</pre>
<h4 id="create-directory">Create project directory and virtualenv</h4>
<pre class="console">
$ # make project directory
$ mkdir travelog-api
$ cd travelog-api
$ # make virtualenv
$ python3 -m venv venv
$ # activate virtualenv
$ source venv/bin/activate
$ # upgrade pip
$ pip install --upgrade pip
</pre>
<h4 id="django">Install Django and create a Django project</h4>
<p>
(Still in the <code>travelog-api</code> directory with virtualenv activated)
</p>
<ul>
<li>
Install Django 2.1.7:
<pre class="console">$ pip install Django </pre>
</li>
<li>
Create Django project:
<pre class="console">$ django-admin startproject travelog_api ./</pre>
</li>
<li>
Run migrations and run the server:
<pre class="console">
$ ./manage.py migrate
$ ./manage.py runserver
$ # go to http://localhost:8000 in the browser</pre
>
</li>
</ul>
<h4 id="postgres-docker">Run Postgres in Docker</h4>
<p>
(Still in the <code>travelog-api</code> directory with virtualenv activated)
</p>
<ul>
<li>
<a href="https://docs.docker.com/docker-for-mac/install/"
>Install Docker for Mac</a
>
</li>
<li>
Create a new file, <code>travelog-api/docker-compose.yml</code>:
<pre>
version: "3.7"
services:
db:
image: "postgres:11.2"
container_name: "travelog_postgres1"
ports:
- "54321:5432"
volumes:
- postgres_data1:/var/lib/postgresql/data
volumes:
postgres_data1:
name: travelog_postgres_data1</pre
>
</li>
<li>
Start Postgres
<pre class="console">
$ docker-compose up -d
$ docker-compose logs</pre
>
</li>
</ul>
<h4 id="create-database">Create a database</h4>
<ul>
<li>
Start psql:
<pre class="console">
$ docker exec -it travelog_postgres1 psql -U postgres</pre
>
</li>
<li>
Create a database (be sure to include the semicolon):
<pre>postgres=# create database travelog;</pre>
</li>
<li>
Create user:
<pre>
postgres=# create user traveloguser with password 'mypassword';
postgres=# grant all privileges on database travelog to traveloguser;</pre
>
</li>
<li>
Exit psql:
<pre>postgres=# \q</pre>
</li>
</ul>
<h4 id="django-postgres">Configure Django to use Postgres</h4>
<p>
(Still in the <code>travelog-api</code> directory with virtualenv activated)
</p>
<ul>
<li>
Install <code>psycopg2</code> 2.7.7:
<pre class="console">$ pip install psycopg2-binary</pre>
</li>
<li>
Edit <code>travelog-api/travelog_api/settings.py</code>:
<pre class="python">
DATABASES = {
"default": {
"ENGINE": "django.db.backends.postgresql_psycopg2",
"NAME": "travelog",
"USER": "traveloguser",
"PASSWORD": "mypassword,
"HOST": "localhost",
"PORT": "54321",
}
}</pre
>
</li>
<li>
Run database migrations and run the server:
<pre class="console">
$ ./manage.py migrate
$ ./manage.py runserver
$ # go to http://localhost:8000 in the browser</pre
>
</li>
</ul>
<h4 id="graphene">Install and configure Graphene</h4>
<p>
(Still in the <code>travelog-api</code> directory with virtualenv activated)
</p>
<ul>
<li>
Install graphene-django 2.2.0
<pre class="console">$ pip install graphene-django</pre>
</li>
<li>
Edit the <code>INSTALLED_APPS</code> setting in
<code>travelog-api/travelog_api/settings.py</code>:
<pre class="python">
INSTALLED_APPS = (
# After the default packages
"graphene_django",
)</pre
>
</li>
</ul>
<h4 id="django-app">Create a new Django app and add a model</h4>
<p>
(Still in the <code>travelog-api</code> directory with virtualenv activated)
</p>
<ul>
<li>
Create a new Django app:
<pre class="console">$ ./manage.py startapp geo</pre>
</li>
<li>
Edit <code>travelog_api/settings.py</code>:
<pre class="python">
INSTALLED_APPS = (
# After the default packages
'graphene_django',
'geo',
)</pre
>
</li>
<li>
Edit <code>travelog-api/geo/models.py</code>:
<pre class="python">
from django.db.models import DateTimeField, FloatField, Model, TextField
class Location(Model):
created_at = DateTimeField(auto_now_add=True)
lat = FloatField()
lon = FloatField()
name = TextField(blank=True)
updated_at = DateTimeField(auto_now=True)</pre
>
</li>
<li>
Make and run migrations:
<pre class="console">
$ ./manage.py makemigrations
$ ./manage.py migrate</pre
>
</li>
</ul>
<h4 id="graphql">GraphQL all the things</h4>
<p>
(Still in the <code>travelog-api</code> directory with virtualenv activated)
</p>
<ul>
<li>
Create a new file <code>travelog-api/geo/schema.py</code>:
<pre class="python">
import graphene
from graphene_django.types import DjangoObjectType
from .models import Location
class LocationType(DjangoObjectType):
class Meta:
model = Location
class Query(object):
all_locations = graphene.List(LocationType)
def resolve_all_locations(self, info, **kwargs):
return Location.objects.all()
class CreateLocation(graphene.Mutation):
location = graphene.Field(LocationType)
class Arguments:
lat = graphene.Float()
lon = graphene.Float()
name = graphene.String()
def mutate(self, info, lat, lon, name):
loc = Location(lat=lat, lon=lon, name=name)
loc.save()
return CreateLocation(location=loc)
class Mutation(graphene.ObjectType):
create_location = CreateLocation.Field()</pre
>
</li>
<li>
Create a new file <code>travelog-api/travelog_api/schema.py</code>:
<pre class="python">
import graphene
import geo.schema
class Query(geo.schema.Query, graphene.ObjectType):
pass
class Mutation(geo.schema.Mutation, graphene.ObjectType):
pass
schema = graphene.Schema(query=Query, mutation=Mutation)</pre
>
</li>
<li>
Edit <code>travelog-api/travelog_api/urls.py</code>:
<pre class="python">
from django.contrib import admin
from django.urls import path
from graphene_django.views import GraphQLView
from .schema import schema
urlpatterns = [
path("admin/", admin.site.urls),
path("graphql/", GraphQLView.as_view(graphiql=True, schema=schema)),
]</pre
>
</li>
</ul>
<h4 id="try">Try it using the GraphiQL explorer</h4>
<p>
(Still in the <code>travelog-api</code> directory with virtualenv activated)
</p>
<ul>
<li>
Run the server:
<pre class="console">$ ./manage.py runserver</pre>
</li>
<li>
Go to
<a href="http://localhost:8000/graphql/"> http://localhost:8000/graphql/</a>
and you should see the
<a href="https://github.com/graphql/graphiql">GraphiQL</a> interactive
GraphQL explorer.
</li>
<li>
Create a location. Enter this mutation in the left pane and hit CTRL+ENTER:
<pre>
mutation {
createLocation(name: "my first location", lat: 1, lon: 2) {
location {
id
}
}
}</pre
>
See the response:
<pre>
{
"data": {
"createLocation": {
"location": {
"id": "1"
}
}
}
}</pre
>
</li>
<li>
Query all locations:
<pre>
query {
allLocations {
createdAt
id
lat
lon
name
}
}</pre
>
See the response:
<pre>
{
"data": {
"allLocations": [
{
"createdAt": "2019-02-22T06:39:08.512197+00:00",
"id": "1",
"lat": 1,
"lon": 2,
"name": "my first location"
}
]
}
}</pre
>
</li>
</ul>
<h4 id="references">References / See also</h4>
<ul>
<li>
<a href="https://www.howtographql.com/graphql-python/0-introduction/">
graphql-python Tutorial - How to GraphQL
</a>
</li>
<li>
<a href="https://docs.graphene-python.org/en/latest/"
>Graphene documentation</a
>
</li>
<li>
<a href="http://docs.graphene-python.org/projects/django/en/latest/"
>Graphene-Django documentation</a
>
</li>
<li>
<a
href="https://www.digitalocean.com/community/tutorials/how-to-use-postgresql-with-your-django-application-on-ubuntu-14-04"
>
https://www.digitalocean.com/community/tutorials/how-to-use-postgresql-with-your-django-application-on-ubuntu-14-04
</a>
</li>
<li>
<a href="https://www.postgresql.org/docs/11/static/sql-createuser.html">
https://www.postgresql.org/docs/11/static/sql-createuser.html
</a>
</li>
<li>
<a href="https://www.postgresql.org/docs/10/static/sql-createrole.html">
https://www.postgresql.org/docs/10/static/sql-createrole.html
</a>
</li>
<li>
<a href="https://hub.docker.com/_/postgres/">
https://hub.docker.com/_/postgres/
</a>
</li>
<li>
<a href="/blog/2019/01/how-run-postgresql-docker-mac-local-development/">
How to run PostgreSQL in Docker on Mac (for local development)</a
>
</li>
<li>
<a href="https://www.saltycrane.com/blog/2017/08/docker-cheat-sheet/">
Docker cheat sheet
</a>
</li>
</ul>
How to run PostgreSQL in Docker on Mac (for local development)
2019-01-09T22:47:12-08:00https://www.saltycrane.com/blog/2019/01/how-run-postgresql-docker-mac-local-development/<p>
These are my notes for running Postgres in a Docker container for use with a
local Django or Rails development server running on the host machine (not in
Docker). Running in Docker allows keeping my database environment isolated
from the rest of my system and allows running multiple versions and instances.
(I previously had a problem where Homebrew upgraded Postgres when I didn't
expect it to and my existing database became incompatible. Admittedly, I
didn't know Homebrew well, but it was frustrating.) Disadvantages of Docker
are it's another layer of abstraction to learn and interact with. We use
Docker extensively at work, so from a mental overhead point of view, it's
something I wanted to learn anyways. Currently I use the Homebrew Postgres for
work, and Postgres in Docker for personal projects. I also wrote some
<a href="/blog/2017/07/postgres-mac-os-homebrew-notes/"
>notes on Postgres and Homebrew here</a
>.
</p>
<h4 id="install-docker">Install Docker</h4>
<p>
Install Docker for Mac:
<a href="https://docs.docker.com/docker-for-mac/install/"
>https://docs.docker.com/docker-for-mac/install/</a
>.
</p>
<p>
Alternatively, you can install Docker using Homebrew:
<code>brew install homebrew/cask/docker</code>
</p>
<h4 id="using-single-docker-command">
OPTION 1: Run Postgres using a single Docker command
</h4>
<b>Run a postgres container</b>
<ul>
<li>
uses the
<a href="https://hub.docker.com/_/postgres/"
>official docker postgres 13 image</a
>
</li>
<li>
uses a <a href="https://docs.docker.com/storage/volumes/">named volume</a>,
<code>my_dbdata</code>, to store postgres data
</li>
<li>
exposes port 54320 to the host using
<a
href="https://github.com/docker/cli/blob/master/docs/reference/commandline/run.md#publish-or-expose-port--p---expose"
><code>-p</code></a
>
</li>
<li>sets the container name to <code>my_postgres</code></li>
<li>uses the <code>-d</code> flag to run in the background</li>
<li>
sets the postgres superuser password to <code>"my_password"</code> using
<a
href="https://github.com/docker/cli/blob/master/docs/reference/commandline/run.md#set-environment-variables--e---env---env-file"
><code>-e</code></a
>
and the
<a
href="https://github.com/docker-library/docs/blob/master/postgres/README.md#postgres_password"
><code>POSTGRES_PASSWORD</code> environment variable</a
>.
</li>
</ul>
<pre class="console">
$ docker run -d --name my_postgres -v my_dbdata:/var/lib/postgresql/data -p 54320:5432 -e POSTGRES_PASSWORD=my_password postgres:13
</pre>
<h4 id="using-docker-compose">OPTION 2: Run Postgres using Docker Compose</h4>
<b>Create a <code>docker-compose.yml</code> file</b>
<pre class="console">
$ mkdir /tmp/myproject
$ cd /tmp/myproject
</pre>
<p>Create a new file <code>docker-compose.yml</code>:</p>
<pre>
version: "3"
services:
db:
image: "postgres:13"
container_name: "my_postgres"
environment:
POSTGRES_PASSWORD: "my_password"
ports:
- "54320:5432"
volumes:
- my_dbdata:/var/lib/postgresql/data
volumes:
my_dbdata:
</pre>
<ul>
<li>uses docker compose file version 3</li>
<li>
sets up a service named <code>"db"</code> (this name can be used with
<code>docker-compose</code> commands)
</li>
<li>
uses the
<a href="https://hub.docker.com/_/postgres/"
><code>postgres:13</code> image from hub.docker.com</a
>
using the
<a
href="https://docs.docker.com/compose/compose-file/compose-file-v3/#image"
><code>image</code> key</a
>
</li>
<li>
creates a container named <code>"my_postgres"</code> using the
<a
href="https://docs.docker.com/compose/compose-file/compose-file-v3/#container_name"
><code>container_name</code> key</a
>
</li>
<li>
sets the postgres superuser password to <code>"my_password"</code> using the
<a
href="https://docs.docker.com/compose/compose-file/compose-file-v3/#environment"
><code>environment</code> key</a
>
and the
<a
href="https://github.com/docker-library/docs/blob/master/postgres/README.md#postgres_password"
><code>POSTGRES_PASSWORD</code> environment variable</a
>
</li>
<li>
connects port 5432 inside Docker as port 54320 on the host machine using the
<a
href="https://docs.docker.com/compose/compose-file/compose-file-v3/#ports"
><code>ports</code> key</a
>
</li>
<li>
uses a named volume, <code>"my_dbdata"</code>, for storing the database data
using the
<a
href="https://docs.docker.com/compose/compose-file/compose-file-v3/#volumes"
><code>volumes</code> key</a
>. Even if the container and image are deleted, the volume will remain
unless explicitly deleted using <code>docker volume rm</code>.
</li>
<li>
for more information, see the
<a href="https://docs.docker.com/compose/compose-file/"
>Docker Compose file reference</a
>
</li>
</ul>
<b>Start Postgres</b>
<p>
Pull the postgres image from hub.docker.com, create a container named
"my_postgres", and start it in the background:
</p>
<pre class="console">
$ docker-compose up -d
</pre>
<h4 id="see-working">See that it's working</h4>
<p>See the logs:</p>
<pre class="console">
$ docker logs -f my_postgres
</pre>
<p>Try running psql:</p>
<pre class="console">
$ docker exec -it my_postgres psql -U postgres
</pre>
<p>hit CTRL+D to exit</p>
<p>
For other commands such as starting, stopping, listing or deleting, see my
<a href="/cheat-sheets/docker/">Docker cheat sheet</a>.
</p>
<h4 id="create-database">Create a database</h4>
<pre class="console">
$ docker exec -it my_postgres psql -U postgres -c "create database my_database"
</pre>
<h4 id="connect-using-python">
Connect using Python and <code>psycopg2</code>
</h4>
<pre class="console">
$ python3 -m venv myenv
$ source myenv/bin/activate
$ pip install psycopg2-binary
</pre>
<p>Create a new file named <code>myscript.py</code></p>
<pre class="python">
import psycopg2
conn = psycopg2.connect(
host='localhost',
port=54320,
dbname='my_database',
user='postgres',
password='my_password',
)
cur = conn.cursor()
cur.execute("CREATE TABLE IF NOT EXISTS test (id serial PRIMARY KEY, num integer, data varchar);")
cur.execute("INSERT INTO test (num, data) VALUES (%s, %s)", (100, "abcdef"))
cur.execute("SELECT * FROM test;")
result = cur.fetchone()
print(result)
conn.commit()
cur.close()
conn.close()
</pre>
<p>Run it</p>
<pre class="console">
$ python myscript.py
(1, 100, 'abcdef')
</pre>
<h4 id="errors">Errors</h4>
<ul>
<li>
<pre><code>docker: Error response from daemon: Conflict. The container name "/my_postgres" is already in use by container "b27594a414db369ec4876a07021c9ea738a55b3bc0a3ad5117158367131b99a2". You have to remove (or rename) that container to be able to reuse that name.</code></pre>
<p>If you get the above error, you can remove the container by running:</p>
<pre class="console">$ docker rm my_postgres</pre>
</li>
<li>
<pre><code>Error response from daemon: You cannot remove a running container 7e94d205b6f4ef40ff885987f11e825e94eddbcd061481e591e07c87ed7cf86e. Stop the container before attempting removal or force remove</code></pre>
<p>If you get the above error, you can stop the container by running:</p>
<pre class="console">$ docker stop my_postgres</pre>
</li>
</ul>
<h4 id="see-also">See also</h4>
<ul>
<li>
<a href="/blog/2017/07/postgres-mac-os-homebrew-notes/">
My Postgres Homebrew notes
</a>
</li>
<li>
<a href="https://www.saltycrane.com/blog/2017/08/docker-cheat-sheet/"
>My Docker cheat sheet</a
>
</li>
<li>
<a href="https://hub.docker.com/_/postgres/"
>postgres image on Docker Hub</a
>
</li>
<li>
<a href="https://docs.docker.com/engine/reference/commandline/run/">
<code>docker run</code> command reference
</a>
</li>
<li>
<a href="https://docs.docker.com/compose/compose-file/">
Docker Compose file reference
</a>
</li>
<li>
<a href="https://docs.docker.com/storage/volumes/"
>Documentation on using volumes</a
>
</li>
</ul>
Docker cheat sheet
2017-08-23T08:05:22-07:00https://www.saltycrane.com/blog/2017/08/docker-cheat-sheet/<style>
.docker-notes h5 {
margin-top: -4px;
margin-bottom: 4px;
}
.docker-notes a,
.docker-notes a code {
color: #337ab7;
}
.docker-notes a:hover,
.docker-notes a:hover code {
color: #399bef;
}
.docker-notes blockquote {
padding: 8px 24px 10px 0;
margin: 0;
}
.docker-notes blockquote footer {
display: inline-block;
margin-left: 8px;
}
.docker-notes .term {
font-weight: 600;
}
.docker-notes li {
margin: 0 0 4px 0;
padding: 2px 0 4px 4px;
}
.docker-notes li small {
font-size: 10px;
margin-left: 12px;
}
.docker-notes .compose {
background-color: #eee;
border-radius: 2px;
}
.docker-notes span.compose {
padding: 0 4px 4px 4px;
}
.docker-notes li.compose code {
background-color: #eee;
}
.docker-notes code {
font-weight: 200;
}
</style>
<div class="row docker-notes">
<!-- Column 1 -->
<div class="col-md-5">
<blockquote>
An <span class="term">image</span> is a read-only template with
instructions for creating a Docker container.
<footer>
<a href="https://docs.docker.com/engine/docker-overview/#docker-objects"
>docs.docker.com/engine/docker-overview/#docker-objects</a
>
</footer>
</blockquote>
<blockquote>
A <span class="term">Dockerfile</span> is a text document that contains
all the commands a user could call on the command line to assemble an
image.
<footer>
<a href="https://docs.docker.com/engine/reference/builder/"
>docs.docker.com/engine/reference/builder/</a
>
</footer>
</blockquote>
</div>
<!-- Column 2 -->
<div class="col-md-7">
<blockquote>
A <span class="term">container</span> is a runnable instance of an image.
You can create, start, stop, move, or delete a container. A container is a
process which runs on a <span class="term">host</span>. ...the container
process that runs is isolated in that it has its own file system, its own
networking, and its own isolated process tree separate from the host.
<footer>
<a href="https://docs.docker.com/engine/docker-overview/#docker-objects"
>docs.docker.com/engine/docker-overview/#docker-objects</a
>,
<a href="https://docs.docker.com/engine/reference/run/"
>docs.docker.com/engine/reference/run/</a
>
</footer>
</blockquote>
</div>
</div>
<div class="row docker-notes">
<!-- Column 1 -->
<div class="col-md-5">
<h4>Listing</h4>
<ul>
<li>
List images:<br />
<a href="https://docs.docker.com/engine/reference/commandline/image_ls/"
><code>docker image ls</code></a
><br />
<small>— OR —</small><br />
<a href="https://docs.docker.com/engine/reference/commandline/images/"
><code>docker images</code></a
>
</li>
<li>
List containers:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/container_ls/"
><code>docker container ls -a</code></a
><br />
<small>— OR —</small><br />
<a href="https://docs.docker.com/engine/reference/commandline/ps/"
><code>docker ps -a</code></a
>
</li>
<li>
List volumes:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/volume_ls/"
><code>docker volume ls</code></a
>
</li>
<li>
List networks:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/network_ls/"
><code>docker network ls</code></a
>
</li>
<li class="compose">
List all containers (Compose):<br />
<a href=""><code>docker-compose ps</code></a>
</li>
</ul>
<h4>Removing</h4>
<ul>
<li>
Remove a container:<br />
<a href="https://docs.docker.com/engine/reference/commandline/rm/"
><code>docker rm my_container</code></a
>
</li>
<li>
Remove all stopped containers:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/container_prune/"
><code>docker container prune</code></a
>
</li>
<li>
Remove an image:<br />
<a href="https://docs.docker.com/engine/reference/commandline/image_rm/"
><code>docker image rm my-image</code></a
><br />
<small>— OR —</small><br />
<a href="https://docs.docker.com/engine/reference/commandline/rmi/"
><code>docker rmi my-image</code></a
>
</li>
<li>
Remove dangling images:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/image_prune/"
><code>docker image prune</code></a
>
</li>
<li>
Remove all images:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/image_prune/"
><code>docker image prune -a</code></a
>
</li>
<li>
Remove a volume:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/volume_rm/"
><code>docker volume rm my_volume</code></a
>
</li>
<li>
Remove all volumes:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/volume_prune/"
><code>docker volume prune</code></a
>
</li>
<li class="compose">
Stop and remove containers and networks in
<code>docker-compose.yml</code>. (This does not remove volumes.) <br />
<a href="https://docs.docker.com/compose/reference/down/"
><code>docker-compose down</code></a
>
</li>
<li>
Remove everything:<br />
<strong
>Preferences -> Uninstall / Reset -> Reset to factory defaults</strong
>
</li>
</ul>
<h4>Pulling images</h4>
<ul>
<li>
Pull and image from the Docker registry (<a
href="https://hub.docker.com/"
>hub.docker.com</a
>):<br />
<a href="https://docs.docker.com/engine/reference/commandline/pull/"
><code>docker pull my-image</code></a
>
</li>
</ul>
<h4>Publishing images</h4>
<p>
See also:
<a href="https://docs.docker.com/get-started/part2/#share-your-image"
>Get started with Docker - Share you image</a
>
</p>
<ul>
<li>
Login to the Docker registry (<a href="https://hub.docker.com/"
>hub.docker.com</a
>):<br />
<a href="https://docs.docker.com/engine/reference/commandline/login/"
><code>docker login</code></a
>
</li>
<li>
Tag an image:<br />
<a href="https://docs.docker.com/engine/reference/commandline/tag/"
><code>docker tag my-image my-username/my-repo:my-tag</code></a
>
</li>
<li>
Push an image to the Docker registry:<br />
<a href="https://docs.docker.com/engine/reference/commandline/push/"
><code>docker push my-username/my-repo:my-tag</code></a
>
</li>
</ul>
<h4>Building images from Dockerfiles</h4>
<ul>
<li>
Build an image from a Dockerfile in the current directory:<br />
<a href="https://docs.docker.com/engine/reference/commandline/build/"
><code>docker build -t my-image .</code></a
>
</li>
<li>
Build an image using a differently-named Dockerfile:<br />
<a href="https://docs.docker.com/engine/reference/commandline/build/"
><code>docker build -f Dockerfile-other -t my-image .</code></a
>
</li>
<li class="compose">
Rebuild all images (Compose):<br />
<a href="https://docs.docker.com/compose/reference/build/"
><code>docker-compose build</code></a
>
</li>
<li class="compose">
Rebuild a specific image (Compose):<br />
<a href="https://docs.docker.com/compose/reference/build/"
><code>docker-compose build my_service</code></a
>
<ul>
<li>
where <code>my_service</code> is one of the services listed in the
<code>docker-compose.yml</code> file
</li>
</ul>
</li>
</ul>
<h4>Creating containers</h4>
<ul>
<li>
Create a new container from an image:<br />
<a href="https://docs.docker.com/engine/reference/commandline/create/"
><code>docker create my-image</code></a
>
</li>
<li class="compose">
Build new images and create all containers (Compose). (This will not
rebuild images if a Dockerfile changes.)
<a href="https://docs.docker.com/compose/reference/up/"
><code>docker-compose up --no-start</code></a
>
</li>
</ul>
<h4>Starting / stopping containers</h4>
<ul>
<li>
Start a container:<br />
<a href="https://docs.docker.com/engine/reference/commandline/start/"
><code>docker start my_container</code></a
>
</li>
<li>
Stop a container:<br />
<a href="https://docs.docker.com/engine/reference/commandline/stop/"
><code>docker stop my_container</code></a
>
</li>
<li class="compose">
Start all containers (Compose):<br />
<a href="https://docs.docker.com/compose/reference/start/"
><code>docker-compose start</code></a
>
</li>
<li class="compose">
Stop all containers (Compose):<br />
<a href="https://docs.docker.com/compose/reference/stop/"
><code>docker-compose stop</code></a
>
</li>
</ul>
</div>
<!-- Column 2 -->
<div class="col-md-7">
<h4>Running containers</h4>
<p>
<code>docker run</code> is a combination of (optionally)
<code>docker pull</code>, <code>docker create</code>, and
<code>docker start</code>. See also
<a href="https://docs.docker.com/engine/reference/run/"
>Docker run reference</a
>.
</p>
<ul>
<li>
Create a container from <code>my-image</code>, and run the default
command:<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run my-image</code></a
>
</li>
<li>
Run the <code>echo</code> command in the container instead of the
default command:<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run my-image echo "hello"</code></a
>
</li>
<li>
Run container in the background:<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run -d my-image</code></a
>
</li>
<li>
Run and remove the container after it exits:<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run --rm my-image</code></a
>
</li>
<li>
Create a container from <code>my-image</code>, named
<code>my_container</code>, and start it:<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run --name my_container my-image</code></a
>
</li>
<li>
Run a container, setting an environment variable in the container:<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run --env MY_ENVIRONMENT_VAR=myvalue my-image</code></a
>
</li>
<li class="compose">
Build new images, create all containers, and start all containers
(Compose). (This will not rebuild images if a Dockerfile changes.)<br />
<a href="https://docs.docker.com/compose/reference/up/"
><code>docker-compose up</code></a
>
</li>
<li class="compose">
Build, create, and start all in the background (Compose):<br />
<a href="https://docs.docker.com/compose/reference/up/"
><code>docker-compose up -d</code></a
>
</li>
<li class="compose">
<em>Rebuild all images</em>, create all containers, and start all
containers (Compose):<br />
<a href="https://docs.docker.com/compose/reference/up/"
><code>docker-compose up --build</code></a
>
</li>
<li class="compose">
Create a new container for <code>my_service</code> in
<code>docker-compose.yml</code> and run the <code>echo</code> command
instead of the specified command:<br />
<a href="https://docs.docker.com/compose/reference/run/"
><code>docker-compose run my_service echo "hello"</code></a
>
</li>
</ul>
<h5>volumes</h5>
<ul>
<li>
Run a container with a volume named <code>my_volume</code> mounted at
<code>/my/path</code> in the Docker container. (The volume will be
created if it doesn't already exist.) See the
<a href="https://docs.docker.com/storage/volumes/"
>Volumes documentation</a
>
for more information. <br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code
>docker run --mount source=my_volume,target=/my/path my-image</code
></a
><br />
<small>— OR —</small><br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run -v my_volume:/my/path my-image</code></a
>
</li>
<li id="back-1">
Copy all data from a volume named <code>my_volume_1</code> to a volume
named <code>my_volume_2</code> by running the <code>cp</code> command in
a temporary container created from a <code>debian</code> image<sup
><a href="#footnote-1">1</a></sup
>:<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>
docker run --rm -it -v my_volume_1:/from -v my_volume_2:/to debian
bash -c "cp -av /from/. /to"
</code></a
><br />
<small>— OR —</small><br />
using <code>rsync</code> (after installing it first):<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>
docker run --rm -it -v my_volume_1:/from -v my_volume_2:/to debian
bash -c "apt update && apt install -y rsync && rsync -avz /from/
/to/"
</code></a
>
</li>
</ul>
<h5>ports & networking</h5>
<ul>
<li>
Run and bind port 80 inside Docker to port 9090 on the host (outside
Docker):<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run -p 9090:80 my-image</code></a
>
</li>
<li>
Run and access <code>localhost</code> on the host (outside Docker) using
the special DNS name <code>host.docker.internal</code>. Requires Docker
18.03 or greater. (Before 18.03,
<code>docker.for.mac.localhost</code> was available on Mac only.) See
<a
href="https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds"
>https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds</a
>
<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run my-image ping host.docker.internal</code></a
>
</li>
<li>
Create a network:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/network_create/"
><code>docker network create my_network</code></a
><br />
Run a container using a network:<br />
<a href="https://docs.docker.com/engine/reference/commandline/run/"
><code>docker run --network=my_network my-image</code></a
>
</li>
</ul>
<h4>Interacting with containers</h4>
<ul>
<li>
Interact with a running container using bash:<br />
<a href="https://docs.docker.com/engine/reference/commandline/exec/"
><code>docker exec -it my_container bash</code></a
>
</li>
<li>
Show logs for a container:<br />
<a href="https://docs.docker.com/engine/reference/commandline/logs/"
><code>docker logs -f my_container</code></a
>
</li>
<li>
Copy <code>my-file.txt</code> from the host current directory to the
<code>/tmp</code> directory in <code>my_container</code>:<br />
<a href="https://docs.docker.com/engine/reference/commandline/cp/"
><code>docker cp ./my-file.txt my_container:/tmp/my-file.txt</code></a
>
</li>
<li class="compose">
Interact with a running container using bash (Compose):<br />
<a href="https://docs.docker.com/compose/reference/logs/">
<code>docker-compose exec my_service bash</code>
</a>
</li>
<li class="compose">
Show all logs (Compose):<br />
<a href="https://docs.docker.com/compose/reference/logs/"
><code>docker-compose logs -f</code></a
>
</li>
</ul>
<h4>Getting information</h4>
<ul>
<li>
Show the Docker version:<br />
<a href="https://docs.docker.com/engine/reference/commandline/version/">
<code>docker version</code>
</a>
</li>
<li>
Show the commands used to build an image:<br />
<a href="https://docs.docker.com/engine/reference/commandline/history/">
<code>docker history my-image</code></a
>
</li>
<li>
Show I/O, CPU, and Memory usage for a container:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/container_stats/"
>
<code>docker stats my_container</code></a
>
</li>
<li>
Show files that have changed in a container:<br />
<a
href="https://docs.docker.com/engine/reference/commandline/container_diff/"
>
<code>docker diff my_container</code></a
>
</li>
<li>
Show running processes in a container:<br />
<a href="https://docs.docker.com/engine/reference/commandline/top/">
<code>docker top my_container</code></a
>
</li>
<li>
Show a JSON blob of information about a image, container, volume,
etc:<br />
<a href="https://docs.docker.com/engine/reference/commandline/inspect/">
<code>docker inspect my-image</code><br />
<code>docker inspect my_container</code> </a
><br />
etc.
</li>
<li class="compose">
Show all running processes (Compose):<br />
<a href="https://docs.docker.com/compose/reference/top/">
<code>docker-compose top</code></a
>
</li>
</ul>
</div>
<div class="col-md-12">
<p>
* <code>docker-compose</code> commands are shaded in
<span class="compose">gray</span>. They assume a
<code>docker-compose.yml</code> file in the current directory.
</p>
</div>
</div>
<h4>See also</h4>
<ul>
<li>
<a href="https://docs.docker.com/engine/reference/builder/"
>Dockerfile reference</a
>
</li>
<li>
<a href="https://docs.docker.com/compose/compose-file/"
>Docker Compose file reference</a
>
</li>
<li><a href="/cheat-sheets/typescript/latest/">TypeScript cheat sheet</a></li>
<li>
<a href="/cheat-sheets/typescript/react/latest/"
>TypeScript React cheat sheet</a
>
</li>
</ul>
<hr />
<small>
<ol>
<li id="footnote-1">
Reference:
<a href="https://www.guidodiepen.nl/2016/05/cloning-docker-data-volumes/">
Cloning Docker data volumes </a
>. See also:
<a
href="https://docs.docker.com/storage/volumes/#backup-restore-or-migrate-data-volumes"
>
Docker volumes guide: Backup, restore, or migrate data volumes </a
>. <a href="#back-1">[back]</a>
</li>
</ol>
</small>