README.md

# GitLab CI template for Google Cloud Platform

This project implements a generic GitLab CI template for [Google Cloud Platform](https://cloud.google.com/) environments.

## Overview

This template implements continuous delivery/continuous deployment for projects hosted on Google Cloud platforms.

It provides several features, usable in different modes.

### Review environments

The template supports **review** environments: those are dynamic and ephemeral environments to deploy your
_ongoing developments_ (a.k.a. _feature_ or _topic_ branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment.
It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's [Review Apps](https://docs.gitlab.com/ee/ci/review_apps/) feature.

It also comes with a _cleanup_ job (accessible either from the _environments_ page, or from the pipeline view).

### Integration environment

If you're using a Git Workflow with an integration branch (such as [Gitflow](https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow)),
the template supports an **integration** environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment.
It is only active for your integration branch (`develop` by default).

### Production environments

Lastly, the template supports 2 environments associated to your production branch (`master` by default):

* a **staging** environment (an iso-prod environment meant for testing and validation purpose),
* the **production** environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

* **continuous deployment**: automatic deployment to production (when the upstream pipeline is successful),
* **continuous delivery**: deployment to production can be triggered manually (when the upstream pipeline is successful).

## Usage

### Include

In order to include this template in your project, add the following to your `gitlab-ci.yml`:

```yaml
include:
  - project: 'to-be-continuous/gcloud'
    ref: '1.4.1'
    file: '/templates/gitlab-ci-gcloud.yml'
```

### Global configuration

The Google Cloud template uses some global configuration used throughout all jobs.

| Name                     | description                            | default value     |
| ------------------------ | -------------------------------------- | ----------------- |
| `GCP_CLI_IMAGE`          | the Docker image used to run Google Cloud CLI commands| `google/cloud-sdk:latest` **it is highly recommended to set the CLI version compatible with your Google Cloud server** |
| :lock: `GCP_KEY_FILE`    | Default [Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) | _none_ |
| `GCP_BASE_APP_NAME`      | Base application name                  | `$CI_PROJECT_NAME` ([see GitLab doc](https://docs.gitlab.com/ee/ci/variables/predefined_variables.html)) |
| `GCP_SCRIPTS_DIR`        | Directory where Google Cloud scripts (deploy & cleanup) are located | `.` _(root project dir)_ |

### Secrets management

Here are some advices about your **secrets** (variables marked with a :lock:): 

1. Manage them as [project or group CI/CD variables](https://docs.gitlab.com/ee/ci/variables/#create-a-custom-variable-in-the-ui):
    * [**masked**](https://docs.gitlab.com/ee/ci/variables/#mask-a-custom-variable) to prevent them from being inadvertently 
      displayed in your job logs,
    * [**protected**](https://docs.gitlab.com/ee/ci/variables/#protect-a-custom-variable) if you want to secure some secrets 
      you don't want everyone in the project to have access to (for instance production secrets).
2. In case a secret contains [characters that prevent it from being masked](https://docs.gitlab.com/ee/ci/variables/#masked-variable-requirements), 
  simply define its value as the [Base64](https://en.wikipedia.org/wiki/Base64) encoded value prefixed with `@b64@`: 
  it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: `$` -> `$$`).

### Environments configuration

As seen above, the Google Cloud template may support up to 4 environments (`review`, `integration`, `staging` and `production`).

Each deployment job produces _output variables_ that are propagated to downstream jobs (using [dotenv artifacts](https://docs.gitlab.com/ee/ci/pipelines/job_artifacts.html#artifactsreportsdotenv)):

* `environment_type`: set to the type of environment (`review`, `integration`, `staging` or `production`),
* `environment_name`: the application name (see below),
* `environment_url`: set to `$CI_ENVIRONMENT_URL`.

They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

Here are configuration details for each environment.

#### Review environments

Review environments are dynamic and ephemeral environments to deploy your _ongoing developments_ (a.k.a. _feature_ or 
_topic_ branches).

They are **disabled by default** and can be enabled by setting the `GCP_REVIEW_PROJECT` variable (see below).

Here are variables supported to configure review environments:

| Name                     | description                            | default value     |
| ------------------------ | -------------------------------------- | ----------------- |
| `GCP_REVIEW_PROJECT`     | Google Cloud project ID for `review` env | _none_ (disabled) |
| :lock: `GCP_REVIEW_KEY_FILE`| [Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) to authenticate on `review` env  _(only define if different from default)_    | `$GCP_KEY_FILE` |
| `GCP_REVIEW_APP_NAME`    | Application name for `review` env      | `"${GCP_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}"` (ex: `myproject-review-fix-bug-12`) |
| `GCP_REVIEW_ENVIRONMENT_SCHEME`| The review environment protocol scheme | `https` |
| `GCP_REVIEW_ENVIRONMENT_DOMAIN`| The review environment domain. | _none_ |

Note: By default, review `environment.url` will be built as `${GCP_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${GCP_REVIEW_ENVIRONMENT_DOMAIN}`

#### Integration environment

The integration environment is the environment associated to your integration branch (`develop` by default).

It is **disabled by default** and can be enabled by setting the `GCP_INTEG_PROJECT` variable (see below).

Here are variables supported to configure the integration environment:

| Name                     | description                            | default value     |
| ------------------------ | -------------------------------------- | ----------------- |
| `GCP_INTEG_PROJECT`      | Google Cloud project ID for `integration` env | _none_ (disabled) |
| :lock: `GCP_INTEG_KEY_FILE`|[Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) to authenticate on `integration` env  _(only define if different from default)_    | `$GCP_KEY_FILE` |
| `GCP_INTEG_APP_NAME`     | Application name for `integration` env | `${GCP_BASE_APP_NAME}-integration` |
| `GCP_INTEG_ENVIRONMENT_URL`| The integration environment url **including scheme** (ex: `https://my-application-integration.nonpublic.domain.com`). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |

#### Staging environment

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production 
branch (`master` by default).

It is **disabled by default** and can be enabled by setting the `GCP_STAGING_PROJECT` variable (see below).

Here are variables supported to configure the staging environment:

| Name                     | description                            | default value     |
| ------------------------ | -------------------------------------- | ----------------- |
| `GCP_STAGING_PROJECT`    | Google Cloud project ID for `staging` env | _none_ (disabled) |
| :lock: `GCP_STAGING_KEY_FILE`|[Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) to authenticate on `staging` env  _(only define if different from default)_    | `$GCP_KEY_FILE` |
| `GCP_STAGING_APP_NAME`   | Application name for `staging` env     | `${GCP_BASE_APP_NAME}-staging` |
| `GCP_STAGING_ENVIRONMENT_URL` | The staging environment url **including scheme** (ex: `https://my-application-staging.nonpublic.domain.com`). Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |

#### Production environment

The production environment is the final deployment environment associated with your production branch (`master` by default).

It is **disabled by default** and can be enabled by setting the `GCP_PROD_PROJECT` variable (see below).

Here are variables supported to configure the production environment:

| Name                      | description                            | default value     |
| ------------------------- | -------------------------------------- | ----------------- |
| `GCP_PROD_PROJECT`        | Google Cloud project ID for `production` env | _none_ (disabled) |
| :lock: `GCP_PROD_KEY_FILE`|[Service Account key file](https://cloud.google.com/bigquery/docs/authentication/service-account-file) to authenticate on `production` env  _(only define if different from default)_    | `$GCP_KEY_FILE` |
| `GCP_PROD_APP_NAME`       | Application name for `production` env  | `$GCP_BASE_APP_NAME` |
| `GCP_PROD_ENVIRONMENT_URL`| The production environment url **including scheme** (ex: `https://my-application.public.domain.com`) Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |
| `AUTODEPLOY_TO_PROD`      | Set this variable to auto-deploy to production. If not set deployment to production will be `manual` (default behaviour). | _none_ (disabled) |

### Deployment jobs

The GitLab CI template for Google Cloud requires you to provide a shell script that fully implements your application
deployment using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud).

The deployment script is searched as follows:

1. look for a specific `gcp-deploy-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-deploy-staging.sh` for staging environment),
2. if not found: look for a default `gcp-deploy.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
3. if not found: the deployment job will fail.

Your script(s) shall use available [dynamic variables](#dynamic-variables).

### Cleanup jobs

The GitLab CI template for Google Cloud requires you to provide a shell script that fully implements your application
cleanup using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud).

The cleanup script is searched as follows:

1. look for a specific `gcp-cleanup-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-cleanup-staging.sh` for staging environment),
2. if not found: look for a default `gcp-cleanup.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
3. if not found: the cleanup job will fail.

Your script(s) shall use available [dynamic variables](#dynamic-variables).

### Dynamic Variables

You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments 
(`review`, `integration`, `staging` and `production`), each with different application names, exposed routes, settings, ...

Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).

In order to be able to implement some **genericity** in your scripts and templates, you should use available environment variables:

1. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables)
    (ex: `${CI_ENVIRONMENT_URL}` to retrieve the actual environment exposed route) 
2. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#custom-environment-variables)
    (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
3. **dynamic variables** set by the template:
    * `${appname}`: the application target name to use in this environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)
    * `${env}`: the environment type (`review`, `integration`, `staging` or `production`)
    * `${hostname}`: the environment hostname, extracted from `${CI_ENVIRONMENT_URL}` (has to be explicitly declared as [`environment:url`](https://docs.gitlab.com/ee/ci/yaml/#environmenturl) in your `.gitlab-ci.yml` file)
    * `${gcp_project_id}`: the current Google Cloud project ID associated to your environment

## Examples

### Google AppEngine application

#### Context

Let's imagine a backend service:

* named **coockedoodledoo**,
* developped in whichever language,
* part of project named **farmvoices**
* hosted on Google AppEngine with project ID `farmvoices-12345`
* with review, staging and production environments enabled.

#### `.gitlab-ci.yml`

```yaml
include:
  # Include Google Cloud template
  - project: 'to-be-continuous/gcloud'
    ref: '1.4.1'
    file: '/templates/gitlab-ci-gcloud.yml'
  ...

# Global variables
variables:
  ...

  # Google Cloud
  # GCP_KEY_FILE defined as secret CI/CD variable
  GCP_REVIEW_PROJECT: "farm-12345" # enable review env
  GCP_STAGING_PROJECT: "farm-12345" # enable staging env
  GCP_PROD_PROJECT: "farm-12345" # enable production env
  GCP_STAGING_ENVIRONMENT_URL: "https://staging-dot-coockedoodledoo-dot-farmvoices-12345.ew.r.appspot.com"
  GCP_PROD_ENVIRONMENT_URL: "https://coockedoodledoo-dot-farmvoices-12345.ew.r.appspot.com"

  # Postman
  REVIEW_ENABLED: "true"

# Pipeline steps
stages:
  - build
  - test
  - deploy
  - acceptance
  - publish
  - production

# define review environment url (uses $CI_ENVIRONMENT_SLUG as app version)
gcp-review:
  environment:
    url: "https://$CI_ENVIRONMENT_SLUG-dot-coockedoodledoo-dot-farmvoices-12345.ew.r.appspot.com"
```

#### AppEngine manifest

```yaml
# Google AppEngine manifest
# see: https://cloud.google.com/appengine/docs/standard/java11/config/appref
runtime: TODO # depends on languages
instance_class: F2
service: coockedoodledoo

...

variables:
  # this is an example of hardcoded (non-sensitive) configuration variable
  SOME_CONFIG: "some-value"
  # this is an example of variabilized (secret) configuration variable
  # will be replaced programmatically during deployment
  SOME_SECRET: "${SOME_SECRET}"
```

#### hook scripts

##### `gcp-deploy.sh`

This script is executed by the template to perform the application(s) deployment based on `gcloud` CLI. 

```bash
#!/usr/bin/env bash
echo "[gcp-deploy] Deploy burger/$CI_ENVIRONMENT_SLUG..."

# prepare GAE deployment directory (copy build output)
mkdir -p gae
cp build/* gae

# copy manifest with variables substitution
awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);gsub("[$]{"var"}",ENVIRON[var])}}1' < src/app.yaml > gae/app.yaml

# gcloud deploy
# use $CI_ENVIRONMENT_SLUG as the version
cd gae

if [[ "$CI_ENVIRONMENT_SLUG" == "production" ]]
then
  promote_opt="--promote"
else
  promote_opt="--no-promote"
fi

gcloud --quiet app deploy --project=${gcp_project_id} --version=${CI_ENVIRONMENT_SLUG} $promote_opt
```

##### `gcp-cleanup.sh`

This script is executed by the template to perform the application(s) cleanup based on `gcloud` CLI (review env only).

```bash
#!/usr/bin/env bash
echo "[gcp-cleanup] Cleanup burger/$CI_ENVIRONMENT_SLUG..."

# use $CI_ENVIRONMENT_SLUG as the version
gcloud --quiet app versions delete --project=${gcp_project_id} --service=coockedoodledoo ${CI_ENVIRONMENT_SLUG}
```

## Gitlab compatibility

:information_source: This template is actually tested and validated on GitLab Community Edition instance version 13.12.11