GitLab CI template for OpenShift

This project implements a generic GitLab CI template for OpenShift environments.

Overview

This template implements continuous delivery/continuous deployment for projects hosted on OpenShift platforms.

It provides several features, usable in different modes.

Choose your deployment technique

OpenShift supports various ways of deploying your application. As a result, this GitLab CI template supports 2 techniques:

• either script-based (with shell script of your own using oc CLI),
• or template-based (with OpenShift templates provided in your project).

Review environments

The template supports review environments: those are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's Review Apps feature.

It also comes with a cleanup job (accessible either from the environments page, or from the pipeline view).

Integration environment

If you're using a Git Workflow with an integration branch (such as Gitflow), the template supports an integration environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (develop by default).

Production environments

Lastly, the template supports 2 environments associated to your production branch (master by default):

• a staging environment (an iso-prod environment meant for testing and validation purpose),
• the production environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

• continuous deployment: automatic deployment to production (when the upstream pipeline is successful),
• continuous delivery: deployment to production can be triggered manually (when the upstream pipeline is successful).

Usage

Include

In order to include this template in your project, add the following to your gitlab-ci.yml:

include:
  - project: 'to-be-continuous/openshift'
    ref: '1.2.7'
    file: '/templates/gitlab-ci-openshift.yml'

Global configuration

The OpenShift template uses some global configuration used throughout all jobs.

Secrets management

Here are some advices about your secrets (variables marked with a 🔒):

1. Manage them as project or group CI/CD variables:
   • masked to prevent them from being inadvertently displayed in your job logs,
   • protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).
2. In case a secret contains characters that prevent it from being masked, simply define its value as the Base64 encoded value prefixed with @b64@: it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: $ -> $$).

Token generation

Tokens associated with OpenShift user accounts are only valid for 24h. To generate a token that never expires you need to create a new service account.

Follow these steps:

oc create serviceaccount cicd
oc policy add-role-to-user <role_name> system:serviceaccount:<your_project_name>:cicd -n <your_project_name>
oc serviceaccounts get-token cicd

⚠️ don't forget to replace <your_project_name> with your OpenShift project name and <role_name> with the appropriate role (ask your OpenShift support).

Environments configuration

As seen above, the OpenShift template may support up to 4 environments (review, integration, staging and production).

Each deployment job produces output variables that are propagated to downstream jobs (using dotenv artifacts):

• environment_type: set to the type of environment (review, integration, staging or production),
• environment_name: the application name (see below),
• environment_url: set to $CI_ENVIRONMENT_URL.

They may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

Here are configuration details for each environment.

Review environments

Review environments are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

They are disabled by default and can be enabled by setting the OS_REVIEW_PROJECT variable (see below).

Here are variables supported to configure review environments:

Note: By default, review environment.url will be built as ${OS_REVIEW_ENVIRONMENT_SCHEME}://${$CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${OS_REVIEW_ENVIRONMENT_DOMAIN}

Integration environment

The integration environment is the environment associated to your integration branch (develop by default).

It is disabled by default and can be enabled by setting the OS_INTEG_PROJECT variable (see below).

Here are variables supported to configure the integration environment:

Staging environment

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (master by default).

It is disabled by default and can be enabled by setting the OS_STAGING_PROJECT variable (see below).

Here are variables supported to configure the staging environment:

Production environment

The production environment is the final deployment environment associated with your production branch (master by default).

It is disabled by default and can be enabled by setting the OS_PROD_PROJECT variable (see below).

Here are variables supported to configure the production environment:

Deployment jobs

The GitLab CI template for OpenShift supports two policies for deploying your code:

1. script-based deployment
2. template-based deployment

1: script-based deployment

In this mode, you only have to provide a shell script that fully implements the deployment using the oc CLI.

The deployment script is searched as follows:

1. look for a specific os-deploy-$env.sh in the $OS_SCRIPTS_DIR directory in your project (e.g. os-deploy-staging.sh for staging environment),
2. if not found: look for a default os-deploy.sh in the $OS_SCRIPTS_DIR directory in your project,
3. if not found: the GitLab CI template assumes you're using the template-based deployment policy.

Your script(s) may use dynamic variables.

2: template-based deployment

In this mode, you have to provide a OpenShift templates in your project structure, and let the GitLab CI template oc apply it.

The template processes the following steps:

1. optionally executes the os-pre-apply.sh script in your project to perform specific environment pre-initialization (for e.g. create required services),
2. looks for your OpenShift template file, substitutes variables and oc apply it,
   1. look for a specific openshift-$env.yml in your project (e.g. openshift-staging.yml for staging environment),
   2. fallbacks to default openshift.yml.
3. optionally executes the os-post-apply.sh script in your project to perform specific environment post-initialization stuff,
4. optionally executes the os-readiness-check to wait & check for the application to be ready (if not found, the template assumes the application was successfully started).

All scripts and OpenShift template files may use dynamic variables.

Deployment jobs process the selected template with the following labels:

• app: the application target name to use in this environment
  Can be overridden with $OS_APP_LABEL.
• env: the env type (one of review|integration|staging|production)
  Can be overridden with $OS_ENV_LABEL.

Cleanup jobs

The GitLab CI template for OpenShift supports two policies for destroying an environment (actually only review environments):

1. script-based cleanup
2. template-based cleanup

1: script-based cleanup

In this mode, you only have to provide a shell script that fully implements the environment cleanup using the oc CLI.

The a deployment script is searched as follows:

1. look for a specific os-cleanup-$env.sh in the $OS_SCRIPTS_DIR directory in your project (e.g. os-cleanup-staging.sh for staging environment),
2. if not found: look for a default os-cleanup.sh in the $OS_SCRIPTS_DIR directory in your project,
3. if not found: the GitLab CI template assumes you're using the template-based cleanup policy.

Your script(s) may/shall use variables.

TIP: a nice way to implement environment cleanup is to declare the label app=${appname} on every OpenShift object associated to your environment. Then environment cleanup can be implemented very easily with command oc delete all,pvc,is,secret -l "app=${appname}"

2: template-based cleanup

In this mode, you mainly let OpenShift delete all objects from your OpenShift deployment file.

The template processes the following steps:

1. optionally executes the os-pre-cleanup.sh script in your project to perform specific environment pre-cleanup stuff,
2. deletes all objects with label app=${appname}
   works well with template-based deployment as this label is forced during oc apply
3. optionally executes the os-post-cleanup.sh script in your project to perform specific environment post-cleanup (for e.g. delete bound services).

All script files may use variables.

Cleanup job limitations

When using this template, you have to be aware of one limitation (bug) with the cleanup job.

By default, the cleanup job triggered automatically on branch deletion will fail due to not being able to fetch the Git branch prior to executing the job (sounds obvious as the branch was just deleted). This is pretty annoying, but as you may see above, deleting an env may require scripts from the project...

So, what can be done about that?

1. if your project doesn't require any delete script (in other words deleting all objects with label app=${appname} is enough to clean-up everything): you could simply override the cleanup job Git strategy to prevent from fetching the branch code:

   os-cleanup-review:
     variables:
       GIT_STRATEGY: none

2. in any other case, we're just sorry about this bug, but there is not much we can do:
   • remind to delete your review env manually before deleting the branch
   • otherwise you'll have to do it afterwards from your computer (using oc CLI) or from the OpenShift console.

Environment Variables

You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments (review, integration, staging and production), each with different application names, exposed routes, settings, ...

Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).

In order to be able to implement some genericity in your scripts and templates, you should use environment variables:

1. any GitLab CI variable (ex: ${CI_ENVIRONMENT_URL} to retrieve the actual environment exposed route)
2. any custom variable (ex: ${SECRET_TOKEN} that you have set in your project CI/CD variables)
3. dynamic variables set by the template:
   • ${project}: the OpenShift project in which application is deployed (ex: my-openshift-project)
   • ${appname}: the application target name to use in this environment (ex: myproject-review-fix-bug-12 or myproject-staging)
   • ${appname_ssc}: the application target name in SCREAMING_SNAKE_CASE format (ex: MYPROJECT_REVIEW_FIX_BUG_12 or MYPROJECT_STAGING)
   • ${env}: the environment type (review, integration, staging or production)
   • ${hostname}: the environment hostname, extracted from ${CI_ENVIRONMENT_URL} (got from environment:url - see OS_REVIEW_ENVIRONMENT_SCHEME, OS_REVIEW_ENVIRONMENT_DOMAIN, OS_STAGING_ENVIRONMENT_URL and OS_PROD_ENVIRONMENT_URL)

OpenShift template parameters evaluation

Your OpenShift templates shall be variabilized using parameters.

Parameters are evaluated in the following order:

1. from a (optional) specific openshift-$env.env file found in the $OS_SCRIPTS_DIR directory of your project,
2. from the (optional) default openshift.env file found in the $OS_SCRIPTS_DIR directory of your project,
3. from the environment (either predefined GitLab CI, custom or dynamic variables).

For example, with the following parameters in your template:

parameters:
  - name: appname
    description: "the application target name to use in this environment (provided by GitLab CI template)"
    required: true
  - name: hostname
    description: "the environment hostname (provided by GitLab CI template)"
    required: true
  - name: MEMORY
    description: "Pod memory (depends on the environment)"
    required: true
  - name: INSTANCES
    description: "Number of pods (depends on the environment)"
    required: true
  - name: SECRET_TOKEN
    description: "A secret that should not be managed in Git !"
    required: true

With a default openshift.env file:

INSTANCE=1
MEMORY=2Gi

And a specific openshift-production.env file:

INSTANCE=3

And finally SECRET_TOKEN variable defined in your project CI/CD variables.

Then, when deploying to production, the parameters will be evaluated as follows:

os-cleanup-all-review job

This job allows destroying all review environments at once (in order to save cloud resources).

It is disabled by default and can be controlled using the $CLEANUP_ALL_REVIEW variable:

1. automatically executed if $CLEANUP_ALL_REVIEW set to force,
2. manual job enabled from any master branch pipeline if $CLEANUP_ALL_REVIEW set to true (or any other value),

The first value force can be used in conjunction with a scheduled pipeline to cleanup cloud resources for instance everyday at 6pm or on friday evening.

The second one simply enables the (manual) cleanup job on the master branch pipeline.

Anyway destroyed review environments will be automatically re-created the next time a developer pushes a new commit on a feature branch.

⚠️ in case of scheduling the cleanup, you'll probably have to create an almost empty branch without any other template (no need to build/test/analyse your code if your only goal is to cleanup environments).

About multi-line parameters

The template manages multiline parameters passed through environment (ex: a TLS certificate in PEM format).

Unfortunately it doesn't support multiline parameters from dotenv files (OpenShift limitation), but you might use the following technique.

Exemple in your default openshift.env file:

# define SSL_CERT template param using a GitLab CI secret variable
TLS_CERT=${DEV_TLS_CERT}

We could imagine the openshift-production.env as follows:

# define SSL_CERT template param using a GitLab CI secret variable
TLS_CERT=${PROD_TLS_CERT}

The template will take care of expanding variables contained in your dotenv files (requires DEV_TLS_CERT and PROD_TLS_CERT are defined in your environment).

Extra functions

The template provides extra scripts that can be called in your .gitlab-ci.yml or hook scripts for extra treatments.

Variants

Vault variant

This variant allows delegating your secrets management to a Vault server.

Configuration

In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:

Usage

Then you may retrieve any of your secret(s) from Vault using the following syntax:

@url@http://vault-secrets-provider/api/secrets/{secret_path}?field={field}

With:

Example

include:
  # main template
  - project: 'to-be-continuous/openshift'
    ref: '1.2.7'
    file: '/templates/gitlab-ci-openshift.yml'
  # Vault variant
  - project: 'to-be-continuous/openshift'
    ref: '1.2.7'
    file: '/templates/gitlab-ci-openshift-vault.yml'

variables:
    # Secrets managed by Vault
    OS_TOKEN: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/openshift/noprod?field=token"
    OS_PROD_TOKEN: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/openshift/noprod?field=token"
    VAULT_BASE_URL: "https://vault.acme.host/v1"
    # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable

Examples

Back-end application

Context

• review & staging environments enabled on Kermit no prod,
• production environment enabled on Kermit prod,
• implements automated acceptance (functional) tests: manual on review env, auto on staging.

.gitlab-ci.yml

include:
  - project: 'to-be-continuous/openshift'
    ref: '1.2.7' 
    file: '/templates/gitlab-ci-openshift.yml'

variables:
  OS_URL: "https://openshift-noprod.acme.host" # noprod cluster is default (review & staging)
  OS_PROD_URL: "https://openshift-prod.acme.host/" # prod cluster for prod env only
  # OS_TOKEN and OS_PROD_TOKEN are defined as a protected project variable
  OS_REVIEW_PROJECT: "myproj-noprod" # activates 'review' env in CI pipeline
  OS_STAGING_PROJECT: "myproj-noprod" # activates 'staging' env in CD pipeline
  OS_PROD_PROJECT: "myproj"
  OS_REVIEW_ENVIRONMENT_DOMAIN: "apps-noprod.acme.host" # intranet route
  OS_STAGING_ENVIRONMENT_URL: "https://myproj-staging.apps-noprod.acme.host" # internet route
  OS_PROD_ENVIRONMENT_URL: "https://myproj.apps.acme.com" # internet route


# Pipeline steps
stages:
  - build
  - test
  - deploy
  - acceptance
  - production

OpenShift template

# This generic template instantiates all required OpenShift objects
# It uses the following parameters that will be dynamically replaced by the deployment script:
# - ${appname}
# - ${appname_ssc}
# - ${hostname}
# - ${appname}
apiVersion: v1
kind: Template
metadata:
  name: my-application-template
  description: an OpenShift template for my application
# template parameters
parameters:
  - name: appname
    description: "the application target name to use in this environment (provided by GitLab CI template)"
    required: true
  - name: appname_ssc
    description: "the application target name in SCREAMING_SNAKE_CASE format (provided by GitLab CI template)"
    required: true
  - name: hostname
    description: "the environment hostname (provided by GitLab CI template)"
    required: true
  - name: docker_image
    description: "the Docker image build in upstream stages (provided by the Docker template)"
    required: true
objects:
# === Service
- apiVersion: v1
  kind: Service
  metadata:
    annotations:
      description: Exposes and load balances the application pods.
    labels:
      app: ${appname}
    name: ${appname}
  spec:
    ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
    selector:
      app: ${appname}
# === DeploymentConfig
- apiVersion: apps.openshift.io/v1
  kind: DeploymentConfig
  metadata:
    annotations:
      description: The deployment configuration of application.
    labels:
      app: ${appname}
    name: ${appname}
  spec:
    replicas: 1
    revisionHistoryLimit: 2
    selector:
      app: ${appname}
    strategy:
      type: Rolling
      rollingParams:
        timeoutSeconds: 3600
    template:
      metadata:
        labels:
          app: ${appname}
      spec:
        containers:
        - image: ${docker_image}
          imagePullPolicy: Always
          name: spring-boot
          ports:
          - containerPort: 8080
            name: http
            protocol: TCP
          securityContext:
            privileged: false
    triggers:
    - type: ConfigChange
# === Route
- apiVersion: route.openshift.io/v1
  kind: Route
  metadata:
    annotations:
      description: The route exposes the service at a hostname.
    labels:
      app: ${appname}
    name: ${appname}
  spec:
    host: ${hostname}
    port:
      targetPort: 8080
    to:
      kind: Service
      name: ${appname}

hook scripts

os-post-apply.sh

This script - when found by the template - is executed after running oc apply, to perform specific environment post-initialization (for e.g. start build).

#!/bin/bash

set -e

# create a source-to-image binary build if does not exist
oc get buildconfig "$appname" 2> /dev/null || oc new-build openshift/redhat-openjdk18-openshift:1.4 --binary="true" --name="$appname" --labels="app=$appname"

# prepare build resources
mkdir -p target/openshift/deployments && cp target/my-application-1.0.0-SNAPSHOT.jar target/openshift/deployments/

# trigger build: this will trigger a deployment
oc start-build "$appname" --from-dir=target/openshift --wait --follow

# example for force_rollout
force_rollout $appname

os-readiness-check.sh

This script - when found by the template - is used to wait & check for the application to be ready.

It uses the GitLab CI variable $CI_ENVIRONMENT_URL to build absolute urls to the application.

It is supposed to exit with status 0 on success (the template will go on with deployment), or any non-0 value in case of error (the template will stop and as much as possible revert the ongoing deployment).

#!/bin/bash
for attempt in {1..20}
do
    echo "Testing application readiness ($attempt/20)..."
    if wget --no-check-certificate -T 2 --tries 1 "$CI_ENVIRONMENT_URL/healthcheck"
    then
        echo "[INFO] healthcheck response: OK"
        exit 0
    fi
    sleep 5
done

echo "[ERROR] max attempts reached: failed"
exit 1