README.md

GitLab CI template for Helm

This project implements a GitLab CI/CD template to build your Helm Charts and/or deploy your application to a Kubernetes platform using Helm.

Usage

In order to include this template in your project, add the following to your gitlab-ci.yml:

include:
  - project: 'to-be-continuous/helm'
    ref: '5.0.0'
    file: '/templates/gitlab-ci-helm.yml'

Understand

This chapter introduces key notions and principle to understand how this template works.

Managed deployment environments

This template implements continuous delivery/continuous deployment based on Helm for projects hosted on Kubernetes platforms.

Review environments

The template supports review environments: those are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

When enabled, it deploys the result from upstream build stages to a dedicated and temporary environment. It is only active for non-production, non-integration branches.

It is a strict equivalent of GitLab's Review Apps feature.

It also comes with a cleanup job (accessible either from the environments page, or from the pipeline view).

Integration environment

If you're using a Git Workflow with an integration branch (such as Gitflow), the template supports an integration environment.

When enabled, it deploys the result from upstream build stages to a dedicated environment. It is only active for your integration branch (develop by default).

Production environments

Lastly, the template supports 2 environments associated to your production branch (master by default):

• a staging environment (an iso-prod environment meant for testing and validation purpose),
• the production environment.

You're free to enable whichever or both, and you can also choose your deployment-to-production policy:

• continuous deployment: automatic deployment to production (when the upstream pipeline is successful),
• continuous delivery: deployment to production can be triggered manually (when the upstream pipeline is successful).

Deployment context variables

In order to manage the various deployment environments, this template provides a couple of dynamic variables that you might use in your hook scripts and Helm charts (as values):

environment variable	template directive	description
$environment_name	{{ .Release.Name }}	a generated application name to use for the current deployment environment (ex: myproject-review-fix-bug-12 or myproject-staging). This is used as the Helm release name in deploy & delete jobs - details below
$environment_type	{{ .Values.environmentType }}	the current deployment environment type (review, integration, staging or production)
$hostname	{{ .Values.hostname }}	the environment hostame, extracted from the environment URL (if you specified the environment url statically)

Generated environment name

The ${environment_name} variable is generated to designate each deployment environment with a unique and meaningful application name. By construction, it is suitable for inclusion in DNS, URLs, Kubernetes labels... It is built from:

• the application base name (defaults to $CI_PROJECT_NAME but can be overridden globally and/or per deployment environment - see configuration variables)
• GitLab predefined $CI_ENVIRONMENT_SLUG variable (sluggified name, truncated to 24 characters)

The ${environment_name} variable is then evaluated as:

• <app base name> for the production environment
• <app base name>-$CI_ENVIRONMENT_SLUG for all other deployment environments
• ${environment_name} can also be overriden per environment with the appropriate configuration variable

Examples (with an application's base name myapp):

$environment_type	Branch	$CI_ENVIRONMENT_SLUG	$environment_name
review	feat/blabla	review-feat-bla-xmuzs6	myapp-review-feat-bla-xmuzs6
integration	develop	integration	myapp-integration
staging	main	staging	myapp-staging
production	main	production	myapp

Deployment and cleanup scripts

The Helm template requires you to provide a Helm chart (either in the project or located in an external repository) to deploy and delete the application.

The environment deployment is processed as follows:

1. optionally executes the helm-pre-deploy.sh script in your project to perform specific environment pre-initialization (for e.g. create required services),
2. helm upgrade the chart with the configured parameters, using $environment_name as release name,
3. optionally executes the helm-post-deploy.sh script in your project to perform specific environment post-initialization stuff,

The environment deletion is processed as follows:

1. optionally executes the helm-pre-delete.sh script in your project to perform specific environment pre-cleanup stuff,
2. helm uninstall, using $environment_name as release name,
3. optionally executes the helm-post-delete.sh script in your project to perform specific environment post-cleanup (for e.g. delete bound services).

each of the above hook scripts needs to be executable, you can add flag execution with: git update-index --chmod=+x helm-pre-cleanup.sh

Using variables

You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments (review, integration, staging and production), each with different application names, exposed routes, settings, ... Part of this complexity can be handled by the lookup policies described above (ex: one script/manifest per env) and also by using available environment variables in your scripts and values files:

1. deployment context variables provided by the template,
2. any GitLab CI variable
3. any custom variable (ex: ${SECRET_TOKEN} that you have set in your project CI/CD variables)

While your scripts may simply use any of those variables, your values files can use variable substitution with the syntax ${VARIABLE_NAME}. Each of those patterns will be dynamically replaced in your files by the template right before using it.

In order to be properly replaced, variables in your YAML value files shall be written with curly braces (ex: ${MYVAR} and not $MYVAR). Multiline variables must be surrounded by double quotes and you might have to disable line-length rule of yamllint as they are rewritten on a single line.

 tlsKey: "${MYKEY}"  # yamllint disable-line rule:line-length

Environments URL management

The Helm template supports two ways of providing your environments url:

• a static way: when the environments url can be determined in advance, probably because you're exposing your routes through a DNS you manage,
• a dynamic way: when the url cannot be known before the deployment job is executed.

The static way can be implemented simply by setting the appropriate configuration variable(s) depending on the environment (see environments configuration chapters):

• $HELM_ENVIRONMENT_URL to define a default url pattern for all your envs,
• $HELM_REVIEW_ENVIRONMENT_URL, $HELM_INTEG_ENVIRONMENT_URL, $HELM_STAGING_ENVIRONMENT_URL and $HELM_PROD_ENVIRONMENT_URL to override the default.

Each of those variables support a late variable expansion mechanism with the %{somevar} syntax, allowing you to use any dynamically evaluated variables such as ${environment_name}.

Example:

variables:
  HELM_BASE_APP_NAME: "wonderapp"
  # global url for all environments
  HELM_ENVIRONMENT_URL: "https://%{environment_name}.nonprod.acme.domain"
  # override for prod (late expansion of $HELM_BASE_APP_NAME not needed here)
  HELM_PROD_ENVIRONMENT_URL: "https://$HELM_BASE_APP_NAME.acme.domain"
  # override for review (using separate resource paths)
  HELM_REVIEW_ENVIRONMENT_URL: "https://wonderapp-review.nonprod.acme.domain/%{environment_name}"

To implement the dynamic way, your deployment script shall simply generate a environment_url.txt file in the working directory, containing only the dynamically generated url. When detected by the template, it will use it as the newly deployed environment url.

Deployment output variables

Each deployment job produces output variables that are propagated to downstream jobs (using dotenv artifacts):

• $environment_type: set to the type of environment (review, integration, staging or production),
• $environment_name: the application name (see below),
• $environment_url: set to the environment URL (whether determined statically or dynamically).

Those variables may be freely used in downstream jobs (for instance to run acceptance tests against the latest deployed environment).

Working with repositories & OCI-based registries

The Helm template supports indifferently the use of chart repositories and OCI-based registries (requires Helm 3 or above).

Those can be used both for pulling and/or pushing charts.

Configuring pull repositories

The pulling repositories/registries can be configured with the $HELM_REPOS. The value is expected as a (whitespace-separated) list of repo_name@repo_url (defaults to stable@https://charts.helm.sh/stable bitnami@https://charts.bitnami.com/bitnami).

When using OCI-based registries, simply prefix the url with oci://.

The Helm template also supports user/password authentication for each, simply by defining HELM_REPO_<NAME>_USER and HELM_REPO_<NAME>_PASSWORD (as project or group secret variables).

The <NAME> part is the repo_name transformed in SCREAMING_SNAKE_CASE (uppercase words separated by underscores).

Example: declare the GitLab chart repository from another GitLab project

variables:
  HELM_REPOS: "stable@https://charts.helm.sh/stable bitnami@https://charts.bitnami.com/bitnami other-proj@${CI_API_V4_URL}/projects/1234/packages/helm/release"
  HELM_REPO_OTHER_PROJ_USER: "gitlab-token"
  # HELM_REPO_OTHER_PROJ_PASSWORD set as a project secret variables

Configuring the push repository

All configuration parameters are extensively documented in the helm-publish job chapter.

Configuration reference

Secrets management

Here are some advices about your secrets (variables marked with a ):

1. Manage them as project or group CI/CD variables:
   • masked to prevent them from being inadvertently displayed in your job logs,
   • protected if you want to secure some secrets you don't want everyone in the project to have access to (for instance production secrets).
2. In case a secret contains characters that prevent it from being masked, simply define its value as the Base64 encoded value prefixed with @b64@: it will then be possible to mask it and the template will automatically decode it prior to using it.
3. Don't forget to escape special characters (ex: $ -> $$).

Global configuration

The Helm template uses some global configuration used throughout all jobs.

Name	description	default value
HELM_CLI_IMAGE	The Docker image used to run Helm set the version required by your Kubernetes server	registry.hub.docker.com/alpine/helm:latest
HELM_CHART_DIR	The folder where the Helm chart is located	. (root project dir)
HELM_SCRIPTS_DIR	The folder where hook scripts are located	. (root project dir)
HELM_COMMON_VALUES	Common values file (used for all environments, overridden by specific per-env values files)	undefined (none)
HELM_ENV_VALUE_NAME	The name of the Helm value containing the environment type	environmentType
HELM_HOSTNAME_VALUE_NAME	The name of the Helm value containing the environment hostname (extracted from the environment URL)	hostname
KUBE_NAMESPACE	The default Kubernetes namespace to use	none but this variable is automatically set by GitLab Kubernetes integration when enabled
HELM_DEFAULT_KUBE_CONFIG	The default kubeconfig to use (either content or file variable)	$KUBECONFIG (thus supports the GitLab Kubernetes integration when enabled)
HELM_DEPLOY_ARGS	The Helm command with options to deploy the application (without dynamic arguments such as release name and chart)	upgrade --install --atomic --timeout 120s
HELM_DELETE_ARGS	The Helm command with options to cleanup the application (without dynamic arguments such as release name)	uninstall
HELM_DEPLOY_CHART	The Helm chart to deploy. Only required if you want to deploy an external chart.	none
HELM_REPOS	The Helm chart repositories to use (formatted as repo_name_1@repo_url_1 repo_name_2@repo_url_2 ...)	stable@https://charts.helm.sh/stable bitnami@https://charts.bitnami.com/bitnami
HELM_BASE_APP_NAME	Base application name	$CI_PROJECT_NAME (see GitLab doc)
HELM_ENVIRONMENT_URL	Default environments url (only define for static environment URLs declaration) supports late variable expansion (ex: https://%{environment_name}.helm.acme.com)	none

Review environments configuration

Review environments are dynamic and ephemeral environments to deploy your ongoing developments (a.k.a. feature or topic branches).

They are disabled by default and can be enabled by setting the HELM_REVIEW_ENABLED variable (see below).

Here are variables supported to configure review environments:

Name	description	default value
HELM_REVIEW_ENABLED	Set to true to enable review env	none (enabled)
HELM_REVIEW_APP_NAME	Application name for review env	"${HELM_BASE_APP_NAME}-${CI_ENVIRONMENT_SLUG}" (ex: myproject-review-fix-bug-12)
HELM_REVIEW_ENVIRONMENT_URL	The review environments url (only define for static environment URLs declaration and if different from default)	$HELM_ENVIRONMENT_URL
HELM_REVIEW_NAMESPACE	The Kubernetes namespace to use for review env (only define to override default)	$KUBE_NAMESPACE
HELM_REVIEW_KUBE_CONFIG	Specific kubeconfig for review env (only define to override default)	$HELM_DEFAULT_KUBE_CONFIG
HELM_REVIEW_VALUES	The Values file to use with review environments	none

Integration environment configuration

The integration environment is the environment associated to your integration branch (develop by default).

It is disabled by default and can be enabled by setting the HELM_INTEG_ENABLED variable (see below).

Here are variables supported to configure the integration environment:

Name	description	default value
HELM_INTEG_ENABLED	Set to true to enable integration env	none (enabled)
HELM_INTEG_APP_NAME	Application name for integration env	$HELM_BASE_APP_NAME-integration
HELM_INTEG_ENVIRONMENT_URL	The integration environment url (only define for static environment URLs declaration and if different from default)	$HELM_ENVIRONMENT_URL
HELM_INTEG_NAMESPACE	The Kubernetes namespace to use for integration env (only define to override default)	$KUBE_NAMESPACE
HELM_INTEG_KUBE_CONFIG	Specific kubeconfig for integration env (only define to override default)	$HELM_DEFAULT_KUBE_CONFIG
HELM_INTEG_VALUES	The Values file to use with the integration environment	none

Staging environment configuration

The staging environment is an iso-prod environment meant for testing and validation purpose associated to your production branch (master by default).

It is disabled by default and can be enabled by setting the HELM_STAGING_ENABLED variable (see below).

Here are variables supported to configure the staging environment:

Name	description	default value
HELM_STAGING_ENABLED	Set to true to enable staging env	none (enabled)
HELM_STAGING_APP_NAME	Application name for staging env	$HELM_BASE_APP_NAME-staging
HELM_STAGING_ENVIRONMENT_URL	The staging environment url (only define for static environment URLs declaration and if different from default)	$HELM_ENVIRONMENT_URL
HELM_STAGING_NAMESPACE	The Kubernetes namespace to use for staging env (only define to override default)	$KUBE_NAMESPACE
HELM_STAGING_KUBE_CONFIG	Specific kubeconfig for staging env (only define to override default)	$HELM_DEFAULT_KUBE_CONFIG
HELM_STAGING_VALUES	The Values file to use with the staging environment	none

Production environment configuration

The production environment is the final deployment environment associated with your production branch (master by default).

It is disabled by default and can be enabled by setting the HELM_PROD_ENABLED variable (see below).

Here are variables supported to configure the production environment:

Name	description	default value
HELM_PROD_ENABLED	Set to true to enable production env	none (enabled)
HELM_PROD_APP_NAME	Application name for production env	$HELM_BASE_APP_NAME
HELM_PROD_ENVIRONMENT_URL	The production environment url (only define for static environment URLs declaration and if different from default)	$HELM_ENVIRONMENT_URL
HELM_PROD_NAMESPACE	The Kubernetes namespace to use for production env (only define to override default)	$KUBE_NAMESPACE
HELM_PROD_KUBE_CONFIG	Specific kubeconfig for production env (only define to override default)	$HELM_DEFAULT_KUBE_CONFIG
HELM_PROD_DEPLOY_STRATEGY	Defines the deployment to production strategy. One of manual (i.e. one-click) or auto.	manual
HELM_PROD_VALUES	The Values file to use with the production environment	none

helm-lint job

This job examines your chart for possible issues and uses the following variables:

Name	description	default value
HELM_LINT_DISABLED	Set to true to disable Helm lint	none (enabled)
HELM_LINT_ARGS	The Helm command with options to trigger the analysis (without dynamic arguments such as the chart path)	lint --strict
HELM_DEPENDENCY_ARGS	The Helm command with options to update on-disk the chart dependencies (without dynamic arguments such as the chart path)	dependency update

helm-values-*-lint job

These jobs perform a Yaml Lint of your Helm values file and uses the following variables:

Name	description	default value
HELM_YAMLLINT_IMAGE	The Docker image used to run YamlLint test	registry.hub.docker.com/cytopia/yamllint
HELM_YAMLLINT_DISABLED	Set to true to disable Yaml lint	none (enabled)
HELM_YAMLLINT_CONFIG	Config used with the yamllint tool	{extends: relaxed, rules: {line-length: {max: 160}}}
HELM_YAMLLINT_ARGS	Arguments used by the lint job	-f colored --strict

helm-*-score job

This job runs Kube-Score on the resources to be created by Helm and uses the following variables:

Name	description	default value
HELM_KUBE_SCORE_DISABLED	Set to true to disable Kube-Score	none (enabled)
HELM_KUBE_SCORE_IMAGE	The Docker image used to run Kube-Score	registry.hub.docker.com/zegl/kube-score
HELM_KUBE_SCORE_ARGS	Arguments used by the helm-score job	none

helm-package job

This job packages the Helm chart. It uses the following variables:

Name	description	default value
HELM_PACKAGE_ARGS	The Helm command with options to perform the packaging (without dynamic arguments such as the chart path)	package --dependency-update
HELM_PUBLISH_SNAPSHOT_ENABLED	Set to true to enable publishing the snapshot (untested) chart during the packaging step	none (disabled)
HELM_SEMREL_RELEASE_DISABLED	Set to true to disable usage of semantic-release release info for helm package (see next chapter)	none (enabled)

semantic-release integration

If you activate the semantic-release-info job from the semantic-release template, the helm-package job will automatically use the generated next version info for the chart version (--version).

If no next version info is determined by semantic-release, the package will be created, but without versioning info.

Note: You can disable the semantic-release integration described herebefore the HELM_SEMREL_RELEASE_DISABLED variable.

Chart version management

Depending on the branch and the step in the CI/CD pipeline, the chart will be packaged with a different version.

The general version format will be <x.y.z>-<label>:

• <x.y.z>:
  • if semantic-release integration is enabled: uses the version determined by semantic-release,
  • otherwise uses the version from the chart file
• <label>:
  • on the production branch (main or master by default), no trailing label is used
  • on any other branch, $CI_COMMIT_REF_SLUG is used as trailing label
    (ex: review-feature-12 on branch review/feature-12)
  • when HELM_PUBLISH_SNAPSHOT_ENABLED is enabled, the chart is additionally packaged (and published) with a label suffixed with snapshot
    (ex: snapshot on production branch and review-feature-12-snapshot on branch review/feature-12)

Package output variables

The helm-package job produces output variables that are propagated to downstream jobs (using dotenv artifacts):

• $helm_package_file: the packaged chart (tgz file),
• $helm_package_name: the chart/package name,
• $helm_package_version: the package version.

If HELM_PUBLISH_SNAPSHOT_ENABLED is set to true, extra variables are produced:

• $helm_snapshot_package_name: the snapshot package name,
• $helm_snapshot_package_version: the snapshot package version,
• $helm_snapshot_package_remote_url: the remote url where the snapshot package was published.

Those variables may be freely used in downstream jobs.

helm-publish job

This job publishes the packaged chart to a chart repository or OCI-based registry. It uses the following variables:

Name	description	default value
HELM_PUBLISH_METHOD	Method to use to publish the packaged chart (one of auto, push, post, put, custom, disabled)	auto
HELM_PUBLISH_USER	Helm registry username	$CI_REGISTRY_USER
HELM_PUBLISH_PASSWORD	Helm registry password	$CI_REGISTRY_PASSWORD
HELM_PUBLISH_URL	The URL of the Helm repository to publish your Helm package. Supports both chart repository or OCI-based registry (url must be prefixed with oci://)	oci://$CI_REGISTRY/$CI_PROJECT_PATH/charts (GitLab's container registry)
HELM_PUBLISH_ON	Defines on which branch(es) the publish job shall be enabled (prod to enable on production branch only, protected to enable on protected references and all to enable on all Git references)	prod
HELM_PUBLISH_STRATEGY	Defines the publish strategy. One of manual (i.e. one-click), auto or none (disabled).	manual
HELM_CM_PUSH_PLUGIN_VERSION	cm-push plugin version to install (only when using push method with a regular chart repository)	none (latest)

Supported publish methods

The Helm publish supports several methods, configurable with the $HELM_PUBLISH_URL variable:

Value	description
auto (default)	tries to auto-detect the most appropriate method
disabled	disables the helm-publish job
push	if publishing to an OCI-based registry, publishes with helm push command; else uses the cm-push plugin
post	publishes the package using http POST method (compatible with GitLab packages repository)
put	publishes the package using http PUT method
custom	forces the use of a custom publish script

The default configuration will use GitLab's container registry to publish your charts

If you wish to use GitLab's Helm package repository instead, simply override:

variables:
  # use channel 'release' (can be changed)
  HELM_PUBLISH_URL: "${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/packages/helm/release"

and leave default $HELM_PUBLISH_USER/$HELM_PUBLISH_PASSWORD values.

Custom publish script

If supported methods don't fit your needs, you may provide a helm-publish.sh script (with execution permissions) in your $HELM_SCRIPTS_DIR directory to implement the required publish method.

This script may use the following variables:

• $helm_package: the packaged chart to publish,
• $HELM_PUBLISH_USER, $HELM_PUBLISH_PASSWORD and $HELM_PUBLISH_URL (see above).

helm-test job

This job runs Helm tests. The job definition must contain the helm test hook annotation: helm.sh/hook: test You are welcome to nest your test suite under a tests/ directory like $HELM_CHART_DIR/templates/tests/ for more isolation.

It is disabled by default and can be enabled by setting the HELM_TEST_ENABLED variable (see below).

It uses the following variables:

Name	description	default value
HELM_TEST_ENABLED	Set to true to enable Helm test	none (disabled)
HELM_TEST_ARGS	The Helm command with options to perform acceptance test (without dynamic arguments such as the chart path)	test

Variants

Vault variant

This variant allows delegating your secrets management to a Vault server.

Configuration

In order to be able to communicate with the Vault server, the variant requires the additional configuration parameters:

Name	description	default value
TBC_VAULT_IMAGE	The Vault Secrets Provider image to use (can be overridden)	$CI_REGISTRY/to-be-continuous/tools/vault-secrets-provider:master
VAULT_BASE_URL	The Vault server base API url	none
VAULT_ROLE_ID	The AppRole RoleID	must be defined
VAULT_SECRET_ID	The AppRole SecretID	must be defined

Usage

Then you may retrieve any of your secret(s) from Vault using the following syntax:

@url@http://vault-secrets-provider/api/secrets/{secret_path}?field={field}

With:

Name	description
secret_path (path parameter)	this is your secret location in the Vault server
field (query parameter)	parameter to access a single basic field from the secret JSON payload

Example

include:
  # main template
  - project: 'to-be-continuous/helm'
    ref: '5.0.0'
    file: '/templates/gitlab-ci-helm.yml'
  # Vault variant
  - project: 'to-be-continuous/helm'
    ref: '5.0.0'
    file: '/templates/gitlab-ci-helm-vault.yml'

variables:
    # Secrets managed by Vault
    HELM_DEFAULT_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/helm/noprod?field=kube_config"
    HELM_PROD_KUBE_CONFIG: "@url@http://vault-secrets-provider/api/secrets/b7ecb6ebabc231/my-app/helm/prod?field=kube_config"
    VAULT_BASE_URL: "https://vault.acme.host/v1"
    # $VAULT_ROLE_ID and $VAULT_SECRET_ID defined as a secret CI/CD variable