Merge branch 'feat/dynamic-env-urls' into 'master'

manage static or dynamic environment urls

See merge request to-be-continuous/gcloud!5

README.md

@@ -78,6 +78,62 @@ Here are some advices about your **secrets** (variables marked with a :lock:):
   it will then be possible to mask it and the template will automatically decode it prior to using it.
 3. Don't forget to escape special characters (ex: `$` -> `$$`).
 
+### Deployment and cleanup jobs
+
+The GitLab CI template for Google Cloud requires you to provide a shell script that fully implements your application
+deployment and cleanup using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud).
+
+#### Lookup policy
+
+The deployment script is searched as follows:
+
+1. look for a specific `gcp-deploy-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-deploy-staging.sh` for staging environment),
+2. if not found: look for a default `gcp-deploy.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
+3. if not found: the deployment job will fail.
+
+The cleanup script is searched as follows:
+
+1. look for a specific `gcp-cleanup-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-cleanup-staging.sh` for staging environment),
+2. if not found: look for a default `gcp-cleanup.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
+3. if not found: the cleanup job will fail.
+
+Your script(s) shall use available [dynamic variables](#dynamic-variables).
+
+#### Dynamic Variables
+
+You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments 
+(`review`, `integration`, `staging` and `production`), each with different application names, exposed routes, settings, ...
+
+Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).
+
+In order to be able to implement some **genericity** in your scripts and templates, you should use available environment variables:
+
+1. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables)
+    (ex: `${CI_ENVIRONMENT_URL}` to retrieve the actual environment exposed route) 
+2. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#custom-environment-variables)
+    (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
+3. **dynamic variables** set by the template:
+    * `${appname}`: the application target name to use in this environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)
+    * `${env}`: the environment type (`review`, `integration`, `staging` or `production`)
+    * `${hostname}`: the environment hostname, extracted from `${CI_ENVIRONMENT_URL}` (has to be explicitly declared as [`environment:url`](https://docs.gitlab.com/ee/ci/yaml/#environmenturl) in your `.gitlab-ci.yml` file)
+    * `${gcp_project_id}`: the current Google Cloud project ID associated to your environment
+
+#### Static vs. Dynamic environment URLs
+
+The Google Cloud template supports two ways of defining your environments url:
+
+* a **static way**: when you know your environments url in advance, probably because you're exposing your routes through a DNS you manage,
+* a [**dynamic way**](https://docs.gitlab.com/ee/ci/environments/#set-dynamic-environment-urls-after-a-job-finishes): when the url cannot be known before the
+  deployment job is executed.
+
+The static way can be implemented simply by setting the appropriate configuration variables depending on the environments (see environments configuration chapters below):
+
+* `$GCP_REVIEW_ENVIRONMENT_SCHEME` and`$GCP_REVIEW_ENVIRONMENT_DOMAIN` for the review environments, 
+* `$GCP_INTEG_ENVIRONMENT_URL`, `$GCP_STAGING_ENVIRONMENT_URL` and `$GCP_PROD_ENVIRONMENT_URL` for others.
+
+To implement the dynamic way, your deployment script shall simply generate a `environment_url.txt` file, containing only
+the dynamically generated url.
+
 ### Environments configuration
 
 As seen above, the Google Cloud template may support up to 4 environments (`review`, `integration`, `staging` and `production`).
@@ -158,51 +214,6 @@ Here are variables supported to configure the production environment:
 | `GCP_PROD_ENVIRONMENT_URL`| The production environment url **including scheme** (ex: `https://my-application.public.domain.com`) Do not use variable inside variable definition as it will result in a two level cascade variable and gitlab does not allow that. | _none_ |
 | `AUTODEPLOY_TO_PROD`      | Set this variable to auto-deploy to production. If not set deployment to production will be `manual` (default behaviour). | _none_ (disabled) |
 
-### Deployment jobs
-
-The GitLab CI template for Google Cloud requires you to provide a shell script that fully implements your application
-deployment using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud).
-
-The deployment script is searched as follows:
-
-1. look for a specific `gcp-deploy-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-deploy-staging.sh` for staging environment),
-2. if not found: look for a default `gcp-deploy.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
-3. if not found: the deployment job will fail.
-
-Your script(s) shall use available [dynamic variables](#dynamic-variables).
-
-### Cleanup jobs
-
-The GitLab CI template for Google Cloud requires you to provide a shell script that fully implements your application
-cleanup using the [`gcloud` CLI](https://cloud.google.com/sdk/gcloud).
-
-The cleanup script is searched as follows:
-
-1. look for a specific `gcp-cleanup-$env.sh` in the `$GCP_SCRIPTS_DIR` directory in your project (e.g. `gcp-cleanup-staging.sh` for staging environment),
-2. if not found: look for a default `gcp-cleanup.sh` in the `$GCP_SCRIPTS_DIR` directory in your project,
-3. if not found: the cleanup job will fail.
-
-Your script(s) shall use available [dynamic variables](#dynamic-variables).
-
-### Dynamic Variables
-
-You have to be aware that your deployment (and cleanup) scripts have to be able to cope with various environments 
-(`review`, `integration`, `staging` and `production`), each with different application names, exposed routes, settings, ...
-
-Part of this complexity can be handled by the lookup policies described above (ex: one resource per env).
-
-In order to be able to implement some **genericity** in your scripts and templates, you should use available environment variables:
-
-1. any [GitLab CI variable](https://docs.gitlab.com/ee/ci/variables/#predefined-environment-variables)
-    (ex: `${CI_ENVIRONMENT_URL}` to retrieve the actual environment exposed route) 
-2. any [custom variable](https://docs.gitlab.com/ee/ci/variables/#custom-environment-variables)
-    (ex: `${SECRET_TOKEN}` that you have set in your project CI/CD variables)
-3. **dynamic variables** set by the template:
-    * `${appname}`: the application target name to use in this environment (ex: `myproject-review-fix-bug-12` or `myproject-staging`)
-    * `${env}`: the environment type (`review`, `integration`, `staging` or `production`)
-    * `${hostname}`: the environment hostname, extracted from `${CI_ENVIRONMENT_URL}` (has to be explicitly declared as [`environment:url`](https://docs.gitlab.com/ee/ci/yaml/#environmenturl) in your `.gitlab-ci.yml` file)
-    * `${gcp_project_id}`: the current Google Cloud project ID associated to your environment
-
 ## Examples
 
 ### Google AppEngine application

templates/gitlab-ci-gcloud.yml

@@ -241,8 +241,9 @@ stages:
     export env=$1
     export appname=$2
     export gcp_project_id=$3
-    # extract hostname from $CI_ENVIRONMENT_URL
-    hostname=$(echo "$CI_ENVIRONMENT_URL" | awk -F[/:] '{print $4}')
+    export environment_url=$4
+    # extract hostname from $environment_url
+    hostname=$(echo "$environment_url" | awk -F[/:] '{print $4}')
     export hostname
 
     log_info "--- \\e[32mdeploy\\e[0m (env: \\e[33;1m${env}\\e[0m)"
@@ -251,6 +252,12 @@ stages:
     log_info "--- \$hostname: \\e[33;1m${hostname}\\e[0m"
     log_info "--- \$gcp_project_id: \\e[33;1m${gcp_project_id}\\e[0m"
 
+    # unset any upstream deployment env & artifacts
+    unset environment_name
+    unset environment_type
+    rm -f gcloud.env
+    rm -f environment_url.txt
+
     deployscript=$(ls -1 "$GCP_SCRIPTS_DIR/gcp-deploy-${env}.sh" 2>/dev/null || ls -1 "$GCP_SCRIPTS_DIR/gcp-deploy.sh" 2>/dev/null || echo "")
     if [[ -f "$deployscript" ]]
     then
@@ -262,8 +269,15 @@ stages:
     fi
 
     # finally persist environment url
-    echo "$CI_ENVIRONMENT_URL" > environment_url.txt
-    echo -e "environment_type=$env\\nenvironment_name=$appname\\nenvironment_url=$CI_ENVIRONMENT_URL" > gcloud.env
+    if [[ -f environment_url.txt ]]
+    then
+      environment_url=$(cat environment_url.txt)
+      export environment_url
+      log_info "--- dynamic environment url found: (\\e[33;1m$environment_url\\e[0m)"
+    else
+      echo "$environment_url" > environment_url.txt
+    fi
+    echo -e "environment_type=$env\\nenvironment_name=$appname\\nenvironment_url=$environment_url" > gcloud.env
   }
 
   # environment cleanup function
@@ -335,6 +349,7 @@ stages:
 # @arg ENV_APP_SUFFIX: env-specific application suffix
 # @arg ENV_PROJECT   : env-specific GCP Project ID
 # @arg ENV_KEY_FILE  : env-specific GCP API key file (JSON)
+# @arg ENV_URL       : env-specific application url
 .gcp-deploy:
   extends: .gcp-base
   stage: deploy
@@ -346,13 +361,15 @@ stages:
     - assert_defined "${ENV_KEY_FILE:-$GCP_KEY_FILE}" 'Missing required GCP key file (JSON)'
     - gcloud auth activate-service-account --key-file ${ENV_KEY_FILE:-$GCP_KEY_FILE}
   script:
-    - deploy "$ENV_TYPE" "${ENV_APP_NAME:-${GCP_BASE_APP_NAME}${ENV_APP_SUFFIX}}" "$ENV_PROJECT"
+    - deploy "$ENV_TYPE" "${ENV_APP_NAME:-${GCP_BASE_APP_NAME}${ENV_APP_SUFFIX}}" "$ENV_PROJECT" "$ENV_URL"
   artifacts:
     name: "$ENV_TYPE env url for $CI_PROJECT_NAME on $CI_COMMIT_REF_SLUG"
     paths:
       - environment_url.txt
     reports:
       dotenv: gcloud.env
+  environment:
+    url: "$environment_url" # can be either static or dynamic
 
 # Cleanup job prototype
 # Can be extended for each deletable environment
@@ -389,9 +406,9 @@ gcp-review:
     ENV_APP_NAME: "$GCP_REVIEW_APP_NAME"
     ENV_PROJECT: "$GCP_REVIEW_PROJECT"
     ENV_KEY_FILE: "$GCP_REVIEW_KEY_FILE"
+    ENV_URL: "${GCP_REVIEW_ENVIRONMENT_SCHEME}://${CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${GCP_REVIEW_ENVIRONMENT_DOMAIN}"
   environment:
     name: review/$CI_COMMIT_REF_NAME
-    url: "${GCP_REVIEW_ENVIRONMENT_SCHEME}://${CI_PROJECT_NAME}-${CI_ENVIRONMENT_SLUG}.${GCP_REVIEW_ENVIRONMENT_DOMAIN}"
     on_stop: gcp-cleanup-review
   resource_group: review/$CI_COMMIT_REF_NAME
   rules:
@@ -439,9 +456,9 @@ gcp-integration:
     ENV_APP_NAME: "$GCP_INTEG_APP_NAME"
     ENV_PROJECT: "$GCP_INTEG_PROJECT"
     ENV_KEY_FILE: "$GCP_INTEG_KEY_FILE"
+    ENV_URL: "${GCP_INTEG_ENVIRONMENT_URL}"
   environment:
     name: integration
-    url: "${GCP_INTEG_ENVIRONMENT_URL}"
   resource_group: integration
   rules:
     # exclude merge requests
@@ -458,9 +475,9 @@ gcp-staging:
     ENV_APP_NAME: "$GCP_STAGING_APP_NAME"
     ENV_PROJECT: "$GCP_STAGING_PROJECT"
     ENV_KEY_FILE: "$GCP_STAGING_KEY_FILE"
+    ENV_URL: "${GCP_STAGING_ENVIRONMENT_URL}"
   environment:
     name: staging
-    url: "${GCP_STAGING_ENVIRONMENT_URL}"
   resource_group: staging
   rules:
     # exclude merge requests
@@ -479,9 +496,9 @@ gcp-production:
     ENV_APP_NAME: "$GCP_PROD_APP_NAME"
     ENV_PROJECT: "$GCP_PROD_PROJECT"
     ENV_KEY_FILE: "$GCP_PROD_KEY_FILE"
+    ENV_URL: "${GCP_PROD_ENVIRONMENT_URL}"
   environment:
     name: production
-    url: "${GCP_PROD_ENVIRONMENT_URL}"
   resource_group: production
   rules:
     # exclude merge requests