Production release policy

Last updated: 2023-12-12 13:15

This page describes the policy for releasing new versions of the Healthdata Keycloak production environment (sso.healthdata.be).

Preparation

Before a new version of sso.healthdata.be can be released, the following steps should be taken:

  1. IS4U deploys the change on sso-test.healthdata.be and sso-acc.healthdata.be
  2. IS4U validates the change by contacting the involved parties and reviewing the Keycloak logs
  3. IS4U communicates the following to Jason and Jeroen (for example see Example release below):
    • Description of the changes in the production release
    • Impact analysis
    • Does the rollback plan require to restore a database backup or not (see Rollback plan)
  4. Jason / Jeroen follow the procedures of the release board to request their approval
  5. The release is planned (usually the next Monday after the approval)
  6. Jeroen communicates the release to end users via mail
  7. If the rollback plan for this change requires restoring a database backup, then on the day of the production release, Devops performs the following:
    • Validate the latest backup of the Keycloak production database
    • Check that an engineer is on stand-by during the upgrade to restore the database if needed (see Rollback plan)

Release steps

  1. IS4U prepares the needed changes on https://github.com/Sciensano-Healthdata/hd-keycloak
  2. IS4U deploys the change via the Github Action Deploy (see https://docs.healthdata.be/documentation/dc-operations-internal/keycloak-github-actions)
  3. Once the Github action has completed, the Keycloak pods will restart automatically
    • If the release involves an upgrade of the Keycloak core version, then the restarted Keycloak pods will also trigger an update of the database schema
  4. If needed, IS4U updates the configuration in the Keycloak admin console
  5. IS4U validates the release via kubectl describe, kubectl logs and logging in and out to the integrated applications

Rollback plan

Despite the above precautions, it is always possible that there are unforeseen issues during the production release. The release can be rolled back as follows:

Rollback plan A (if restoring database backup is not needed):

  1. IS4U uses step 2 and 4 of the Upgrade path to redeploy the old Keycloak version and restore the old configuration

Rollback plan B (if restoring database backup is needed) :

  1. IS4U gets on a call with a DevOps engineer (see the above Preparationn - step 6)
  2. IS4U takes the Keycloak environment down by reducing the amount of Keycloak pods to 0
  3. DevOps restores the latest backup of the production Keycloak database
  4. IS4U uses step 2 of the Release steps to redeploy the old Keycloak version
    • This will also restore the old amount of Keycloak pods. Once the pods have started succesfully, the Keycloak environment should be available again.
  5. IS4U validates the rollback via kubectl describe, kubectl logs and logging in and out to the integrated applications

Example release

  • Description: Upgrade sso-healthdata.be to hd-keycloak 2.2.0:
  • Impact analysis
    • The upgrade was already validated on sso-test.healthdata.be and sso-acc.healthdata.be, but it is still possible that an unforeseen issue emerges. This would impact logging in/out to the integrated applications, ranging from minor issues to login/logout being completely unavailable.
    • The applications integrated with sso.healthdata.be are Healthstat, SAS Viya, Limesurvey and adm.healthdata.be
  • Rolling back this change (if needed) requires a DevOps engineer for restoring the latest backup of the Keycloak production database (for more information, see https://docs.healthdata.be/documentation/dc-operations-internal/production-release-policy.)