Policy for production releases

Policy for production releases

Last updated: 2023-11-28 11:53

Preparation

Before a new version of sso.healthdata.be can be released, various steps need to be completed in the following order:

  1. IS4U deploys the change on sso-test.healthdata.be and sso-acc.healthdata.be
  2. IS4U contacts the involved parties and reviews the Keycloak logs to validate the change
  3. IS4U communicates the following to Jason and Jeroen:
    • Description of the changes in the production release
    • Impact analysis
    • Is restoring a database backup needed for rolling back this change (see Rollback plan)
  4. Jason / Jeroen follow the procedures of the release board to request their approval
  5. The release is planned (usually the next Monday after the approval)
  6. If the rollback plan for this change requires restoring a database backup, then on the day of the production release, Devops performs the following:
    • Validate the latest backup of the Keycloak production database
    • Check that an engineer is on stand-by during the upgrade to restore the database if needed (see Rollback plan)

Upgrade path

  1. IS4U prepares the needed changes on https://github.com/Sciensano-Healthdata/hd-keycloak
  2. IS4U deploys the change via the Github Action Deploy (see https://docs.healthdata.be/documentation/dc-operations-internal/keycloak-github-actions)
  3. Once the Github action has completed, the Keycloak pods will restart automatically
    • If the release involves an upgrade of the Keycloak core version, then the restarted Keycloak pods will also trigger an update of the database schema
  4. If needed, IS4U updates the configuration in the Keycloak admin console
  5. IS4U validates the release via kubectl describe, kubectl logs and logging in and out to the integrated applications

Rollback plan

Despite the above precautions, it is always possible that there are unforeseen issues during the production release. Depending on the change, one of the rollback plans below can followed:

Rollback plan A (if restoring database backup is not needed):

  1. IS4U uses step 2 and 4 of the Upgrade path to redeploy the old Keycloak version or restore the old configuration

Rollback plan B (if restoring database backup is needed) :

  1. IS4U gets on a call with the DevOps engineer from step 6 of Preparation
  2. IS4U takes the Keycloak environment down by reducing the amount of Keycloak pods to 0
  3. DevOps restores the latest backup of the production Keycloak database
  4. IS4U uses step 2 of the Upgrade path to redeploy the old Keycloak version
  5. IS4U validates the release via kubectl describe, kubectl logs and logging in and out to the integrated applications