Bring Your Own Kubernetes (BYOK8s) Upgrade

​

Prerequisites

• You must have access to the Administrative Server.
• You must verify the Service Account used for the upgrade has the correct permissions.
• Air-gapped servers must have the Docker images added to their Docker image repository.

​

Upgrading

1. Log in to the service account on the Administrative Server running your Workbench software.
2. Create a backup of your cluster configurations.
   This backup is the only reliable way to undo this upgrade later. Some Workbench components run forward database migrations when they start under the new chart version (confirmed for the git-storage service), and helm rollback does not reverse those migrations. Once a migration has run, rolling the release back to the prior version leaves that component unable to start against the now-newer database schema.Confirm the backup completed successfully before continuing. If you need to undo this upgrade after it has run, restore from this backup rather than relying on helm rollback alone.
3. Unpack the Helm charts provided by Anaconda onto the Administrative Server:
   tar xvzf <HELM_CHARTS>
   
4. Save your current configurations with the extract_config.sh script delivered with your Helm chart:
   NAMESPACE=<NAMESPACE> ./extract_config.sh
   
   The extract_config.sh script creates a file called helm_values.yaml and saves it in the directory where the script was run.
5. Verify the information captured in the helm_values.yaml file is correct and contains all of your current cluster configuration settings.
6. Begin the upgrade:
   helm upgrade --values ./helm_values.yaml anaconda-enterprise ./Anaconda-Enterprise/
   
   If the upgrade is successful, your output will look like this:
   Release "anaconda-enterprise" has been upgraded. Happy Helming!
   NAME: anaconda-enterprise
   LAST DEPLOYED: Wed Dec  7 21:52:34 2022
   NAMESPACE: <YOUR_NAMESPACE>
   STATUS: deployed
   REVISION: 10
   TEST SUITE: None
   
   The duration of the upgrade depends on both the cluster node count and the speed of the new Docker images loading into each node.

​

Verifying your upgrade

1. Verify all pods are running:
   sudo kubectl get pods
   
2. Open a web browser and navigate to your Authentication Center.
   https://<FQDN>/auth/admin
   
3. Select Users from the left-hand navigation, then click View all users and verify your users’ data is present.
4. Open a new tab in your web browser and navigate to your Workbench URL. Log in using the same credentials you used for your previous installation.
5. Review the Projects list to verify that all project data has been restored.
6. Verify your deployments have returned to a started state.

​

Troubleshooting a failed rollback

Error: conflict occurred while applying object default/kubernetes-dashboard-csrf /v1, Kind=Secret: Apply failed with 1 conflict: conflict with "dashboard" using v1: .data.csrf

1. Find the revision number you want to roll back to:
   helm history anaconda-enterprise --name <NAMESPACE>
   
2. Delete the conflicting secret, then retry the rollback:
   kubectl delete secret kubernetes-dashboard-csrf --name <NAMESPACE>
   helm rollback anaconda-enterprise <REVISION> --name <NAMESPACE>
   
   Kubernetes and Helm recreate the secret cleanly once there is no longer a live value to conflict with.
3. Restart the Workbench Dashboard pod:
   kubectl delete pod --name <NAMESPACE> -l app.kubernetes.io/name=anaconda-enterprise-ops-dashboard
   
   The Dashboard process reads and caches its CSRF token only once, at startup, so it will not notice that the secret changed until it restarts. If this label selector does not match any pods in your environment, find the pod name first and delete it directly:
   kubectl get pods --name <NAMESPACE> | grep dashboard
   kubectl delete pod --name <NAMESPACE> <POD_NAME>
   
4. Confirm the rollback succeeded:
   kubectl get secret kubernetes-dashboard-csrf --name <NAMESPACE> -o jsonpath='{.data.csrf}'
   kubectl get pods --name <NAMESPACE> | grep dashboard
   
   The csrf field should return a non-empty base64 value, and the Dashboard pod should show 1/1 Running.