Skip to main content
Anaconda’s Data Science & AI Workbench provides detailed logs and monitoring information related to the Kubernetes services and containers it uses. Gravity-based installations can use the provided Operations Center and Kubernetes CLI to access this information, to help diagnose and debug errors that you or other users might encounter while using the platform.

​
Gravity troubleshooting

Administrators can access the Operations Center to configure, monitor, and troubleshoot the platform. To access the Operations Center:
  1. Log in to Workbench, select Menu in the top right corner, and click the Administrative Console link displayed at the bottom of the slideout menu.
  2. Click Manage Resources.
  3. Log in to the Operations Center using the Administrator credentials configured after installation.
To view resource utilization:
  1. Select Servers in the menu on the left.
  2. Click the Private IP address of the Workbench master node, and select SSH login as root.
    The SSH login as root option on the master node
  3. To display the current resource utilization of each node in the cluster, run this command:
    Output of kubectl top nodes showing CPU and memory utilization per node
    This is actual resource utilization, not limits or requests.
  4. To view utilization and requests for a particular node, run the kubectl describe node command against the IP address for the node (listed under NAME). For example:
    Output of kubectl describe node showing resource requests and limits
  5. To view the resource utilization per pod, run this command:
    Output of kubectl top pods showing CPU and memory utilization per pod
  6. To view the current status of all pods in the cluster, run kubectl get pods.
    Output of kubectl get pods showing pod names, status, and restarts
    The following table summarizes common pod states:
  7. To view information for a particular pod, run the kubectl describe pod command against the pod (listed under NAME). For example:
    Output of kubectl describe pod showing pod events and container details
    You can also use the Operations Center Logs to gain insights into pod behavior and troubleshoot issues. See Logging for more information.

​
User errors

If a user experiences issues within a Notebook session, have them send you the name of the pod associated with their project session. They can obtain this information by running the hostname command from within a Jupyter Notebook or terminal window. The hostname appears in the Notebook cell output:
Running the hostname command in a Jupyter Notebook cell
Or from a terminal window:
Running the hostname command in a terminal session
You can then use the commands described above or the Operations Center’s Monitoring and Logs features to investigate the issue. See Monitoring sessions and deployments for more information.
The Operations Center Monitoring view showing pod status
As an Administrator, you can also use the Authentication Center to impersonate a user to try to reproduce the problem they are experiencing.
To access the Authentication Center:
  1. Log in to Workbench, click Menu in the top right corner, then click the Administrative Console link in the bottom of the slideout menu.
  2. Click Manage Users.
  3. In the Manage menu on the left, click Users.
  4. On the Lookup tab, click View all users to list every user in the system, or search the user database for all users that match the criteria you enter, based on their first name, last name, or email address.
    The Users list with an Impersonate action button in each row
  5. Click Impersonate in the row of Actions for the user to display a table of all Applications this user has interacted with on the platform, including editor sessions and deployments.
    The Applications table for an impersonated user
  6. Click the Anaconda Platform link to interact with Workbench as the user.
See Managing users for more information on managing users.

​
Editor sessions

To troubleshoot issues with editor sessions, it helps to understand what happens behind the scenes.
  • When a user starts a session, Workbench launches the appropriate editor for them to work with their project files. In the background, the editor environment and other services are running in Docker containers.
  • To improve startup time for projects, the editor container includes conda environments for each of the project template environments provided by the platform. These environments are stored in /opt/continuum/anaconda/envs, along with any custom environments created during the editor session.
  • The project repository is cloned into /opt/continuum/project. (Only changes to files in this directory can be saved to the repository.)
  • The anaconda-project prepare command runs, scans the project’s anaconda-project.yml file for new packages and environments, and installs them into the running session. During this phase, you can monitor the progress by watching the output of /opt/continuum/preparing. When this process completes, the /opt/continuum/prepare.log is created.
Any changes made to the container image will be lost when the session stops, so any packages installed from the command line are available during the current session only. To persist package installs across sessions, they must be added to the project’s anaconda-project.yml file.

​
Helm rollback fails with a kubernetes-dashboard-csrf conflict

Running helm rollback anaconda-enterprise <REVISION> can fail with an error similar to:
This occurs because the kubernetes-dashboard-csrf secret is managed by both the Helm chart and the Kubernetes Dashboard application, which writes its own CSRF token to the same secret at runtime. helm rollback reapplies the secret content captured at the time of the target revision, which is typically out of date by the time a rollback is needed, causing Kubernetes to reject the conflicting write. This affects Workbench chart versions prior to ae-helm-chart-1.1-364, on both K3s and BYOK8s installations. To resolve it:
  1. Delete the conflicting secret and retry the rollback:
  2. Restart the Workbench Dashboard pod so it picks up the recreated secret (it caches its token at startup and will not notice the change otherwise):
    If no pods match this selector in your environment, locate the pod by name and delete it directly:
  3. Confirm the secret and pod are healthy:
    The csrf field should return a non-empty base64 value, and the Dashboard pod should show 1/1 Running.
See the Bring Your Own Kubernetes (BYOK8s) Upgrade guide for the full walkthrough, including how to find the revision to roll back to.