Procedure

1. Run must-gather:

   $ oc adm must-gather --image=registry.access.redhat.com/rhdh/rhdh-must-gather-rhel9:1.10

   Output: ./must-gather.local.<timestamp>

   Note

   The must-gather tool automatically detects and collects data from all RHDH instances in the cluster, regardless of whether they were deployed using the Operator or Helm chart.

   For advanced scenarios, you can limit data collection:

   1. Skip Helm-based deployments: -- /usr/bin/gather --without-helm
   2. Skip Operator-based deployments: -- /usr/bin/gather --without-operator

Procedure

1. Install must-gather:

   $ helm upgrade --install my-rhdh-must-gather redhat-developer-hub-must-gather \
     --repo https://charts.openshift.io \
     --namespace rhdh-diagnostics \
     --create-namespace

   Note

   The must-gather tool automatically detects and collects data from all RHDH instances in the cluster, regardless of whether they were deployed using the Operator or Helm chart.

   For advanced scenarios, you can limit data collection using Helm values:

   1. Skip Helm-based deployments: --set gather.withHelm=false
   2. Skip Operator-based deployments: --set gather.withOperator=false

2. Wait for collection:

   $ kubectl wait --for=condition=ready pod \
     -l app.kubernetes.io/instance=my-rhdh-must-gather,app.kubernetes.io/component=gather \
     --timeout=3600s -n rhdh-diagnostics

3. Extract data:

   $ kubectl exec deploy/my-rhdh-must-gather -c data-holder -n rhdh-diagnostics -- \
     tar czf - -C /must-gather . > rhdh-must-gather-output.tar.gz

4. Clean up:

   $ helm uninstall my-rhdh-must-gather -n rhdh-diagnostics

Procedure

1. Choose your mirroring workflow based on network access:

   Partially disconnected:

   Your local machine can access both the internet and the internal registry.

   Fully disconnected:

   Requires a bastion host or approved file transfer method to move images.

2. Mirror the must-gather image:

   For partially disconnected environments:

   $ skopeo copy \
     docker://registry.access.redhat.com/rhdh/rhdh-must-gather-rhel9:*<version>* \
     docker://<internal-registry>/rhdh/rhdh-must-gather:*<version>*

   For fully disconnected environments:

   On a machine with internet access, pull the must-gather image:

   $ skopeo copy \
     docker://registry.access.redhat.com/rhdh/rhdh-must-gather-rhel9:*<version>* \
     dir:./rhdh-must-gather-<version>

   Pull the Helm chart:

   $ helm pull redhat-developer-hub-must-gather --repo https://charts.openshift.io

   Transfer the image directory and the redhat-developer-hub-must-gather-<version>.tgz chart file to your bastion host. Push the image to the internal registry:

   $ skopeo copy \
     dir:./rhdh-must-gather-<version> \
     docker://<internal-registry>/rhdh/rhdh-must-gather:*<version>*

   Replace <version> with the must-gather version that matches your RHDH deployment and <internal-registry> with your internal registry hostname.

3. Configure image pull authentication:

   Note

   If your internal registry requires authentication, you must configure image pull secrets before running must-gather. Without proper credentials, the must-gather pod fails with ImagePullBackOff errors.

   On OpenShift Container Platform:

   Add your internal registry credentials to the cluster-wide pull secret. For instructions, see Updating the global cluster pull secret in the Red Hat OpenShift Container Platform documentation.

   On Kubernetes:

   Create a docker-registry secret:

   $ kubectl create secret docker-registry must-gather-pull-secret \
     --docker-server=<internal-registry> \
     --docker-username=<username> \
     --docker-password=<password> \
     -n rhdh-diagnostics

4. Run must-gather using the mirrored image:

   On OpenShift Container Platform:

   $ oc adm must-gather --image=<internal-registry>/rhdh/rhdh-must-gather:*<version>*

   On Kubernetes (partially disconnected):

   $ helm upgrade --install my-rhdh-must-gather redhat-developer-hub-must-gather \
     --repo https://charts.openshift.io \
     --namespace rhdh-diagnostics \
     --create-namespace \
     --set image.registry=<internal-registry> \
     --set image.repository=rhdh/rhdh-must-gather \
     --set image.tag=<version> \
     --set imagePullSecrets[0].name=must-gather-pull-secret

   On Kubernetes (fully disconnected):

   $ helm upgrade --install my-rhdh-must-gather /path/to/redhat-developer-hub-must-gather-<version>.tgz \
     --repo https://charts.openshift.io \
     --namespace rhdh-diagnostics \
     --create-namespace \
     --set image.registry=<internal-registry> \
     --set image.repository=rhdh/rhdh-must-gather \
     --set image.tag=<version> \
     --set imagePullSecrets[0].name=must-gather-pull-secret

Procedure

1. Choose the heap dump collection method:

   Inspector protocol (default):

   • Works out of the box
   • Use unless your deployment sets --disable-sigusr1 in NODE_OPTIONS

   SIGUSR2 signal:

   • Requires configuring NODE_OPTIONS in your deployment first
   • Use when the inspector protocol is not available

2. If using SIGUSR2, configure NODE_OPTIONS:

   Note

   Skip this step if using the inspector protocol (default method).

   Operator deployments:

   apiVersion: rhdh.redhat.com/v1alpha1
   kind: Backstage
   metadata:
     name: my-rhdh
   spec:
     application:
       extraEnvs:
         - name: NODE_OPTIONS
           value: "--heapsnapshot-signal=SIGUSR2 --diagnostic-dir=/tmp"

   Apply the change:

   $ oc apply -f backstage-cr.yaml

   Helm deployments:

   upstream:
     backstage:
       extraEnvVars:
         - name: NODE_OPTIONS
           value: "--heapsnapshot-signal=SIGUSR2 --diagnostic-dir=/tmp"

   Update the release:

   $ helm upgrade my-rhdh redhat-developer-hub/backstage -f values.yaml -n <namespace>

   Wait for pods to restart before proceeding.

3. Increase the liveness probe timeout to prevent pod restarts:

   Important

   RHDH stops responding during heap dump collection, and large memory footprints (1GB+) can cause liveness probe failures. Plan collection during maintenance windows or low-traffic periods.

   Operator deployments:

   Update the Backstage custom resource to patch the deployment:

   apiVersion: rhdh.redhat.com/v1alpha1
   kind: Backstage
   metadata:
     name: my-rhdh
   spec:
     deployment:
       patch:
         spec:
           template:
             spec:
               containers:
                 - name: backstage-backend
                   livenessProbe:
                     failureThreshold: 180

   Apply the change:

   $ oc apply -f backstage-cr.yaml

   Helm deployments:

   Update your Helm values:

   upstream:
     backstage:
       livenessProbe:
         failureThreshold: 180

   Apply the change:

   $ helm upgrade my-rhdh redhat-developer-hub/backstage -f values.yaml -n <namespace>

   Wait for pods to restart before proceeding.

4. Run must-gather with heap dumps enabled:

   On OpenShift Container Platform:

   For inspector protocol (default):

   $ oc adm must-gather --image=registry.access.redhat.com/rhdh/rhdh-must-gather-rhel9:1.10 -- /usr/bin/gather --with-heap-dumps

   For SIGUSR2 method:

   $ oc adm must-gather --image=registry.access.redhat.com/rhdh/rhdh-must-gather-rhel9:1.10 -- /usr/bin/gather --with-heap-dumps --heap-dump-method sigusr2

   On supported Kubernetes platforms:

   For inspector protocol (default):

   gather:
     heapDump:
       enabled: true
       method: "inspector"

   For SIGUSR2 method:

   gather:
     heapDump:
       enabled: true
       method: "sigusr2"

   Collection takes 5-15 minutes depending on pod count and memory size.