Monitoring and logging
Tracking performance and capturing insights with monitoring and logging tools in Red Hat Developer Hub
Abstract
1. Enabling observability for Red Hat Developer Hub on OpenShift Container Platform
In OpenShift Container Platform, metrics are exposed through an HTTP service endpoint under the /metrics
canonical name. You can create a ServiceMonitor
custom resource (CR) to scrape metrics from a service endpoint in a user-defined project.
1.1. Enabling metrics monitoring in a Red Hat Developer Hub Operator installation on an OpenShift Container Platform cluster
You can enable and view metrics for an Operator-installed Red Hat Developer Hub instance from the Developer perspective of the OpenShift Container Platform web console.
Prerequisites
- Your OpenShift Container Platform cluster has monitoring for user-defined projects enabled.
- You have installed Red Hat Developer Hub on OpenShift Container Platform using the Red Hat Developer Hub Operator.
-
You have installed the OpenShift CLI (
oc
).
Procedure
Currently, the Red Hat Developer Hub Operator does not support creating a ServiceMonitor
custom resource (CR) by default. You must complete the following steps to create a ServiceMonitor
CR to scrape metrics from the endpoint.
Create the
ServiceMonitor
CR as a YAML file:apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: <developer_hub_service_monitor_name> 1 namespace: <rhdh_namespace_name> 2 labels: app.kubernetes.io/instance: <rhdh_cr_name> 3 app.kubernetes.io/name: Backstage spec: namespaceSelector: matchNames: - <rhdh_namespace_name> 4 selector: matchLabels: app.kubernetes.io/instance: <deployment_name> 5 app.kubernetes.io/name: <rhdh_cr_type> 6 endpoints: - port: http-metrics path: '/metrics'
- 1
- The name of your
ServiceMonitor
resource, for example,developer_hub_service_monitor
. - 2
- The namespace where your
ServiceMonitor
will live, for example,my-rhdh-project
. - 3
- The label name identifying the
ServiceMonitor
CR instance, for example,my-rhdh-custom-resource
. - 4
- The namespace where your RHDH instance is installed, for example,
my-rhdh-project
. - 5
- The name of your RHDH deployment, for example,
developer-hub
. - 6
- The name of your RHDH application, for example,
backstage
.
Notespec.selector.matchLabels
configuration must match the labels of your RHDH installation.Apply the
ServiceMonitor
CR by running the following command:oc apply -f <filename>
Verification
- From the Developer perspective in the OpenShift Container Platform web console, select the Observe view.
- Click the Metrics tab to view metrics for Red Hat Developer Hub pods.
-
From the Developer perspective in the OpenShift Container Platform web console, click Project > Services and verify the labels for
backstage-developer-hub
.
1.2. Enabling metrics monitoring in a Helm chart installation on an OpenShift Container Platform cluster
You can enable and view metrics for a Red Hat Developer Hub Helm deployment from the Developer perspective of the OpenShift Container Platform web console.
Prerequisites
- Your OpenShift Container Platform cluster has monitoring for user-defined projects enabled.
- You have installed Red Hat Developer Hub on OpenShift Container Platform using the Helm chart.
Procedure
- From the Developer perspective in the OpenShift Container Platform web console, select the Topology view.
Click the overflow menu of the Red Hat Developer Hub Helm chart, and select Upgrade.
On the Upgrade Helm Release page, select the YAML view option in Configure via, then configure the
metrics
section in the YAML, as shown in the following example:upstream: # ... metrics: serviceMonitor: enabled: true path: /metrics port: http-metrics # ...
- Click Upgrade.
Verification
- From the Developer perspective in the OpenShift Container Platform web console, select the Observe view.
- Click the Metrics tab to view metrics for Red Hat Developer Hub pods.
Additional resources
2. Monitoring and logging Red Hat Developer Hub on Amazon Web Services (AWS)
You can configure Red Hat Developer Hub to use Amazon CloudWatch for real-time monitoring and Amazon Prometheus for comprehensive logging. This is convenient when hosting Developer Hub on Amazon Web Services (AWS) infrastructure.
2.1. Monitoring with Amazon Prometheus
You can configure Red Hat Developer Hub to use Amazon Prometheus for comprehensive logging. Amazon Prometheus extracts data from pods that have specific pod annotations.
2.1.1. Prerequisites
- You configured Prometheus for your Elastic Kubernetes Service (EKS) clusters.
- You created an Amazon managed service for the Prometheus workspace.
- You configured Prometheus to import the Developer Hub metrics.
- You ingested Prometheus metrics into the created workspace.
2.1.2. Configuring annotations for monitoring with Amazon Prometheus by using the Red Hat Developer Hub Operator
To enable logging to Amazon Prometheus, you can configure the required pod annotations by using the Red Hat Developer Hub Operator.
Procedure
As an administrator of the Red Hat Developer Hub Operator, edit the default configuration to add Prometheus annotations as follows:
# Update OPERATOR_NS accordingly $ OPERATOR_NS=rhdh-operator $ kubectl edit configmap backstage-default-config -n "${OPERATOR_NS}"
Find the
deployment.yaml
key in the config map and add the annotations to thespec.template.metadata.annotations
field as follows:deployment.yaml: |- apiVersion: apps/v1 kind: Deployment # --- truncated --- spec: template: # --- truncated --- metadata: labels: rhdh.redhat.com/app: # placeholder for 'backstage-<cr-name>' # --- truncated --- annotations: prometheus.io/scrape: 'true' prometheus.io/path: '/metrics' prometheus.io/port: '9464' prometheus.io/scheme: 'http' # --- truncated ---
- Save your changes.
Verification
To verify if the scraping works:
Use
kubectl
to port-forward the Prometheus console to your local machine as follows:$ kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090
-
Open your web browser and navigate to
http://localhost:9090
to access the Prometheus console. -
Monitor relevant metrics, such as
process_cpu_user_seconds_total
.
2.1.3. Configuring annotations for monitoring with Amazon Prometheus by using the Red Hat Developer Hub Helm chart
To enable logging to Amazon Prometheus, you can configure the required pod annotations by using the Red Hat Developer Hub Helm chart.
Procedure
To annotate the backstage pod for monitoring, update your
values.yaml
file as follows:upstream: backstage: # --- TRUNCATED --- podAnnotations: prometheus.io/scrape: 'true' prometheus.io/path: '/metrics' prometheus.io/port: '9464' prometheus.io/scheme: 'http'
Verification
To verify if the scraping works:
Use
kubectl
to port-forward the Prometheus console to your local machine as follows:kubectl --namespace=prometheus port-forward deploy/prometheus-server 9090
-
Open your web browser and navigate to
http://localhost:9090
to access the Prometheus console. -
Monitor relevant metrics, such as
process_cpu_user_seconds_total
.
2.2. Logging with Amazon CloudWatch
Logging within the Red Hat Developer Hub relies on the Winston library. The default logging level is info
. To have more detailed logs, set the LOG_LEVEL
environment variable to debug
in your Red Hat Developer Hub instance.
2.2.1. Configuring the application log level by using the Red Hat Developer Hub Operator
You can configure the application log level by using the Red Hat Developer Hub Operator.
Procedure
Modify the logging level by including the environment variable
LOG_LEVEL
in your custom resource as follows:spec: # Other fields omitted application: extraEnvs: envs: - name: LOG_LEVEL value: debug
2.2.2. Configuring the application log level by using the Red Hat Developer Hub Helm chart
You can configure the application log level by using the Red Hat Developer Hub Helm chart.
Procedure
Modify the logging level by adding the environment variable
LOG_LEVEL
to your Helm chartvalues.yaml
file:upstream: backstage: # --- Truncated --- extraEnvVars: - name: LOG_LEVEL value: debug
2.2.3. Retrieving logs from Amazon CloudWatch
Prerequisites
- CloudWatch Container Insights is used to capture logs and metrics for Amazon Elastic Kubernetes Service. For more information, see Logging for Amazon Elastic Kubernetes Service documentation.
- To capture the logs and metrics, install the Amazon CloudWatch Observability EKS add-on in your cluster. Following the setup of Container Insights, you can access container logs using Logs Insights or Live Tail views.
CloudWatch names the log group where all container logs are consolidated in the following manner:
/aws/containerinsights/<cluster_name>/application
Procedure
To retrieve logs from the Developer Hub instance, run a query such as:
fields @timestamp, @message, kubernetes.container_name | filter kubernetes.container_name in ["install-dynamic-plugins", "backstage-backend"]
3. Monitoring and logging with Azure Kubernetes Services (AKS) in Red Hat Developer Hub
Monitoring and logging are integral aspects of managing and maintaining Azure Kubernetes Services (AKS) in Red Hat Developer Hub. With features like Managed Prometheus Monitoring and Azure Monitor integration, administrators can efficiently monitor resource utilization, diagnose issues, and ensure the reliability of their containerized workloads.
3.1. Enabling Azure Monitor metrics
To enable managed Prometheus monitoring, use the -enable-azure-monitor-metrics
option within either the az aks create
or az aks update
command, depending on whether you’re creating a new cluster or updating an existing one, such as:
az aks create/update --resource-group <your-ResourceGroup> --name <your-Cluster> --enable-azure-monitor-metrics
The previous command installs the metrics add-on, which gathers Prometheus metrics. Using the previous command, you can enable monitoring of Azure resources through both native Azure Monitor metrics. You can also view the results in the portal under Monitoring → Insights. For more information, see Monitor Azure resources with Azure Monitor.
Furthermore, metrics from both the Managed Prometheus service and Azure Monitor can be accessed through Azure Managed Grafana service. For more information, see Link a Grafana workspace section.
By default, Prometheus uses the minimum ingesting profile, which optimizes ingestion volume and sets default configurations for scrape frequency, targets, and metrics collected. The default settings can be customized through custom configuration. Azure offers various methods, including using different ConfigMaps, to provide scrape configuration and other metric add-on settings. For more information about default configuration, see Default Prometheus metrics configuration in Azure Monitor and Customize scraping of Prometheus metrics in Azure Monitor managed service for Prometheus documentation.
3.2. Configuring annotations for monitoring
You can configure the annotations for monitoring Red Hat Developer Hub specific metrics in both Helm deployment and Operator-backed deployment.
- Helm deployment
To annotate the backstage pod for monitoring, update your
values.yaml
file as follows:upstream: backstage: # --- TRUNCATED --- podAnnotations: prometheus.io/scrape: 'true' prometheus.io/path: '/metrics' prometheus.io/port: '9464' prometheus.io/scheme: 'http'
- Operator-backed deployment
Procedure
As an administrator of the operator, edit the default configuration to add Prometheus annotations as follows:
# Update OPERATOR_NS accordingly OPERATOR_NS=rhdh-operator kubectl edit configmap backstage-default-config -n "${OPERATOR_NS}"
Find the
deployment.yaml
key in the ConfigMap and add the annotations to thespec.template.metadata.annotations
field as follows:deployment.yaml: |- apiVersion: apps/v1 kind: Deployment # --- truncated --- spec: template: # --- truncated --- metadata: labels: rhdh.redhat.com/app: # placeholder for 'backstage-<cr-name>' # --- truncated --- annotations: prometheus.io/scrape: 'true' prometheus.io/path: '/metrics' prometheus.io/port: '9464' prometheus.io/scheme: 'http' # --- truncated ---
- Save your changes.
Verification
To verify if the scraping works, navigate to the corresponding Azure Monitor Workspace and view the metrics under Monitoring → Metrics.
3.3. Viewing logs with Azure Kubernetes Services (AKS)
You can access live data logs generated by Kubernetes objects and collect log data in Container Insights within AKS.
Prerequisites
- You have deployed Developer Hub on AKS.
For more information, see Installing Red Hat Developer Hub on Microsoft Azure Kubernetes Service.
Procedure
- View live logs from your Developer Hub instance
- Navigate to the Azure Portal.
-
Search for the resource group
<your-ResourceGroup>
and locate your AKS cluster<your-Cluster>
. - Select Kubernetes resources → Workloads from the menu.
-
Select the
<your-rhdh-cr>-developer-hub
(in case of Helm Chart installation) or<your-rhdh-cr>-backstage
(in case of Operator-backed installation) deployment. - Click Live Logs in the left menu.
Select the pod.
NoteThere must be only single pod.
Live log data is collected and displayed.
- View real-time log data from the Container Engine
- Navigate to the Azure Portal.
-
Search for the resource group
<your-ResourceGroup>
and locate your AKS cluster<your-Cluster>
. - Select Monitoring → Insights from the menu.
- Go to the Containers tab.
- Find the backend-backstage container and click it to view real-time log data as it’s generated by the Container Engine.