Procedure

1. Locate your Developer Hub application configuration file.

   The location depends on your deployment method:

   • For Operator deployments: The configuration is in a ConfigMap, typically named my-rhdh-app-config.
   • For Helm deployments: The configuration is in the values.yaml file or a custom configuration file referenced in your Helm values.

2. Add the orchestrator.kafka configuration section to your app-config.yaml file.

   orchestrator:
     kafka:
       clientId: my-rhdh-orchestrator
       brokers:
         - kafka-broker-1.example.com:9092
         - kafka-broker-2.example.com:9092
         - kafka-broker-3.example.com:9092
       # logLevel override for the orchestrator kafka services. Defaults to INFO which is 4
       # logLevel values based on KafkaJS values https://kafka.js.org/docs/configuration#logging
       # logLevel: 5 // DEBUG
       logLevel: 4

   where:

   clientId

   Unique identifier for the RHDH Kafka client. This identifier is displayed in Kafka broker logs and metrics.

   brokers

   Array of Kafka broker URLs. Include multiple brokers for high availability.

   logLevel

   Optional. Kafka client logging level. Valid numeric values based on KafkaJS values are 0 (NOTHING), 1 (ERROR), 2 (WARN), 4 (INFO), or 5 (DEBUG). Default is 4 (INFO).

3. Apply the configuration changes.

   • For Operator deployments: Update the ConfigMap and restart the RHDH instance by scaling the deployment to zero and back to the target replica count, or by deleting the pod to trigger a restart.

     Replace <my_deployment_name> with the name of your deployment:

     $ oc rollout restart deployment/<my_deployment_name>

   • For Helm deployments: Upgrade the Helm release with the updated configuration.

     $ helm upgrade my-rhdh-custom-resource redhat-developer/backstage -f values.yaml -n my-rhdh-project

Verification

1. Check the orchestrator-backend plugin logs for Kafka connection messages.

   Replace <my_deployment_name> with the name of your deployment:

   $ oc logs deployment/<my_deployment_name> | grep -i kafka

   Successful connection logs include messages indicating the Kafka client has connected to the broker cluster.

2. Navigate to the Orchestrator plugin in the RHDH UI.

3. Verify that the Run as Event button is displayed next to workflows.

   The button is only visible when Kafka connectivity is successfully configured.

Procedure

1. In the RHDH UI, navigate to the Orchestrator plugin.
2. In the workflows list, locate the workflow you want to trigger.

3. Click the Run as Event button next to the workflow.

   Note

   The Run as Event button appears only when you have configured Kafka connectivity and the workflow supports event-driven execution.

4. If the workflow requires input data, complete the workflow input form.

   The form fields correspond to the workflow’s input schema. The CloudEvent data payload includes the values you provide.

5. Click Submit to send the CloudEvent to Kafka.

   The RHDH UI transmits a CloudEvent to the configured Kafka broker with the workflow input data. The workflow starts when the Kafka broker delivers the event to the SonataFlow engine.

6. Monitor the workflow status.

   After submitting the CloudEvent, one of two outcomes occurs:

   Immediate start

   If the workflow starts before the UI timeout period, the UI navigates to the workflow instance detail page, where you can monitor progress.

   Delayed start

   If the workflow has not started when the UI timeout expires, the UI displays an informational message indicating that it sent the event to Kafka as a kafkaEvent. The UI navigates to the workflow runs list, where the workflow instance appears when the workflow starts.

7. If the workflow does not start immediately, locate the workflow in the workflow runs list.

   The workflow instance appears in the list when the Kafka broker delivers the CloudEvent and the workflow engine starts the workflow. Depending on Kafka broker latency and workflow engine processing time, this can take several seconds.

Procedure

1. In your plugin that implements OrchestratorFormApi, import the required types:

   import type {
     OrchestratorFormApi,
     ReviewComponentProps,
   } from '@red-hat-developer-hub/backstage-plugin-orchestrator-form-api';

2. Import the helper utilities from the orchestrator-form-react package:

   import {
     generateReviewTableData,
     schemaHasUiHiddenFields,
     ReviewHiddenParametersAlert,
     NestedReviewTable,
   } from '@red-hat-developer-hub/backstage-plugin-orchestrator-form-react';

   These utilities handle hidden fields, password masking, and nested data structures in your custom review page.

3. Create your custom review page component:

   import React from 'react';
   import { Button, Box, Typography } from '@mui/material';
   
   export const CustomReviewPage = (props: ReviewComponentProps) => {
     const { busy, schema, data, handleBack, handleExecute } = props;
     const [showHiddenFields, setShowHiddenFields] = React.useState(false);
   
     const reviewData = React.useMemo(
       () => generateReviewTableData(schema, data, {
         includeHiddenFields: showHiddenFields
       }),
       [schema, data, showHiddenFields]
     );
   
     const hasHiddenFields = schemaHasUiHiddenFields(schema);
   
     return (
       <Box>
         <Typography variant="h5">Review Your Workflow Data</Typography>
   
         {hasHiddenFields && (
           <ReviewHiddenParametersAlert
             showHiddenFields={showHiddenFields}
             onShowHiddenFieldsChange={setShowHiddenFields}
           />
         )}
   
         <NestedReviewTable data={reviewData} />
   
         <Box sx={{ mt: 2, display: 'flex', gap: 1 }}>
           <Button onClick={handleBack} disabled={busy}>
             Back
           </Button>
           <Button
             variant="contained"
             onClick={handleExecute}
             disabled={busy}
           >
             Execute Workflow
           </Button>
         </Box>
       </Box>
     );
   };

4. Add the getReviewComponent() method to your OrchestratorFormApi implementation:

   export class MyFormApi implements OrchestratorFormApi {
     getReviewComponent() {
       return CustomReviewPage;
     }
   
     // ... other OrchestratorFormApi methods
   }

5. Register your custom form API with the Orchestrator plugin according to your plugin’s extension mechanism.

Verification

1. Open the Orchestrator plugin in the Developer Hub web interface.
2. Select a workflow and complete the workflow form.
3. Proceed to the review step.
4. Confirm that your custom review page displays with the correct layout and styling.
5. Click Back and confirm that the workflow form is populated.
6. Click Execute Workflow and verify that the workflow runs successfully.

Procedure

1. To enable the Orchestrator plugins with default settings, set disabled: false for the corresponding packages:

   - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator:{{inherit}}"
    disabled: false
   - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend:{{inherit}}"
     disabled: false
     dependencies:
       - ref: sonataflow
   - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator:{{inherit}}"
     disabled: false
   - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets:{{inherit}}"
     disabled: false

   Note

   When you enable the plugins, the pre-loaded plugin configuration are used. Additionally, the ref: sonataflow field installs the OpenShift Serverless and OpenShift Serverless Logic resources. This happens automatically when you are using the Operator.

   The following example shows a complete configuration of the Orchestrator plugin:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: orchestrator-plugin
   data:
       dynamic-plugins.yaml: |
         includes:
           - dynamic-plugins.default.yaml
         plugins:
           - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator:{{inherit}}"
             disabled: false
           - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend:{{inherit}}"
             disabled: false
             dependencies:
               - ref: sonataflow
           - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator:{{inherit}}"
             disabled: false
           - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets:{{inherit}}"
             disabled: false
   ---
   apiVersion: rhdh.redhat.com/v1alpha5
   kind: Backstage
   metadata:
     name: orchestrator
   spec:
     application:
       appConfig:
         configMaps:
           - name: app-config-rhdh
       dynamicPluginsConfigMapName: orchestrator-plugin

2. Create a secret containing the BACKEND_SECRET value as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: app-config-rhdh
   data:
     app-config.yaml: |-
       auth:
         environment: development
         providers:
           guest:
             # using the guest user to query the '/api/dynamic-plugins-info/loaded-plugins' endpoint.
             dangerouslyAllowOutsideDevelopment: true
       backend:
         auth:
           externalAccess:
             - type: static
               options:
                 token: ${BACKEND_SECRET}
                 subject: orchestrator
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: backend-auth-secret
   stringData:
     # generated with the command below (from https://backstage.io/docs/auth/service-to-service-auth/#setup):
     # node -p 'require("crypto").randomBytes(24).toString("base64")'
     # notsecret
     BACKEND_SECRET: "R2FxRVNrcmwzYzhhN3l0V1VRcnQ3L1pLT09WaVhDNUEK"

3. Configure your Backstage CR to update the secret name in the extraEnvs field as shown in the following example:

   apiVersion: rhdh.redhat.com/v1alpha5
   kind: Backstage
   metadata:
     name: orchestrator
   spec:
     application:
       appConfig:
         configMaps:
           - name: app-config-rhdh
       dynamicPluginsConfigMapName: orchestrator-plugin
       extraEnvs:
         secrets:
             # secret that contains the BACKEND_SECRET key
           - name: backend-auth-secret

Procedure

1. Delete the previous logic-operator-rhel8 subscription and apply the following configuration to install the logic-operator subscription:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-serverless-logic
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-serverless-logic
     namespace: openshift-serverless-logic
   spec:
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: logic-operator
     namespace: openshift-serverless-logic
   spec:
     channel: stable  #  channel of an operator package to subscribe to
     installPlanApproval: Automatic #  whether the update should be installed automatically
     name: logic-operator  #  name of the operator package
     source: redhat-operators  #  name of the catalog source
     sourceNamespace: openshift-marketplace
     startingCSV: logic-operator.v1.37.2  # The initial version of the operator

2. Optional: If your configuration uses an external PostgreSQL database with SSL, add the required datasource environment variables to the jobService specification in the SonataflowPlatform custom resource as shown in the following configuration:

   jobService:
     enabled: true
     persistence:
       dbMigrationStrategy: service
       postgresql:
         # no additional url params here. We only have currentSchema=jobs-service.
         jdbcUrl: 'jdbc:postgresql://postgress-external-db-primary.postgress-external-db.svc.cluster.local:5432/sonataflow?currentSchema=jobs-service'
         secretRef:
           name: postgres-cred
           passwordKey: POSTGRES_PASSWORD
           userKey: POSTGRES_USER
     podTemplate:
       container:
         env:
           # only this two env vars
           - name: QUARKUS_DATASOURCE_REACTIVE_POSTGRESQL_SSL_MODE
             value: allow
           - name: QUARKUS_DATASOURCE_REACTIVE_TRUST_ALL
             value: 'true'

Procedure

1. Open your dynamic-plugins ConfigMap for editing.

2. Update the package references for the Orchestrator plugins to use the 1.9 OCI registry paths as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: dynamic-plugins-rhdh
   data:
     dynamic-plugins.yaml: |
       includes:
         - dynamic-plugins.default.yaml
       plugins:
         - package: 'oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator:{{inherit}}'
           disabled: false
         - package: 'oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend:{{inherit}}'
           disabled: false
           dependencies:
             - ref: sonataflow
         - package: 'oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator:{{inherit}}'
           disabled: false
         - package: 'oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets:{{inherit}}'
           disabled: false

   Note

   The {{inherit}} attribute in your configuration automatically resolves to the 1.9 version provided by the Operator.

3. Save the configuration changes.

Verification

1. Log in to your Red Hat Developer Hub instance.
2. Confirm that the Orchestrator plugins display the version metadata for 1.9.

Procedure

1. Edit the dynamic-plugins ConfigMap to update the Orchestrator plugin version to 1.8.12:

   $ oc edit configmap dynamic-plugins-rhdh -n <your_namespace>

2. Update the plugin versions in the ConfigMap:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: dynamic-plugins-rhdh
   data:
     dynamic-plugins.yaml: |
       includes:
         - dynamic-plugins.default.yaml
       plugins:
         - package: "@redhat/backstage-plugin-orchestrator@1.8.12"
           disabled: false
         - package: "@redhat/backstage-plugin-orchestrator-backend-dynamic@1.8.12"
           disabled: false
           dependencies:
             - ref: sonataflow
         - package: "@redhat/backstage-plugin-scaffolder-backend-module-orchestrator-dynamic@1.8.12"
           disabled: false
         - package: "@redhat/backstage-plugin-orchestrator-form-widgets@1.8.12"
           disabled: false

3. Save and close the ConfigMap. The RHDH pods restart automatically.

Verification

1. Monitor the status of the RHDH pods to ensure they restart:

   $ oc get pods -w

2. Verify that all RHDH pods are in Running status with no errors.

Procedure

1. Identify the workflowId from your workflow definition file:

   id: greeting
   version: '1.0'

2. In your RBAC policy CSV file, define the permissions using the p, role, permission, action, allow format.

   Note

   Generic permissions (for example, orchestrator.workflow) take precedence over specific permissions targeting a workflowId, (for example, orchestrator.workflow.greeting). You cannot grant generic access and then selectively deny a specific ID.

3. Add the following example policies to your CSV file to establish basic user and administrator roles:

   # Minimal user role - can only view and run specific workflows
   p, role:default/workflowUser, orchestrator.workflow.greeting, read, allow
   p, role:default/workflowUser, orchestrator.workflow.use.greeting, update, allow
   
   # Support role - can view all workflows and instances, but not execute
   p, role:default/workflowSupport, orchestrator.workflow, read, allow
   p, role:default/workflowSupport, orchestrator.instanceAdminView, read, allow
   
   # Full admin role - complete access to all Orchestrator functions
   p, role:default/workflowAdmin, orchestrator.workflow, read, allow
   p, role:default/workflowAdmin, orchestrator.workflow.use, update, allow
   p, role:default/workflowAdmin, orchestrator.workflowAdminView, read, allow
   p, role:default/workflowAdmin, orchestrator.instanceAdminView, read, allow
   
   # Assign users to the roles
   g, user:default/example_user, role:default/workflowUser

4. In your RHDH app-config.yaml file, enable permissions by adding the orchestrator plugin to the rbac section and setting policyFileReload to true.

   permission:
     enabled: true
     rbac:
       policies-csv-file: <absolute_path_to_the_policy_file>
       pluginsWithPermission:
         - orchestrator
       policyFileReload: true
       admin:
         users:
           - name: user:default/YOUR_USER

5. Restart the application to apply the changes.

Verification

1. Log in as a user assigned to the workflowUser role.
2. Navigate to the Orchestrator plugin and verify that the workflow appears in the list.

Procedure

1. Manually approve the install plans for the Operators. You must run the oc patch installplan commands provided in the output to approve their installation.

   Important

   By default, Orchestrator Infrastructure for Red Hat Developer Hub Helm chart does not auto-approve the required Serverless Operators. You must manually approve the install plans.

2. As an administrator, install relevant cluster-wide resources.

   $ helm repo add openshift-helm-charts https://charts.openshift.io/
   $ helm install <release_name> openshift-helm-charts/redhat-developer-hub-orchestrator-infra

   Important

   You must be an administrator to install the redhat-developer-hub-orchestrator-infra Helm chart because it deploys additional cluster-scoped OpenShift Serverless and OpenShift Serverless Logic Operators. As an administrator, you must manually approve the install plans for OpenShift Serverless and Serverless Logic Operators.

3. Install the Backstage chart with the orchestrator enabled as shown in the following example:

   $ helm install <release_name> openshift-helm-charts/redhat-developer-hub --version 1.10.0 \
     --set orchestrator.enabled=true

4. (Optional) Enable Notifications and Signals plugins by adding them to the global.dynamic.plugins list in your values.yaml file as shown in the following example:

   global:
     dynamic:
       plugins:
         - disabled: false
           package: "./dynamic-plugins/dist/backstage-plugin-notifications"
         - disabled: false
           package: "./dynamic-plugins/dist/backstage-plugin-signals"
         - disabled: false
           package: "./dynamic-plugins/dist/backstage-plugin-notifications-backend-dynamic"
         - disabled: false
           package: "./dynamic-plugins/dist/backstage-plugin-signals-backend-dynamic"

5. (Optional) You can disable the Serverless Logic and Serverless Operators individually or together by setting their values to false, as shown in the following example:

   $ helm install <release_name> openshift-helm-charts/redhat-developer-hub \
     --version 1.10.0 \
     --set orchestrator.enabled=true \
     --set orchestrator.serverlessOperator=false \
     --set orchestrator.serverlessLogicOperator=false

6. (Optional) If you are using an external database, add the following configuration under orchestrator.sonataflowPlatform in your values.yaml file:

   orchestrator:
     sonataflowPlatform:
       externalDBsecretRef: "<cred-secret>"
       externalDBName: "<database_name>" # The name of the user-configured existing database (Not the database that the orchestrator and sonataflow resources use).
       externalDBHost: "<database_host>"
       externalDBPort: "<database_port>"

   Note

   This step only configures the Orchestrators use of an external database. To configure Red Hat Developer Hub to use an external PostgreSQL instance, follow the steps in Configure a PostgreSQL instance using Helm.

Verification

1. Verify that the Orchestrator plugin is visible in the Red Hat Developer Hub UI.
2. Create and run sample workflows to confirm the orchestration is functioning correctly.

Procedure

1. In the OpenShift Container Platform web console, go to the Helm Charts and verify that the Red Hat Developer Hub Helm chart repository is available.

2. Search for the Orchestrator infrastructure for Red Hat Developer Hub and select Install.

   Important

   You must be an administrator to install the Orchestrator Infrastructure for Red Hat Developer Hub Helm chart because it deploys cluster-scoped resources. As an administrator, you must manually approve the install plans for OpenShift Serverless and Serverless Logic Operators.

   As a regular user, search for the Red Hat Developer Hub chart and install it by setting the value of orchestrator.enabled to true. Otherwise, the Orchestrator will not be deployed.

3. Wait until they are successfully deployed.
4. Monitor the deployment status by navigating to Pods or releases.

1. The orchestrator-related pods are running in the selected namespace.
2. Cluster-wide resources are present.
3. You can start connecting the orchestrator to your Red Hat Developer Hub UI.

1. Default resource limits

1. You can override these values in any of the following ways:

   • With values.yaml
   • With --set flags

2. Override defaults with values.yaml as shown in the following example:

   orchestrator:
     enabled: true
     sonataflowPlatform:
     resources:
         limits:
           cpu: "500m"
           memory: "1Gi"

3. Override with --set as shown in the following example:

   $ helm upgrade --install <release_name>  openshift-helm-charts/redhat-developer-hub \
     --set orchestrator.enabled=true \
     --set orchestrator.sonataflowPlatform.resources.requests.cpu=500m \
     --set orchestrator.sonataflowPlatform.resources.requests.memory=128Mi \
     --set orchestrator.sonataflowPlatform.resources.limits.cpu=1 \
     --set orchestrator.sonataflowPlatform.resources.limits.memory=2Gi

   Note

   The --set setting is applicable only when orchestrator.enabled is true. By default, it is set to false.

Procedure

1. Install the OpenShift Serverless components manually by following the instructions in the Red Hat OpenShift Serverless documentation.

2. (Optional) If required, deploy a custom PostgreSQL database.

   Important

   Prevent workflow context from being lost when the Pod restarts by configuring workflow persistence. You can configure persistence at the namespace level by using the SonataFlowPlatform or SonataFlow custom resources (CR). For more information, check the Managing workflow persistence documentation.

Procedure

1. Create an ImageSetConfiguration file for oc-mirror. You must include the images and operators required by the Serverless Logic Operator in the ImageSetConfiguration file, as shown in the following example:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalimages:
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@<digest>
   
     operators:
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v<ocp-version>
        # For example: registry.redhat.io/redhat/redhat-operator-index:v4.21
         packages:
         - name: logic-operator
           channels:
           - name: stable
             minVersion: 1.37.2
             maxVersion: 1.37.2
         - name: serverless-operator
           channels:
           - name: stable
             minVersion: 1.37.1
             maxVersion: 1.37.1

   where:

   <digest>

   Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

   #!/bin/bash
   
   unpack () {
     local IMAGE="$1"
     DIR="${IMAGE//:/}" DIR="/tmp/${DIR//\//-}" rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}") podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar echo "Unpacked $IMAGE into $DIR" cd $DIR; tree -d -L 3 -I "usr|root|buildinfo" } unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:{product-version)" # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index{product-version)/dynamic-plugins.default.yaml

2. Mirror the images in the ImageSetConfiguration.yaml file by running the oc-mirror command. For example:

   $ oc-mirror --config=ImageSetConfiguration.yaml file:///path/to/mirror-archive --authfile /path/to/authfile  --v2

   Note

   The oc-mirror command generates a local workspace containing the mirror archive files and the required cluster manifests.

3. Transfer the directory specified by /path/to/mirror-archive to a bastion host within your disconnected environment.

4. From the bastion host which has access to the mirror registry, mirror the images from the disk directory to your target registry. For example:

   $ oc-mirror --v2 --from <mirror-archive-file> docker://<target-registry-url:port> --workspace file://<workspace folder> --authfile /path/to/authfile

   where:

   <mirror-archive-file>

   Enter the name of the transferred tar file.

   <target-registry-url:port>

   Enter your local registry, for example, registry.localhost:5000.

5. Apply the cluster-wide resources generated during the push step to redirect all image pulls to your local registry, as shown in the following example:

   $ cd <workspace folder>/working-dir/cluster-resources/
   $ oc apply -f .

6. Install the OpenShift Serverless Operator and OpenShift Serverless Logic Operators using OperatorHub.
7. Create a Backstage custom resource (CR).

8. Configure the Backstage CR for the Orchestrator as described in the Orchestrator plugin dependencies for Operator installation.

   Create all the resources and configure the Backstage instance accordingly.

Verification

1. Restart the RHDH pod and wait for the components to deploy properly.
2. Once stable, go to the RHDH UI, and confirm that the Orchestrator UI is accessible and functioning correctly.

Procedure

1. Create an ImageSetConfiguration file for oc-mirror. You must include the images and operators required by the Serverless Logic Operator in the ImageSetConfiguration file, as shown in the following example:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalimages:
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@<digest>
   
     operators:
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v<ocp-version>
        # For example: registry.redhat.io/redhat/redhat-operator-index:v4.21
         packages:
         - name: logic-operator
           channels:
           - name: stable
             minVersion: 1.37.2
             maxVersion: 1.37.2
         - name: serverless-operator
           channels:
           - name: stable
             minVersion: 1.37.1
             maxVersion: 1.37.1

   Where:

   <digest>

   Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

   #!/bin/bash
   
   unpack () {
     local IMAGE="$1"
     DIR="${IMAGE//:/}" DIR="/tmp/${DIR//\//-}" rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}") podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar echo "Unpacked $IMAGE into $DIR" cd $DIR; tree -d -L 3 -I "usr|root|buildinfo" } unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:{product-version)" # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index{product-version)/dynamic-plugins.default.yaml

2. Mirror the images in the ImageSetConfiguration.yaml file by running the oc-mirror command. For example:

   $ oc-mirror --config=imagesetconfiguration.yaml docker://<registry URL:port> --workspace file://<workspace folder> --authfile /path/to/authfile  --v2
   $ cd <workspace folder>/working-dir/cluster-resources/
   $ oc apply -f .

3. Install the OpenShift Serverless Operator and OpenShift Serverless Logic Operators using OperatorHub.
4. Create a Backstage custom resource (CR).

5. Configure the Backstage CR for the Orchestrator as described in the Orchestrator plugin dependencies for Operator installation.

   Create all the resources and configure the Backstage instance accordingly.

Verification

1. Restart the RHDH pod and wait for the components to deploy properly.
2. Once stable, go to the RHDH UI, and confirm that the Orchestrator UI is accessible and functioning correctly.

Procedure

1. Create an ImageSetConfiguration.yaml file for oc-mirror. You must use an ImageSetConfiguration file to include all mirrored images required, as shown in the following example:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalimages:
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@<digest>
   
     helm:
       repositories:
         - name: openshift-charts
           url: https://charts.openshift.io
           charts:
             - name: redhat-developer-hub
               version: "1.10.0"
             - name: redhat-developer-hub-orchestrator-infra
               version: "1.10.0"
     operators:
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v<ocp-version>
        # For example: registry.redhat.io/redhat/redhat-operator-index:v4.21
         packages:
         - name: logic-operator
           channels:
           - name: stable
             minVersion: 1.37.2
             maxVersion: 1.37.2
         - name: serverless-operator
           channels:
           - name: stable
             minVersion: 1.37.1
             maxVersion: 1.37.1

   where:

   <digest>

   Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

   #!/bin/bash
   
   unpack () {
     local IMAGE="$1"
     DIR="${IMAGE//:/_}"
     DIR="/tmp/${DIR//\//-}"
     rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}")
     podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar
     echo "Unpacked $IMAGE into $DIR"
     cd $DIR; tree -d -L 3 -I "usr|root|buildinfo"
   }
   
   unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:1.9"
   
   # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index_1.9/dynamic-plugins.default.yaml

2. Mirror the images in the ImageSetConfiguration.yaml file by running the oc-mirror command. For example:

   $ oc-mirror --config=ImageSetConfiguration.yaml file:///path/to/mirror-archive --authfile /path/to/authfile  --v2

   Note

   The oc-mirror command pulls the charts listed in the ImageSetConfiguration file and makes them available as tgz archives under the /path/to/mirror-archive directory.

3. Apply the cluster-wide resources generated during the push step to redirect all image pulls to your local registry, as shown in the following example:

   $ cd <workspace folder>/working-dir/cluster-resources/
   $ oc apply -f .

4. Transfer the generated mirror archive file, for example, /path/to/mirror-archive/mirror_000001.tar, to a bastion host within your disconnected environment.

5. From the bastion host in your disconnected environment, which has access to the mirror registry, mirror the images from the archive file to your target registry. For example:

   $ oc-mirror --v2 --from <mirror-archive-file> docker://<target-registry-url:port> --workspace file://<workspace folder> --authfile /path/to/authfile

   where:

   <mirror-archive-file>

   Enter the name of the transferred tar file.

   <target-registry-url:port>

   Enter your local registry, for example, registry.localhost:5000.

6. Apply the redhat-developer-hub-orchestrator-infra Helm chart and approve the install plans. See Air-gapped installation with Helm chart instructions for details.

7. Apply the RHDH 1.9 Helm chart. Include the version 1.10.0 and enable the Orchestrator plugin, as shown in the following example:

   orchestrator.enabled=true

8. The RHDH 1.9 Helm chart defaults to pulling Orchestrator plugins from the official Red Hat OCI registry using full URL references. Override this default behavior to point the chart to your local registry.

   To configure the Orchestrator plugins to use a custom registry, complete the following steps:

9. Open your values.yaml file.

10. List the Orchestrator plugin packages under the orchestrator.plugins section. You must replace the simplified package references with the full URLs that point to your custom OCI registry.

    Important

    You must explicitly include the pluginConfig configuration for each plugin as shown in the following example:

    orchestrator:
      plugins:
        - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@_<digest>_
          disabled: true
          pluginConfig:
            orchestrator:
              dataIndexService:
                url: http://sonataflow-platform-data-index-service
        - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@_<digest>_
          disabled: true
          pluginConfig:
            dynamicPlugins:
              frontend:
                red-hat-developer-hub.backstage-plugin-orchestrator-form-widgets: {}
        - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@_<digest>_
          disabled: true
          pluginConfig:
            dynamicPlugins:
              frontend:
                red-hat-developer-hub.backstage-plugin-orchestrator:
                  appIcons:
                  - importName: OrchestratorIcon
                    name: orchestratorIcon
                  dynamicRoutes:
                  - importName: OrchestratorPage
                    menuItem:
                      icon: orchestratorIcon
                      text: Orchestrator
                      textKey: menuItem.orchestrator
                    path: /orchestrator
                  entityTabs:
                  - path: /workflows
                    title: Workflows
                    titleKey: catalog.entityPage.workflows.title
                    mountPoint: entity.page.workflows
                  mountPoints:
                  - mountPoint: entity.page.workflows/cards
                    importName: OrchestratorCatalogTab
                    config:
                      layout:
                      gridColumn: 1 / -1
                      if:
                        anyOf:
                        - IsOrchestratorCatalogTabAvailable
        - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@_<digest>_
          disabled: true
          pluginConfig:
            orchestrator:
              dataIndexService:
                url: http://sonataflow-platform-data-index-service

    where:

    <custom_registry_url>

    Enter the address of your custom registry where the OCI images have been mirrored.

    <digest>

    Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

    #!/bin/bash
    
    unpack () {
      local IMAGE="$1"
      DIR="${IMAGE//:/}" DIR="/tmp/${DIR//\//-}" rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}") podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar echo "Unpacked $IMAGE into $DIR" cd $DIR; tree -d -L 3 -I "usr|root|buildinfo" } unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:{product-version)" # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index{product-version)/dynamic-plugins.default.yaml

Verification

1. Restart the RHDH Pod and wait for the components to deploy properly.
2. After deployment is complete, go to the RHDH UI and confirm that the Orchestrator UI is accessible and functioning correctly.

Procedure

1. Create an ImageSetConfiguration.yaml file for oc-mirror. You must use an ImageSetConfiguration file to include all mirrored images required, as shown in the following example:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalimages:
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@<digest>
   
     helm:
       repositories:
         - name: openshift-charts
           url: https://charts.openshift.io
           charts:
             - name: redhat-developer-hub
               version: "1.10.0"
             - name: redhat-developer-hub-orchestrator-infra
               version: "1.10.0"
     operators:
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v<ocp-version>
        # For example: registry.redhat.io/redhat/redhat-operator-index:v4.21
         packages:
         - name: logic-operator
           channels:
           - name: stable
             minVersion: 1.37.2
             maxVersion: 1.37.2
         - name: serverless-operator
           channels:
           - name: stable
             minVersion: 1.37.1
             maxVersion: 1.37.1

2. Mirror the images in the ImageSetConfiguration.yaml file by running the oc-mirror command to pull images and charts, and push the images directly to the target registry. For example:

   $ oc-mirror --config=imagesetconfiguration.yaml docker://<registry URL:port> --workspace file://<workspace folder> --authfile /path/to/authfile  --v2

   Note

   The oc-mirror command pulls the charts listed in the ImageSetConfiguration file and makes them available as tgz archives under the <workspace folder> directory.

3. Apply the generated cluster resources to the disconnected cluster. For example:

   $ cd <workspace folder>/working-dir/cluster-resources/
   $ oc apply -f .

4. Apply the redhat-developer-hub-orchestrator-infra Helm chart and approve the install plans. See Air-gapped installation with Helm chart instructions for details.

5. Apply the RHDH 1.9 Helm chart. Include the version 1.10.0 and enable the Orchestrator plugin, as shown in the following example:

   orchestrator.enabled=true

6. The RHDH 1.9 Helm chart defaults to pulling Orchestrator plugins from the official Red Hat OCI registry using full URL references. You must override this behavior to point to your local registry.

   To configure the Orchestrator plugins to use a custom registry, complete the following steps:

   • Open your values.yaml file.

   • Explicitly list the Orchestrator plugin packages under the orchestrator.plugins section.

     You must replace the simplified package references with the full URLs that point to your custom OCI registry. You must explicitly include the pluginConfig configuration for each plugin as shown in the following example:

     orchestrator:
       plugins:
         - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@_<digest>_
           disabled: true
           pluginConfig:
             orchestrator:
               dataIndexService:
                 url: http://sonataflow-platform-data-index-service
         - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@_<digest>_
           disabled: true
           pluginConfig:
             dynamicPlugins:
               frontend:
                 red-hat-developer-hub.backstage-plugin-orchestrator-form-widgets: {}
         - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@_<digest>_
           disabled: true
           pluginConfig:
             dynamicPlugins:
               frontend:
                 red-hat-developer-hub.backstage-plugin-orchestrator:
                   appIcons:
                   - importName: OrchestratorIcon
                     name: orchestratorIcon
                   dynamicRoutes:
                   - importName: OrchestratorPage
                     menuItem:
                       icon: orchestratorIcon
                       text: Orchestrator
                       textKey: menuItem.orchestrator
                     path: /orchestrator
                   entityTabs:
                   - path: /workflows
                     title: Workflows
                     titleKey: catalog.entityPage.workflows.title
                     mountPoint: entity.page.workflows
                   mountPoints:
                   - mountPoint: entity.page.workflows/cards
                     importName: OrchestratorCatalogTab
                     config:
                       layout:
                       gridColumn: 1 / -1
                       if:
                         anyOf:
                         - IsOrchestratorCatalogTabAvailable
         - - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@_<digest>_
           disabled: true
           pluginConfig:
             orchestrator:
               dataIndexService:
                 url: http://sonataflow-platform-data-index-service

     Where:

     <custom_registry_url>

     Enter the address of your custom registry where the OCI images have been mirrored.

     <digest>

     Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

     #!/bin/bash
     
     unpack () {
       local IMAGE="$1"
       DIR="${IMAGE//:/}" DIR="/tmp/${DIR//\//-}" rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}") podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar echo "Unpacked $IMAGE into $DIR" cd $DIR; tree -d -L 3 -I "usr|root|buildinfo" } unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:{product-version)" # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index{product-version)/dynamic-plugins.default.yaml

Verification

1. Restart the RHDH pod and wait for the components to deploy properly.
2. After deployment is complete, go to the RHDH UI and confirm that the Orchestrator UI is accessible and functioning correctly.

Procedure

1. Enable the Loki backend module in the redhat-developer-hub-dynamic-plugins ConfigMap.

   
2. Open the ConfigMap and select the YAML view.

3. Add the Loki backend module to the plugins section:

   - disabled: false
         package: oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend-module-loki:{{inherit}}

   
4. Save the file.

5. In your application app-config.yaml ConfigMap file, add the Loki workflow log provider integration to the orchestrator section:

   
   
   Note

   The base URl might vary depending on where you store and access the Loki logs.

   To obtain the base URL, run the following command:

   LOKI_HOST=$(oc get route logging-loki -n openshift-logging -o jsonpath='{.spec.host}')
   echo "https://$LOKI_HOST/api/logs/v1/application/"

   orchestrator:
     workflowLogProvider:
       loki:
         baseUrl: <LOKI_BASE_URL>
   	token: <AUTH_TOKEN>
   	rejectUnauthorized: false
        # logPipelineFilters:
         # - '| filter1'
         # - '|= filter2'
         # logStreamSelectors:
         #   - label: 'app'
         #     value: '=~".+"'

   where:

   baseUrl: Specifies the base URL of your Loki instance. This value is required.

   token: The access token for authentication. This value is required. To find your current token, run the oc whoami -t command.

   rejectUnauthorized: Set to false if using self-signed certificates.

   Optional Parameters

   logPipelineFilters: Multiple Log Pipeline Filters can be specified in the logPipelineFilters section. See the Loki documentation to learn more about the log pipeline filters and their values and usage.

   logStreamSelectors: Specifies log stream selectors to filter the logs. By default, the plugin retrieves logs with the openshift_log_type="application". For more information about selector syntax, see the Grafana Loki documentation.

6. Save the ConfigMap.
7. Restart the Red Hat Developer Hub pod to apply the changes.

Verification

1. Navigate to the Orchestrator plugin in the RHDH interface.
2. Select a workflow instance.
3. Click View Logs to display the workflow instance logs.

Procedure

1. Update your workflow build configuration to include the JSON logging extension:

   export QUARKUS_EXTENSIONS="${QUARKUS_EXTENSIONS},io.quarkus:quarkus-logging-json"

2. Open the {workflow-name}-props ConfigMap for your workflow.

3. Add the following properties to the application.properties section:

   # Enable JSON logging with Quarkus JSON logging extension
   quarkus.log.console.json=true
   quarkus.log.console.json.pretty-print=false
   
   # Include all MDC context fields in JSON output
   # - processInstanceId: Set automatically by SonataFlow/Kogito
   # - traceId, spanId: Set by Quarkus OpenTelemetry (requires quarkus.otel.enabled=true)
   quarkus.log.console.json.print-details=true
   
   # Set log levels for workflow components
   quarkus.log.category."org.kie.kogito".level=DEBUG
   quarkus.log.category."io.serverlessworkflow".level=INFO
   
   # Optional: Enable additional context logging
   quarkus.log.category."org.kie.kogito.services.context".level=DEBUG

4. Save the ConfigMap and restart the workflow pod.

   The following is an example of a workflow ConfigMap with an enabled JSON logging:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: greetings-props
     namespace: sonataflow-infra
   data:
     application.properties: |
       # JSON logging configuration
       quarkus.log.console.json=true
       quarkus.log.console.json.pretty-print=false
       quarkus.log.console.json.print-details=true
   
       # Log levels
       quarkus.log.category."org.kie.kogito".level=DEBUG
       quarkus.log.category."io.serverlessworkflow".level=INFO

Verification

1. Check the pod logs to verify the JSON format and the presence of the processInstanceId:

   oc logs <workflow_pod_name> | grep processInstanceId

   {"timestamp":"...","level":"INFO","message":"...","mdc":{"processInstanceId":"abc-123-..."}}

Procedure

1. Add the following properties to the workflow ConfigMap to enable file-based JSON output:

   quarkus.log.file.enable=true
   quarkus.log.file.path=/var/log/sonataflow/workflow.log
   quarkus.log.file.json=true

2. Configure log rotation settings to manage disk usage:

   quarkus.log.file.rotation.max-file-size=10M
   quarkus.log.file.rotation.max-backup-index=5
   quarkus.log.file.rotation.rotate-on-boot=true

   This configuration does the following:

3. Rotates logs when they reach 10MB
4. Keeps up to 5 backup files
5. Adds date suffix to rotated files
6. Rotates on application startup

7. Set log level for file output:

   quarkus.log.file.level=INFO

8. Update the SonataFlow custom resource (CR) to mount the volume at the log path:

   spec:
     podTemplate:
       container:
         volumeMounts:
         - name: shared-logs
           mountPath: /var/log/sonataflow
     volumes:
     - name: shared-logs
       emptyDir:
         sizeLimit: 500Mi

9. After applying the configuration, restart your workflow pod and check the log output:

   # Get workflow pod name
   oc get pods -n sonataflow-infra -l sonataflow.org/workflow-app=your-workflow
   
   # Check logs for JSON format
   oc logs -n sonataflow-infra your-workflow-pod-name | head -5

Verification

1. Access the workflow container and verify the log file exists and is receiving JSON data:

   oc exec <pod_name> -- ls -l /var/log/sonataflow/workflow.log

2. Verify that the file contains JSON data:

   oc exec <pod_name> -- head -n 5 /var/log/sonataflow/workflow.log

Procedure

1. Add the OpenTelemetry exporter and service identification properties to your workflow ConfigMap:

   # Enable OpenTelemetry integration
   quarkus.otel.enabled=true
   quarkus.otel.exporter.otlp.traces.endpoint=http://jaeger-collector:4317
   quarkus.otel.service.name=${workflow.name}

2. Set the resource attributes to filter traces in your observability dashboard:

   quarkus.otel.resource.attributes=service.namespace=sonataflow-infra

3. Restart the workflow pod to apply the new configuration.

Verification

1. Trigger a workflow execution.

2. Check the logs for trace identifiers:

   oc logs <pod_name> | grep traceId

3. Make sure the mdc block in the JSON output contains traceId and spanId fields with non-empty values.

Procedure

1. Deploy the PLG stack by using Helm:

   # Add Grafana Helm repository
   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update
   
   # Create namespace
   oc new-project sonataflow-observability
   
   # Deploy Loki stack
   helm install loki-stack grafana/loki-stack \
     --namespace sonataflow-observability \
     --set loki.persistence.enabled=true \
     --set loki.persistence.size=20Gi \
     --set promtail.config.logLevel=info \
     --set grafana.enabled=true

   Note

   For production deployments, use a custom values.yaml file with appropriate resource limits and security contexts.

2. Create a ConfigMap for the Promtail sidecar by selecting the configuration that matches your logging method:

   1. Scrape container stdout

      Use this configuration to collect logs from container stdout by using Kubernetes service discovery:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: promtail-config
        namespace: sonataflow-observability
      data:
        config.yml: |
          server:
            http_listen_port: 3101
      
          clients:
            - url: http://loki:3100/loki/api/v1/push
      
          scrape_configs:
          - job_name: sonataflow-workflows
            kubernetes_sd_configs:
            - role: pod
              namespaces:
                names: ["sonataflow-infra"]
      
            relabel_configs:
            - source_labels: [__meta_kubernetes_pod_label_sonataflow_org_workflow_app]
              action: keep
              regex: (.+)
      
            - source_labels: [__meta_kubernetes_pod_name]
              target_label: pod
      
            - source_labels: [__meta_kubernetes_pod_label_sonataflow_org_workflow_app]
              target_label: workflow
      
            pipeline_stages:
            - json:
                expressions:
                  timestamp: timestamp
                  level: level
                  logger: logger
                  message: message
                  processInstanceId: mdc.processInstanceId
                  traceId: mdc.traceId
                  spanId: mdc.spanId
      
            - labels:
                level:
                logger:
                processInstanceId:
                traceId:

   2. Scrape JSON log files

      If you use [file-based JSON logging](#file-based-json-logging), configure Promtail to read from the shared log volume:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: promtail-sidecar-config
        namespace: sonataflow-infra
      data:
        config.yml: |
          server:
            http_listen_port: 3101
      
          clients:
            - url: http://loki.sonataflow-observability.svc.cluster.local:3100/loki/api/v1/push
      
          positions:
            filename: /var/log/positions.yaml
      
          scrape_configs:
          - job_name: sonataflow-json-files
            static_configs:
            - targets:
                - localhost
              labels:
                job: sonataflow-workflows
                __path__: /var/log/sonataflow/*.log
      
            pipeline_stages:
            - json:
                expressions:
                  timestamp: timestamp
                  level: level
                  logger: loggerName
                  message: message
                  processInstanceId: mdc.processInstanceId
                  traceId: mdc.traceId
                  spanId: mdc.spanId
      
            - labels:
                level:
                logger:
                processInstanceId:
                traceId:
      
            - timestamp:
                source: timestamp
                format: RFC3339Nano

3. Add the Promtail sidecar container to your SonataFlow custom resource:

   apiVersion: sonataflow.org/v1alpha08
   kind: SonataFlow
   metadata:
     name: my-workflow
     namespace: sonataflow-infra
   spec:
     podTemplate:
       container:
         volumeMounts:
         - name: shared-logs
           mountPath: /var/log/sonataflow
       containers:
       - name: promtail-sidecar
         image: grafana/promtail:2.9.0
         args:
           - -config.file=/etc/promtail/config.yml
         volumeMounts:
         - name: shared-logs
           mountPath: /var/log/sonataflow
           readOnly: true
         - name: promtail-config
           mountPath: /etc/promtail
         - name: positions
           mountPath: /var/log
         resources:
           requests:
             cpu: 50m
             memory: 64Mi
           limits:
             cpu: 100m
             memory: 128Mi
       volumes:
       - name: shared-logs
         emptyDir:
           sizeLimit: 500Mi
       - name: promtail-config
         configMap:
           name: promtail-sidecar-config
       - name: positions
         emptyDir: {}

4. Querying logs in Grafana: After deploying the stack, use the following LogQL queries in the Grafana Explore view:

   1. Filter logs by process instance

      {job="sonataflow-workflows"} | json | processInstanceId="abc-123-def-456"

   2. Find workflow errors:

      {job="sonataflow-workflows", workflow="onboarding"} | json | level="ERROR"

   3. Trace correlation:

      {job="sonataflow-workflows"} | json | traceId="4bf92f3577b34da6a3ce929d0e0e4736"

   4. Process instance timeline:

      {job="sonataflow-workflows"} | json | processInstanceId="abc-123-def-456" | line_format "{{.timestamp}} [{{.level}}] {{.message}}"

Verification

1. Access the Grafana Explore view.

2. Run the following LogQL query, replacing <instance_id> with a valid ID:

   {job="sonataflow-workflows"} | json | processInstanceId="<instance_id>"

   Confirm that Grafana displays the log entries associated with the specified process instance.

Procedure

1. Create a configuration file containing the following alert rule groups based on your monitoring requirements:

   • To monitor failure rates:

     - alert: WorkflowHighErrorRate
       expr: rate({job="sonataflow-workflows", level="ERROR"}[5m]) > 0.1
       for: 2m
       labels:
         severity: warning
       annotations:
         summary: "High error rate in SonataFlow workflows"

   • To identify stalled process instances:

     - alert: WorkflowInstanceStuck
       expr: |
         time() - max by (process_instance_id) (
           {job="sonataflow-workflows"} | json | unwrap timestamp[1h]
         ) > 3600
       labels:
         severity: critical

   • To identify long-running workflows:

       - alert: LongRunningWorkflow
         expr: |
           time() - min by (process_instance_id) (
             {job="sonataflow-workflows"} | json | message="Workflow started" | unwrap timestamp[24h]
           ) > 7200
         labels:
           severity: warning
         annotations:
           summary: "Workflow {{ $labels.process_instance_id }} running longer than 2 hours"

2. Apply the alert rules to your cluster.

Verification

1. Access the monitoring dashboard, such as the Prometheus or OpenShift Console.
2. Verify that the alerts appear in the list under the Alerts tab.

Procedure

1. Define a receiver and a routing path in your Alertmanager configuration:

   route:
     group_by: ['alertname', 'workflow']
     group_wait: 10s
     group_interval: 10s
     repeat_interval: 1h
     receiver: 'web.hook'
   
   receivers:
   - name: 'web.hook'
     slack_configs:
     - api_url: 'YOUR_SLACK_WEBHOOK_URL'
       channel: '#workflow-alerts'
       title: 'SonataFlow Alert'
       text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

2. Reload the Alertmanager configuration to apply the changes.

Verification

1. Trigger a test alert in your workflow environment.
2. Monitor the external notification service (for example, the Slack channel #workflow-alerts). A notification appears in the external service containing the summary and details of the triggered alert.

Procedure

1. Verify JSON log formatting.

   If logs appear as plain text instead of structured JSON, verify the following:

2. The io.quarkus:quarkus-logging-json extension is defined in the pom.xml file.
3. The quarkus.log.console.json=true property is set in the {workflow-name}-props ConfigMap.
4. The workflow image was rebuilt and redeployed after adding the extension.
5. The workflow pod was restarted after applying ConfigMap changes.

6. Diagnose missing process instance context.

   If logs are in JSON format but the processInstanceId field is missing or empty, verify the following:

7. Workflow instances are actively running.

8. The following property is set in the workflow ConfigMap:

   quarkus.log.console.json.print-details=true

9. The SonataFlow version in use supports automatic Mapped Diagnostic Context (MDC) population.

10. Resolve log collection failures in Loki.

    If logs are generated but do not appear in Loki or the aggregation dashboard, verify the following:

11. The Promtail or Fluent Bit label selector matches the workflow pod labels.
12. The collector has the required Role-Based Access Control (RBAC) permissions to read logs from the workflow namespace.
13. The scrape_configs in the collector configuration include the correct namespace.

14. Check the collector logs for permission errors:

    oc logs -l app=promtail -n sonataflow-observability

15. Mitigate high resource usage.

    If JSON logging causes performance degradation or high storage costs, implement the following changes:

16. Increase the log level for verbose categories to reduce output volume:

    quarkus.log.category."org.kie.kogito".level=WARN

17. Enable asynchronous logging to reduce the impact on workflow execution time:

    quarkus.log.console.async=true

18. Configure log rotation and retention policies in the aggregation backend.

Verification

1. After applying a fix, trigger a workflow execution.

2. Inspect the latest log entries. The logs appear in JSON format and include valid processInstanceId, traceId, and spanId fields:

   oc logs <workflow_pod_name> --tail=10

Procedure

1. Add the OpenTelemetry addon to the QUARKUS_EXTENSIONS environment variable during the image build process:

   export QUARKUS_EXTENSIONS="${QUARKUS_EXTENSIONS},org.apache.kie.sonataflow:sonataflow-addons-quarkus-opentelemetry"

2. Open the {workflow-name}-props ConfigMap for your workflow.

3. In the application.properties section, enable the OpenTelemetry integration and configure the service attributes:

   # Application Identity
   quarkus.application.name=my-workflow
   quarkus.application.version=1.0.0
   
   # OpenTelemetry Configuration
   quarkus.otel.enabled=true
   quarkus.otel.traces.enabled=true
   quarkus.otel.metrics.enabled=true
   quarkus.otel.logs.enabled=true
   
   # Service Resource Attributes
   quarkus.otel.resource.attributes=\
     service.name=my-workflow,\
     service.namespace=workflows,\
     service.version=1.0.0,\
     deployment.environment=production
   
   # SonataFlow Specific Configuration
   # Master switch for SonataFlow OpenTelemetry integration
   sonataflow.otel.enabled=true
   # Service identification (uses Quarkus application name/version as defaults)
   sonataflow.otel.service-name=${quarkus.application.name:kogito-workflow-service}
   sonataflow.otel.service-version=${quarkus.application.version:unknown}
   # Enable span creation for workflow states
   sonataflow.otel.spans.enabled=true
   # Enable process lifecycle events (start, complete, error, state transitions)
   sonataflow.otel.events.enabled=true

4. Save the ConfigMap and restart the workflow pod to apply the changes.

Verification

1. Verify that the OpenTelemetry addon is loaded by checking the pod logs:

   oc logs -n workflows deployment/onboarding-workflow | grep "sonataflow-addons-quarkus-opentelemetry"

2. Verify the trace report status:

   oc logs -n workflows deployment/greeting | grep -i "export\|batch"

3. Confirm that the observability backend, such as Jaeger, is receiving data:

   oc logs -n observability deployment/jaeger | grep -i "span\|trace"

Procedure

1. Configure an export strategy based on your environment requirements:

   • Configure the OTLP exporter with batch processing (Recommended)

     For production environments, use an OTLP exporter with batch processing to reduce network overhead and improve performance:

     # OTLP exporter - Direct to Jaeger
     quarkus.otel.exporter.otlp.endpoint=http://jaeger-collector.observability.svc.cluster.local:4317
     quarkus.otel.exporter.otlp.protocol=grpc
     quarkus.otel.traces.exporter=cdi
     
     # Batch processing for production
     quarkus.otel.bsp.schedule.delay=5s
     quarkus.otel.bsp.max.export.batch.size=512
     quarkus.otel.bsp.export.timeout=2s
     quarkus.otel.bsp.max.queue.size=2048

   • Configure direct export to an external platform

     For development or simple integrations, use a direct export configuration:

     # Example: Direct export to Jaeger
     quarkus.otel.exporter.otlp.endpoint=http://jaeger-collector:4317
     quarkus.otel.exporter.otlp.protocol=grpc
     quarkus.otel.traces.exporter=cdi

2. Externalize the configuration for production deployments by using environment variables. This ensures that your deployment remains secure and flexible across environments.

   # Externalized Configuration
   quarkus.otel.exporter.otlp.endpoint=${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4317}
   quarkus.otel.exporter.otlp.headers=${OTEL_EXPORTER_OTLP_HEADERS:}
   quarkus.application.name=${OTEL_SERVICE_NAME:my-workflow}
   quarkus.otel.resource.attributes=${OTEL_RESOURCE_ATTRIBUTES:deployment.environment=dev}

Procedure

1. Clone the project as shown in the following example:

   git clone git@github.com:rhdhorchestrator/orchestrator-demo.git
   cd orchestrator-demo

2. Check the help menu of the script:

   ./scripts/build.sh --help

3. Run the build.sh script, providing the required flags, for example, the image path (-i), workflow source directory (-w), and manifests output directory (-m).

   Important

   You must specify the full target image path with a tag as shown in the following example:

   ./scripts/build.sh --image=quay.io/orchestrator/demo-basic:test -w 01_basic/ -m 01_basic/manifests

Procedure

1. Use the kubectl create command specifying the target namespace to apply the Kubernetes manifests as shown in the following example:

   $ kubectl create -n <your_namespace> -f ./01_basic/manifests/.

2. After deployment, monitor the status of the workflow pods as shown in the following example:

   $ kubectl get pods -n <your_namespace> -l app=basic

   The pod may initially appear in an Error state because of missing or incomplete configuration in the Secret or ConfigMap.

3. Inspect the Pod logs as shown in the following example:

   $ oc logs -n <your_namespace> basic-f7c6ff455-vwl56

   The following code is an example of the output:

   SRCFG00040: The config property quarkus.openapi-generator.notifications.auth.BearerToken.bearer-token is defined as the empty String ("") which the following Converter considered to be null: io.smallrye.config.Converters$BuiltInConverter
   java.lang.RuntimeException: Failed to start quarkus
   ...
   Caused by: io.quarkus.runtime.configuration.ConfigurationException: Failed to read configuration properties

   The error indicates a missing property: quarkus.openapi-generator.notifications.auth.BearerToken.bearer-token.

4. In such a case where the logs show the ConfigurationException: Failed to read configuration properties error or indicate a missing value, retrieve the ConfigMap as shown in the following example:

   $ oc get -n <your_namespace> configmaps basic-props -o yaml

   The following code is an example of the sample output:

   apiVersion: v1
   data:
     application.properties: |
       # Backstage notifications service
       quarkus.rest-client.notifications.url=${BACKSTAGE_NOTIFICATIONS_URL}
       quarkus.openapi-generator.notifications.auth.BearerToken.bearer-token=${NOTIFICATIONS_BEARER_TOKEN}
   ...

   Resolve the placeholders using values provided using a Secret.

5. You must edit the corresponding Secret and provide appropriate base64-encoded values to resolve the placeholders in application.properties as shown in the following example:

   $ kubectl edit secrets -n <your_namespace> basic-secrets

6. Restart the workflow Pod for Secret changes to take effect in OpenShift Serverless Logic 1.37.2.

Verification

1. Verify the deployment status by checking the Pods again as shown in the following example:

   $ oc get pods -n <your_namespace> -l app=basic

   The expected status for a successfully deployed workflow Pod is as shown in the following example:

   NAME                    READY   STATUS    RESTARTS   AGE
   basic-f7c6ff455-grkxd   1/1     Running   0          47s

2. Once the Pod is in the Running state, the workflow now appears in the Orchestrator plugin inside the Red Hat Developer Hub.

Procedure

1. Install the orchestrator-software-templates-infra chart.
2. Install the orchestrator-software-templates chart.

Procedure

1. Prepare the image registry. Before creating the template, configure the target repository in Quay.io.

   1. Log in to your Quay.io organization (for example, orchestrator-testing).
   2. Create a new repository (for example, serverless-workflow-demo).
   3. Add robot account permissions to the repository settings.

2. Open the Red Hat Developer Hub Catalog.

   
3. Select the Basic workflow bootstrap project template and click Launch Template.
4. Follow the template form to enter required details, including the GitHub organization, source code repository name, and a unique Workflow ID.
5. For the CI/CD method, select Tekton with Argo CD to generate GitOps resources.
6. Set the Workflow Namespace to rhdh and the GitOps Namespace to orchestrator-gitops.
7. Enter your Quay.io registry details.

8. Click Review, then click Create.

   
9. Optional: Enable persistence and provide database connection details if the workflow requires a database schema.

Procedure

1. Open the generated GitOps repository.

2. Clone the repository and navigate to the bootstrap directory:

   $ git clone https://token:<PAT>@${{ values.gitHost }}/${{ values.orgName }}/${{ values.repoName }}.git
   cd <repo_name>/bootstrap

   Note

   If you are not authenticated, you must use a personal access token (PAT) in the clone URL. Make sure the PAT has repository access permissions.

3. Open ${{values.workflowId}}-argocd-repo.yaml and replace the REPLACE_SSH_PRIVATE_KEY string with your SSH private key.

4. Apply the manifests to the cluster:

   $ kubectl apply -f .

   Applying these manifests triggers the following automated sequence:

   1. CI Pipeline (Tekton): Builds the workflow image and pushes it to your Quay.io registry.
   2. CD Pipeline (Argo CD): Deploys the serverless workflow manifests to the cluster.

Procedure

1. For CI:

   1. In the RHDH Catalog, select your source code repository component (for example, onboardings).
   2. Click the CI tab and verify that the pipeline run status is Succeeded.
   3. If the pipeline status does not appear in the Red Hat Developer Hub console, verify the CI status directly in your Git provider (GitHub or GitLab).
   4. If the pipeline fails, click the run name to view the logs and identify build errors.

2. For CD:

   1. Open the GitOps Resources Repository component in the Catalog (for example, onboarding-gitops).

   2. Click the CD tab and make sure the Kubernetes resources are synced and healthy. This confirms that ArgoCD deployed the workflow to the cluster.

      

1. Visibility issues

   Missing Orchestrator plugin

   If Orchestrator features do not appear in RHDH, make sure you have updated the RHDH Helm chart with the required plugins.

   Software templates not appearing

   Make sure the orchestrator-software-templates chart is installed and the orchestrator-auth-secret exists in the correct namespace.

2. Pipeline failure (CI)

   GitHub or GitLab actions failure

   The GitOps automation includes a GitHub Action or GitLab CI step that creates a PipelineRun manifest from a PipelineRun template. Examine the failed GitHub or GitLab actions logs. Failures often occur due to invalid Git credentials or misconfigured runner permissions. You can also create the PipelineRun file manually to bypass automation issues.

   Build or push issues

   Check the CI tab in the RHDH Catalog.

   If RHDH does not display the status, use the OpenShift Container Platform console to monitor pipeline instances and triggered jobs. Navigate to Pipelines > PipelineRuns for detailed logs.

   If the Tekton pipeline fails during the build or push stages:

   • Verify that your Quay.io robot account has Write permissions.
   • Ensure the docker-registry-credentials secret exists in the rhdh namespace.

3. Resource visibility and Sync issues (CD)

   Pipeline succeeds but workflows are missing

   If the CI pipeline succeeds but the workflow does not appear in the CD tab:

   • Make sure the target namespace is labeled for Argo CD:

     $ oc label ns <target_namespace> rhdh.redhat.com/argocd-namespace=true

   • Make sure the ArgoCD ServiceAccount has the required permissions to manage resources in the rhdh namespace.

   Argo CD sync failure

   If resources appear but remain in an OutOfSync state, click Refresh in the Argo CD UI or verify that the AppProject exists in the orchestrator-gitops namespace.

   PostgreSQL authentication failures in Argo CD

   If the Orchestrator fails to connect to the PostgreSQL database when you deploy by using Argo CD, the failure is often due to a mismatch in password generation.

   The Orchestrator Helm chart uses the Helm lookup function to check for an existing PostgreSQL secret. Because Argo CD uses helm template to render manifests, it cannot query the live cluster. Consequently, the chart generates a new, random password instead of retrieving the existing one, resulting in an authentication failure.

   To resolve this failure, you must complete the following steps:

   1. Create the database secret manually with the correct credentials:

      $ kubectl create secret generic <backstage-postgresql-svcbind-postgres> --from-literal=password=<your_password>

   2. Update your Helm configuration (for example, in values.yaml) to disable automatic service binding generation:

      upstream:
        postgresql:
          serviceBindings:
            enabled: false
          auth:
            username: postgres
            database: backstage
            existingSecret: backstage-postgresql-svcbind-postgres
            secretKeys:
              adminPasswordKey: password
              userPasswordKey: password

   3. Sync the application in Argo CD to apply the changes.

Procedure

1. If the workflow operation fails, examine the container log of the specific workflow instance to determine the cause by running the following command:

   $ oc logs my-workflow-xy73lj

2. If the workflow fails to reach an HTTPS endpoint, check the pod log for an SSL certificate verification failure. This occurs if the target endpoint uses a Certificate Authority (CA) that the workflow cannot verify. The resulting error resembles the following:

   sun.security.provider.certpath.SunCertPathBuilderException - unable to find valid certification path to requested target

3. To resolve the SSL certificate error, load the additional CA certificate into the running workflow container.

Procedure

1. Identify required namespaces.
2. Retrieve the namespace value where RHDH is running using oc get backstage -A.

3. Identify the SonataFlow Services Namespace by checking for either a sonataflowclusterplatform or sonataflowplatform instance.

   Note

   By default, the SonataFlow namespace must be the same as the RHDH namespace.

4. If the workflow is deployed to a namespace outside the core SonataFlow services, configure network policies to permit the necessary inter-namespace traffic.

   # Example NetworkPolicy configuration to ingress traffic into the workflow namespace
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: {{ .Release.Name }}-allow-infra-ns-to-workflow-ns
     # SonataFlow and Workflows are using the RHDH target namespace.
     namespace: {{ .Release.Namespace | quote }}
   spec:
     podSelector: {}
     ingress:
       - from:
         - namespaceSelector:
             matchLabels:
               # Allow knative events to be delivered to workflows.
               kubernetes.io/metadata.name: knative-eventing
         - namespaceSelector:
             matchLabels:
               # Allow auxiliary knative function for workflow (such as m2k-save-transformation)
               kubernetes.io/metadata.name: knative-serving
         - namespaceSelector:
             matchLabels:
               # Allow communication between the serverless logic operator and the workflow namespace.
               kubernetes.io/metadata.name: openshift-serverless-logic

5. Add SonataFlowClusterPlatform Custom Resource as shown in the following configuration:

   oc create -f - <<EOF
   apiVersion: sonataflow.org/v1alpha08
   kind: SonataFlowClusterPlatform
   metadata:
     name: cluster-platform
   spec:
     platformRef:
       name: sonataflow-platform
       namespace: $RHDH_NAMESPACE

6. To allow communication between RHDH namespace and the workflow namespace, create the following network policies:

   1. Allow RHDH services to accept traffic from workflows. Create an additional network policy within the RHDH instance namespace as shown in the following configuration::

      oc create -f - <<EOF
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-external-workflows-to-rhdh
        # Namespace where network policies are deployed
        namespace: $RHDH_NAMESPACE
      spec:
        podSelector: {}
        ingress:
          - from:
            - namespaceSelector:
                matchLabels:
                  # Allow SonataFlow services to communicate with new/additional workflow namespace.
                  kubernetes.io/metadata.name: $ADDITIONAL_WORKFLOW_NAMESPACE

   2. Allow traffic from RHDH, SonataFlow and Knative. Create a network policy within the additional workflow namespace as shown in the following configuration:

      oc create -f - <<EOF
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-rhdh-and-knative-to-workflows
        namespace: $ADDITIONAL_WORKFLOW_NAMESPACE
      spec:
        podSelector: {}
        ingress:
          - from:
            - namespaceSelector:
                matchLabels:
                  # Allows traffic from pods in the RHDH namespace.
                  kubernetes.io/metadata.name: $RHDH_NAMESPACE
            - namespaceSelector:
                matchLabels:
                  # Allows traffic from pods in the Knative Eventing namespace.
                  kubernetes.io/metadata.name: knative-eventing
            - namespaceSelector:
                matchLabels:
                  # Allows traffic from pods in the Knative Serving namespace.
                  kubernetes.io/metadata.name: knative-serving

7. (Optional) Create an allow-intra-namespace policy in the workflow namespace to enable unrestricted communication among all pods within that namespace.

8. If workflow persistence is required, perform the following configuration steps:

   1. Create a dedicated PostgreSQL Secret containing database credentials within the workflow namespace as shown in the following configuration:

      oc get secret sonataflow-psql-postgresql -n <your_namespace> -o yaml > secret.yaml
      sed -i '/namespace: <your_namespace>/d' secret.yaml
      oc apply -f secret.yaml -n $ADDITIONAL_NAMESPACE

   2. Configure the workflow serviceRef property to correctly reference the PostgreSQL service namespace as shown in the following configuration:

      apiVersion: sonataflow.org/v1alpha08
      kind: SonataFlow
        ...
      spec:
        ...
        persistence:
          postgresql:
            secretRef:
              name: sonataflow-psql-postgresql
              passwordKey: postgres-password
              userKey: postgres-username
            serviceRef:
              databaseName: sonataflow
              databaseSchema: greeting
              name: sonataflow-psql-postgresql
              namespace: $POSTGRESQL_NAMESPACE
              port: 5432

      namespace

      Enter the namespace where the PostgreSQL server is deployed.

9. If the sonataflow-platform-data-index-service cannot connect to the PostgreSQL database on startup, perform the following diagnostic checks:

   1. Verify that the PostgreSQL Pod has fully transitioned to a running and operational status. Allow additional time for database initialization before expecting related service pods (DataIndex, JobService) to establish a connection.
   2. If the PostgreSQL Server operates in a dedicated namespace (for example, outside RHDH), verify that network policies are configured to allow ingress traffic from the SonataFlow services namespace. Network policies might prevent the Data Index and Job Service pods from connecting to the database.

Procedure

1. Verify if the workflow uses GitOps profile. The RHDH Orchestrator UI displays only the workflows that use this profile. Make sure the workflow definition and the SonataFlow manifests use the GitOps profile.

2. Verify that the workflow pod has started and is ready. The readiness of a workflow pod depends on its successful registration with the Data Index. When a workflow initializes, it performs the following actions:

   1. It attempts to create its schema in the database (if persistence is active).

   2. It attempts to register itself to the Data Index. The workflow pod remains in an unready state until it successfully registers to the Data Index.

      Check the workflow deployment for additional status and error messages that might be unavailable in the pod log.

3. Check if the workflow pod can reach the Data Index service. Connect to the workflows pod and send the following GraphQL request to the Data Index:

   curl -g -k  -X POST  -H "Content-Type: application/json" \
                       -d '{"query":"query{ ProcessDefinitions  { id, serviceUrl, endpoint } }"}' \
                       http://sonataflow-platform-data-index-service.<your_namespace>/graphql

   Use the Data Index service and namespace as defined in your environment. By default, this is the same namespace where RHDH is installed. If your SonataFlow resources are installed in a separate namespace, use <your_namespace>. Check if the RHDH pod can reach the workflow service by running the following command:

   curl http://<workflow_service>.<workflow_namespace>/management/processes

4. Connect to the RHDH pod. Verify its connection to the Data Index service and inspect the RHDH pod logs for messages from the Orchestrator plugin.

   To inspect the logs, identify the RHDH pod and run the following oc logs command:

   oc get pods -n <your_namespace>
   oc logs <rhdh_pod_name> -n <your_namespace>

   You must find messages indicating it is attempting to fetch workflow information from the Data Index, similar to the following:

   {"level":"\u001b[32minfo\u001b[39m","message":"fetchWorkflowInfos() called: http://sonataflow-platform-data-index-service.<your_namespace>","plugin":"orchestrator","service":"backstage","span_id":"fca4ab29f0a7aef9","timestamp":"2025-08-04 17:58:26","trace_flags":"01","trace_id":"5408d4b06373ff8fb34769083ef771dd"}

   Notice the "plugin":"orchestrator" that can help to filter the messages.

5. Make sure the Data Index properties are set in the -managed-props ConfigMap of the workflow as shown in the following configuration:

   kogito.data-index.health-enabled = true
   kogito.data-index.url = http://sonataflow-platform-data-index-service.<your_namespace>
   ...
   mp.messaging.outgoing.kogito-processdefinitions-events.url = http://sonataflow-platform-data-index-service.<your_namespace>/definitions
   mp.messaging.outgoing.kogito-processinstances-events.url = http://sonataflow-platform-data-index-service.<your_namespace>/processes

   Note

   The -managed-props ConfigMap is located in the same namespace as the workflow and is generated by the OpenShift Serverless Logic (OSL) Operator.

   These properties, along with similar settings for the Job Services, indicate that the (OSL) Operator successfully registered the Data Index service.

6. Confirm that the workflow is registered in the Data Index database. Connect to the database used by the Data Index and run the following command from the PSQL instance pod:

   PGPASSWORD=<psql password> psql -h localhost -p 5432 -U < user> -d sonataflow

   Replace <psql password> and <user> with your database credentials.

   Run the following SQL commands to query the registered workflow definitions:

   sonataflow=# SET search_path TO "sonataflow-platform-data-index-service";
   sonataflow=# select id, name from definitions;

   You must see your workflows listed in the query results.

7. Make sure you have enabled Data Index and Job Service in the SonataFlowPlatform custom resource (CR) as shown in the following configuration:

   services:
       dataIndex:
         enabled: true
       jobService:
         enabled: true

   If you fail to enable the Data Index and the Job Services in the SonataFlowPlatform custom resource (CR), the Orchestrator plugin fails to fetch the available workflows.

   Note

   You can also manually edit the SonataFlowPlatform CR instance to trigger the re-creation of workflow-related manifests.

8. Configure role-based access control (RBAC) permissions to ensure workflows are visible in the Orchestrator UI.

   Note

   When the RBAC plugin is enabled, the Orchestrator UI does not display workflows by default. You must explicitly grant read permissions.

   1. Check your RHDH app-config.yaml file to confirm if the RBAC plugin is enabled.
   2. Confirm your user or role has the orchestrator.workflow permission with the read action.

   3. If this permission is missing, add the following to your RBAC CSV (rbac-policy.csv) file:

      p, role:default/workflowUser, orchestrator.workflow, read, allow

   4. Make sure policyFileReload is set to true in your configuration, or restart the RHDH application:

      permission:
        enabled: true
        rbac:
          policyFileReload: true

Procedure

1. Identify duplicate workflows in the Orchestrator UI:

   1. Navigate to the Orchestrator plugin in RHDH.
   2. Review the workflow list for entries that appear multiple times with the same workflow name.

   3. Note the version information displayed in the version column of the workflow list and on the workflow details page to distinguish between duplicate entries.

      Note

      The version column displays metadata from the workflow definition, retrieved from the Data Index GraphQL schema. This information helps you identify which workflows share the same ID but does not prevent the duplicate entries. If a workflow definition does not specify a version, the field appears empty in the UI. Duplicate entries can occur when you deploy the same workflow ID to different runtime servers over time, because the Data Index records all executions.

2. Verify the workflow IDs in your workflow definitions:

   1. Locate the workflow definition files (.sw.yaml or .sw.json files).
   2. Check the id field in each workflow definition.
   3. Identify workflows that use the same id value, even if they have different version values.

   4. Review the version field in each workflow definition to understand how workflows appear in the UI.

      Example of problematic workflow definitions:

      # First deployment
      id: customer-onboarding
      version: "1.0"
      name: Customer Onboarding
      
      # Second deployment (causes duplicate)
      id: customer-onboarding
      version: "2.0"
      name: Customer Onboarding

3. Determine which workflow version to retain:

   1. Review the workflow instances and their execution history.
   2. Identify which version is currently in active use.
   3. Check for any running instances of older versions that must complete before removal.

4. Update workflow definitions with unique IDs:

   1. For the new workflow version, modify the id field to include a version identifier:

      id: customer-onboarding-v2
      version: "2.0"
      name: Customer Onboarding

   2. Maintain the original workflow ID for the current deployment.
   3. Build and deploy the updated workflow definition.

5. Remove outdated workflow deployments:

   1. After confirming the new workflow operates correctly, remove the old workflow deployment.
   2. Verify that all instances of the old workflow have completed.

   3. Delete the workflow resources from your cluster:

      oc delete sonataflow <old-workflow-name> -n <workflow-namespace>

      Note

      Deleting the workflow deployment removes it from the cluster but preserves historical execution records in the Data Index. Users can still view past workflow runs in RHDH.

6. Clean historical data if necessary:

   If duplicate entries persist in the UI after you remove the workflow deployments, the Data Index database has historical records from earlier workflow executions. These records preserve the execution history of workflows that ran on different runtime servers over time.

   Important

   Back up your workflow execution records before you remove historical data from the Data Index database. Removing this data permanently prevents access to past execution records.

   1. Connect to the Data Index database to verify the duplicate entries.

   2. Query the workflow definitions to identify duplicate entries:

      SET search_path TO "sonataflow-platform-data-index-service";
      SELECT id, version, name FROM definitions;

   3. Evaluate whether to remove the historical data. You can keep the historical data to retain past workflow execution records, which allows you to view the execution history and results of completed workflow instances. Alternatively, contact your system administrator or Red Hat Support for guidance on safely removing historical duplicate entries from the Data Index without affecting active workflow operations.

Verification

1. Navigate to the Orchestrator plugin in RHDH.
2. Confirm that the UI shows only one entry for each workflow.
3. Verify that the version information displays correctly for each workflow.
4. Test workflow execution to confirm the correct version runs.

Procedure

1. Download the plugin-infra.sh script as shown in the following example:

   $ curl -sSLO https://raw.githubusercontent.com/redhat-developer/rhdh-operator/refs/heads/release-1.9/config/profile/rhdh/plugin-infra/plugin-infra.sh # Specify the Red Hat Developer Hub version in the URL or use main

2. Run the script:

   $ ./plugin-infra.sh