Procedure

1. To enable the Orchestrator plugins with default settings, set disabled: false for the corresponding packages:

   - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator:{{inherit}}"
    disabled: false
   - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend:{{inherit}}"
     disabled: false
     dependencies:
       - ref: sonataflow
   - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator:{{inherit}}"
     disabled: false
   - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets:{{inherit}}"
     disabled: false

   Note

   When you enable the plugins, the pre-loaded plugin configuration are used. Additionally, the ref: sonataflow field installs the OpenShift Serverless and OpenShift Serverless Logic resources. This happens automatically when you are using the Operator.

   The following example shows a complete configuration of the Orchestrator plugin:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: orchestrator-plugin
   data:
       dynamic-plugins.yaml: |
         includes:
           - dynamic-plugins.default.yaml
         plugins:
           - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator:{{inherit}}"
             disabled: false
           - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend:{{inherit}}"
             disabled: false
             dependencies:
               - ref: sonataflow
           - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator:{{inherit}}"
             disabled: false
           - package: "oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets:{{inherit}}"
             disabled: false
   ---
   apiVersion: rhdh.redhat.com/v1alpha5
   kind: Backstage
   metadata:
     name: orchestrator
   spec:
     application:
       appConfig:
         configMaps:
           - name: app-config-rhdh
       dynamicPluginsConfigMapName: orchestrator-plugin

2. Create a secret containing the BACKEND_SECRET value as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: app-config-rhdh
   data:
     app-config.yaml: |-
       auth:
         environment: development
         providers:
           guest:
             # using the guest user to query the '/api/dynamic-plugins-info/loaded-plugins' endpoint.
             dangerouslyAllowOutsideDevelopment: true
       backend:
         auth:
           externalAccess:
             - type: static
               options:
                 token: ${BACKEND_SECRET}
                 subject: orchestrator
   ---
   apiVersion: v1
   kind: Secret
   metadata:
     name: backend-auth-secret
   stringData:
     # generated with the command below (from https://backstage.io/docs/auth/service-to-service-auth/#setup):
     # node -p 'require("crypto").randomBytes(24).toString("base64")'
     # notsecret
     BACKEND_SECRET: "R2FxRVNrcmwzYzhhN3l0V1VRcnQ3L1pLT09WaVhDNUEK"

3. Configure your Backstage CR to update the secret name in the extraEnvs field as shown in the following example:

   apiVersion: rhdh.redhat.com/v1alpha5
   kind: Backstage
   metadata:
     name: orchestrator
   spec:
     application:
       appConfig:
         configMaps:
           - name: app-config-rhdh
       dynamicPluginsConfigMapName: orchestrator-plugin
       extraEnvs:
         secrets:
             # secret that contains the BACKEND_SECRET key
           - name: backend-auth-secret

Procedure

1. Delete the previous logic-operator-rhel8 subscription and apply the following configuration to install the logic-operator subscription:

   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-serverless-logic
   ---
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: openshift-serverless-logic
     namespace: openshift-serverless-logic
   spec:
   ---
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: logic-operator
     namespace: openshift-serverless-logic
   spec:
     channel: stable  #  channel of an operator package to subscribe to
     installPlanApproval: Automatic #  whether the update should be installed automatically
     name: logic-operator  #  name of the operator package
     source: redhat-operators  #  name of the catalog source
     sourceNamespace: openshift-marketplace
     startingCSV: logic-operator.v1.37.2  # The initial version of the operator

2. Optional: If your configuration uses an external PostgreSQL database with SSL, add the required datasource environment variables to the jobService specification in the SonataflowPlatform custom resource as shown in the following configuration:

   jobService:
     enabled: true
     persistence:
       dbMigrationStrategy: service
       postgresql:
         # no additional url params here. We only have currentSchema=jobs-service.
         jdbcUrl: 'jdbc:postgresql://postgress-external-db-primary.postgress-external-db.svc.cluster.local:5432/sonataflow?currentSchema=jobs-service'
         secretRef:
           name: postgres-cred
           passwordKey: POSTGRES_PASSWORD
           userKey: POSTGRES_USER
     podTemplate:
       container:
         env:
           # only this two env vars
           - name: QUARKUS_DATASOURCE_REACTIVE_POSTGRESQL_SSL_MODE
             value: allow
           - name: QUARKUS_DATASOURCE_REACTIVE_TRUST_ALL
             value: 'true'

Procedure

1. Open your dynamic-plugins ConfigMap for editing.

2. Update the package references for the Orchestrator plugins to use the 1.9 OCI registry paths as shown in the following example:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: dynamic-plugins-rhdh
   data:
     dynamic-plugins.yaml: |
       includes:
         - dynamic-plugins.default.yaml
       plugins:
         - package: 'oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator:{{inherit}}'
           disabled: false
         - package: 'oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend:{{inherit}}'
           disabled: false
           dependencies:
             - ref: sonataflow
         - package: 'oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator:{{inherit}}'
           disabled: false
         - package: 'oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets:{{inherit}}'
           disabled: false

   Note

   The {{inherit}} attribute in your configuration automatically resolves to the 1.9 version provided by the Operator.

3. Save the configuration changes.

Verification

1. Log in to your Red Hat Developer Hub instance.
2. Confirm that the Orchestrator plugins display the version metadata for 1.9.

Procedure

1. Identify the workflowId from your workflow definition file:

   id: greeting
   version: '1.0'

2. In your RBAC policy CSV file, define the permissions using the p, role, permission, action, allow format.

   Note

   Generic permissions (for example, orchestrator.workflow) take precedence over specific permissions targeting a workflowId, (for example, orchestrator.workflow.greeting). You cannot grant generic access and then selectively deny a specific ID.

3. Add the following example policies to your CSV file to establish basic user and administrator roles:

   # Minimal user role - can only view and run specific workflows
   p, role:default/workflowUser, orchestrator.workflow.greeting, read, allow
   p, role:default/workflowUser, orchestrator.workflow.use.greeting, update, allow
   
   # Support role - can view all workflows and instances, but not execute
   p, role:default/workflowSupport, orchestrator.workflow, read, allow
   p, role:default/workflowSupport, orchestrator.instanceAdminView, read, allow
   
   # Full admin role - complete access to all Orchestrator functions
   p, role:default/workflowAdmin, orchestrator.workflow, read, allow
   p, role:default/workflowAdmin, orchestrator.workflow.use, update, allow
   p, role:default/workflowAdmin, orchestrator.workflowAdminView, read, allow
   p, role:default/workflowAdmin, orchestrator.instanceAdminView, read, allow
   
   # Assign users to the roles
   g, user:default/example_user, role:default/workflowUser

4. In your RHDH app-config.yaml file, enable permissions by adding the orchestrator plugin to the rbac section and setting policyFileReload to true.

   permission:
     enabled: true
     rbac:
       policies-csv-file: <absolute_path_to_the_policy_file>
       pluginsWithPermission:
         - orchestrator
       policyFileReload: true
       admin:
         users:
           - name: user:default/YOUR_USER

5. Restart the application to apply the changes.

Verification

1. Log in as a user assigned to the workflowUser role.
2. Navigate to the Orchestrator plugin and verify that the workflow appears in the list.

Procedure

1. Manually approve the install plans for the Operators. You must run the oc patch installplan commands provided in the output to approve their installation.

   Important

   By default, Orchestrator Infrastructure for Red Hat Developer Hub Helm chart does not auto-approve the required Serverless Operators. You must manually approve the install plans.

2. As an administrator, install relevant cluster-wide resources.

   $ helm repo add openshift-helm-charts https://charts.openshift.io/
   $ helm install <release_name> openshift-helm-charts/redhat-developer-hub-orchestrator-infra

   Important

   You must be an administrator to install the redhat-developer-hub-orchestrator-infra Helm chart because it deploys additional cluster-scoped OpenShift Serverless and OpenShift Serverless Logic Operators. As an administrator, you must manually approve the install plans for OpenShift Serverless and Serverless Logic Operators.

3. Install the Backstage chart with the orchestrator enabled as shown in the following example:

   $ helm install <release_name> openshift-helm-charts/redhat-developer-hub --version 1.9.0 \
     --set orchestrator.enabled=true

4. (Optional) Enable Notifications and Signals plugins by adding them to the global.dynamic.plugins list in your values.yaml file as shown in the following example:

   global:
     dynamic:
       plugins:
         - disabled: false
           package: "./dynamic-plugins/dist/backstage-plugin-notifications"
         - disabled: false
           package: "./dynamic-plugins/dist/backstage-plugin-signals"
         - disabled: false
           package: "./dynamic-plugins/dist/backstage-plugin-notifications-backend-dynamic"
         - disabled: false
           package: "./dynamic-plugins/dist/backstage-plugin-signals-backend-dynamic"

5. (Optional) You can disable the Serverless Logic and Serverless Operators individually or together by setting their values to false, as shown in the following example:

   $ helm install <release_name> openshift-helm-charts/redhat-developer-hub \
     --version 1.9.0 \
     --set orchestrator.enabled=true \
     --set orchestrator.serverlessOperator=false \
     --set orchestrator.serverlessLogicOperator=false

6. (Optional) If you are using an external database, add the following configuration under orchestrator.sonataflowPlatform in your values.yaml file:

   orchestrator:
     sonataflowPlatform:
       externalDBsecretRef: "<cred-secret>"
       externalDBName: "<database_name>" # The name of the user-configured existing database (Not the database that the orchestrator and sonataflow resources use).
       externalDBHost: "<database_host>"
       externalDBPort: "<database_port>"

   Note

   This step only configures the Orchestrators use of an external database. To configure Red Hat Developer Hub to use an external PostgreSQL instance, follow the steps in Configure a PostgreSQL instance using Helm.

Verification

1. Verify that the Orchestrator plugin is visible in the Red Hat Developer Hub UI.
2. Create and run sample workflows to confirm the orchestration is functioning correctly.

Procedure

1. In the OpenShift Container Platform web console, go to the Helm Charts and verify that the Red Hat Developer Hub Helm chart repository is available.

2. Search for the Orchestrator infrastructure for Red Hat Developer Hub and select Install.

   Important

   You must be an administrator to install the Orchestrator Infrastructure for Red Hat Developer Hub Helm chart because it deploys cluster-scoped resources. As an administrator, you must manually approve the install plans for OpenShift Serverless and Serverless Logic Operators.

   As a regular user, search for the Red Hat Developer Hub chart and install it by setting the value of orchestrator.enabled to true. Otherwise, the Orchestrator will not be deployed.

3. Wait until they are successfully deployed.
4. Monitor the deployment status by navigating to Pods or releases.

1. The orchestrator-related pods are running in the selected namespace.
2. Cluster-wide resources are present.
3. You can start connecting the orchestrator to your Red Hat Developer Hub UI.

1. Default resource limits

1. You can override these values in any of the following ways:

   • With values.yaml
   • With --set flags

2. Override defaults with values.yaml as shown in the following example:

   orchestrator:
     enabled: true
     sonataflowPlatform:
     resources:
         limits:
           cpu: "500m"
           memory: "1Gi"

3. Override with --set as shown in the following example:

   $ helm upgrade --install <release_name>  openshift-helm-charts/redhat-developer-hub \
     --set orchestrator.enabled=true \
     --set orchestrator.sonataflowPlatform.resources.requests.cpu=500m \
     --set orchestrator.sonataflowPlatform.resources.requests.memory=128Mi \
     --set orchestrator.sonataflowPlatform.resources.limits.cpu=1 \
     --set orchestrator.sonataflowPlatform.resources.limits.memory=2Gi

   Note

   The --set setting is applicable only when orchestrator.enabled is true. By default, it is set to false.

Procedure

1. Install the OpenShift Serverless components manually by following the instructions in the Red Hat OpenShift Serverless documentation.

2. (Optional) If required, deploy a custom PostgreSQL database.

   Important

   Prevent workflow context from being lost when the Pod restarts by configuring workflow persistence. You can configure persistence at the namespace level by using the SonataFlowPlatform or SonataFlow custom resources (CR). For more information, check the Managing workflow persistence documentation.

Procedure

1. Create an ImageSetConfiguration file for oc-mirror. You must include the images and operators required by the Serverless Logic Operator in the ImageSetConfiguration file, as shown in the following example:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalimages:
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@<digest>
   
     operators:
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v<ocp-version>
        # For example: registry.redhat.io/redhat/redhat-operator-index:v4.20
         packages:
         - name: logic-operator
           channels:
           - name: stable
             minVersion: 1.37.2
             maxVersion: 1.37.2
         - name: serverless-operator
           channels:
           - name: stable
             minVersion: 1.37.1
             maxVersion: 1.37.1

   where:

   <digest>

   Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

   #!/bin/bash
   
   unpack () {
     local IMAGE="$1"
     DIR="${IMAGE//:/_}"
     DIR="/tmp/${DIR//\//-}"
     rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}")
     podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar
     echo "Unpacked $IMAGE into $DIR"
     cd $DIR; tree -d -L 3 -I "usr|root|buildinfo"
   }
   
   unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:1.9"
   
   # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index_1.9/dynamic-plugins.default.yaml

2. Mirror the images in the ImageSetConfiguration.yaml file by running the oc-mirror command. For example:

   $ oc-mirror --config=ImageSetConfiguration.yaml file:///path/to/mirror-archive --authfile /path/to/authfile  --v2

   Note

   The oc-mirror command generates a local workspace containing the mirror archive files and the required cluster manifests.

3. Transfer the directory specified by /path/to/mirror-archive to a bastion host within your disconnected environment.

4. From the bastion host which has access to the mirror registry, mirror the images from the disk directory to your target registry. For example:

   $ oc-mirror --v2 --from <mirror-archive-file> docker://<target-registry-url:port> --workspace file://<workspace folder> --authfile /path/to/authfile

   where:

   <mirror-archive-file>

   Enter the name of the transferred tar file.

   <target-registry-url:port>

   Enter your local registry, for example, registry.localhost:5000.

5. Apply the cluster-wide resources generated during the push step to redirect all image pulls to your local registry, as shown in the following example:

   $ cd <workspace folder>/working-dir/cluster-resources/
   $ oc apply -f .

6. Install the OpenShift Serverless Operator and OpenShift Serverless Logic Operators using OperatorHub.
7. Create a Backstage custom resource (CR).

8. Configure the Backstage CR for the Orchestrator as described in the Orchestrator plugin dependencies for Operator installation.

   Create all the resources and configure the Backstage instance accordingly.

Verification

1. Restart the RHDH pod and wait for the components to deploy properly.
2. Once stable, go to the RHDH UI, and confirm that the Orchestrator UI is accessible and functioning correctly.

Procedure

1. Create an ImageSetConfiguration file for oc-mirror. You must include the images and operators required by the Serverless Logic Operator in the ImageSetConfiguration file, as shown in the following example:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalimages:
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@<digest>
   
     operators:
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v<ocp-version>
        # For example: registry.redhat.io/redhat/redhat-operator-index:v4.20
         packages:
         - name: logic-operator
           channels:
           - name: stable
             minVersion: 1.37.2
             maxVersion: 1.37.2
         - name: serverless-operator
           channels:
           - name: stable
             minVersion: 1.37.1
             maxVersion: 1.37.1

   Where:

   <digest>

   Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

   #!/bin/bash
   
   unpack () {
     local IMAGE="$1"
     DIR="${IMAGE//:/_}"
     DIR="/tmp/${DIR//\//-}"
     rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}")
     podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar
     echo "Unpacked $IMAGE into $DIR"
     cd $DIR; tree -d -L 3 -I "usr|root|buildinfo"
   }
   
   unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:1.9"
   
   # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index_1.9/dynamic-plugins.default.yaml

2. Mirror the images in the ImageSetConfiguration.yaml file by running the oc-mirror command. For example:

   $ oc-mirror --config=imagesetconfiguration.yaml docker://<registry URL:port> --workspace file://<workspace folder> --authfile /path/to/authfile  --v2
   $ cd <workspace folder>/working-dir/cluster-resources/
   $ oc apply -f .

3. Install the OpenShift Serverless Operator and OpenShift Serverless Logic Operators using OperatorHub.
4. Create a Backstage custom resource (CR).

5. Configure the Backstage CR for the Orchestrator as described in the Orchestrator plugin dependencies for Operator installation.

   Create all the resources and configure the Backstage instance accordingly.

Verification

1. Restart the RHDH pod and wait for the components to deploy properly.
2. Once stable, go to the RHDH UI, and confirm that the Orchestrator UI is accessible and functioning correctly.

Procedure

1. Create an ImageSetConfiguration.yaml file for oc-mirror. You must use an ImageSetConfiguration file to include all mirrored images required, as shown in the following example:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalimages:
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@<digest>
   
     helm:
       repositories:
         - name: openshift-charts
           url: https://charts.openshift.io
           charts:
             - name: redhat-developer-hub
               version: "1.9.0"
             - name: redhat-developer-hub-orchestrator-infra
               version: "1.9.0"
     operators:
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v<ocp-version>
        # For example: registry.redhat.io/redhat/redhat-operator-index:v4.20
         packages:
         - name: logic-operator
           channels:
           - name: stable
             minVersion: 1.37.2
             maxVersion: 1.37.2
         - name: serverless-operator
           channels:
           - name: stable
             minVersion: 1.37.1
             maxVersion: 1.37.1

   where:

   <digest>

   Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

   #!/bin/bash
   
   unpack () {
     local IMAGE="$1"
     DIR="${IMAGE//:/_}"
     DIR="/tmp/${DIR//\//-}"
     rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}")
     podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar
     echo "Unpacked $IMAGE into $DIR"
     cd $DIR; tree -d -L 3 -I "usr|root|buildinfo"
   }
   
   unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:1.9"
   
   # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index_1.9/dynamic-plugins.default.yaml

2. Mirror the images in the ImageSetConfiguration.yaml file by running the oc-mirror command. For example:

   $ oc-mirror --config=ImageSetConfiguration.yaml file:///path/to/mirror-archive --authfile /path/to/authfile  --v2

   Note

   The oc-mirror command pulls the charts listed in the ImageSetConfiguration file and makes them available as tgz archives under the /path/to/mirror-archive directory.

3. Apply the cluster-wide resources generated during the push step to redirect all image pulls to your local registry, as shown in the following example:

   $ cd <workspace folder>/working-dir/cluster-resources/
   $ oc apply -f .

4. Transfer the generated mirror archive file, for example, /path/to/mirror-archive/mirror_000001.tar, to a bastion host within your disconnected environment.

5. From the bastion host in your disconnected environment, which has access to the mirror registry, mirror the images from the archive file to your target registry. For example:

   $ oc-mirror --v2 --from <mirror-archive-file> docker://<target-registry-url:port> --workspace file://<workspace folder> --authfile /path/to/authfile

   where:

   <mirror-archive-file>

   Enter the name of the transferred tar file.

   <target-registry-url:port>

   Enter your local registry, for example, registry.localhost:5000.

6. Apply the redhat-developer-hub-orchestrator-infra Helm chart and approve the install plans. See Air-gapped installation with Helm chart instructions for details.

7. Apply the RHDH 1.9 Helm chart. Include the version 1.9.0 and enable the Orchestrator plugin, as shown in the following example:

   orchestrator.enabled=true

8. The RHDH 1.9 Helm chart defaults to pulling Orchestrator plugins from the official Red Hat OCI registry using full URL references. Override this default behavior to point the chart to your local registry.

   To configure the Orchestrator plugins to use a custom registry, complete the following steps:

9. Open your values.yaml file.

10. List the Orchestrator plugin packages under the orchestrator.plugins section. You must replace the simplified package references with the full URLs that point to your custom OCI registry.

    Important

    You must explicitly include the pluginConfig configuration for each plugin as shown in the following example:

    orchestrator:
      plugins:
        - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@_<digest>_
          disabled: true
          pluginConfig:
            orchestrator:
              dataIndexService:
                url: http://sonataflow-platform-data-index-service
        - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@_<digest>_
          disabled: true
          pluginConfig:
            dynamicPlugins:
              frontend:
                red-hat-developer-hub.backstage-plugin-orchestrator-form-widgets: {}
        - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@_<digest>_
          disabled: true
          pluginConfig:
            dynamicPlugins:
              frontend:
                red-hat-developer-hub.backstage-plugin-orchestrator:
                  appIcons:
                  - importName: OrchestratorIcon
                    name: orchestratorIcon
                  dynamicRoutes:
                  - importName: OrchestratorPage
                    menuItem:
                      icon: orchestratorIcon
                      text: Orchestrator
                      textKey: menuItem.orchestrator
                    path: /orchestrator
                  entityTabs:
                  - path: /workflows
                    title: Workflows
                    titleKey: catalog.entityPage.workflows.title
                    mountPoint: entity.page.workflows
                  mountPoints:
                  - mountPoint: entity.page.workflows/cards
                    importName: OrchestratorCatalogTab
                    config:
                      layout:
                      gridColumn: 1 / -1
                      if:
                        anyOf:
                        - IsOrchestratorCatalogTabAvailable
        - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@_<digest>_
          disabled: true
          pluginConfig:
            orchestrator:
              dataIndexService:
                url: http://sonataflow-platform-data-index-service

    where:

    <custom_registry_url>

    Enter the address of your custom registry where the OCI images have been mirrored.

    <digest>

    Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

    #!/bin/bash
    
    unpack () {
      local IMAGE="$1"
      DIR="${IMAGE//:/_}"
      DIR="/tmp/${DIR//\//-}"
      rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}")
      podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar
      echo "Unpacked $IMAGE into $DIR"
      cd $DIR; tree -d -L 3 -I "usr|root|buildinfo"
    }
    
    unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:1.9"
    
    # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index_1.9/dynamic-plugins.default.yaml

Verification

1. Restart the RHDH Pod and wait for the components to deploy properly.
2. After deployment is complete, go to the RHDH UI and confirm that the Orchestrator UI is accessible and functioning correctly.

Procedure

1. Create an ImageSetConfiguration.yaml file for oc-mirror. You must use an ImageSetConfiguration file to include all mirrored images required, as shown in the following example:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalimages:
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@<digest>
     - name: registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@<digest>
   
     helm:
       repositories:
         - name: openshift-charts
           url: https://charts.openshift.io
           charts:
             - name: redhat-developer-hub
               version: "1.9.0"
             - name: redhat-developer-hub-orchestrator-infra
               version: "1.9.0"
     operators:
       - catalog: registry.redhat.io/redhat/redhat-operator-index:v<ocp-version>
        # For example: registry.redhat.io/redhat/redhat-operator-index:v4.20
         packages:
         - name: logic-operator
           channels:
           - name: stable
             minVersion: 1.37.2
             maxVersion: 1.37.2
         - name: serverless-operator
           channels:
           - name: stable
             minVersion: 1.37.1
             maxVersion: 1.37.1

2. Mirror the images in the ImageSetConfiguration.yaml file by running the oc-mirror command to pull images and charts, and push the images directly to the target registry. For example:

   $ oc-mirror --config=imagesetconfiguration.yaml docker://<registry URL:port> --workspace file://<workspace folder> --authfile /path/to/authfile  --v2

   Note

   The oc-mirror command pulls the charts listed in the ImageSetConfiguration file and makes them available as tgz archives under the <workspace folder> directory.

3. Apply the generated cluster resources to the disconnected cluster. For example:

   $ cd <workspace folder>/working-dir/cluster-resources/
   $ oc apply -f .

4. Apply the redhat-developer-hub-orchestrator-infra Helm chart and approve the install plans. See Air-gapped installation with Helm chart instructions for details.

5. Apply the RHDH 1.9 Helm chart. Include the version 1.9.0 and enable the Orchestrator plugin, as shown in the following example:

   orchestrator.enabled=true

6. The RHDH 1.9 Helm chart defaults to pulling Orchestrator plugins from the official Red Hat OCI registry using full URL references. You must override this behavior to point to your local registry.

   To configure the Orchestrator plugins to use a custom registry, complete the following steps:

   • Open your values.yaml file.

   • Explicitly list the Orchestrator plugin packages under the orchestrator.plugins section.

     You must replace the simplified package references with the full URLs that point to your custom OCI registry. You must explicitly include the pluginConfig configuration for each plugin as shown in the following example:

     orchestrator:
       plugins:
         - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@_<digest>_
           disabled: true
           pluginConfig:
             orchestrator:
               dataIndexService:
                 url: http://sonataflow-platform-data-index-service
         - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-form-widgets@_<digest>_
           disabled: true
           pluginConfig:
             dynamicPlugins:
               frontend:
                 red-hat-developer-hub.backstage-plugin-orchestrator-form-widgets: {}
         - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator@_<digest>_
           disabled: true
           pluginConfig:
             dynamicPlugins:
               frontend:
                 red-hat-developer-hub.backstage-plugin-orchestrator:
                   appIcons:
                   - importName: OrchestratorIcon
                     name: orchestratorIcon
                   dynamicRoutes:
                   - importName: OrchestratorPage
                     menuItem:
                       icon: orchestratorIcon
                       text: Orchestrator
                       textKey: menuItem.orchestrator
                     path: /orchestrator
                   entityTabs:
                   - path: /workflows
                     title: Workflows
                     titleKey: catalog.entityPage.workflows.title
                     mountPoint: entity.page.workflows
                   mountPoints:
                   - mountPoint: entity.page.workflows/cards
                     importName: OrchestratorCatalogTab
                     config:
                       layout:
                       gridColumn: 1 / -1
                       if:
                         anyOf:
                         - IsOrchestratorCatalogTabAvailable
         - - package: oci://<custom_registry_url>/rhdh/red-hat-developer-hub-backstage-plugin-scaffolder-backend-module-orchestrator@_<digest>_
           disabled: true
           pluginConfig:
             orchestrator:
               dataIndexService:
                 url: http://sonataflow-platform-data-index-service

     Where:

     <custom_registry_url>

     Enter the address of your custom registry where the OCI images have been mirrored.

     <digest>

     Locate the image digests for your version of RHDH in the dynamic-plugins.default.yaml file. You can extract this file from the plugin catalog index image to verify the default settings for your specific release:

     #!/bin/bash
     
     unpack () {
       local IMAGE="$1"
       DIR="${IMAGE//:/_}"
       DIR="/tmp/${DIR//\//-}"
       rm -fr "$DIR"; mkdir -p "$DIR"; container_id=$(podman create "${IMAGE}")
       podman export $container_id -o /tmp/image.tar && tar xf /tmp/image.tar -C "${DIR}/"; podman rm $container_id; rm -f /tmp/image.tar
       echo "Unpacked $IMAGE into $DIR"
       cd $DIR; tree -d -L 3 -I "usr|root|buildinfo"
     }
     
     unpack "registry.access.redhat.com/rhdh/plugin-catalog-index:1.9"
     
     # you can then find the dynamic-plugins.default.yaml under /tmp/registry.access.redhat.com/rhdh/plugin-catalog-index_1.9/dynamic-plugins.default.yaml

Verification

1. Restart the RHDH pod and wait for the components to deploy properly.
2. After deployment is complete, go to the RHDH UI and confirm that the Orchestrator UI is accessible and functioning correctly.

Procedure

1. Enable the Loki backend module in the redhat-developer-hub-dynamic-plugins ConfigMap.

   
2. Open the ConfigMap and select the YAML view.

3. Add the Loki backend module to the plugins section:

   - disabled: false
         package: oci://registry.access.redhat.com/rhdh/red-hat-developer-hub-backstage-plugin-orchestrator-backend-module-loki:{{inherit}}

   
4. Save the file.

5. In your application app-config.yaml ConfigMap file, add the Loki workflow log provider integration to the orchestrator section:

   
   
   Note

   The base URl might vary depending on where you store and access the Loki logs.

   To obtain the base URL, run the following command:

   LOKI_HOST=$(oc get route logging-loki -n openshift-logging -o jsonpath='{.spec.host}')
   echo "https://$LOKI_HOST/api/logs/v1/application/"

   orchestrator:
     workflowLogProvider:
       loki:
         baseUrl: <LOKI_BASE_URL>
   	token: <AUTH_TOKEN>
   	rejectUnauthorized: false
        # logPipelineFilters:
         # - '| filter1'
         # - '|= filter2'
         # logStreamSelectors:
         #   - label: 'app'
         #     value: '=~".+"'

   where:

   baseUrl: Specifies the base URL of your Loki instance. This value is required.

   token: The access token for authentication. This value is required. To find your current token, run the oc whoami -t command.

   rejectUnauthorized: Set to false if using self-signed certificates.

   Optional Parameters

   logPipelineFilters: Multiple Log Pipeline Filters can be specified in the logPipelineFilters section. See the Loki documentation to learn more about the log pipeline filters and their values and usage.

   logStreamSelectors: Specifies log stream selectors to filter the logs. By default, the plugin retrieves logs with the openshift_log_type="application". For more information about selector syntax, see the Grafana Loki documentation.

6. Save the ConfigMap.
7. Restart the Red Hat Developer Hub pod to apply the changes.

Verification

1. Navigate to the Orchestrator plugin in the RHDH interface.
2. Select a workflow instance.
3. Click View Logs to display the workflow instance logs.

Procedure

1. Update your workflow build configuration to include the JSON logging extension:

   export QUARKUS_EXTENSIONS="${QUARKUS_EXTENSIONS},io.quarkus:quarkus-logging-json"

2. Open the {workflow-name}-props ConfigMap for your workflow.

3. Add the following properties to the application.properties section:

   # Enable JSON logging with Quarkus JSON logging extension
   quarkus.log.console.json=true
   quarkus.log.console.json.pretty-print=false
   
   # Include all MDC context fields in JSON output
   # - processInstanceId: Set automatically by SonataFlow/Kogito
   # - traceId, spanId: Set by Quarkus OpenTelemetry (requires quarkus.otel.enabled=true)
   quarkus.log.console.json.print-details=true
   
   # Set log levels for workflow components
   quarkus.log.category."org.kie.kogito".level=DEBUG
   quarkus.log.category."io.serverlessworkflow".level=INFO
   
   # Optional: Enable additional context logging
   quarkus.log.category."org.kie.kogito.services.context".level=DEBUG

4. Save the ConfigMap and restart the workflow pod.

   The following is an example of a workflow ConfigMap with an enabled JSON logging:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: greetings-props
     namespace: sonataflow-infra
   data:
     application.properties: |
       # JSON logging configuration
       quarkus.log.console.json=true
       quarkus.log.console.json.pretty-print=false
       quarkus.log.console.json.print-details=true
   
       # Log levels
       quarkus.log.category."org.kie.kogito".level=DEBUG
       quarkus.log.category."io.serverlessworkflow".level=INFO

Procedure

1. Add the following properties to your workflow ConfigMap to enable file-based JSON output:

   quarkus.log.file.enable=true
   quarkus.log.file.path=/var/log/sonataflow/workflow.log
   quarkus.log.file.json=true

2. Configure log rotation settings to manage disk usage:

   quarkus.log.file.rotation.max-file-size=10M
   quarkus.log.file.rotation.max-backup-index=5
   quarkus.log.file.rotation.rotate-on-boot=true

   This configuration does the following:

   • Rotates logs when they reach 10MB
   • Keeps up to 5 backup files
   • Adds date suffix to rotated files
   • Rotates on application startup

3. Set log level for file output:

   quarkus.log.file.level=INFO

4. Update your SonataFlow custom resource to mount a volume at the log path:

   spec:
     podTemplate:
       container:
         volumeMounts:
         - name: shared-logs
           mountPath: /var/log/sonataflow
     volumes:
     - name: shared-logs
       emptyDir:
         sizeLimit: 500Mi

1. After applying the configuration, restart your workflow pod and check the log output:

   # Get workflow pod name
   oc get pods -n sonataflow-infra -l sonataflow.org/workflow-app=your-workflow
   
   # Check logs for JSON format
   oc logs -n sonataflow-infra your-workflow-pod-name | head -5

Procedure

1. Add the OpenTelemetry exporter and service identification properties to your workflow ConfigMap:

   # Enable OpenTelemetry integration
   quarkus.otel.enabled=true
   quarkus.otel.exporter.otlp.traces.endpoint=http://jaeger-collector:14268/api/traces
   quarkus.otel.service.name=${workflow.name}

2. Set the resource attributes to help filter traces in your dashboard:

   quarkus.otel.resource.attributes=service.namespace=sonataflow-infra

3. Restart the workflow pod.

Procedure

1. For a quick start, deploy the PLG stack using Helm:

   # Add Grafana Helm repository
   helm repo add grafana https://grafana.github.io/helm-charts
   helm repo update
   
   # Create namespace
   oc new-project sonataflow-observability
   
   # Deploy Loki stack
   helm install loki-stack grafana/loki-stack \
     --namespace sonataflow-observability \
     --set loki.persistence.enabled=true \
     --set loki.persistence.size=20Gi \
     --set promtail.config.logLevel=info \
     --set grafana.enabled=true

2. For production deployment, use the complete Helm values configuration, ../observability/helm-values/ with proper resource limits, security contexts, and OpenShift-specific settings.

3. Create a ConfigMap for the Promtail sidecar to parse the JSON logs. You can choose between the following options:

   • Scrape container stdout (default)

   • Custom JSON log files

     Scrape Container Stdout (Default)

     This configuration uses Kubernetes service discovery to collect logs from container stdout:

     apiVersion: v1
     kind: ConfigMap
     metadata:
       name: promtail-config
       namespace: sonataflow-observability
     data:
       config.yml: |
         server:
           http_listen_port: 3101
     
         clients:
           - url: http://loki:3100/loki/api/v1/push
     
         scrape_configs:
         - job_name: sonataflow-workflows
           kubernetes_sd_configs:
           - role: pod
             namespaces:
               names: ["sonataflow-infra"]
     
           relabel_configs:
           - source_labels: [__meta_kubernetes_pod_label_sonataflow_org_workflow_app]
             action: keep
             regex: (.+)
     
           - source_labels: [__meta_kubernetes_pod_name]
             target_label: pod
     
           - source_labels: [__meta_kubernetes_pod_label_sonataflow_org_workflow_app]
             target_label: workflow
     
           pipeline_stages:
           - json:
               expressions:
                 timestamp: timestamp
                 level: level
                 logger: logger
                 message: message
                 processInstanceId: mdc.processInstanceId
                 traceId: mdc.traceId
                 spanId: mdc.spanId
     
           - labels:
               level:
               logger:
               processInstanceId:
               traceId:

     Scrape JSON log files

     When using [file-based JSON logging](#file-based-json-logging), configure Promtail as a sidecar to read from the shared log volume:

     apiVersion: v1
     kind: ConfigMap
     metadata:
       name: promtail-sidecar-config
       namespace: sonataflow-infra
     data:
       config.yml: |
         server:
           http_listen_port: 3101
     
         clients:
           - url: http://loki.sonataflow-observability.svc.cluster.local:3100/loki/api/v1/push
     
         positions:
           filename: /var/log/positions.yaml
     
         scrape_configs:
         - job_name: sonataflow-json-files
           static_configs:
           - targets:
               - localhost
             labels:
               job: sonataflow-workflows
               __path__: /var/log/sonataflow/*.log
     
           pipeline_stages:
           - json:
               expressions:
                 timestamp: timestamp
                 level: level
                 logger: loggerName
                 message: message
                 processInstanceId: mdc.processInstanceId
                 traceId: mdc.traceId
                 spanId: mdc.spanId
     
           - labels:
               level:
               logger:
               processInstanceId:
               traceId:
     
           - timestamp:
               source: timestamp
               format: RFC3339Nano

4. Add the Promtail sidecar container to your SonataFlow custom resource:

   apiVersion: sonataflow.org/v1alpha08
   kind: SonataFlow
   metadata:
     name: my-workflow
     namespace: sonataflow-infra
   spec:
     podTemplate:
       container:
         volumeMounts:
         - name: shared-logs
           mountPath: /var/log/sonataflow
       containers:
       - name: promtail-sidecar
         image: grafana/promtail:2.9.0
         args:
           - -config.file=/etc/promtail/config.yml
         volumeMounts:
         - name: shared-logs
           mountPath: /var/log/sonataflow
           readOnly: true
         - name: promtail-config
           mountPath: /etc/promtail
         - name: positions
           mountPath: /var/log
         resources:
           requests:
             cpu: 50m
             memory: 64Mi
           limits:
             cpu: 100m
             memory: 128Mi
       volumes:
       - name: shared-logs
         emptyDir:
           sizeLimit: 500Mi
       - name: promtail-config
         configMap:
           name: promtail-sidecar-config
       - name: positions
         emptyDir: {}

1. Filter logs by process instance

   {job="sonataflow-workflows"} | json | processInstanceId="abc-123-def-456"

2. Find workflow errors

   {job="sonataflow-workflows", workflow="onboarding"} | json | level="ERROR"

3. Trace correlation

   {job="sonataflow-workflows"} | json | traceId="4bf92f3577b34da6a3ce929d0e0e4736"

4. Process instance timeline

   {job="sonataflow-workflows"} | json | processInstanceId="abc-123-def-456" | line_format "{{.timestamp}} [{{.level}}] {{.message}}"

Procedure

1. Update your configuration with the following rule groups:

   • To monitor failure rates:

     - alert: WorkflowHighErrorRate
       expr: rate({job="sonataflow-workflows", level="ERROR"}[5m]) > 0.1
       for: 2m
       labels:
         severity: warning
       annotations:
         summary: "High error rate in SonataFlow workflows"

   • To identify stuck process instances:

     - alert: WorkflowInstanceStuck
       expr: |
         time() - max by (process_instance_id) (
           {job="sonataflow-workflows"} | json | unwrap timestamp[1h]
         ) > 3600
       labels:
         severity: critical

   • For long-running processes:

       - alert: LongRunningWorkflow
         expr: |
           time() - min by (process_instance_id) (
             {job="sonataflow-workflows"} | json | message="Workflow started" | unwrap timestamp[24h]
           ) > 7200
         labels:
           severity: warning
         annotations:
           summary: "Workflow {{ $labels.process_instance_id }} running longer than 2 hours"

2. Apply the configuration to your cluster.

Procedure

1. Edit your configuration to define a receiver and a routing path:

   route:
     group_by: ['alertname', 'workflow']
     group_wait: 10s
     group_interval: 10s
     repeat_interval: 1h
     receiver: 'web.hook'
   
   receivers:
   - name: 'web.hook'
     slack_configs:
     - api_url: 'YOUR_SLACK_WEBHOOK_URL'
       channel: '#workflow-alerts'
       title: 'SonataFlow Alert'
       text: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'

2. Reload the configuration.

Procedure

1. Verify JSON log formatting

   If logs appear as plain text instead of structured JSON, perform the following checks:

   • Make sure the io.quarkus:quarkus-logging-json extension is defined in your pom.xml workflow.
   • Confirm that quarkus.log.console.json=true is set in the {workflow-name}-props ConfigMap.
   • Rebuild and redeploy the workflow image to ensure the extension is active.
   • Restart the workflow pod to apply ConfigMap changes.

2. Diagnose missing process instance context

   If logs are in JSON format but the processInstanceId field is missing or empty:

   • Verify that workflow instances are actively running.

   • Check the workflow ConfigMap for the following property:

     quarkus.log.console.json.print-details=true

   • Consult the release notes for your SonataFlow version to confirm if Mapped Diagnostic Context (MDC) is automatically populated.

3. Resolve log collection failures in Loki

   If logs are generated but do not appear in Loki or your log aggregation dashboard:

   • Verify the Promtail or Fluent Bit label selector matches the labels on your workflow pods.

   • Check the collector logs for permission errors:

     oc logs -l app=promtail -n sonataflow-observability

   • Make sure that the collector has the necessary RBAC permissions to read logs from the workflow namespace.
   • Verify that the scrape_configs in the collector configuration include the correct namespace.

4. Mitigate high resource usage

   If JSON logging causes performance degradation or high storage costs:

   • Increase the log level for verbose categories to reduce output volume:

     quarkus.log.category."org.kie.kogito".level=WARN

   • Enable asynchronous logging to reduce the impact on workflow execution time:

     quarkus.log.console.async=true

   • Implement log rotation and set appropriate retention policies in your aggregation backend.
   • Monitor network bandwidth and storage usage to identify anomalies.

Procedure

1. Add the OpenTelemetry addon to the QUARKUS_EXTENSIONS environment variable during the image build process:

   export QUARKUS_EXTENSIONS="${QUARKUS_EXTENSIONS},org.apache.kie.sonataflow:sonataflow-addons-quarkus-opentelemetry"

2. Open the {workflow-name}-props ConfigMap for your workflow.

3. In the application.properties section, enable the OpenTelemetry integration and configure the service attributes:

   # Application Identity
   quarkus.application.name=my-workflow
   quarkus.application.version=1.0.0
   
   # OpenTelemetry Configuration
   quarkus.otel.enabled=true
   quarkus.otel.traces.enabled=true
   quarkus.otel.metrics.enabled=true
   quarkus.otel.logs.enabled=true
   
   # Service Resource Attributes
   quarkus.otel.resource.attributes=\
     service.name=my-workflow,\
     service.namespace=workflows,\
     service.version=1.0.0,\
     deployment.environment=production
   
   # SonataFlow Specific Configuration
   # Master switch for SonataFlow OpenTelemetry integration
   sonataflow.otel.enabled=true
   # Service identification (uses Quarkus application name/version as defaults)
   sonataflow.otel.service-name=${quarkus.application.name:kogito-workflow-service}
   sonataflow.otel.service-version=${quarkus.application.version:unknown}
   # Enable span creation for workflow states
   sonataflow.otel.spans.enabled=true
   # Enable process lifecycle events (start, complete, error, state transitions)
   sonataflow.otel.events.enabled=true

4. Save the ConfigMap and restart the workflow pod to apply the changes.

Procedure

1. Define your export strategy.

   Choose an export strategy that matches your observability platform requirements:

   • OTLP exporter with batch processing (Recommended)

     For production environments, use an OTLP exporter with batch processing to reduce network overhead and improve performance:

     # OTLP exporter - Direct to Jaeger
     quarkus.otel.exporter.otlp.endpoint=http://jaeger-collector.observability.svc.cluster.local:4317
     quarkus.otel.exporter.otlp.protocol=grpc
     quarkus.otel.traces.exporter=cdi
     
     # Batch processing for production
     quarkus.otel.bsp.schedule.delay=5s
     quarkus.otel.bsp.max.export.batch.size=512
     quarkus.otel.bsp.export.timeout=2s
     quarkus.otel.bsp.max.queue.size=2048

   • Direct export to an external platform

     For development or simple integrations, use a direct export configuration:

     # Example: Direct export to Jaeger
     quarkus.otel.exporter.otlp.endpoint=http://jaeger-collector:4317
     quarkus.otel.exporter.otlp.protocol=grpc
     quarkus.otel.traces.exporter=cdi

2. Externalize the configuration for production deployments.

   Use environment variables to externalize your OpenTelemetry configuration. This ensures that your deployment remains secure and flexible across environments.

   # Externalized Configuration
   quarkus.otel.exporter.otlp.endpoint=${OTEL_EXPORTER_OTLP_ENDPOINT:http://localhost:4317}
   quarkus.otel.exporter.otlp.headers=${OTEL_EXPORTER_OTLP_HEADERS:}
   quarkus.application.name=${OTEL_SERVICE_NAME:my-workflow}
   quarkus.otel.resource.attributes=${OTEL_RESOURCE_ATTRIBUTES:deployment.environment=dev}

1. Jaeger distributed tracing

   Jaeger provides distributed tracing visualization for SonataFlow workflows.

   Jaeger all-in-one deployment (development and testing)

   apiVersion: v1
   kind: Namespace
   metadata:
     name: jaeger-system
   ---
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: jaeger
     namespace: jaeger-system
     labels:
       app: jaeger
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: jaeger
     template:
       metadata:
         labels:
           app: jaeger
       spec:
         containers:
         - name: jaeger
           image: jaegertracing/all-in-one:1.59
           env:
           - name: COLLECTOR_OTLP_ENABLED
             value: "true"
           ports:
           - containerPort: 16686
             name: query
           - containerPort: 4317
             name: otlp-grpc
           - containerPort: 4318
             name: otlp-http
           resources:
             requests:
               memory: "256Mi"
               cpu: "100m"
             limits:
               memory: "512Mi"
               cpu: "500m"
           readinessProbe:
             httpGet:
               path: /
               port: 14269
             initialDelaySeconds: 5
           livenessProbe:
             httpGet:
               path: /
               port: 14269
             initialDelaySeconds: 10
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: jaeger-collector
     namespace: jaeger-system
     labels:
       app: jaeger
   spec:
     selector:
       app: jaeger
     ports:
     - name: otlp-grpc
       port: 4317
       targetPort: 4317
     - name: otlp-http
       port: 4318
       targetPort: 4318
     type: ClusterIP
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: jaeger-query
     namespace: jaeger-system
     labels:
       app: jaeger
   spec:
     selector:
       app: jaeger
     ports:
     - name: query-http
       port: 16686
       targetPort: 16686
     type: ClusterIP

   OpenShift route for UI access

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: jaeger-query
     namespace: jaeger-system
   spec:
     to:
       kind: Service
       name: jaeger-query
     port:
       targetPort: query-http
     tls:
       termination: edge
       insecureEdgeTerminationPolicy: Redirect

   Workflow configuration for Jaeger

   Add these properties to the application.properties file of your workflow:

   # Direct connection to Jaeger
   quarkus.otel.exporter.otlp.endpoint=http://jaeger-collector.jaeger-system.svc.cluster.local:4317
   quarkus.otel.exporter.otlp.protocol=grpc
   quarkus.otel.traces.exporter=cdi
   
   # Additional Jaeger-specific propagation
   quarkus.otel.propagators=tracecontext,baggage,jaeger

   Production deployment with Elasticsearch

   For production environments, use the Jaeger Operator with Elasticsearch storage:

   apiVersion: jaegertracing.io/v1
   kind: Jaeger
   metadata:
     name: jaeger-production
     namespace: observability
   spec:
     strategy: production
     storage:
       type: elasticsearch
       elasticsearch:
         nodeCount: 3
         storage:
           storageClassName: gp3
           size: 50Gi
         resources:
           requests:
             cpu: 500m
             memory: 4Gi
           limits:
             cpu: 1000m
             memory: 8Gi
     collector:
       replicas: 2
       resources:
         requests:
           cpu: 200m
           memory: 256Mi
         limits:
           cpu: 500m
           memory: 512Mi

2. Loki log aggregation

   Loki supports OpenTelemetry Protocol (OTLP) for direct log ingestion from SonataFlow workflows.

   Loki configuration for OpenTelemetry

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: loki-config
     namespace: observability
   data:
     loki-config.yaml: |
       auth_enabled: false
   
       server:
         http_listen_port: 3100
         grpc_listen_port: 9096
   
       common:
         path_prefix: /loki
         storage:
           filesystem:
             chunks_directory: /loki/chunks
             rules_directory: /loki/rules
         replication_factor: 1
         ring:
           instance_addr: 127.0.0.1
           kvstore:
             store: inmemory
   
       distributor:
         otlp_config:
           # Default resource attributes as index labels
           default_resource_attributes_as_index_labels:
             - service.name
             - service.namespace
             - deployment.environment
             - k8s.namespace.name
             - k8s.cluster.name
   
       limits_config:
         # Enable structured metadata (default in Loki 3.0+)
         allow_structured_metadata: true
         # Maximum number of index labels per stream
         max_label_names_per_series: 15
   
       schema_config:
         configs:
           - from: 2024-01-01
             store: tsdb
             object_store: filesystem
             schema: v13  # Required for OTLP support
             index:
               prefix: index_
               period: 24h

   Loki deployment

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: loki
     namespace: observability
     labels:
       app: loki
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: loki
     template:
       metadata:
         labels:
           app: loki
       spec:
         securityContext:
           fsGroup: 10001
           runAsUser: 10001
           runAsNonRoot: true
         containers:
         - name: loki
           image: grafana/loki:3.0.0
           args:
             - -config.file=/etc/loki/loki-config.yaml
           ports:
           - containerPort: 3100
             name: http-metrics
           - containerPort: 9096
             name: grpc
           resources:
             requests:
               cpu: 500m
               memory: 1Gi
             limits:
               cpu: 1000m
               memory: 2Gi
           volumeMounts:
           - name: config
             mountPath: /etc/loki
           - name: storage
             mountPath: /loki
           livenessProbe:
             httpGet:
               path: /ready
               port: 3100
             initialDelaySeconds: 45
           readinessProbe:
             httpGet:
               path: /ready
               port: 3100
             initialDelaySeconds: 45
         volumes:
         - name: config
           configMap:
             name: loki-config
         - name: storage
           emptyDir: {}
   ---
   apiVersion: v1
   kind: Service
   metadata:
     name: loki
     namespace: observability
     labels:
       app: loki
   spec:
     selector:
       app: loki
     ports:
     - name: http-metrics
       port: 3100
       targetPort: 3100
     - name: grpc
       port: 9096
       targetPort: 9096
     type: ClusterIP

   Workflow configuration for Loki and Jaeger

   To route logs to Loki and traces to Jaeger, use the following configuration:

   # OpenTelemetry Configuration
   quarkus.otel.enabled=true
   quarkus.otel.traces.enabled=true
   quarkus.otel.metrics.enabled=true
   quarkus.otel.logs.enabled=true
   
   # OTLP Exporter - Send logs to Loki, traces to Jaeger
   quarkus.otel.exporter.otlp.logs.endpoint=http://loki.observability.svc.cluster.local:3100/otlp
   quarkus.otel.exporter.otlp.traces.endpoint=http://jaeger-collector.observability.svc.cluster.local:4317
   quarkus.otel.exporter.otlp.protocol=grpc
   
   # JSON Logging for better structure
   quarkus.log.console.json=true
   quarkus.log.console.json.pretty-print=false
   
   # Include trace correlation in logs
   quarkus.log.console.format=%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p [%c{3.}] (%t) traceId=%X{traceId}, spanId=%X{spanId} %s%e%n
   
   # Resource attributes for Loki labels
   quarkus.otel.resource.attributes=\
     service.name=greeting-workflow,\
     service.namespace=workflows,\
     deployment.environment=production

3. Optional: OpenTelemetry collector for advanced processing

   Deploy an OpenTelemetry collector between workflows and backends for advanced log processing, filtering, and multi-destination export.

   Collector pipeline configuration

   # Collector routes to both Jaeger and Loki
   exporters:
     otlp/jaeger:
       endpoint: jaeger-collector:4317
     otlphttp/loki:
       endpoint: http://loki:3100/otlp
   
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: [batch]
         exporters: [otlp/jaeger]
       logs:
         receivers: [otlp]
         processors: [batch]
         exporters: [otlphttp/loki]

Diagnosing missing traces

1. Verify that OpenTelemetry is enabled in the workflow ConfigMap:

   kubectl get cm {workflow-name}-props -n workflows -o yaml

2. Check the pod logs for initialization errors:

   kubectl logs deployment/{deployment-name} -n workflows | grep -i "otel"

3. Test the connection to the Jaeger collector from within the workflow pod:

   kubectl exec deployment/{deployment-name} -n workflows -- curl -v http://jaeger-collector.observability.svc.cluster.local:4317

Procedure

1. Clone the project as shown in the following example:

   git clone git@github.com:rhdhorchestrator/orchestrator-demo.git
   cd orchestrator-demo

2. Check the help menu of the script:

   ./scripts/build.sh --help

3. Run the build.sh script, providing the required flags, for example, the image path (-i), workflow source directory (-w), and manifests output directory (-m).

   Important

   You must specify the full target image path with a tag as shown in the following example:

   ./scripts/build.sh --image=quay.io/orchestrator/demo-basic:test -w 01_basic/ -m 01_basic/manifests

Procedure

1. Use the kubectl create command specifying the target namespace to apply the Kubernetes manifests as shown in the following example:

   $ kubectl create -n <your_namespace> -f ./01_basic/manifests/.

2. After deployment, monitor the status of the workflow pods as shown in the following example:

   $ kubectl get pods -n <your_namespace> -l app=basic

   The pod may initially appear in an Error state because of missing or incomplete configuration in the Secret or ConfigMap.

3. Inspect the Pod logs as shown in the following example:

   $ oc logs -n <your_namespace> basic-f7c6ff455-vwl56

   The following code is an example of the output:

   SRCFG00040: The config property quarkus.openapi-generator.notifications.auth.BearerToken.bearer-token is defined as the empty String ("") which the following Converter considered to be null: io.smallrye.config.Converters$BuiltInConverter
   java.lang.RuntimeException: Failed to start quarkus
   ...
   Caused by: io.quarkus.runtime.configuration.ConfigurationException: Failed to read configuration properties

   The error indicates a missing property: quarkus.openapi-generator.notifications.auth.BearerToken.bearer-token.

4. In such a case where the logs show the ConfigurationException: Failed to read configuration properties error or indicate a missing value, retrieve the ConfigMap as shown in the following example:

   $ oc get -n <your_namespace> configmaps basic-props -o yaml

   The following code is an example of the sample output:

   apiVersion: v1
   data:
     application.properties: |
       # Backstage notifications service
       quarkus.rest-client.notifications.url=${BACKSTAGE_NOTIFICATIONS_URL}
       quarkus.openapi-generator.notifications.auth.BearerToken.bearer-token=${NOTIFICATIONS_BEARER_TOKEN}
   ...

   Resolve the placeholders using values provided using a Secret.

5. You must edit the corresponding Secret and provide appropriate base64-encoded values to resolve the placeholders in application.properties as shown in the following example:

   $ kubectl edit secrets -n <your_namespace> basic-secrets

6. Restart the workflow Pod for Secret changes to take effect in OpenShift Serverless Logic 1.37.2.

Verification

1. Verify the deployment status by checking the Pods again as shown in the following example:

   $ oc get pods -n <your_namespace> -l app=basic

   The expected status for a successfully deployed workflow Pod is as shown in the following example:

   NAME                    READY   STATUS    RESTARTS   AGE
   basic-f7c6ff455-grkxd   1/1     Running   0          47s

2. Once the Pod is in the Running state, the workflow now appears in the Orchestrator plugin inside the Red Hat Developer Hub.

Procedure

1. Install the orchestrator-software-templates-infra chart.
2. Install the orchestrator-software-templates chart.

Procedure

1. Prepare the image registry. Before creating the template, configure the target repository in Quay.io.

   1. Log in to your Quay.io organization (for example, orchestrator-testing).
   2. Create a new repository (for example, serverless-workflow-demo).
   3. Add robot account permissions to the repository settings.

2. Open the Red Hat Developer Hub Catalog.

   
3. Select the Basic workflow bootstrap project template and click Launch Template.
4. Follow the template form to enter required details, including the GitHub organization, source code repository name, and a unique Workflow ID.
5. For the CI/CD method, select Tekton with Argo CD to generate GitOps resources.
6. Set the Workflow Namespace to rhdh and the GitOps Namespace to orchestrator-gitops.
7. Enter your Quay.io registry details.

8. Click Review, then click Create.

   
9. Optional: Enable persistence and provide database connection details if the workflow requires a database schema.

Procedure

1. Open the generated GitOps repository.

2. Clone the repository and navigate to the bootstrap directory:

   $ git clone https://token:<PAT>@${{ values.gitHost }}/${{ values.orgName }}/${{ values.repoName }}.git
   cd <repo_name>/bootstrap

   Note

   If you are not authenticated, you must use a personal access token (PAT) in the clone URL. Make sure the PAT has repository access permissions.

3. Open ${{values.workflowId}}-argocd-repo.yaml and replace the REPLACE_SSH_PRIVATE_KEY string with your SSH private key.

4. Apply the manifests to the cluster:

   $ kubectl apply -f .

   Applying these manifests triggers the following automated sequence:

   1. CI Pipeline (Tekton): Builds the workflow image and pushes it to your Quay.io registry.
   2. CD Pipeline (Argo CD): Deploys the serverless workflow manifests to the cluster.

Procedure

1. For CI:

   1. In the RHDH Catalog, select your source code repository component (for example, onboardings).
   2. Click the CI tab and verify that the pipeline run status is Succeeded.
   3. If the pipeline status does not appear in the Red Hat Developer Hub console, verify the CI status directly in your Git provider (GitHub or GitLab).
   4. If the pipeline fails, click the run name to view the logs and identify build errors.

2. For CD:

   1. Open the GitOps Resources Repository component in the Catalog (for example, onboarding-gitops).

   2. Click the CD tab and make sure the Kubernetes resources are synced and healthy. This confirms that ArgoCD deployed the workflow to the cluster.

      

1. Visibility issues

   Missing Orchestrator plugin

   If Orchestrator features do not appear in RHDH, make sure you have updated the RHDH Helm chart with the required plugins.

   Software templates not appearing

   Make sure the orchestrator-software-templates chart is installed and the orchestrator-auth-secret exists in the correct namespace.

2. Pipeline failure (CI)

   GitHub or GitLab actions failure

   The GitOps automation includes a GitHub Action or GitLab CI step that creates a PipelineRun manifest from a PipelineRun template. Examine the failed GitHub or GitLab actions logs. Failures often occur due to invalid Git credentials or misconfigured runner permissions. You can also create the PipelineRun file manually to bypass automation issues.

   Build or push issues

   Check the CI tab in the RHDH Catalog.

   If RHDH does not display the status, use the OpenShift Container Platform console to monitor pipeline instances and triggered jobs. Navigate to Pipelines > PipelineRuns for detailed logs.

   If the Tekton pipeline fails during the build or push stages:

   • Verify that your Quay.io robot account has Write permissions.
   • Ensure the docker-registry-credentials secret exists in the rhdh namespace.

3. Resource visibility and Sync issues (CD)

   Pipeline succeeds but workflows are missing

   If the CI pipeline succeeds but the workflow does not appear in the CD tab:

   • Make sure the target namespace is labeled for Argo CD:

     $ oc label ns <target_namespace> rhdh.redhat.com/argocd-namespace=true

   • Make sure the ArgoCD ServiceAccount has the required permissions to manage resources in the rhdh namespace.

   Argo CD sync failure

   If resources appear but remain in an OutOfSync state, click Refresh in the Argo CD UI or verify that the AppProject exists in the orchestrator-gitops namespace.

   PostgreSQL authentication failures in Argo CD

   If the Orchestrator fails to connect to the PostgreSQL database when you deploy by using Argo CD, the failure is often due to a mismatch in password generation.

   The Orchestrator Helm chart uses the Helm lookup function to check for an existing PostgreSQL secret. Because Argo CD uses helm template to render manifests, it cannot query the live cluster. Consequently, the chart generates a new, random password instead of retrieving the existing one, resulting in an authentication failure.

   To resolve this failure, you must complete the following steps:

   1. Create the database secret manually with the correct credentials:

      $ kubectl create secret generic <backstage-postgresql-svcbind-postgres> --from-literal=password=<your_password>

   2. Update your Helm configuration (for example, in values.yaml) to disable automatic service binding generation:

      upstream:
        postgresql:
          serviceBindings:
            enabled: false
          auth:
            username: postgres
            database: backstage
            existingSecret: backstage-postgresql-svcbind-postgres
            secretKeys:
              adminPasswordKey: password
              userPasswordKey: password

   3. Sync the application in Argo CD to apply the changes.

Procedure

1. If the workflow operation fails, examine the container log of the specific workflow instance to determine the cause by running the following command:

   $ oc logs my-workflow-xy73lj

2. If the workflow fails to reach an HTTPS endpoint, check the pod log for an SSL certificate verification failure. This occurs if the target endpoint uses a Certificate Authority (CA) that the workflow cannot verify. The resulting error resembles the following:

   sun.security.provider.certpath.SunCertPathBuilderException - unable to find valid certification path to requested target

3. To resolve the SSL certificate error, load the additional CA certificate into the running workflow container.

Procedure

1. Identify required namespaces.
2. Retrieve the namespace value where RHDH is running using oc get backstage -A.

3. Identify the SonataFlow Services Namespace by checking for either a sonataflowclusterplatform or sonataflowplatform instance.

   Note

   By default, the SonataFlow namespace must be the same as the RHDH namespace.

4. If the workflow is deployed to a namespace outside the core SonataFlow services, configure network policies to permit the necessary inter-namespace traffic.

   # Example NetworkPolicy configuration to ingress traffic into the workflow namespace
   apiVersion: networking.k8s.io/v1
   kind: NetworkPolicy
   metadata:
     name: {{ .Release.Name }}-allow-infra-ns-to-workflow-ns
     # SonataFlow and Workflows are using the RHDH target namespace.
     namespace: {{ .Release.Namespace | quote }}
   spec:
     podSelector: {}
     ingress:
       - from:
         - namespaceSelector:
             matchLabels:
               # Allow knative events to be delivered to workflows.
               kubernetes.io/metadata.name: knative-eventing
         - namespaceSelector:
             matchLabels:
               # Allow auxiliary knative function for workflow (such as m2k-save-transformation)
               kubernetes.io/metadata.name: knative-serving
         - namespaceSelector:
             matchLabels:
               # Allow communication between the serverless logic operator and the workflow namespace.
               kubernetes.io/metadata.name: openshift-serverless-logic

5. Add SonataFlowClusterPlatform Custom Resource as shown in the following configuration:

   oc create -f - <<EOF
   apiVersion: sonataflow.org/v1alpha08
   kind: SonataFlowClusterPlatform
   metadata:
     name: cluster-platform
   spec:
     platformRef:
       name: sonataflow-platform
       namespace: $RHDH_NAMESPACE

6. To allow communication between RHDH namespace and the workflow namespace, create the following network policies:

   1. Allow RHDH services to accept traffic from workflows. Create an additional network policy within the RHDH instance namespace as shown in the following configuration::

      oc create -f - <<EOF
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-external-workflows-to-rhdh
        # Namespace where network policies are deployed
        namespace: $RHDH_NAMESPACE
      spec:
        podSelector: {}
        ingress:
          - from:
            - namespaceSelector:
                matchLabels:
                  # Allow SonataFlow services to communicate with new/additional workflow namespace.
                  kubernetes.io/metadata.name: $ADDITIONAL_WORKFLOW_NAMESPACE

   2. Allow traffic from RHDH, SonataFlow and Knative. Create a network policy within the additional workflow namespace as shown in the following configuration:

      oc create -f - <<EOF
      apiVersion: networking.k8s.io/v1
      kind: NetworkPolicy
      metadata:
        name: allow-rhdh-and-knative-to-workflows
        namespace: $ADDITIONAL_WORKFLOW_NAMESPACE
      spec:
        podSelector: {}
        ingress:
          - from:
            - namespaceSelector:
                matchLabels:
                  # Allows traffic from pods in the RHDH namespace.
                  kubernetes.io/metadata.name: $RHDH_NAMESPACE
            - namespaceSelector:
                matchLabels:
                  # Allows traffic from pods in the Knative Eventing namespace.
                  kubernetes.io/metadata.name: knative-eventing
            - namespaceSelector:
                matchLabels:
                  # Allows traffic from pods in the Knative Serving namespace.
                  kubernetes.io/metadata.name: knative-serving

7. (Optional) Create an allow-intra-namespace policy in the workflow namespace to enable unrestricted communication among all pods within that namespace.

8. If workflow persistence is required, perform the following configuration steps:

   1. Create a dedicated PostgreSQL Secret containing database credentials within the workflow namespace as shown in the following configuration:

      oc get secret sonataflow-psql-postgresql -n <your_namespace> -o yaml > secret.yaml
      sed -i '/namespace: <your_namespace>/d' secret.yaml
      oc apply -f secret.yaml -n $ADDITIONAL_NAMESPACE

   2. Configure the workflow serviceRef property to correctly reference the PostgreSQL service namespace as shown in the following configuration:

      apiVersion: sonataflow.org/v1alpha08
      kind: SonataFlow
        ...
      spec:
        ...
        persistence:
          postgresql:
            secretRef:
              name: sonataflow-psql-postgresql
              passwordKey: postgres-password
              userKey: postgres-username
            serviceRef:
              databaseName: sonataflow
              databaseSchema: greeting
              name: sonataflow-psql-postgresql
              namespace: $POSTGRESQL_NAMESPACE
              port: 5432

      namespace

      Enter the namespace where the PostgreSQL server is deployed.

9. If the sonataflow-platform-data-index-service cannot connect to the PostgreSQL database on startup, perform the following diagnostic checks:

   1. Verify that the PostgreSQL Pod has fully transitioned to a running and operational status. Allow additional time for database initialization before expecting related service pods (DataIndex, JobService) to establish a connection.
   2. If the PostgreSQL Server operates in a dedicated namespace (for example, outside RHDH), verify that network policies are configured to allow ingress traffic from the SonataFlow services namespace. Network policies might prevent the Data Index and Job Service pods from connecting to the database.

Procedure

1. Verify if the workflow uses GitOps profile. The RHDH Orchestrator UI displays only the workflows that use this profile. Make sure the workflow definition and the SonataFlow manifests use the GitOps profile.

2. Verify that the workflow pod has started and is ready. The readiness of a workflow pod depends on its successful registration with the Data Index. When a workflow initializes, it performs the following actions:

   1. It attempts to create its schema in the database (if persistence is active).

   2. It attempts to register itself to the Data Index. The workflow pod remains in an unready state until it successfully registers to the Data Index.

      Check the workflow deployment for additional status and error messages that might be unavailable in the pod log.

3. Check if the workflow pod can reach the Data Index service. Connect to the workflows pod and send the following GraphQL request to the Data Index:

   curl -g -k  -X POST  -H "Content-Type: application/json" \
                       -d '{"query":"query{ ProcessDefinitions  { id, serviceUrl, endpoint } }"}' \
                       http://sonataflow-platform-data-index-service.<your_namespace>/graphql

   Use the Data Index service and namespace as defined in your environment. By default, this is the same namespace where RHDH is installed. If your SonataFlow resources are installed in a separate namespace, use <your_namespace>. Check if the RHDH pod can reach the workflow service by running the following command:

   curl http://<workflow_service>.<workflow_namespace>/management/processes

4. Connect to the RHDH pod. Verify its connection to the Data Index service and inspect the RHDH pod logs for messages from the Orchestrator plugin.

   To inspect the logs, identify the RHDH pod and run the following oc logs command:

   oc get pods -n <your_namespace>
   oc logs <rhdh_pod_name> -n <your_namespace>

   You must find messages indicating it is attempting to fetch workflow information from the Data Index, similar to the following:

   {"level":"\u001b[32minfo\u001b[39m","message":"fetchWorkflowInfos() called: http://sonataflow-platform-data-index-service.<your_namespace>","plugin":"orchestrator","service":"backstage","span_id":"fca4ab29f0a7aef9","timestamp":"2025-08-04 17:58:26","trace_flags":"01","trace_id":"5408d4b06373ff8fb34769083ef771dd"}

   Notice the "plugin":"orchestrator" that can help to filter the messages.

5. Make sure the Data Index properties are set in the -managed-props ConfigMap of the workflow as shown in the following configuration:

   kogito.data-index.health-enabled = true
   kogito.data-index.url = http://sonataflow-platform-data-index-service.<your_namespace>
   ...
   mp.messaging.outgoing.kogito-processdefinitions-events.url = http://sonataflow-platform-data-index-service.<your_namespace>/definitions
   mp.messaging.outgoing.kogito-processinstances-events.url = http://sonataflow-platform-data-index-service.<your_namespace>/processes

   Note

   The -managed-props ConfigMap is located in the same namespace as the workflow and is generated by the OpenShift Serverless Logic (OSL) Operator.

   These properties, along with similar settings for the Job Services, indicate that the (OSL) Operator successfully registered the Data Index service.

6. Confirm that the workflow is registered in the Data Index database. Connect to the database used by the Data Index and run the following command from the PSQL instance pod:

   PGPASSWORD=<psql password> psql -h localhost -p 5432 -U < user> -d sonataflow

   Replace <psql password> and <user> with your database credentials.

   Run the following SQL commands to query the registered workflow definitions:

   sonataflow=# SET search_path TO "sonataflow-platform-data-index-service";
   sonataflow=# select id, name from definitions;

   You must see your workflows listed in the query results.

7. Make sure you have enabled Data Index and Job Service in the SonataFlowPlatform custom resource (CR) as shown in the following configuration:

   services:
       dataIndex:
         enabled: true
       jobService:
         enabled: true

   If you fail to enable the Data Index and the Job Services in the SonataFlowPlatform custom resource (CR), the Orchestrator plugin fails to fetch the available workflows.

   Note

   You can also manually edit the SonataFlowPlatform CR instance to trigger the re-creation of workflow-related manifests.

8. Configure role-based access control (RBAC) permissions to ensure workflows are visible in the Orchestrator UI.

   Note

   When the RBAC plugin is enabled, the Orchestrator UI does not display workflows by default. You must explicitly grant read permissions.

   1. Check your RHDH app-config.yaml file to confirm if the RBAC plugin is enabled.
   2. Confirm your user or role has the orchestrator.workflow permission with the read action.

   3. If this permission is missing, add the following to your RBAC CSV (rbac-policy.csv) file:

      p, role:default/workflowUser, orchestrator.workflow, read, allow

   4. Make sure policyFileReload is set to true in your configuration, or restart the RHDH application:

      permission:
        enabled: true
        rbac:
          policyFileReload: true

Procedure

1. Download the plugin-infra.sh script as shown in the following example:

   $ curl -sSLO https://raw.githubusercontent.com/redhat-developer/rhdh-operator/refs/heads/release-1.9/config/profile/rhdh/plugin-infra/plugin-infra.sh # Specify the Red Hat Developer Hub version in the URL or use main

2. Run the script:

   $ ./plugin-infra.sh