Red Hat Developer Hub 1.10

Integrating Red Hat Developer Hub with your Git provider

Configuring integration to Git providers in Red Hat Developer Hub

Red Hat Customer Content Services

Abstract

As a Red Hat Developer Hub platform engineer, you can integrate Developer Hub with GitHub and GitLab Git providers.

1. Integrate with GitHub
- 1.1. Enable GitHub repository discovery
- 1.2. Bulk importing from GitHub
2. Integrate with GitLab
- 2.1. Bulk importing from GitLab

Connect Red Hat Developer Hub to GitHub or GitLab to discover repositories automatically and import multiple projects efficiently using bulk import capabilities.

1. Integrate with GitHub

Connect Red Hat Developer Hub to GitHub to discover repositories automatically and import multiple projects efficiently using bulk import capabilities.

1.1. Enable GitHub repository discovery

Configure automatic GitHub repository discovery to import repositories containing catalog-info.yaml files into the Red Hat Developer Hub catalog without manual registration.

Prerequisites

You have added a custom Developer Hub application configuration.
You have sufficient permissions in GitHub to create and manage a GitHub App.
To allow users to access GitHub templates or plugins that require GitHub authentication, you have configured GitHub either as an auxiliary authentication provider or as your main authentication provider.

Procedure

Create a GitHub App to allow Developer Hub to access the GitHub API. Opt for a GitHub App instead of an OAuth app to use fine-grained permissions, gain more control over which repositories the application can access, and use short-lived tokens.
1. Register a GitHub App with the following configuration:
  GitHub App name
  Enter a unique name identifying your GitHub App, such as integrating-with-rhdh-<GUID>.
  Homepage URL
  Enter your Developer Hub URL: https://<my_developer_hub_domain>.
  Authorization callback URL
  Enter your Developer Hub authentication backend URL: https://<my_developer_hub_domain>/api/auth/github/handler/frame.
  Webhook
  Clear "Active", as this is not needed for authentication and catalog providers.
  App permissions
  Select permissions to define the level of access for the app. Adapt permissions to your needs:
  Reading software components
  Contents
  Read-only
  Commit statuses
  Read-only
  Reading organization data
  Members
  Read-only
  Publishing software templates
  Set permissions if you intend to use the same GitHub App for software templates.
  Administration
  Read & write (for creating repositories)
  Contents
  Read & write
  Metadata
  Read-only
  Pull requests
  Read & write
  Issues
  Read & write
  Workflows
  Read & write (if templates include GitHub workflows)
  Variables
  Read & write (if templates include GitHub Action Repository Variables)
  Secrets
  Read & write (if templates include GitHub Action Repository Secrets)
  Environments
  Read & write (if templates include GitHub Environments)
  Organization permissions
  Members
  Read-only
  Where can this GitHub App be installed?
  Select Only on this account.
2. In the General → Clients secrets section, click Generate a new client secret.
3. In the General → Private keys section, click Generate a private key.
4. In the Install App tab, choose an account to install your GitHub App on.
5. Save the following values for the next step:
App ID
Client ID
Client secret
Private key
Add your GitHub credentials to Developer Hub by adding the following key/value pairs to your Developer Hub secrets. You can use these secrets in the Developer Hub configuration files by using the corresponding environment variable name for each secret.
GITHUB_APP_APP_ID
Enter the saved App ID.
GITHUB_APP_CLIENT_ID_INTEGRATION
Enter the saved Client ID.
GITHUB_APP_CLIENT_SECRET_INTEGRATION
Enter the saved Client Secret.
GITHUB_APP_PRIVATE_KEY
Enter the saved Private key.
GITHUB_URL
Enter the GitHub host domain: https://github.com.
GITHUB_ORG
Enter your GitHub organization name, such as <your_github_organization_name>.
Enable the plugin-catalog-backend-module-github plugin in your dynamic-plugins.yaml file.
This plugin discovers catalog entities by scanning repositories within a GitHub organization for catalog-info.yaml files. It provides an automated alternative to manually registering components via catalog.locations. When a repository contains a catalog-info.yaml file, the entity is ingested into the catalog as a component.
dynamic-plugins.yaml file fragment
```
plugins:
  - package: './dynamic-plugins/dist/backstage-plugin-catalog-backend-module-github'
    disabled: false
```

Configure the GitHub integration, by adding the catalog.providers.github and the integrations.github sections to your custom Developer Hub app-config.yaml configuration file:

app-config.yaml file fragment with mandatory fields to enable GitHub integration

catalog:
  providers:
    github:
      providerId:
        organization: "${GITHUB_ORG}"
        schedule:
          frequency:
            minutes: 30
          initialDelay:
            seconds: 15
          timeout:
            minutes: 15
integrations:
  github:
    - host: ${GITHUB_URL}
      apps:
        - appId: ${GITHUB_APP_APP_ID}
          clientId: ${GITHUB_APP_CLIENT_ID_INTEGRATION}
          clientSecret: ${GITHUB_APP_CLIENT_SECRET_INTEGRATION}
          privateKey: |
            ${GITHUB_APP_PRIVATE_KEY}

1.2. Bulk importing from GitHub

Automate onboarding of GitHub repositories to the Red Hat Developer Hub catalog, and monitor import status by using bulk import capabilities.

Important

These features are for Technology Preview only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs), might not be functionally complete, and Red Hat does not recommend using them for production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information on Red Hat Technology Preview features, see Technology Preview Features Scope.

1.2.1. Repository visibility in Bulk Import

View and bulk-import only the repositories you have permission to access and have not yet added to your catalog.

1.2.1.1. User-scoped repository access

When you use Bulk Import, Red Hat Developer Hub retrieves the list of available repositories using your authenticated user credentials through OAuth. Repository and organization listing API calls use your OAuth token for each configured source code management (SCM) host, ensuring that results match your personal access permissions rather than server-wide integration credentials.

This approach provides several benefits:

User-scoped access: You see only repositories you can access in GitHub or GitLab, matching your experience in those platforms.
Enhanced security: Import operations use your personal access token, maintaining audit trails tied to your user account.
Permission alignment: Repository visibility respects your organizational access policies and role-based permissions.

This differs from catalog discovery providers, which use service account credentials to automatically discover and import repositories containing catalog-info.yaml files across an entire organization.

1.2.1.2. Technical implementation

Bulk Import requires user authentication for all repository and organization listing operations. The following API endpoints require a valid OAuth token:

GET /repositories - List all accessible repositories
GET /organizations/{org}/repositories - List repositories in a specific organization

These endpoints use the X-SCM-Tokens header to pass user OAuth tokens for GitHub and GitLab hosts. There is no fallback to server-wide integration credentials (GitHub App, PAT, or GitLab token) for these listing operations. Requests without valid user OAuth tokens are rejected with HTTP 401 Unauthorized.

Important

Deployments that previously relied on integration-only listing must configure SCM authentication providers for user authentication. See the Bulk Import plugin documentation for configuration details.

1.2.1.3. Automatic filtering of imported repositories

The Bulk Import interface automatically excludes repositories that are already present in your Developer Hub catalog, showing only repositories that have not yet been imported. This filtering helps you:

Avoid duplicate catalog entries.
Focus on new repositories that need onboarding.
Identify which repositories remain to be imported from your Git provider.

Note

If a repository is removed from the catalog, it will reappear in the Bulk Import repository list and can be imported again.

1.2.1.4. Prerequisites for repository visibility

User authentication with OAuth is a mandatory requirement for the Bulk Import feature. To view and import repositories through Bulk Import, you must:

Configure GitHub or GitLab authentication using one of the following approaches:
- As your primary authentication provider for Developer Hub.
- As an auxiliary authentication provider if you use another primary provider, for example, OIDC or Red Hat build of Keycloak.
Configure SCM authentication providers with OAuth support for your Git hosts
Have the bulk.import permission configured in RBAC policies.
Maintain an active OAuth session with your Git provider.

Warning

Repository and organization listing operations require user OAuth tokens. There is no fallback to server-wide integration credentials. Deployments without user authentication configured will receive HTTP 401 errors when accessing Bulk Import.

1.2.1.5. Troubleshooting repository visibility

If you cannot see expected repositories in the Bulk Import interface, verify the following:

User access: Your user account has access to those repositories in GitHub or GitLab
Catalog status: The repositories are not already imported into the Developer Hub catalog
Session validity: Your Developer Hub OAuth session has not expired
Authentication configuration: GitHub or GitLab authentication provider is correctly configured
ScmAuthApi registration: The ScmAuthApi is properly registered for your SCM hosts

If you receive HTTP 401 Unauthorized errors when accessing the Bulk Import page, this indicates that user OAuth tokens are not available. Verify that:

You have authenticated to Developer Hub using GitHub or GitLab as your authentication provider
Your authentication provider includes OAuth scopes required for repository access
The SCM authentication provider is configured and registered in your Developer Hub deployment

1.2.2. Enable and authorize Bulk Import capabilities in Red Hat Developer Hub

Enable Bulk Import plugins and configure RBAC permissions to allow users to import multiple GitHub repositories and GitLab projects into the catalog.

Prerequisites

You have configured user authentication with GitHub or GitLab as your authentication provider. This is a mandatory requirement for Bulk Import, as repository listings use user OAuth tokens.
For GitHub only: You have enabled GitHub repository discovery.

Important

Bulk Import requires user OAuth tokens for all repository and organization listing operations. Deployments without user authentication configured will receive HTTP 401 Unauthorized errors when accessing the Bulk Import feature.

Procedure

The Bulk Import plugins are installed but disabled by default. To enable the ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import-backend-dynamic and ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import plugins, edit your dynamic-plugins.yaml with the following content:
```
plugins:
  - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import-backend-dynamic
    disabled: false
  - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import
    disabled: false
```
See Installing and viewing plugins in Red Hat Developer Hub.
Configure the required bulk.import RBAC permission for the users who are not administrators as shown in the following code:
rbac-policy.csv fragment
```
p, role:default/bulk-import, bulk.import, use, allow
g, user:default/<your_user>, role:default/bulk-import
```
Note that only Developer Hub administrators or users with the bulk.import permission can use the Bulk Import feature. See Permission policies in Red Hat Developer Hub.

Verification

The sidebar displays a Bulk Import option.
The Bulk Import page shows a list of added GitHub repositories and GitLab projects.

1.2.3. Import multiple GitHub repositories

Select and import multiple GitHub repositories that you can access into the Red Hat Developer Hub catalog, automatically creating pull requests with required catalog-info.yaml files.

Prerequisites

Note

The Bulk Import feature displays only repositories that your GitHub user account can access and that are not already imported into the Developer Hub catalog. Repository visibility is determined by your GitHub permissions, not the Developer Hub GitHub App configuration.

Procedure

Click Bulk Import in the Developer Hub left sidebar.
If your RHDH instance has multiple source control tools configured, select GitHub from the Source control tool list.
The interface displays GitHub repositories that your authenticated user account can access, excluding repositories already present in the catalog.
Select the repositories to import, and click Add.
Developer Hub creates a pull request in each selected repository to add the required catalog-info.yaml file using your GitHub credentials.
For each repository to import, click PR to review and merge the changes in GitHub.

Verification

Click Bulk Import in the Developer Hub left sidebar.
Verify that each imported GitHub repository in the Selected repositories list has the status Waiting for approval or Imported.
For each Waiting for approval repository, click the pull request link to review and merge the catalog-info.yaml file in the corresponding repository.
The pull request is created using your GitHub user account, ensuring proper attribution in the repository history.

1.2.4. Monitor Bulk Import actions using audit logs

Review Bulk Import backend plugin audit log events to monitor repository import operations, track API requests, and troubleshoot import issues.

Procedure

Access your Developer Hub backend logs where audit log events are recorded.
Review the following Bulk Import audit log events to monitor repository operations:
BulkImportUnknownEndpoint
Tracks requests to unknown endpoints.
BulkImportPing
Tracks GET requests to the /ping endpoint, which allows us to make sure the bulk import backend is up and running.
BulkImportFindAllOrganizations
Tracks GET requests to the /organizations endpoint, which returns the list of organizations accessible from all configured GitHub Integrations.
BulkImportFindRepositoriesByOrganization
Tracks GET requests to the /organizations/:orgName/repositories endpoint, which returns the list of repositories for the specified organization (accessible from any of the configured GitHub Integrations).
BulkImportFindAllRepositories
Tracks GET requests to the /repositories endpoint, which returns the list of repositories accessible from all configured GitHub Integrations.
BulkImportFindAllImports
Tracks GET requests to the /imports endpoint, which returns the list of existing import jobs along with their statuses.
BulkImportCreateImportJobs
Tracks POST requests to the /imports endpoint, which allows to submit requests to bulk-import one or many repositories into the Developer Hub catalog, by eventually creating import pull requests in the target repositories.
BulkImportFindImportStatusByRepo
Tracks GET requests to the /import/by-repo endpoint, which fetches details about the import job for the specified repository.
BulkImportDeleteImportByRepo
Tracks DELETE requests to the /import/by-repo endpoint, which deletes any existing import job for the specified repository, by closing any open import pull request that could have been created.
Example audit log output:
```
{
  "actor": {
    "actorId": "user:default/myuser",
    "hostname": "localhost",
    "ip": "::1",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
  },
  "eventName": "BulkImportFindAllOrganizations",
  "isAuditLog": true,
  "level": "info",
  "message": "'get /organizations' endpoint hit by user:default/myuser",
  "meta": {},
  "plugin": "bulk-import",
  "request": {
    "body": {},
    "method": "GET",
    "params": {},
    "query": {
      "pagePerIntegration": "1",
      "sizePerIntegration": "5"
    },
    "url": "/api/bulk-import/organizations?pagePerIntegration=1&sizePerIntegration=5"
  },
  "response": {
    "status": 200
  },
  "service": "backstage",
  "stage": "completion",
  "status": "succeeded",
  "timestamp": "2024-08-26 16:41:02"
}
```

1.2.5. Input parameters for Bulk Import Scaffolder template

Define Scaffolder template parameters such as repository URL, name, organization, and branch details to customize bulk import automation workflows for your repositories.

As an administrator, you can use the Bulk Import plugin to run a Scaffolder template task with specified parameters, which you must define within the template.

The Bulk Import plugin analyzes Git repository information and provides the following parameters for the Scaffolder template task:

repoUrl

Normalized repository URL in the following format:

  ${gitProviderHost}?owner=${owner}&repo=${repository-name}

name

The repository name.

organization

The repository owner, which can be a user nickname or organization name.

branchName

The proposed repository branch. By default, the proposed repository branch is bulk-import-catalog-entity.

targetBranchName

The default branch of the Git repository.

gitProviderHost

The Git provider host parsed from the repository URL. You can use this parameter to write Git-provider-agnostic templates.

Example of a Scaffolder template:

parameters:
  - title: Repository details
    required:
      - repoUrl
      - branchName
      - targetBranchName
      - name
      - organization
    properties:
      repoUrl:
        type: string
        title: Repository URL ({product-short} format)
        description: github.com?owner=Org&repo=repoName
      organization:
        type: string
        title: Owner of the repository
      name:
        type: string
        title: Name of the repository
      branchName:
        type: string
        title: Branch to add the catalog entity to
      targetBranchName:
        type: string
        title: Branch to target the PR/MR to
      gitProviderHost:
        type: string
        title: Git provider host

1.2.6. Set up a custom Scaffolder workflow for Bulk Import

Create custom Scaffolder templates aligned with your organization’s repository conventions to automate bulk import tasks such as entity imports, pull request creation, and webhook integration.

As an administrator, you can create a custom Scaffolder template inline with the repository conventions of your organization and add the template into the Red Hat Developer Hub catalog for use by the Bulk Import plugin on many selected repositories.

You can define various custom tasks, including, but not limited to the following:

Importing existing catalog entities from a repository
Creating pull requests for cleanup
Calling webhooks for external system integration

Prerequisites

You created a custom Scaffolder template for the Bulk Import plugin.
You have run your RHDH instance with the following environment variable enabled to allow the use of the Scaffolder functionality:
```
export NODE_OPTIONS=--no-node-snapshot
```

Procedure

Configure your app-config.yaml configuration to instruct the Bulk Import plugin to use your custom template as shown in the following example:
```
bulkImport:
  importTemplate: <your_template_entity_reference_or_template_name>
  importAPI: `open-pull-requests` | `scaffolder`;
```
where:
importTemplate:
Enter your Scaffolder template entity reference.
importAPI
Set the API to 'scaffolder' to trigger the defined workflow for high-fidelity automation. This field defines the import workflow and currently supports two following options:
open-pull-requests
This is the default import workflow, which includes the logic for creating pull requests for every selected repository.
scaffolder
This workflow uses an import scenario defined in the Scaffolder template to create import jobs. Select this option to use the custom import scenario defined in your Scaffolder template.
Optional: You can direct the Bulk Import plugin to hand off the entire list of selected repositories to a custom Orchestrator workflow.
Important
The Scaffolder template must be generic and not specific to a single repository if you want your custom Scaffolder template to run successfully for every repository in the bulk list.

Verification

The Bulk Import plugin runs the custom Scaffolder template for the list of repositories using the /task-imports API endpoint.

1.2.7. Run Orchestrator workflows for bulk imports

Configure Bulk Import to use Orchestrator workflows for advanced bulk operations across multiple repositories, enabling automated pull request creation and configuration publishing at scale.

As a platform engineer, you can configure the Bulk Import plugin to run Orchestrator workflows for bulk import operations. This mode uses the Orchestrator engine to provide advanced capabilities, such as creating pull requests or publishing configurations across multiple repositories.

Prerequisites

You have installed and configured the Orchestrator plugin in your Developer Hub instance.
You have registered a generic custom workflow (for example, universal-pr) in the Orchestrator plugin.
You have added a custom Developer Hub application configuration.

Procedure

Configure the Bulk Import plugin by editing your app-config.yaml file to enable Orchestrator mode.
```
bulkImport:
  orchestratorWorkflow: your_workflow_id
  importAPI: 'orchestrator'
```
where:
orchestratorWorkflow
The ID of the workflow to run for each repository.
importAPI
The execution mode for the workflow. Enter orchestrator to enable workflow execution.
Verify that the Orchestrator workflow receives the following input:
```
{
  "inputData": {
    "owner": "redhat-developer",
    "repo": "rhdh-plugins",
    "baseBranch": "main",
    "targetBranch": "bulk-import-orchestrator"
  },
  "authTokens": [
    {
      "token": "<github_token>",
      "provider": "github"
    }
  ]
}
```
where:
owner
Specifies the repository owner (organization or user name).
repo
Specifies the repository name.
baseBranch
Specifies the default branch of the Git repository (for example, main).
targetBranch
Specifies the target branch for the import operation. By default, this is set to bulk-import-orchestrator.
authTokens
Specifies the authentication tokens for the Git provider:
For GitHub: { token: <github_token>, provider: github }
For GitLab: { token: <gitlab_token>, provider: gitlab }
Navigate to the Bulk Import page in the sidebar and complete the following steps:
1. Select your Git provider (for example, GitHub or GitLab).
2. Select the projects you want to import.
Click import to run the workflow.

Verification

Locate your repository and confirm status is COMPLETED.

1.2.8. Data handoff and custom workflow design

Design Scaffolder templates to receive repository data as parameters and automate repository-specific tasks when using Scaffolder mode for bulk imports.

When you configure the Bulk Import plugin by setting the importAPI field to scaffolder, the Bulk Import Backend passes all necessary context directly to the Scaffolder API.

As an administrator, you can define the Scaffolder template workflow and structure the workflow to do the following:

Define template parameters to consume input: Structure the Scaffolder template to receive the repository data as template parameters for the current workflow run. The template must be generic, and not specific to a single repository, so that it can successfully run for every repository in the bulk list.
Automate processing for each repository: Implement the custom logic needed for a single repository within the template. The Orchestrator iterates through the repository list, launching the template once for each repository and passes only the data for that single repository to the template run. This allows you to automate tasks such as creating the catalog-info.yaml, running compliance checks, or registering the entity with the catalog.

2. Integrate with GitLab

Connect Red Hat Developer Hub to GitLab and import multiple projects efficiently using bulk import capabilities.

2.1. Bulk importing from GitLab

Automate onboarding of GitLab projects to the Red Hat Developer Hub catalog, and monitor import status by using bulk import capabilities.

Important

For more information on Red Hat Technology Preview features, see Technology Preview Features Scope.

2.1.1. Repository visibility in Bulk Import

View and bulk-import only the repositories you have permission to access and have not yet added to your catalog.

2.1.1.1. User-scoped repository access

This approach provides several benefits:

User-scoped access: You see only repositories you can access in GitHub or GitLab, matching your experience in those platforms.
Enhanced security: Import operations use your personal access token, maintaining audit trails tied to your user account.
Permission alignment: Repository visibility respects your organizational access policies and role-based permissions.

2.1.1.2. Technical implementation

Bulk Import requires user authentication for all repository and organization listing operations. The following API endpoints require a valid OAuth token:

GET /repositories - List all accessible repositories
GET /organizations/{org}/repositories - List repositories in a specific organization

Important

Deployments that previously relied on integration-only listing must configure SCM authentication providers for user authentication. See the Bulk Import plugin documentation for configuration details.

2.1.1.3. Automatic filtering of imported repositories

Avoid duplicate catalog entries.
Focus on new repositories that need onboarding.
Identify which repositories remain to be imported from your Git provider.

Note

If a repository is removed from the catalog, it will reappear in the Bulk Import repository list and can be imported again.

2.1.1.4. Prerequisites for repository visibility

User authentication with OAuth is a mandatory requirement for the Bulk Import feature. To view and import repositories through Bulk Import, you must:

Configure GitHub or GitLab authentication using one of the following approaches:
- As your primary authentication provider for Developer Hub.
- As an auxiliary authentication provider if you use another primary provider, for example, OIDC or Red Hat build of Keycloak.
Configure SCM authentication providers with OAuth support for your Git hosts
Have the bulk.import permission configured in RBAC policies.
Maintain an active OAuth session with your Git provider.

Warning

2.1.1.5. Troubleshooting repository visibility

If you cannot see expected repositories in the Bulk Import interface, verify the following:

User access: Your user account has access to those repositories in GitHub or GitLab
Catalog status: The repositories are not already imported into the Developer Hub catalog
Session validity: Your Developer Hub OAuth session has not expired
Authentication configuration: GitHub or GitLab authentication provider is correctly configured
ScmAuthApi registration: The ScmAuthApi is properly registered for your SCM hosts

If you receive HTTP 401 Unauthorized errors when accessing the Bulk Import page, this indicates that user OAuth tokens are not available. Verify that:

You have authenticated to Developer Hub using GitHub or GitLab as your authentication provider
Your authentication provider includes OAuth scopes required for repository access
The SCM authentication provider is configured and registered in your Developer Hub deployment

2.1.2. Enable and authorize Bulk Import capabilities in Red Hat Developer Hub

Enable Bulk Import plugins and configure RBAC permissions to allow users to import multiple GitHub repositories and GitLab projects into the catalog.

Prerequisites

You have configured user authentication with GitHub or GitLab as your authentication provider. This is a mandatory requirement for Bulk Import, as repository listings use user OAuth tokens.
For GitHub only: You have enabled GitHub repository discovery.

Important

Procedure

The Bulk Import plugins are installed but disabled by default. To enable the ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import-backend-dynamic and ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import plugins, edit your dynamic-plugins.yaml with the following content:
```
plugins:
  - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import-backend-dynamic
    disabled: false
  - package: ./dynamic-plugins/dist/red-hat-developer-hub-backstage-plugin-bulk-import
    disabled: false
```
See Installing and viewing plugins in Red Hat Developer Hub.
Configure the required bulk.import RBAC permission for the users who are not administrators as shown in the following code:
rbac-policy.csv fragment
```
p, role:default/bulk-import, bulk.import, use, allow
g, user:default/<your_user>, role:default/bulk-import
```
Note that only Developer Hub administrators or users with the bulk.import permission can use the Bulk Import feature. See Permission policies in Red Hat Developer Hub.

Verification

The sidebar displays a Bulk Import option.
The Bulk Import page shows a list of added GitHub repositories and GitLab projects.

2.1.3. Import multiple GitLab repositories

Select and import multiple GitLab projects that you can access into the Red Hat Developer Hub catalog, automatically creating merge requests with required catalog-info.yaml files.

Important

For more information on Red Hat Technology Preview features, see Technology Preview Features Scope.

Prerequisites

You have enabled the Bulk Import feature and configured access permissions.
You have authenticated to Developer Hub using GitLab as your authentication provider.
You have set up a GitLab personal access token (PAT).
You configured the GitLab integration by adding the following section to your RHDH app-config.yaml file:
```
integrations:
  gitlab:
    - host: ${GITLAB_HOST}
      token: ${GITLAB_TOKEN}
```

You enabled the GitLab catalog provider plugin in your dynamic-plugins.yaml file to import GitLab users and groups:

plugins:
  - package: './dynamic-plugins/dist/backstage-plugin-catalog-backend-module-gitlab-org-dynamic'
    disabled: false

Note

The Bulk Import feature displays only projects that your GitLab user account can access and that are not already imported into the Developer Hub catalog. Project visibility is determined by your GitLab permissions, not the Developer Hub GitLab integration token configuration.

Procedure

In the Developer Hub left sidebar, click Bulk Import.
If your RHDH instance has multiple source control tools configured, select GitLab from the Source control tool list.
The interface displays GitLab projects that your authenticated user account can access, excluding projects already present in the catalog.
Select the projects to import, and click Add.
Developer Hub creates a merge request in each selected project to add the required catalog-info.yaml file using your GitLab credentials.
For each project to import, click MR to review and merge the changes in GitLab.

Verification

Click Bulk Import in the Developer Hub left sidebar.
Verify that each imported GitLab project in the Selected projects list has the status Waiting for approval or Imported.
For projects with the Waiting for approval status, click the merge request link to review and add the catalog-info.yaml file to the project repository.
The merge request is created using your GitLab user account, ensuring proper attribution in the project history.

2.1.4. Monitor Bulk Import actions using audit logs

Review Bulk Import backend plugin audit log events to monitor repository import operations, track API requests, and troubleshoot import issues.

Procedure

Access your Developer Hub backend logs where audit log events are recorded.
Review the following Bulk Import audit log events to monitor repository operations:
BulkImportUnknownEndpoint
Tracks requests to unknown endpoints.
BulkImportPing
Tracks GET requests to the /ping endpoint, which allows us to make sure the bulk import backend is up and running.
BulkImportFindAllOrganizations
Tracks GET requests to the /organizations endpoint, which returns the list of organizations accessible from all configured GitHub Integrations.
BulkImportFindRepositoriesByOrganization
Tracks GET requests to the /organizations/:orgName/repositories endpoint, which returns the list of repositories for the specified organization (accessible from any of the configured GitHub Integrations).
BulkImportFindAllRepositories
Tracks GET requests to the /repositories endpoint, which returns the list of repositories accessible from all configured GitHub Integrations.
BulkImportFindAllImports
Tracks GET requests to the /imports endpoint, which returns the list of existing import jobs along with their statuses.
BulkImportCreateImportJobs
Tracks POST requests to the /imports endpoint, which allows to submit requests to bulk-import one or many repositories into the Developer Hub catalog, by eventually creating import pull requests in the target repositories.
BulkImportFindImportStatusByRepo
Tracks GET requests to the /import/by-repo endpoint, which fetches details about the import job for the specified repository.
BulkImportDeleteImportByRepo
Tracks DELETE requests to the /import/by-repo endpoint, which deletes any existing import job for the specified repository, by closing any open import pull request that could have been created.
Example audit log output:
```
{
  "actor": {
    "actorId": "user:default/myuser",
    "hostname": "localhost",
    "ip": "::1",
    "userAgent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36"
  },
  "eventName": "BulkImportFindAllOrganizations",
  "isAuditLog": true,
  "level": "info",
  "message": "'get /organizations' endpoint hit by user:default/myuser",
  "meta": {},
  "plugin": "bulk-import",
  "request": {
    "body": {},
    "method": "GET",
    "params": {},
    "query": {
      "pagePerIntegration": "1",
      "sizePerIntegration": "5"
    },
    "url": "/api/bulk-import/organizations?pagePerIntegration=1&sizePerIntegration=5"
  },
  "response": {
    "status": 200
  },
  "service": "backstage",
  "stage": "completion",
  "status": "succeeded",
  "timestamp": "2024-08-26 16:41:02"
}
```

2.1.5. Input parameters for Bulk Import Scaffolder template

Define Scaffolder template parameters such as repository URL, name, organization, and branch details to customize bulk import automation workflows for your repositories.

As an administrator, you can use the Bulk Import plugin to run a Scaffolder template task with specified parameters, which you must define within the template.

The Bulk Import plugin analyzes Git repository information and provides the following parameters for the Scaffolder template task:

repoUrl

Normalized repository URL in the following format:

  ${gitProviderHost}?owner=${owner}&repo=${repository-name}

name

The repository name.

organization

The repository owner, which can be a user nickname or organization name.

branchName

The proposed repository branch. By default, the proposed repository branch is bulk-import-catalog-entity.

targetBranchName

The default branch of the Git repository.

gitProviderHost

The Git provider host parsed from the repository URL. You can use this parameter to write Git-provider-agnostic templates.

Example of a Scaffolder template:

parameters:
  - title: Repository details
    required:
      - repoUrl
      - branchName
      - targetBranchName
      - name
      - organization
    properties:
      repoUrl:
        type: string
        title: Repository URL ({product-short} format)
        description: github.com?owner=Org&repo=repoName
      organization:
        type: string
        title: Owner of the repository
      name:
        type: string
        title: Name of the repository
      branchName:
        type: string
        title: Branch to add the catalog entity to
      targetBranchName:
        type: string
        title: Branch to target the PR/MR to
      gitProviderHost:
        type: string
        title: Git provider host

2.1.6. Set up a custom Scaffolder workflow for Bulk Import

Create custom Scaffolder templates aligned with your organization’s repository conventions to automate bulk import tasks such as entity imports, pull request creation, and webhook integration.

You can define various custom tasks, including, but not limited to the following:

Importing existing catalog entities from a repository
Creating pull requests for cleanup
Calling webhooks for external system integration

Prerequisites

You created a custom Scaffolder template for the Bulk Import plugin.
You have run your RHDH instance with the following environment variable enabled to allow the use of the Scaffolder functionality:
```
export NODE_OPTIONS=--no-node-snapshot
```

Procedure

Configure your app-config.yaml configuration to instruct the Bulk Import plugin to use your custom template as shown in the following example:
```
bulkImport:
  importTemplate: <your_template_entity_reference_or_template_name>
  importAPI: `open-pull-requests` | `scaffolder`;
```
where:
importTemplate:
Enter your Scaffolder template entity reference.
importAPI
Set the API to 'scaffolder' to trigger the defined workflow for high-fidelity automation. This field defines the import workflow and currently supports two following options:
open-pull-requests
This is the default import workflow, which includes the logic for creating pull requests for every selected repository.
scaffolder
This workflow uses an import scenario defined in the Scaffolder template to create import jobs. Select this option to use the custom import scenario defined in your Scaffolder template.
Optional: You can direct the Bulk Import plugin to hand off the entire list of selected repositories to a custom Orchestrator workflow.
Important
The Scaffolder template must be generic and not specific to a single repository if you want your custom Scaffolder template to run successfully for every repository in the bulk list.

Verification

The Bulk Import plugin runs the custom Scaffolder template for the list of repositories using the /task-imports API endpoint.

2.1.7. Run Orchestrator workflows for bulk imports

Configure Bulk Import to use Orchestrator workflows for advanced bulk operations across multiple repositories, enabling automated pull request creation and configuration publishing at scale.

Prerequisites

You have installed and configured the Orchestrator plugin in your Developer Hub instance.
You have registered a generic custom workflow (for example, universal-pr) in the Orchestrator plugin.
You have added a custom Developer Hub application configuration.

Procedure

Configure the Bulk Import plugin by editing your app-config.yaml file to enable Orchestrator mode.
```
bulkImport:
  orchestratorWorkflow: your_workflow_id
  importAPI: 'orchestrator'
```
where:
orchestratorWorkflow
The ID of the workflow to run for each repository.
importAPI
The execution mode for the workflow. Enter orchestrator to enable workflow execution.
Verify that the Orchestrator workflow receives the following input:
```
{
  "inputData": {
    "owner": "redhat-developer",
    "repo": "rhdh-plugins",
    "baseBranch": "main",
    "targetBranch": "bulk-import-orchestrator"
  },
  "authTokens": [
    {
      "token": "<github_token>",
      "provider": "github"
    }
  ]
}
```
where:
owner
Specifies the repository owner (organization or user name).
repo
Specifies the repository name.
baseBranch
Specifies the default branch of the Git repository (for example, main).
targetBranch
Specifies the target branch for the import operation. By default, this is set to bulk-import-orchestrator.
authTokens
Specifies the authentication tokens for the Git provider:
For GitHub: { token: <github_token>, provider: github }
For GitLab: { token: <gitlab_token>, provider: gitlab }
Navigate to the Bulk Import page in the sidebar and complete the following steps:
1. Select your Git provider (for example, GitHub or GitLab).
2. Select the projects you want to import.
Click import to run the workflow.

Verification

Locate your repository and confirm status is COMPLETED.

2.1.8. Data handoff and custom workflow design

Design Scaffolder templates to receive repository data as parameters and automate repository-specific tasks when using Scaffolder mode for bulk imports.

When you configure the Bulk Import plugin by setting the importAPI field to scaffolder, the Bulk Import Backend passes all necessary context directly to the Scaffolder API.

As an administrator, you can define the Scaffolder template workflow and structure the workflow to do the following:

Define template parameters to consume input: Structure the Scaffolder template to receive the repository data as template parameters for the current workflow run. The template must be generic, and not specific to a single repository, so that it can successfully run for every repository in the bulk list.
Automate processing for each repository: Implement the custom logic needed for a single repository within the template. The Orchestrator iterates through the repository list, launching the template once for each repository and passes only the data for that single repository to the template run. This allows you to automate tasks such as creating the catalog-info.yaml, running compliance checks, or registering the entity with the catalog.

Legal Notice

The text of and illustrations in this document are licensed by Red Hat under a Creative Commons Attribution–Share Alike 3.0 Unported license ("CC-BY-SA"). An explanation of CC-BY-SA is available at http://creativecommons.org/licenses/by-sa/3.0/. In accordance with CC-BY-SA, if you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, Red Hat Enterprise Linux, the Shadowman logo, the Red Hat logo, JBoss, OpenShift, Fedora, the Infinity logo, and RHCE are trademarks of Red Hat, Inc., registered in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

Java® is a registered trademark of Oracle and/or its affiliates.

XFS® is a trademark of Silicon Graphics International Corp. or its subsidiaries in the United States and/or other countries.

MySQL® is a registered trademark of MySQL AB in the United States, the European Union and other countries.

Node.js® is an official trademark of Joyent. Red Hat is not formally related to or endorsed by the official Joyent Node.js open source or commercial project.

The OpenStack® Word Mark and OpenStack logo are either registered trademarks/service marks or trademarks/service marks of the OpenStack Foundation, in the United States and other countries and are used with the OpenStack Foundation's permission. We are not affiliated with, endorsed or sponsored by the OpenStack Foundation, or the OpenStack community.

All other trademarks are the property of their respective owners.