Procedure

1. Configure the logging deployment, including both the CPU and memory limits for each logging component.

   For more information, see Red Hat OpenShift Container Platform - Configuring your Logging deployment.

2. To configure the logging collector, configure the spec.collection stanza in the ClusterLogging custom resource (CR) to use a supported modification to the log collector and collect logs from STDOUT.

   For more information, see Red Hat OpenShift Container Platform - Configuring the logging collector.

3. To configure log forwarding, send logs to specific endpoints inside and outside your OpenShift Container Platform cluster by specifying a combination of outputs and pipelines in a ClusterLogForwarder CR.

   For more information, see Red Hat OpenShift Container Platform - Enabling JSON log forwarding and Red Hat OpenShift Container Platform - Configuring log forwarding.

Procedure

1. Log in to your OpenShift Container Platform cluster.

2. Install the OpenShift Logging Operator in the openshift-logging namespace and switch to the namespace:

   $ oc project openshift-logging

3. Create a serviceAccount named log-collector:

   $ oc create sa log-collector

4. Bind the collect-application-logs role to the serviceAccount:

   $ oc create clusterrolebinding log-collector --clusterrole=collect-application-logs --serviceaccount=openshift-logging:log-collector

5. Generate a hecToken in your Splunk instance.

6. Create a key/value secret in the openshift-logging namespace and verify the secret:

   $ oc -n openshift-logging create secret generic splunk-secret --from-literal=hecToken=<HEC_Token>

   $ oc -n openshift-logging get secret/splunk-secret -o yaml

7. Create a basic `ClusterLogForwarder`resource YAML file as follows:

   apiVersion: logging.openshift.io/v1
   kind: ClusterLogForwarder
   metadata:
     name: instance
     namespace: openshift-logging

   For more information, see Creating a log forwarder.

8. Define the following ClusterLogForwarder configuration using OpenShift web console or OpenShift CLI:

   1. Specify the log-collector as serviceAccount in the YAML file:

      serviceAccount:
        name: log-collector

   2. Configure inputs to specify the type and source of logs to forward. The following configuration enables the forwarder to capture logs from all applications in a provided namespace:

      inputs:
        - name: my-app-logs-input
          type: application
          application:
            includes:
              - namespace: my-rhdh-project
            containerLimit:
              maxRecordsPerSecond: 100

      For more information, see Forwarding application logs from specific pods.

   3. Configure outputs to specify where to send the captured logs. In this step, focus on the splunk type. You can either use tls.insecureSkipVerify option if the Splunk endpoint uses self-signed TLS certificates (not recommended) or provide the certificate chain using a Secret.

      outputs:
        - name: splunk-receiver-application
          type: splunk
          splunk:
            authentication:
              token:
                key: hecToken
                secretName: splunk-secret
            index: main
            url: 'https://my-splunk-instance-link'
            rateLimit:
              maxRecordsPerSecond: 250

      For more information, see Forwarding logs to Splunk in OpenShift Container Platform documentation.

   4. Optional: Filter logs to include only audit logs:

      filters:
        - name: audit-logs-only
          type: drop
          drop:
            - test:
              - field: .message
                notMatches: isAuditEvent

      For more information, see Filtering logs by content in OpenShift Container Platform documentation.

   5. Configure pipelines to route logs from specific inputs to designated outputs. Use the names of the defined inputs and outputs to specify multiple inputRefs and outputRefs in each pipeline:

      Example pipelines configuration

      pipelines:
        - name: my-app-logs-pipeline
          detectMultilineErrors: true
          inputRefs:
            - my-app-logs-input
          outputRefs:
            - splunk-receiver-application
          filterRefs:
            - audit-logs-only

9. Run the following command to apply the ClusterLogForwarder configuration:

   Example command to apply ClusterLogForwarder configuration

   $ oc apply -f <ClusterLogForwarder-configuration.yaml>

10. Optional: To reduce the risk of log loss, configure your ClusterLogForwarder pods using the following options:

    1. Define the resource requests and limits for the log collector as follows:

       Example collector configuration

       collector:
         resources:
           requests:
             cpu: 250m
             memory: 64Mi
             ephemeral-storage: 250Mi
           limits:
             cpu: 500m
             memory: 128Mi
             ephemeral-storage: 500Mi

    2. Define tuning options for log delivery, including delivery, compression, and RetryDuration. You can apply tuning per output as needed.

       Example tuning configuration

       tuning:
         delivery: AtLeastOnce
         compression: none
         minRetryDuration: 1s
         maxRetryDuration: 10s

       AtLeastOnce

       The AtLeastOnce delivery mode ensures that if the log forwarder crashes or restarts, the forwarder re-sends any logs read but not yet delivered to their destination. The forwarder might duplicate some logs after a crash.

Verification

1. Verify that your Splunk instance receives logs by viewing them in the Splunk dashboard.
2. Troubleshoot any issues using OpenShift Container Platform and Splunk logs as needed.

Procedure

1. From the Developer perspective of the OpenShift Container Platform web console, click the Topology tab.
2. From the Topology view, click the pod that you want to view audit log data for.
3. From the pod panel, click the Resources tab.
4. From the Pods section of the Resources tab, click View logs.
5. From the Logs view, enter isAuditEvent into the Search field to filter audit logs from other log types. You can use the arrows to browse the logs containing the isAuditEvent field.