Async SaaS integrations

DSR 3.0 must be enabled for asynchronous support

Overview

Fides offers support for async integrations where an access or erasure request cannot be fulfilled immediately, but can support posting results to a callback URL upon completion.

This will typically involve uploading a custom template that denotes the endpoint's async configuration (callback only supported) as well as specifying reply-to and reply-to-token values in request headers so Fides can pass along instructions for sending results asynchronously back to Fides.

Prerequisite: Enable DSR 3.0

DSR 3.0 task scheduling added the ability to run tasks in parallel and introduced task persistence to the application database to support async integrations. To enable DSR 3.0, you must be running a celery worker as part of your Fides deployment, in addition to the environment variables below:

FIDES__CELERY__TASK_ALWAYS_EAGER=false
FIDES__EXECUTION__USE_DSR_3_0=true

Further, you can optionally update these configuration variables which specify how long data lives in your system and the time to wait between status updates. See Configuration variables for more information.

FIDES__REDIS__DEFAULT_TTL_SECONDS=604800
FIDES__EXECUTION__REQUEST_TASK_TTL=604800
FIDES__EXECUTION__STATE_POLLING_INTERVAL=30

Example Async SaaS Config Endpoint

In the example below, we describe how a third-party custom integration can denote its callback async strategy, as well as add reply-to and reply-to-token values to request headers to tell the third party service how to send the results asynchronously back to Fides.

saas_config:
  fides_key: saas_async_config
  name: Async Callback Example Custom Connector
  type: async_callback_example
  description: Test Async Config
  version: 0.0.1
 
  connector_params:
    - name: domain
    - name: api_token
      label: API token
 
  client_config:
    protocol: http
    host: <domain>
    authentication:
      strategy: bearer
      configuration:
        token: <api_token>
 
  test_request:
    method: GET
    path: /
 
  endpoints:
    - name: user
      requests:
        read:
          method: GET
          path: /api/v1/user
          query_params:
            - name: query
              value: <email>
          param_values:
            - name: email
              identity: email
          async_config:
            strategy: callback
          headers:
            - name: reply-to
              value: <reply_to>
            - name: reply-to-token
              value: <reply_to_token>

Callback endpoint

Asynchronous Access Results

Asynchronous access results can be posted back to the Fides reply-to endpoint as a list of rows, using the reply-to-token as the Bearer Token. Note that these results may be filtered and returned to the end user and/or used to make downstream dependent requests.

    curl -X 'POST' \
      'http://localhost:8080/api/v1/request-task/callback' \
      -H 'accept: application/json' \
      -H 'Authorization: Bearer <reply_to_token>' \
      -H 'Content-Type: application/json' \
      -d '{
          "access_results": [
            {"id": "user_id", "system_id": "test_system_id", "state": "test_state"}
          ]
      }'

Asynchronous Erasure Results

Asynchronous erasure results can be posted back to the Fides reply-to endpoint as a count of affected rows, using the reply-to-token as the Bearer Token. The rows masked are not passed along to the end user but may be useful for internal troubleshooting.

    curl -X 'POST' \
      'http://localhost:8080/api/v1/request-task/callback' \
      -H 'accept: application/json' \
      -H 'Authorization: Bearer <reply_to_token>' \
      -H 'Content-Type: application/json' \
      -d '{
          "rows_masked": 3
      }'

Asynchronous Completion Status

If you do not need to pass on asynchronous results to the end user or downstream collections or have no use for tracking rows masked, posting an empty json object will signify the request has been asynchronously completed which will allow Privacy Request processing to resume.

    curl -X 'POST' \
      'http://localhost:8080/api/v1/request-task/callback' \
      -H 'accept: application/json' \
      -H 'Authorization: Bearer <reply_to_token>' \
      -H 'Content-Type: application/json' \
      -d '{}'

Postprocessing Hubspot