Skip to content
Storage destinations

Configure Storage Destinations

What is a storage destination?

Data subject access requests (DSARs) produce a package of returned data upon completion. This data is then uploaded to a storage destination (e.g. an S3 bucket) before being sent to the user.

Fides never stores privacy request results locally. At least one storage destination must be configured if you wish to process access requests.

Storage destinations are associated with privacy request policy rules, allowing multiple storage destinations to be configured per privacy request policy.

Storage Destinations

Create a storage destination

Configure your storage method

To configure a Storage destination, first choose a method to store your results. Fides currently supports the following methods of storage:

  • local - This saves upload packages locally, generating a fides_uploads directory at the root of your project. This destination type should only be used for testing purposes, and not to process real-world access requests.
  • S3 - Files are uploaded to an S3 bucket of your choosing upon completion of an access request. Use S3 if you need a place to store those files.
  • Google Cloud Storage - Files are uploaded to a GCS bucket.

Create your storage destination

Storage destinations are created and managed via the API. To create a new Storage destination, use the following endpoint:

PATCH /api/v1/storage/config
[
  {
    "name": "{{storage_destination_name}}",
    "key": "{{storage_destination_key}}",
    "type": "s3" | "local",
    "format": "json" | "csv" | "html",
    "details": { 
      "naming": "request_id",                       // default value
      "auth_method": "secret_keys" | "automatic",   // for cloud storage destinations
      "bucket": "{{bucket_name}}",                  // for cloud storage destinations
    }
  }
]

Destination attributes

AttributeDescription
nameA unique user-friendly name for your storage destination.
keyOptional. A unique key used to manage your storage destination. This is auto-generated from name if left blank. Accepted values are alphanumeric, _, and ..
typeType of storage destination. Supported types include s3, and local. You may configure multiple destinations of the same type.
formatThe format of uploaded data. Supported formats include json, csv, and html.
details.namingThis value should be request_id for all your storage destinations. It will name the access package according to the request_id.
details.auth_methodThe authentication method for creating a session with S3. Either automatic or secret_keys.
details.bucketThe name of the bucket in S3.

Additional attributes for s3 buckets

AttributeDescription
auth_methodThe authentication method for creating a session with S3. Either automatic or secret_keys.
bucketThe name of the bucket in S3.
namingThis defines how the uploaded files will be named. Currently, Fides only supports upload file naming by request_id. Use this value for all your storage destinations.

Additional attributes for Google Cloud Storage

AttributeDescription
auth_methodThe authentication method for creating a GCS client. Either adc (Application Default Credentials) or service_account_keys.
bucketThe name of the bucket in GCS.
namingThis defines how the uploaded files will be named. Currently, Fides only supports upload file naming by request_id. Use this value for all your storage destinations.

Additional attributes for local storage

AttributeDescription
namingThis defines how the uploaded files will be named. Currently, Fides supports upload file naming by request_id. Use this value for all your storage destinations.

On success, the response from the above endpoint will include a storage_key for each destination, which can be used when defining privacy request policy rules.

Example response
{
  "items": [
    {
      "id": "sto_fe4e4dc0-b5d3-4ac1-bfcd-86e60e9891b9",
      "name": "s3 storage 2",
      "type": "s3",
      "details": {
        "auth_method": "secret_keys",
        "bucket": "my-bucket",
        "naming": "request_id",
        "object_name": "requests"
      },
      "key": "s3_storage_2"
    }
  ],
  "total": 1,
  "page": 1,
  "size": 1
}

Authenticate with your destination

Fides requires authenticated access to update and erase (or mask) data in your storage destination.

Use config_key returned during your storage creation to provide access credentials:

PUT /api/v1/storage/config/{config_key}/secret
{
  "aws_access_key_id": "{{aws_access_key_id}}",
  "aws_secret_access_key": "{{aws_secret_access_key}}"
}

Secrets are not saved if credentials fail authentication with the given storage destination.

Additional attributes for S3 buckets

Fides supports automatically creating a session for S3. If your auth_method is set to automatic, no secrets need to be provided. Boto3 will look for credentials on the server.
AttributeDescription
aws_access_key_idAWS access key id, obtained from AWS console.
aws_secret_access_keyAWS secret access key, obtained from AWS console.

Additional attributes for Google Cloud Storage

If your auth_method is set to adc, no secrets need to be provided. When creating the client, the system will look for the credentials defined on the server in the file specified in GOOGLE_APPLICATION_CREDENTIALS.
AttributeDescription
typeType of the credential. Always "service_account" for service account keys.
project_idThe ID of the Google Cloud project that owns the service account.
private_key_idIdentifier for the private key. Useful for key rotation or revocation.
private_keyThe actual private RSA key (PEM format). This is used to sign authentication tokens.
client_emailThe service account's email address. This acts as the identity of the application when making API requests.
client_idOAuth2 client ID associated with the service account.
auth_uriThe URL for starting the OAuth2 flow (mostly relevant for user credentials, not service accounts). Always https://accounts.google.com/o/oauth2/auth.
token_uriThe URL used to exchange a signed JWT for an access token. Always https://oauth2.googleapis.com/token.
auth_provider_x509_cert_urlPublic URL where Google’s OAuth2 certificates can be fetched — used to verify tokens issued by Google.
client_x509_cert_urlPublic URL exposing this service account’s X.509 certificate — can be used to verify signatures made by the account.
universe_domainThe Google Cloud "universe" this key is part of — typically googleapis.com.

The fields described are extracted from a Service Account JSON file, which contains all the necessary information for authentication with Google Cloud services. This file is downloaded from the Google Cloud Console.

Test your storage connection

To test your storage configuration, send an API request to the upload data endpoint:

PUT /api/v1/storage/{request_id}
{
  "storage_key": {storage_key},
  "data": {
    // include sample JSON to upload to storage destination
  }
}
AttributeDescription
request_idA privacy request ID.
storage_keyThe key associated with the storage destination.
dataA dictionary of arbitrary data you wish to upload to storage destination.

Extensibility

Fides can be extended to support additional storage destinations by:

  1. Adding destination-specific enums in src/fides/ops/schemas/storage/storage.py
  2. Implementing an authenticator in src/fides/ops/service/storage/storage_authenticator_service.py
  3. Implementing the uploader in src/fides/ops/service/storage/storage_uploader_service.py