Datasets and databases

What is a dataset?

A Dataset is the configuration you provide for a database or other queryable datastore. Fides Datasets are applicable to a wide variety of datastores beyond traditional databases. Within Datasets, the term collection is used to describe an SQL table, mongo database collection, or any other single coherent set values.

Configure a Dataset

Fides uses a YAML manifest file to represent your datastores, and requires information beyond table names and fields to fully configure a Dataset. Datastores connected in this way will be automatically traversed when Fides executes a privacy request, and will either return or update the requested data according to the associated privacy request policy.

Ensure you have created a Connection for the datastore you would like to map. The Dataset defined by the following process should be associated to the Connection.

Describe a datastore

The following is a sample database of customers and addresses. It includes a customer table that has a foreign key of address_id to an address table:

CREATE TABLE CUSTOMER (
  id INT PRIMARY KEY,
  name VARCHAR,
  email VARCHAR,
  address_id int REFERENCES ADDRESS(id)
);
 
CREATE TABLE ADDRESS(
  id INT PRIMARY KEY,
  street VARCHAR,
  city VARCHAR,
  state VARCHAR,
  zip VARCHAR
);

A Fides Dataset contains a map of the database's fields, and metadata describing how those fields are related. Fides uses this relationship information to navigate between different collections and fulfill privacy requests. The Dataset declaration for the above schema looks like:

dataset:
  - fides_key: mydatabase
    name: internal database
    description: our internal database of customer data
    collections:
      - name: address
        fields:
          - name: id
            data_categories: [system.operations]
            fidesops_meta:
              primary_key: True
          - name: street
            data_categories: [user.contact.address.street]
            fidesops_meta:
              data_type: string
          - name: city
            data_categories: [user.contact.address.city]
            fidesops_meta:
              data_type: string
          - name: state
            data_categories: [user.contact.address.state]
            fidesops_meta:
              data_type: string
          - name: zip
            data_categories: [user.contact.address.postal_code]
            fidesops_meta:
              data_type: string
 
      - name: customer
        after: mydatabase.address
        fields:
          - name: address_id
            data_categories: [system.operations]
            fidesops_meta:
              references:
                - dataset: mydatabase
                  field: address.id
                  direction: to
          - name: created
            data_categories: [system.operations]
          - name: email
            data_categories: [user.contact.email]
            fidesops_meta:
              identity: email
              data_type: string
          - name: id
            data_categories: [user.unique_id]
            fidesops_meta:
              primary_key: True
          - name: name
            data_categories: [user.name]
            fidesops_meta:
              data_type: string

Dataset members

Collection members

Field members

Generate a dataset

The Fides CLI allows you to both connect to and generate a blank Dataset for your datastores. This blank Dataset does not include any annotations (e.g., Fides data descriptions) or fidesops_meta information, but can be used to initially map your databases.

Configure a manual Dataset

Not all data can be automatically retrieved. When services have no external API, or when user data is held in a physical location, you can define a Dataset to describe the types of manual fields you plan to upload, as well as any dependencies between these manual collections and other collections.

Describe a manual datastore

In the following example, the manual Dataset is a physical location, which contains one storage_unit collection. email is defined as the unit's identity, which will then be used to retrieve the box_id in the storage unit.

To add a Manual Dataset, first create a Manual Connection. The following Manual Dataset can then be added to the new ConnectionConfig:

dataset:
  - fides_key: manual_input
    name: Manual Dataset
    description: Example of a Dataset whose data must be manually retrieved
    collections:
      - name: storage_unit
        fields:
          - name: box_id
            data_categories: [ user ]
            fidesops_meta:
              primary_key: true
          - name: email
            data_categories: [ user.contact.email ]
            fidesops_meta:
              identity: email
              data_type: string

Resume a paused access privacy request

A privacy request will pause execution when it reaches a manual collection in an access request. An administrator should manually retrieve the data and send it in a POST request. The fields should match the fields on the paused collection.

Erasure requests with manual collections will also need data manually added as well.

[{
    "box_id": 5,
    "email": "customer-1@example.com"
}]

If no manual data can be found, pass in an empty list to resume the privacy request:

[]

Resume a paused erasure privacy request

A privacy request will pause execution when it reaches a manual collection in an erasure request. An administrator should manually mask the records in question, and send confirmation of the rows affected in a POST request.

{"row_count": 2}

If no manual data was destroyed, pass in a count of 0 to resume the privacy request:

{"row_count": 0}