Skip to content
Datasets
Creating datasets
Creating datasets via the CLI

Creating datasets from the CLI

Upload a dataset YAML using fides push

The fides push command allows you to quickly add YAML datasets from your working project directory to the Fides server.

$ fides push --help
> Loaded config from: .fides/fides.toml
                                                                                                                                               
 Usage: fides push [OPTIONS] [MANIFESTS_DIR]                                                                                                                                   
                                                                                                                                      
 Parse local manifest files and upload them to the server.                                                                                                                     
                                                                                                                                                                               
╭─ Arguments ──────────────────────────────────────────────────────────────────────╮
 MANIFESTS_DIR    PATH                                                               
╰──────────────────────────────────────────────────────────────────────────────────╯
╭─ Options ────────────────────────────────────────────────────────────────────────╮
 --dry         Do not upload results to the Fides webserver.                                            
 --diff        Print any diffs between the local & server objects                                  
 --help  -h    Show this message and exit.                                                      
╰──────────────────────────────────────────────────────────────────────────────────╯

The command will upload dataset YAML documents from the .fides/ working directory and then persist any changes to the Fides Server. Visit the push CLI guide for an in-depth review of the command.

Example

$ fides push
Loaded config from: .fides/fides.toml
Loading resource manifests from: .fides/
Taxonomy successfully created.
----------
Processing system resource(s)...
PUSHED 4 system resource(s).
----------
Processing dataset resource(s)...
PUSHED 1 dataset resource(s).
----------

After execution, navigate to the Manage datasets tab in the Fides Admin UI to review your dataset.

Generate from a database connection

The fides generate dataset command allows you to generate a dataset by connecting Fides to a datastore.

fides generate dataset --help
> Loaded config from: .fides/fides.toml
                                                                                                                                                                               
 Usage: fides generate dataset [OPTIONS] COMMAND [ARGS]...                                                                                                                     
                                                                                                                                                                               
 Generate Fides datasets.                                                                                                                                                      
                                                                                                                                                                               
╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────╮
 --help  -h    Show this message and exit.                                                                 
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯
╭─ Commands ────────────────────────────────────────────────────────────────────────────────────────────────╮
 aws  Generate Fides datasets from specific Amazon Web Services.                                           
 db   Generate a Fides dataset by walking a database and recording every schema/table/field.               
 gcp  Generate Fides datasets from Google Cloud Platform.                                                  
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────╯

Visit the generate CLI guide for an in-depth review of the command.

Example

This example generates a dataset from a database using connection details stored in the fides.toml. It will also write a dataset YAML at the location .fides/database.yml.

In this example, pg-credentials is the key that represents the credentials in the configuration file. For more in-depth details on using configuration files, check out the Fides Configuration tutorial.

fides.toml
[pg-credentials]
user = "[DB-USERNAME]"
password = "[PASSWORD]"
server = "127.0.0.1"
port = "5432"
db = "[DATABASE]"
$ fides generate dataset db \
--credentials-id "pg-credentials" .fides/database.yml
 
Loaded config from: .fides/fides.toml
Generated dataset manifest written to .fides/database.yml