Skip to content

Privacera Plugin in Dataproc#

This section covers how you can use Privacera Manager to generate the setup script and Dataproc custom configuration to install Privacera Plugin in the GCP Dataproc environment.

Prerequisites

Ensure the following prerequisites are met:

  • A working Dataproc environment.

  • Privacera services must be up and running.

Configuration

  1. SSH to the instance where Privacera is installed.

  2. Run the following command:

    cd ~/privacera/privacera-manager
    cp config/sample-vars/vars.dataproc.yml config/custom-vars/
    vi config/custom-vars/vars.dataproc.yml
    
  3. Edit the following properties:

    Property Description Example
    DATAPROC_ENABLE Enable Dataproc template creation. true
    DATAPROC_MANAGE_INIT_SCRIPT

    Set this property to upload the init script to GCP Cloud Storage.

    If the value is set to true, then Privacera will upload the init script to the GCP bucket.

    If the value is set to false, then manually upload the init script to a GCP bucket.

    false
    DATAPROC_PRIVACERA_GS_BUCKET Enter the GCP bucket name where the init script will be uploaded. gs://privacera-bucket

  4. Run the update.

    cd ~/privacera/privacera-manager
    ./privacera-manager.sh update
    

    After the update is complete, the setup script setup_dataproc.sh and Dataproc custom configurations privacera_cust_conf.zip will be generated at the path, ~/privacera/privacera-manager/output/dataproc.

  5. If DATAPROC_MANAGE_INIT_SCRIPT is set to false, then copy setup_dataproc.sh and privacera_cust_conf.zip. Both the files should be placed under the same folder.

    cd ~/privacera/privacera-manager/output/dataproc
    GS_BUCKET=<PLEASE_CHANGE>
    gsutil cp setup_dataproc.sh gs://${GS_BUCKET}/privacera/dataproc/init/
    gsutil cp privacera_cust_conf.zip gs://${GS_BUCKET}/privacera/dataproc/init/
    
  6. SSH to the instance where the master node of the Dataproc is installed. Then, enter the GCP bucket name and run the setup script.

    sudo su - 
    mkdir -p /opt/privacera/downloads
    cd /opt/privacera/downloads
    GS_BUCKET=privacera-dev
    gsutil cp gs://${GS_BUCKET}/privacera/dataproc/init/setup_dataproc.sh .
    chmod +x setup_dataproc.sh
    ./setup_dataproc.sh
    

Last update: August 9, 2021