Skip to content

Custom Spark Configuration for Databricks#

Authenticate Databricks Using JWT#

For information on the section, click here.

Add Extra Properties to Spark Config#

To add custom properties in the Databricks cluster init script, do the following:

  1. Create a custom configuration file.

    cd ~/privacera/privacera-manager
    vi config/custom-properties/databricks-spark.conf
    
  2. Now, you can use the file to add any custom spark properties. Just add the properties one below the other, and then save the file.

    For example, you can add the following property and save the file.

    "spark.databricks.delta.formatCheck.enabled"="false"
    

    Note

    Avoid putting comments, extra words, or blank lines in the config file.

  3. Run the following command.

    cd ~/privacera/privacera-manager
    ./privacera-manager.sh update
    

Configure Databricks Cluster Policy#

  1. Add the following two properties in vars.databricks.plugin.yml.

    • DATABRICKS_SQL_CLUSTER_POLICY_SPARK_CONF
    • DATABRICKS_SCALA_CLUSTER_POLICY_SPARK_CONF

    For example,

    DATABRICKS_SQL_CLUSTER_POLICY_SPARK_CONF:
    - Note: first spark conf
        key : "spark.hadoop.first.spark.test"
        value: "test1"
    - Note: second spark first spark conf
        key: "spark.hadoop.first.spark.test2"
        value: "test2"
    DATABRICKS_SCALA_CLUSTER_POLICY_SPARK_CONF:
    - Note: first spark conf
        key : "spark.hadoop.first.spark.test"
        value: "test1"
    - Note: second spark first spark conf
        key: "spark.hadoop.first.spark.test2"
        value: "test2"
    
  2. To add custom properties with Java agent, add the following property in vars.databricks.plugin.yml.

    • DATABRICKS_SPARK_PLUGIN_AGENT_JAR

    For example,

    DATABRICKS_SPARK_PLUGIN_AGENT_JAR: " -Dmy.custom.propery=test -javaagent:/databricks/jars/privacera-agent.jar"
    

Last update: August 24, 2021