Overview:

Databricks supports a wide range of compliance standards to meet the needs of highly regulated industries, including:

• HIPAA (Health Insurance Portability and Accountability Act)
• PCI-DSS (Payment Card Industry Data Security Standard)
• FedRAMP High & Moderate
• DoD IL5
• IRAP (Australia)
• GDPR (EU)
• CCPA (California)

However, I was surprised to read that Databricks Serverless workloads are not covered for PCI-DSS (Databricks PCI DSS Compliance | Databricks) and got curious to find the reason behind it. Based on my research, I managed to convince with the reason and would like to share it over here.

To begin with, let’s understand different Databricks SQL Warehouse types and its capabilities,

Databricks SQL (Classic/Pro):

• In Databricks SQL (Classic/Pro) warehouses, compute resources in customer account will be leveraged
• When running workloads using Databricks SQL (Classic/Pro), data is processed by the compute resources which are managed by the customers
• Customers will have more control and monitoring over the compute resources
• Data getting processed will also reside within network boundary of the customer cloud account

Databricks SQL (Serverless):

• In Databricks SQL (Serverless) warehouse, compute resources in Databricks account will be leveraged
• Serverless compute operates on multi-tenant architecture, where compute resources are shared across different customers
• Compute resources are completed managed by Databricks and customers will have less control and monitoring ability on the networking and compute resources
• Different workload data is processed within compute resources of Databricks account
• Though customers have less control over the compute, they can greatly benefit out of the capabilities that Serverless warehouses exhibit

Final View:

• PCI-DSS requires strict isolation of environments handling cardholder data, which is difficult to guarantee in a shared setup
• It mandates restricted and monitored network access, especially for systems handling payment data
• It requires fine-grained control and auditing, which is more feasible in dedicated or customer-managed environments
• Databricks recommends using classic or pro clusters with dedicated VPCs, private networking, and enhanced security controls for PCI DSS compliant workloads
• Additionally, Databricks dedicates effort to bring in more isolation boundaries within Serverless compute

This post was originally published on this site - UniAspect assumes no responsibility or liability for any errors or omissions in the content of this article. The information contained in this article is provided on an "as is" basis with no guarantees of completeness, accuracy, usefulness or timeliness and without any warranties of any kind, express or implied.