Data Governance with Unity Catalog in Databricks

The data breach report landed on a Tuesday morning. A junior analyst had queried a production table containing customer PII. The query ran without error. No one had restricted the column. No alert had fired. The data had been sitting exposed for four months.

That story plays out in organizations every week. Not because engineers are careless, but because governance was treated as something to add later. Permissions lived in spreadsheets. Access policies were per-workspace, not per-account. When someone asked who had accessed a table last quarter, no one could answer with confidence.

In a Databricks environment without centralized governance, this is the default state. Three workspaces mean three separate Hive metastores, three sets of permissions, no shared lineage, and no audit trail that spans them all. According to a Gartner finding cited by Prolifics, 60% of organizations face management and compliance challenges directly tied to siloed data systems and the absence of centralized oversight.

Unity Catalog is Databricks answer to that problem. It replaces the fragmented workspace-level governance model with a single control plane that spans every workspace in your account. One metastore. One permission system. Lineage and audit logs that follow data wherever it goes. Gartner also projects that by 2026, organizations implementing modern governance frameworks will reduce data and analytics risks by up to 50%.

This article is your practical guide to how it works and how to implement it correctly. If you are new to this series, Modern Data Engineering: The Complete Guide covers the broader landscape before you go deeper here.

What Is Unity Catalog and What Problem Does It Solve?

Before Unity Catalog, a typical multi-team Databricks setup had a real governance problem. Each workspace managed its own metadata, its own Hive metastore, and its own permission system. There was no shared catalog. There was no way to see, from one place, what data existed across workspaces or who had accessed what.

According to Databricks documentation, Unity Catalog is the unified governance layer built into Databricks. When enabled for a workspace, it operates beneath every data interaction automatically: enforcing access control when a table is queried, tracking lineage as data moves, and logging every activity for auditing.

Unity Catalog governs all of this from one central place:

• Tables, views, and materialized views
• Volumes (unstructured files, PDFs, images, ML artifacts)
• Machine learning models and feature store assets
• Functions and user-defined SQL logic
• Delta Sharing shares for external data distribution

The shift is significant. Before Unity Catalog, governance was additive. You built your pipelines and then tried to bolt permissions on top. Unity Catalog makes governance structural. It sits underneath everything. You cannot query a table without Unity Catalog enforcing the policy for that query.

A March 2026 practitioner post on Medium describes the pre-Unity Catalog state directly: three workspaces, three separate Hive metastores, three permission systems, no shared visibility into data movement and governance living in IAM roles, bucket policies, and a spreadsheet someone updated irregularly.

Unity Catalog replaces that entire stack with one metastore, standard SQL-based permissions, and automatic lineage.

How the Three-Level Namespace Works

Understanding the structure of Unity Catalog is the foundation for everything else. All data assets inside Unity Catalog live in a three-level namespace: catalog.schema.table.

Databricks documentation on the permissions model describes the hierarchy clearly:

• Metastore: the top-level container, one per cloud region. Holds all catalogs and all governance metadata for every workspace linked to it.
• Catalog: the first level of the namespace. Typically maps to an environment (prod, staging, dev) or a business domain (finance, marketing, engineering).
• Schema: the second level (also called database in older Spark SQL terminology). Groups related tables, views, volumes, and functions by subject area.
• Table / View / Volume / Function: the actual data objects where permissions are enforced and lineage is tracked.

A real reference looks like: prod.finance.transactions or staging.marketing.campaign_events. This format is how you query, grant permissions, and track lineage throughout the platform.

How Objects Outside the Three-Level Namespace Work

Some objects in Unity Catalog sit directly under the metastore rather than inside a catalog. These include:

• Storage credentials: the cloud identity or service account used to access external cloud storage
• External locations: specific storage paths (S3 buckets, ADLS containers, GCS paths) that Unity Catalog is authorized to access
• Delta Sharing shares: packages of data shared externally with recipients outside your organization

These objects are governed by Unity Catalog but are not addressed using the catalog.schema.object format.

Access Control: How Permissions Work in Unity Catalog

Unity Catalog uses standard ANSI SQL GRANT and REVOKE statements to manage permissions. This is intentional. It makes permissions legible to anyone who knows SQL and auditable in the same way code is auditable.

Databricks best practices documentation states that all principals (users, groups, and service principals) must be defined at the Databricks account level, not the workspace level. This is the most common configuration mistake teams make.

The Core Permission Model

Every object in the hierarchy is a securable object. You grant privileges on securable objects. Common privileges include:

Privilege	What It Controls
USE CATALOG	Lets a principal access the catalog and see its schemas
USE SCHEMA	Lets a principal access the schema and see its tables
SELECT	Lets a principal read data from a table or view
MODIFY	Lets a principal write, update, or delete data in a table
CREATE TABLE	Lets a principal create new tables inside a schema
ALL PRIVILEGES	Grants all applicable privileges on the object

Privileges inherit downward. If you grant USE CATALOG on prod, the user can see all schemas inside prod unless you restrict further. If you grant SELECT on a specific table, the user can read that table only if they also have USE CATALOG and USE SCHEMA on the parent objects.

A minimal permission grant looks like this in SQL:

GRANT USE CATALOG ON CATALOG prod TO `data-analysts`;
GRANT USE SCHEMA ON SCHEMA prod.finance TO `data-analysts`;
GRANT SELECT ON TABLE prod.finance.transactions TO `data-analysts`;

This grants only what is needed. Nothing more. That is the principle of least privilege applied directly through SQL.

Using Groups Instead of Users

Granting permissions to individual users does not scale. A team of 30 analysts means 30 individual grants. Every offboarding requires finding and revoking grants scattered across the catalog hierarchy.

The right pattern is to manage access through groups:

• Define groups in your identity provider (Azure Active Directory, Okta, AWS IAM Identity Center)
• Sync them to Databricks using SCIM provisioning at the account level
• Grant catalog permissions to groups, not individuals
• When someone joins or leaves, update group membership in the IdP. Permissions update automatically.

Databricks best practices explicitly recommends avoiding workspace-level SCIM provisioning and managing provisioning entirely at the account level.

Row-Level and Column-Level Security: Protecting Sensitive Data

Table-level access is not enough for most production use cases. A single customer table might contain email addresses, transaction amounts, and account numbers alongside data that is safe to share broadly. Blocking the whole table is too restrictive. Exposing it fully is not safe.

Unity Catalog handles this with row filters and column masks, both of which reached general availability in 2024 and are now standard tooling for production data teams.

Column Masks: Redacting Sensitive Fields

A column mask is a SQL function applied to a column that controls what each user sees when they query it. Instead of seeing the raw value, users who do not have the appropriate role see a masked version.

Databricks announced general availability of column masks in August 2024 for AWS, Azure, and GCP. You define a masking function and attach it to a specific column on a table.

A simple column mask example:

CREATE FUNCTION finance.mask_card_number(card_number STRING)
RETURN CASE
WHEN is_member('pci-analysts') THEN card_number
ELSE CONCAT('--****-', RIGHT(card_number, 4))
END;

ALTER TABLE prod.finance.transactions
ALTER COLUMN card_number SET MASK finance.mask_card_number;

After this, any user who is not in the pci-analysts group sees only the last four digits. PCI analysts see the full number. The masking happens at query time. There is no duplicate table, no separate pipeline, no data copy.

Row Filters: Restricting Which Rows Each User Sees

A row filter is a SQL function that controls which rows are visible in a query result based on who is running it. A regional manager seeing only their region's data. A vendor seeing only their own records. A healthcare analyst seeing only their hospital's patient data.

CREATE FUNCTION finance.filter_by_region(region STRING)
 RETURN region = session_context('databricks.groupNames')
 OR is_member('global-finance');

ALTER TABLE prod.finance.transactions
 ADD ROW FILTER finance.filter_by_region ON (region);

The function runs transparently at query time. Users do not need to add WHERE clauses. They simply cannot see rows they are not authorized to see. This is governance without friction for the end user.

Automated Data Lineage: Knowing Where Your Data Came From

Data lineage answers a question that most teams cannot answer today: if something changes in a source table, what downstream tables, reports, and models will be affected?

Unity Catalog documentation states that lineage is tracked automatically. Every data access event generates a lineage entry. This covers table-level lineage and column-level lineage across SQL, Python, Spark, and notebooks.

Column-level lineage is particularly valuable. It shows not just that Table B was built from Table A, but specifically that the revenue_usd column in Table B came from the amount column in Table A after a currency conversion. This level of detail is what makes impact analysis fast. When the currency conversion logic changes, engineers know exactly which downstream columns are affected without tracing code manually.

Practical Uses of Lineage in Production

• Impact analysis: before changing a source table schema, run a lineage check to see what breaks downstream before the change happens in production
• Incident investigation: when a dashboard shows wrong numbers, trace the lineage from the dashboard back to the raw source to find where the bad data entered
• Compliance audits: demonstrate to auditors exactly how PII flows from collection through transformation to its current storage location
• Data discovery: new team members can explore how a table was built without reading through pipeline code

Forrester research, cited by Prolifics, finds that organizations with strong data lineage capabilities measurably improve trust in analytics outcomes. When engineers and analysts can see where data came from, they spend less time questioning its accuracy and more time using it.

Audit Logs: A Complete Record of Every Data Access

Every query, every permission grant, every login attempt, every file access generates an audit log entry in Unity Catalog automatically. No configuration required. No log pipeline to build.

Databricks documentation states that audit logs are available as system tables, which means you query them using SQL exactly like any other table.

A basic audit log query to see who accessed a sensitive table:

SELECT
 user_identity.email,
 action_name,
 request_params.table_full_name,
 event_time
FROM system.access.audit
WHERE request_params.table_full_name = 'prod.finance.transactions' AND event_time > DATEADD(day, -30, CURRENT_TIMESTAMP())
ORDER BY event_time DESC;

This returns every user who queried the transactions table in the last 30 days, what action they took, and when. No log parsing. No external SIEM required for basic queries. The data is there and queryable immediately.

What the Audit System Tables Capture

Unity Catalog system tables expose multiple event categories:

System Table	What It Contains
system.access.audit	All data access events, permission changes, login events
system.lineage.table_lineage	Table-level read/write relationships across pipelines
system.lineage.column_lineage	Column-level transformation tracking
system.billing.usage	Compute costs attributed by workspace, job, and user

These tables are the foundation for any internal compliance program. When your security team asks who accessed PII last quarter, the answer is a SQL query, not a ticket to the cloud provider.

Unity Catalog and Compliance: GDPR, HIPAA, and PCI-DSS

Compliance requirements are fundamentally data access and auditability requirements. GDPR requires demonstrating who can access personal data and tracing what happened to it. HIPAA requires access controls on patient data and audit trails. PCI-DSS requires restricting cardholder data to authorized personnel.

Unity Catalog supports all three frameworks, providing the technical controls each one requires. The controls are not separate integrations. They are the same permissions, lineage, and audit features already described in this article, applied to the right data.

The key connection between Unity Catalog and compliance:

• GDPR: use column masks to de-identify PII in analytics views, use audit logs to respond to data subject access requests, use lineage to demonstrate data flows to auditors
• HIPAA: apply row filters to restrict patient records to authorized care teams, use system tables to demonstrate access logging to auditors, use managed tables to enforce storage in HIPAA-compliant storage accounts
• PCI-DSS: use column masks on cardholder data, restrict pci-scope schemas to service principals with dedicated compute, use audit logs to demonstrate access control effectiveness

A November 2025 implementation guide by Kanerika reports that enterprises using Unity Catalog have reduced time spent managing data permissions by up to 40% while improving audit readiness. The time saving comes from replacing manual permission spreadsheets with SQL-based grant management and automated audit tables.

Delta Sharing: Governed Data Sharing Beyond Your Organization

Unity Catalog includes Delta Sharing, an open protocol for sharing data and AI assets with external recipients. Recipients do not need to be on Databricks. They do not need to be on the same cloud. They do not need to copy any data.

Flexera's 2026 Unity Catalog overview notes that Delta Sharing supports sharing data across clouds, regions, and organizations without data duplication. Recipients access data through the open Delta Sharing protocol using any compatible client.

The governance model applies to shares the same way it applies to internal catalogs. You define what data is in the share, who the recipients are, and what audit logs capture. If a share is revoked, recipients immediately lose access. There is no data they downloaded to govern separately.

This is particularly useful for:

• Sharing transformed analytics data with business partners or customers
• Distributing reference datasets to teams outside your organization
• Building data products that external teams subscribe to

Common Mistakes Teams Make with Unity Catalog

Most Unity Catalog problems come from the same small set of decisions made early in the implementation. These are the ones worth knowing before you start.

Granting permissions to individual users instead of groups. This creates an unmanageable permission graph after the first few months. When someone leaves, their access stays until someone manually finds and revokes every grant. Use groups from your identity provider and grant to groups only.

Using workspace-local groups for catalog permissions. Unity Catalog GRANT statements only work with account-level groups. Workspace-local groups are a legacy concept that does not apply here. Databricks best practices explicitly warns against workspace-level SCIM provisioning for this reason.

Running legacy workloads on clusters not configured for Unity Catalog. A March 2026 Medium post notes that clusters set to "No Isolation Shared" mode do not support Unity Catalog features. Legacy libraries, custom Spark configurations, and older R or Scala code may not run on Unity Catalog-enabled clusters due to security sandboxing. Audit your workloads before migrating, not during.

Treating metastore configuration as reversible. A misconfiguration at the metastore level can affect access across every workspace in your account simultaneously. The account-wide scope is a strength for governance but a risk if mistakes happen at the top. Test metastore-level changes in a non-production account before applying them to production.

Skipping automated permission audits. Permissions drift. Analysts request access for a project and it never gets revoked. Service principals accumulate privileges over time. Business Compass governance guidance recommends setting up automated daily checks against governance policies to flag orphaned accounts, excessive permissions, and unusual access patterns before they become security incidents.

How Unity Catalog Connects to the Rest of the Platform

Unity Catalog is not an isolated feature. It is the governance backbone that every other Databricks component inherits from. Understanding these connections matters for how you design your platform.

• Lakeflow Jobs: every job run inherits Unity Catalog identity and lineage automatically. Job-level orchestration appears in the lineage graph alongside table transformations. Workflow Orchestration with Lakeflow Jobs covers how this inheritance works in practice.
• Delta Lake: all managed tables in Unity Catalog are Delta tables. The ACID guarantees, time travel, and Change Data Feed features in Delta Lake sit underneath Unity Catalog governance. Delta Lake Explained for Data Engineers covers the storage layer in detail.
• Data quality monitoring: Data Quality and Reliability Patterns for Databricks Pipelines covers how monitoring connects to Unity Catalog for governed quality checks.
• Medallion architecture: the catalog and schema structure in Unity Catalog is the natural home for bronze, silver, and gold layers. Medallion Architecture in Databricks covers how to map the three-level namespace to the medallion pattern.