Data Quality Layer for Unity Catalog
In today’s data-driven world, poor data quality leads to unreliable insights, stalled growth, and increased compliance risks. Traditional approaches to data quality management are slow, costly, and struggle to keep pace with evolving data landscapes.
DQGateway, seamlessly integrated with Databricks, transforms data quality management by providing real-time observability, automation, and machine learning-driven anomaly detection—eliminating the need for manual intervention or complex coding.
🔹 Real-Time Data Monitoring – Gain end-to-end visibility into your data quality across warehouses, lakes, and pipelines.
🔹 AI-Powered Anomaly Detection – Leverage machine learning to automate data validation and identify issues before they impact business decisions.
🔹 Scalable & Continuous Data Quality – Manage enterprise-wide data quality effortlessly, empowering analytics, AI, and regulatory reporting.
With DQGateway, businesses can confidently scale their data operations, ensuring trusted, high-quality data fuels their analytics, AI, and compliance initiatives—without the bottlenecks. Experience data quality at light speed.
Recommendations for using Databricks Connector
Compute
DQGateway recommends that you use Databricks Serverless SQL Warehouses.
Serverless compute allows you to run workloads without provisioning a cluster. Instead, Databricks automatically allocates and manages the necessary compute resources. This enables you to focus on writing code and analyzing data, without worrying about cluster management or resource utilization.
Databricks SQL delivers optimal price and performance with Serverless SQL Warehouses.
Authentication
Databricks OAuth supports secure credentials and access for resources and operations at the Databricks workspace level and supports fine-grained permissions for authorization.
Databricks also supports Personal Access Tokens (PATs), but recommends you use OAuth instead.
M2M OAuth is the default and preferred authentication mode provided by DQGateway for connecting to Databricks. Personal Access Token (PAT) option is also supported.
Catalog name
Databricks unity catalog name can be entered by the user, while creating a new connection using DQGateway interface.
This catalog name is then set at connection level as the default. This ensures that you can support one catalog set at the connection level.
