The common trap: Starting from the bottom
Many companies approach their Databricks migration by starting at the bottom of the stack: rolling out the platform, re-integrating data sources (often via ODBC/JDBC), and building a bronze layer before modelling and consuming the data. While this method seems logical, it often leaves teams “below the surface” for too long, struggling to demonstrate value as they work through foundational layers.
To avoid this, it’s crucial to rethink how you start. Databricks, for instance, can pull data via JDBC, but its true strength lies in AutoLoader and working with files stored in cost-effective blob storage. Adding change data capture (CDC) capabilities with tools like Debezium can enhance this, but it may also introduce dependencies on platform or infrastructure teams who may not share your timeline or goals.
The quickest unlock: Federate into legacy
If your data already resides in a cloud platform like Azure or AWS, the quickest path to success is leveraging native services such as Azure Data Factory (ADF) or AWS Data Workflow Services (DWS). These can convert CDC streams into Parquet files, which are easily stored on blob storage. By using these existing tools, you simplify the process, reduce dependencies, and get data into Databricks faster.
When this isn’t an option, or if you really want to go fast, Unity Catalog’s Federation capabilities can provide a workaround. By making your SQL Server databases available in Databricks, you can federate queries directly to the source, enabling you to join live data with datasets already in Databricks. Whether it’s staging databases, data warehouses, or data marts, this approach allows you to build on your existing infrastructure while transitioning to a modern platform.
Show business value from day one
Instead of focusing solely on ingestion pipelines and modelling workflows, prioritise moving consumption use cases to Databricks early. By demonstrating business value—almost from day one—you can gain buy-in from stakeholders and justify further investments in the migration process.
Once the immediate needs are met, gradually shift your data sources from staging into a new ingestion pattern that leverages blob storage and AutoLoader. This step-by-step approach ensures a smoother transition while delivering results that matter to your business.
Ready to take the next step?
At RevoData, we specialize in helping organizations unlock the full potential of Databricks. Whether you’re migrating from SQL Server, optimizing your workflows, or building a modern data platform, our consultants are here to guide you every step of the way. Let us show you how Databricks can transform your data strategy and drive real business impact. Contact RevoData today to get started!
Rafal Frydrych
Senior Consultant at RevoData, sharing with you his knowledge in the opinionated series: Migrating from MSBI to Databricks.