Senior Databricks Data Engineer

Location: London (Hybrid)
Contract: 6-Month Contract
Pay: £700 p/d inside IR35

This is a key contract role focused on transforming and migrating legacy SQL Server-based data solutions into a modern Databricks architecture.

The successful candidate will assess the current data landscape, identify dependencies and data quality issues, redesign critical data structures, and support the migration of selected workloads into Databricks. You will work closely with business, accounting, and technology stakeholders to ensure the successful delivery of a scalable and supportable data platform.

Key Responsibilities

• Conduct a detailed inventory of the current data environment, including databases, schemas, tables, views, stored procedures, SQL Agent jobs, SSIS packages, feeds, reports, consumers, permissions, and operational processes.
• Document data ownership, lineage, business keys, relationships, calculations, transformations, refresh frequencies, and downstream dependencies.
• Perform data profiling and quality assessments, including completeness, accuracy, duplication, referential integrity, timeliness, and historical coverage.
• Collaborate with business and technical SMEs to determine which data structures, calculations, and reports should be retained, redesigned, consolidated, enhanced, or retired.
• Convert, redesign, and migrate T-SQL code, stored procedures, SSIS packages, ETL processes, and reporting transformations into the target architecture.
• Design, build, and optimise Databricks solutions, including Delta tables, Spark SQL transformations, pipelines, quality controls, reconciliation processes, and monitoring capabilities.
• Create source-to-target mappings, migration strategies, validation plans, and technical documentation.
• Support business-as-usual activities, including SQL development, feed troubleshooting, reporting enhancements, testing, reconciliation, and production issue resolution.