Smartedge Solutions
Platform Architect /Platform Engineer (Azure Databricks )
England
Posted about 17 hours ago
Get notified of more jobs like this · No spam, ever
How your CV stacks up
1Upload CV
2Analyse CV
3Improve CV
Upload your CV to see how well it fits this job role
?%
Smartedge’s Client is looking for an individual to help with their Platform Architect /Platform Engineer (Azure Databricks ) @ London or Horsham, UK (Hybrid Working )
Role Description
We are seeking a highly experienced, hands-on Azure Databricks Platform Engineer / Architect to enhance and optimise an enterprise Data Platform. This role combines architecture, engineering, operational support, and platform optimisation. The successful candidate must be capable of designing, configuring, developing, troubleshooting, and improving Azure Databricks solutions, rather than operating solely at a strategic or governance level.
The role focuses on three primary outcomes:
- Enabling and optimising Databricks Serverless capabilities.
- Strengthening FinOps practices, cost controls, and platform governance.
- Enhancing the Databricks Discovery Zone to support migration and consolidation of workloads currently delivered through POSIT/RStudio.
Key Responsibilities
Databricks Serverless Enablement and Optimisation
- Assess existing workloads and determine the most appropriate compute model, including Serverless, Jobs Compute, Interactive Compute, or Classic Clusters, based on workload characteristics, SLA requirements, performance, utilisation, and cost.
- Configure and implement Serverless capabilities for notebooks, jobs, SQL workloads, analytical processing, and data pipelines.
- Develop workload placement standards and guidance, identifying scenarios where Serverless may not be the most cost-effective solution for predictable, long-running, or continuously utilised workloads.
- Implement compute policies, autoscaling configurations, quotas, budget controls, and operational guardrails.
- Monitor performance and cost, identifying oversized, underutilised, or idle resources and recommending optimisation opportunities.
FinOps and Enterprise Platform Controls
- Define and implement a practical FinOps operating model covering ownership, accountability, environments, projects, applications, cost centres, and teams.
- Establish mandatory tagging standards and integrate automated validation into CI/CD pipelines to prevent deployment of non-compliant resources.
- Deliver granular cost attribution and reporting across workspaces, projects, applications, workloads, and business teams.
- Configure budgets, spend thresholds, alerts, and usage monitoring to proactively manage platform costs.
- Analyse platform usage and billing data to identify cost anomalies, inefficient workloads, excessive storage consumption, unnecessary data movement, and underutilised resources.
- Support continuous improvement through cost optimisation recommendations and governance controls.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Discovery Zone Enhancement and POSIT/RStudio Migration
- Enhance the Databricks Discovery Zone to support the migration and modernisation of analytics and data science workloads currently running on POSIT/RStudio.
Enable capabilities including:
- Secure API and external system integrations
- Application deployment and operationalisation
- External data ingestion
- BI and reporting connectivity
- Scheduling and orchestration
- Local IDE-based development
- LLM and AI integration
- Operational monitoring and reporting
Define reusable onboarding, migration, and delivery patterns that reduce technology sprawl while improving platform security, supportability, and delivery speed.
Collaborate with stakeholders to support migration planning, solution design, and adoption.
Data Engineering and Integration
- Design, build, and optimise scalable data ingestion and transformation solutions using Python, PySpark, SQL, and Delta Lake.
- Implement batch and incremental processing patterns, including Change Data Capture (CDC), schema evolution, reconciliation processes, data quality controls, and error handling.
- Develop reusable integration frameworks for: REST APIs, SaaS platforms, Databases, Files and object storage, Document repositories, Enterprise systems, External and public data sources.
- Implement secure authentication, secrets management, and credential handling practices.
- Deliver end-to-end data flows from source ingestion through governed and curated data layers, supporting analytics, BI, machine learning, and application consumption.
Required Technical Skills
Azure & Databricks Platform
- Deep hands-on Azure Databricks implementation, administration, and troubleshooting experience.
- Databricks Serverless architecture, workload placement, and compute optimisation.
- Azure networking, identity, security, monitoring, secrets management, and private connectivity.
- Databricks SQL, Delta Lake, and query/workload performance optimisation.
- Strong understanding of platform governance, operational support, and best practices.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Engineering, Delivery & Operations
- Python, PySpark, and SQL development expertise.
- Data engineering and pipeline delivery within enterprise-scale environments.
- Jobs, workflows, orchestration, incremental processing, and CDC implementation.
- Data quality, reconciliation, and operational monitoring.
- REST API integration and external data connectivity patterns.
- FinOps implementation, tagging strategies, cost attribution, monitoring, and budget management.
- Experience supporting production platforms and driving continual optimisation.
Experience and Candidate Profile
- Proven experience delivering and supporting enterprise-scale Azure Databricks platforms in production environments.
- Demonstrated ability to work across architecture, engineering, implementation, optimisation, and operational support without dependence on specialist teams.
- Strong understanding of security, governance, compliance, and controlled delivery within complex or regulated organisations.
- Experience collaborating with data engineers, data scientists, architects, security teams, platform teams, and business stakeholders.
- Excellent communication skills with the ability to document standards, patterns, operational procedures, and architectural decisions.
Highly Desirable
- Experience migrating workloads from POSIT/RStudio to Databricks.
- Migration of analytical and data science solutions, including conversion of R-based workloads and libraries.
- AI/ML experience, including LLM integration, RAG/vector retrieval, model serving, and model lifecycle management.
- Experience delivering large-scale platform transformation programmes.
- Strong Azure and Databricks FinOps experience, including cost optimisation, governance, and chargeback/showback models.
- Experience operating within highly regulated or complex enterprise environments.
If this sounds like a role you would be interested in or if you know someone in this field.
Connect with me or email me at pavitra.b@smartedgesolutions.co.uk
Alternatively, you can call me on Tel: +44(0)203 500 2108.
Trusted by 25,000+ job seekers
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Azure Databricks
Python
PySpark
SQL
Delta Lake
FinOps
Serverless Architecture
Data Engineering
Azure Networking
CI/CD Pipelines
REST API Integration
Cloud Governance
Cost Optimization
RStudio Migration
LLM Integration
Change Data Capture
Location