Cognitive Group | Part of the Focus Cloud Group
SC CLEARED Azure Databricks Technical Consultant
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Azure Databricks Technical Consultant
We are looking for an experienced Azure Databricks Technical Consultant to design and deliver secure, scalable data and analytics solutions within a complex, highly regulated environment.
The consultant will have a strong background in Azure Databricks and modern data engineering, working across large-scale data ingestion, transformation, analytics and AI workloads. The role will involve integrating structured and unstructured data from multiple sources, including telemetry, operational datasets, OSINT and SOCMINT.
Key Responsibilities
- Design, build and maintain scalable data pipelines using Azure Databricks, Spark/PySpark and Azure Data Lake Storage
- Ingest, transform and process large volumes of structured, semi-structured and unstructured data from multiple sources
- Develop secure Lakehouse architectures and reusable data engineering frameworks within Azure
- Design and optimise data models, analytical datasets and feature stores supporting advanced analytics and AI use cases
- Integrate Databricks with Azure Machine Learning, Azure OpenAI and wider Azure data services
- Support AI/ML and agentic AI workloads including RAG, LLMs and autonomous agent architectures
- Develop analytical solutions incorporating machine learning, NLP, graph analytics, clustering, entity resolution and pattern analysis
- Implement MLOps, AIOps and SecML processes covering model monitoring, governance, audit logging, explainability, drift detection and rollback
- Work with graph technologies including Neo4j, Azure Cosmos DB Gremlin API and Microsoft Graph Data Connect
- Build reporting, dashboards and analytical outputs using Power BI and associated Microsoft technologies
- Ensure data platforms meet stringent requirements around security, governance, Zero Trust, compliance and Responsible AI
- Conduct functional, system, performance and non-functional testing across data and AI solutions
- Produce solution designs, technical documentation, implementation plans and operational support documentation
- Collaborate with data scientists, cloud architects, software engineers, analysts and business stakeholders within Agile delivery teams
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Essential Skills & Experience
- Strong hands-on experience with Azure Databricks
- Excellent Python/PySpark and SQL data engineering skills
- Strong experience designing and developing ETL/ELT and large-scale data pipelines
- Experience with Apache Spark and Databricks Lakehouse architectures
- Strong knowledge of Azure Data Lake Storage (ADLS)
- Experience integrating Databricks with wider Azure data and analytics services
- Understanding of data modelling, data quality, governance and performance optimisation
- Experience working with Azure Machine Learning and/or Azure OpenAI
- Strong understanding of cloud security, identity, access controls and Zero Trust principles
- Experience working within Agile/Scrum delivery environments
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Desirable Experience
- Agentic AI, autonomous agents, RAG and Large Language Models
- MLOps and productionisation of machine learning workloads
- Graph databases and graph analytics including Neo4j or Cosmos DB Gremlin API
- Power BI and analytical visualisation
- Feature stores and ML data pipelines
- NLP, clustering, entity resolution and causal inference
- Responsible AI and AI governance frameworks
- Previous experience within Defence, Government, National Security or similarly regulated environments
This is an Inside IR35 PAYE contract role so you must be resident in the UK to be considered for this role and you must hold ACTIVE SC CLEARANCE
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location