Beacon
Senior Data Engineer – Data Products & AI
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
About Beacon
Beacon Intelligence delivers data and insights that help R&D scientists develop better pharmaceuticals, faster than ever before.
Our focus is simple: we curate high-accuracy data enriched and transformed with AI. This is delivered through a range of platforms, increasingly via AI-native products and features that enable customers to extract insights faster than ever before.
This is a rare opportunity to make a meaningful impact on patients’ lives worldwide.
The Role
As a Senior Data/AI Engineer, you will play a central role in building and scaling our data products.
This is not a traditional back-end data engineering role. You will work closely with product, commercial and technical colleagues, transforming datasets into commercial, customer-facing solutions.
You will combine deep technical expertise with a strong understanding of how data can be structured, enriched and operationalised through AI systems.
What You’ll Be Doing
Data Product Development (Core Focus)
- Design and build scalable data products underpinning Beacon’s commercial offerings
- Transform raw and third-party data into structured, enriched, product-ready datasets
- Partner with product and commercial teams to define how data is packaged, accessed and monetised
- Enable delivery via APIs, internal tools and customer-facing platforms
AI & Data Enrichment
- Apply AI and LLM capabilities to enrich and enhance data (e.g. classification, tagging, summarisation, insight generation)
- Design and build LLM-powered product features
- Integrate RAG into pipelines to improve data quality and unlock new features
- Support development of AI-driven products such as recommendation engines, search and insight tools
- Ensure systems are optimised, well-governed and resilient
Platform & Engineering
- Support deployment and lifecycle management of data and AI systems
- Own and evolve the data platform architecture (Databricks, Azure, Airbyte) to support scalability
- Own performance and cost optimisation across Azure Databricks and supporting Azure services, including compute selection, autoscaling, workload monitoring and resource-efficiency improvements.
- Build and maintain robust data pipelines (batch and streaming) delivering reliable, production-ready datasets
- Ensure high standards in data quality, testing and observability
- Improve efficiency through automation, CI/CD and engineering best practice
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Commercial & Stakeholder Impact
- Partner with senior stakeholders to identify high-value data product opportunities
- Translate business needs into practical, scalable data and AI solutions
- Act as a bridge between technical teams and commercial/product stakeholders
- Contribute to prioritisation based on business impact
Leadership & Growth
- Shape the evolution of our data product and AI strategy
- Mentor junior engineers and promote a strong engineering and product mindset
- Help build a high-performing, commercially aware data and AI engineering team
What We’re Looking For
Core Engineering Experience
- Deep, hands-on experience designing, building and operating production data platforms using Azure Databricks and Azure Cloud environments
- Strong experience with Apache Spark, PySpark, Spark SQL and Delta Lake, including incremental processing, schema evolution, performance optimisation and batch and streaming workloads.
- Strong experience implementing Unity Catalog, including catalog and schema design, access control, data lineage, discovery and governance of production data assets.
- Proven experience designing lakehouse and Medallion architectures and converting raw and third-party data into governed, reusable data products.
- Advanced Python and SQL, focused on production-quality, scalable solutions
- Proven experience designing and building data pipelines and models
- Experience working with APIs and data delivery mechanisms (critical for productisation)
- Experience deploying Databricks workloads through automated CI/CD, preferably using Azure DevOps or GitHub Actions, automated testing and Databricks Declarative Automation Bundles.
- Experience integrating Azure Databricks with ADLS Gen2, Azure identity, Key Vault, APIs and external data sources.
Data Product Mindset (Key Differentiator)
- Experience building or supporting data products or customer-facing data solutions
- Strong understanding of how to structure, expose and scale data for end users or clients
- Ability to think beyond pipelines and focus on commercial value
- Experience working closely with product or commercial teams
Get help with your application
Your very own career expert that helps elevate your application to the next level.
AI Engineering
- Familiarity with generative AI / LLMs (Azure OpenAI, LangChain etc.) and experience applying them to real-world use cases
- Experience evaluating, monitoring and optimising AI systems
- Experience working with unstructured data (text, documents, web data)
- Familiarity with embeddings, vector databases or semantic search
- Understanding of AI governance and data ethics
Desirable
- Experience with Airbyte or modern data ingestion tools
- Experience with Terraform or infrastructure-as-code
- Exposure to LLM frameworks (e.g. LangChain, LlamaIndex)
- Experience with experimentation or A/B testing frameworks
- Background in building data-driven / AI-enabled products
- Domain experience in life sciences
How You Work
- Focused on shipping products, not just experimentation
- Think in terms of products, not just pipelines
- Commercially aware and motivated by impact
- Combine technical excellence with pragmatism and pace
- Collaborate effectively across disciplines and influence stakeholders
- Curious about AI and emerging technologies, with a focus on reliable, scalable delivery
Why Join Beacon Intelligence
- Build products, not just infrastructure – directly helping scientists treat and cure diseases
- Shape our AI future – define how we embed AI into products and workflows
- High-impact role – influence product strategy and commercial performance
- Modern, evolving stack – work with cutting-edge tools across data engineering and AI
- Growth and ownership – join early and grow as we scale our capabilities
- Collaborative environment – stakeholders who actively value data and insight
Location & Working Pattern
London (hybrid working) – 2 days per week minimum in office
Flexible working arrangements available
Time Spent (Directional)
- 60% hands-on technical
- 20% management and mentoring
- 20% exploration
Pay Range and Compensation Package
Salary range: £80,000 to £90,000 + Bonus
Equal Opportunity Statement
Beacon Intelligence is committed to diversity and inclusivity in the workplace.
Apply
If you’re excited about turning data into products and building AI-enabled solutions that drive real commercial value, we’d love to hear from you!
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location