AI Retrieval & Data Layer Expertise

This is a senior AI data engineering role, so these are skills you already have and can demonstrate, not areas you will grow into on the job:

• Retrieval literacy. You understand how an AI agent finds and uses information, and how that goes wrong. You know retrieval at the level that lets you debug it when it fails in production.
• Unstructured data engineering. You turn documents, not just databases, into data an agent can read: parsing, layout analysis, table extraction, and chunking strategies that reflect how information will actually be retrieved.
• Retrieval pipelines, end to end. You build the pipeline that fetches the right material the moment an agent needs it — chunk, embed, store, retrieve — and you own every step of it.
• Retrieval quality engineering. You make retrieval return the right answer, not just an answer: hybrid search, rerankers, contextual retrieval, and metadata filtering. You know the difference between a system that works in a demo and one that holds up in production.
• The semantic layer. You build governed metrics so an agent reports approved numbers instead of inventing them. This matters especially in healthcare, where data accuracy is not optional.
• Evals and golden datasets. You engineer the eval sets that define what good looks like and gate quality in CI, building the infrastructure that makes the team confident in every release.
• Securing and governing retrieval. You enforce access at the data layer, so an agent only ever retrieves what a person is allowed to see. In a healthcare environment this is foundational, not a nice-to-have.
• Advanced retrieval. You work with knowledge graphs for multi-hop questions, agent memory, and the interfaces that connect the retrieval layer to the agents that consume it.

Nice to Have

• Databricks experience. Useful context, not a hard requirement.
• Snowflake background. If you have worked with Snowflake and are now applying those skills in an AI context, that translates well.
• Direct experience with AWS Bedrock Knowledge Bases and how data ingestion pipelines feed into them.
• Prior exposure to a regulated or healthcare data environment (for example HL7 / FHIR), though a serious attitude to governance matters more than the domain.

DAY TO DAY

Data Engineering for AI

• Design and build pipelines that transform unstructured enterprise data into clean, structured, agent-consumable formats.
• Develop and refine chunking and parsing strategies that reflect how data will actually be retrieved and used by AI systems in production.
• Build and maintain ingestion pipelines that feed into AWS Bedrock Knowledge Bases and other retrieval infrastructure.
• Instrument the data layer so that quality, freshness, coverage, and access controls are measurable and can be monitored over time.

Working in the Delivery Team

• Pick up and progress work continuously within a Kanban flow, managing your own priorities and communicating blockers early.
• Work closely with AI engineers, architects, and delivery managers to ensure the data layer supports what the AI systems actually need.
• Use AI-assisted development tools as a natural part of how you work, not as an afterthought.
• Pair with and coach data engineers who are moving into the AI data layer, reviewing their work and helping them build retrieval, chunking, eval, and governance skills on real delivery.
• Document what you build: pipeline logic, data contracts, transformation decisions, and lessons learned for the wider team.

Raising the Bar

• Hold your work to high engineering standards: tested, reproducible pipelines with clear ownership and documentation.
• Set the standard for the data layer and actively bring other engineers up to it. Growing the team's AI data engineering capability is part of the job, not a side effect of it — through pairing, code review, and sharing what good looks like.
• Bring ideas from previous roles and toolsets. If you have seen a better way to solve a data problem elsewhere, say so.
• Contribute to how Blackstone& thinks about the data layer in AI delivery. We are building practice here and your input matters.

WHO WILL THRIVE HERE

This role will suit you if you:

• Think about data from an AI-first perspective: not just how to store and query it, but how an agent will consume it.
• Have already owned a production retrieval and data layer, and can point to what you built and what you learned when it broke.
• Have worked across different companies and different tooling stacks, and carry those experiences as an asset rather than a constraint.
• Are comfortable with AI-driven development and use it naturally in your day-to-day work.
• Thrive in a continuous flow environment. You do not need a sprint to get moving.
• Are clear and direct when communicating with technical and non-technical colleagues alike.
• Want ownership of a critical layer in AI delivery, not just a seat at the table.
• Get satisfaction from levelling up the engineers around you, and see growing a strong data engineer into an AI data engineer as a genuine part of the work.
• Are comfortable operating in a sensitive, regulated data environment. Healthcare experience is not required. A serious attitude toward data governance is.

ENGAGEMENT DETAILS

This engagement is offered outside IR35, reflecting the genuine independence and expertise the role requires. Key terms:

• Rate: Competitive, reflecting the seniority and expertise of the role
• Outside IR35 — engaged as an independent contractor via your own limited company or umbrella
• Initial contract length: 6 months, with extensions expected subject to performance and project progression
• Hybrid working — a mix of remote and on-site with the client as the engagement requires
• Start date: to be agreed. We are looking to move quickly for the right person

Blackstone& is an equal opportunities employer. We welcome applications from all backgrounds. www.blackstoneand.com