CyberFortis Consulting
AI Engineer — LLM Orchestration & AI Systems
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
About CyberFortis
CyberFortis Consulting Limited is a UK cybersecurity and technology company developing AI-enabled governance, risk and compliance solutions and expanding its capabilities in secure government and defence technology.
We are seeking an experienced AI Engineer specialising in Large Language Models (LLMs), AI orchestration, retrieval-augmented generation and agentic AI systems to support the development of advanced AI-enabled planning, knowledge exploitation and decision-support capabilities.
This is a hands-on engineering role. The successful candidate will be expected to design, build, integrate and evaluate production-oriented AI systems rather than simply experiment with prompts or consume existing AI APIs.
The role will involve working closely with software engineers, security engineers and company leadership to turn complex requirements into reliable, testable and auditable AI capabilities.
Role Purpose
The AI Engineer will be responsible for designing and implementing the AI application layer, including:
- LLM integration and model selection
- LLM orchestration
- Retrieval-augmented generation (RAG)
- Agentic workflows
- Tool and function calling
- Knowledge extraction and structured reasoning
- AI-powered analytical workflows
- Prompt and context engineering
- Model and workflow evaluation
- AI observability and performance optimisation
- AI security and guardrails
- Integration with enterprise data, APIs and software services
The successful candidate will help develop AI systems capable of processing complex information, reasoning over multiple sources, identifying relationships and dependencies, executing controlled analytical workflows and producing evidence-grounded outputs.
Key Responsibilities
LLM Engineering & Model Integration
- Integrate commercial and open-source LLMs into production applications.
- Evaluate and select models based on accuracy, latency, cost, context and operational constraints.
- Build reliable inference pipelines, structured outputs, fallbacks and context-handling workflows.
- Optimise prompts, monitor model performance and maintain abstraction from underlying models.
LLM Orchestration
- Design multi-step workflows for retrieval, reasoning, tool use and validated outputs.
- Manage state, branching, validation, recovery and deterministic controls.
- Build reusable orchestration components and integrate with APIs, databases and enterprise systems.
- Implement human-in-the-loop and controlled agent/tool execution.
- Advantageous: LangGraph, LangChain, LlamaIndex, Semantic Kernel, DSPy.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Retrieval-Augmented Generation
- Build production-quality RAG pipelines: ingestion, parsing, chunking, embeddings and vector search.
- Implement semantic, keyword, hybrid and access-aware retrieval.
- Improve precision/recall, apply reranking, assemble context and provide source attribution.
- Handle document versions, conflicting information and retrieval failures.
Knowledge Extraction and Structured Intelligence
- Convert unstructured information into structured, machine-readable knowledge.
- Extract facts, entities, relationships, dependencies, assumptions and constraints.
- Maintain provenance, detect contradictions and track version changes.
- Support knowledge graphs where appropriate, prioritising traceability over unverified LLM output.
Agentic AI and Tool Use
- Design controlled agentic workflows and function/tool calling.
- Integrate internal and external APIs with validation before and after execution.
- Set permissions, execution boundaries, failure handling and human approval gates.
- Maintain auditability of agent decisions, tool calls and outputs.
- Multi-agent or agentic system experience is highly desirable.
AI Evaluation and Testing
- Build repeatable evaluation datasets, metrics and automated pipelines.
- Measure accuracy, reasoning, retrieval quality, grounding, citation, hallucination and structured-output validity.
- Test agent/tool reliability and run regression testing after model, prompt or retrieval changes.
- Compare models objectively, maintain baselines and feed failures back into engineering.
- Desirable: RAGAS, DeepEval, Promptfoo, Hugging Face Evaluate.
AI Security and Responsible AI
- Assess prompt injection, indirect injection and unauthorised data disclosure risks.
- Control agent/tool permissions and protect sensitive context and system prompts.
- Implement validation, guardrails, logging, auditability and human oversight.
- Identify malicious sources and context poisoning; validate outputs before consequential use.
- Work with security engineers to ensure secure-by-design AI.
AI Data and Information Pipelines
- Design pipelines integrating structured and unstructured data sources.
- Manage preprocessing, normalisation, metadata, provenance, classification and access control.
- Implement retention/deletion and ensure only authorised data is exposed to models and workflows.
- Control sensitive data in prompts, context, outputs, logs and telemetry.
- Support reproducible ingestion and indexing.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
AI Application Engineering
- Develop production-quality Python services, APIs and microservices, e.g. FastAPI.
- Integrate AI services with existing systems and build asynchronous/event-driven workflows.
- Implement error handling, observability, testing, documentation and Git-based workflows.
- Containerise services and support CI/CD and automated testing.
Performance, Reliability and Cost Optimisation
- Monitor latency and inference performance; optimise token use and context construction.
- Balance model cost against performance and implement caching where appropriate.
- Design resilient workflows with retries, fallbacks and graceful failure handling.
- Identify bottlenecks across retrieval, orchestration, inference and downstream services.
Desirable Experience
- LangGraph, LangChain, LlamaIndex, Semantic Kernel, DSPy.
- RAGAS, DeepEval, Promptfoo, Hugging Face ecosystem.
- IBM watsonx.ai, IBM watsonx Orchestrate.
- Azure OpenAI / Azure AI, Anthropic APIs, OpenAI APIs.
- Open-source LLM deployment.
- Vector databases such as Milvus, Qdrant, Weaviate, pgvector or equivalent.
- Knowledge graphs / graph databases.
- Hybrid search and reranking.
- Multi-agent systems.
- AI security/red teaming.
- Kubernetes and Docker.
- Infrastructure as Code.
- Cloud-based AI architectures.
- Multimodal AI.
- Defence, government or other regulated-sector technology.
- Planning, optimisation, simulation or decision-support systems.
- Experience developing AI evaluation benchmarks.
CyberFortis Consulting Limited is an equal opportunities employer. We are committed to creating an inclusive and diverse workplace and welcome applications from all suitably qualified candidates.
All applicants will receive consideration for employment without regard to age, disability, gender reassignment, marriage or civil partnership, pregnancy or maternity, race, religion or belief, sex, sexual orientation, or any other protected characteristic under the Equality Act 2010.
We will make reasonable adjustments to support candidates throughout the recruitment process.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location