Inference QA Engineer

Applied Computing

Applied Computing was founded in 2024 to build Orbital, a physics-informed foundation model for energy operations. We’re live across oil and gas, refineries, and petrochemicals, working towards our mission: sustainable abundance for a growing planet.

The hydrocarbon industry keeps the world running. But its complexity has left operators tied to legacy systems, making critical decisions on less than 10% of available data. We built Orbital to change that. It’s a foundation model built specifically for energy that lets companies use AI at scale, harnessing all of their operational data and optimising in real time for any metric. Decisions get faster, operations get safer, and carbon intensity falls.

We’ve raised over $32 million, including one of the largest seed rounds for an AI company in the UK. We’re just getting started.

About The Role

We build Orbital, an agentic LLM platform that answers hard operational and analytical questions for enterprise clients in energy and heavy industry. It routes questions across a graph of tools (data retrieval, SQL-generating coders, statistical analysis, anomaly detection, forecasting, RAG) and synthesises answers that engineers act on. When it is right, it compresses hours of analyst work into seconds.

When it is subtly wrong, quiet, or slow, that costs us trust with technical customers who check our numbers.

We are hiring a senior engineer to own inference quality end to end: the person who watches every deployment, decides what “working” means, builds the systems that prove it, and catches regressions before a client does. This is a founding-level QA function. You will define the discipline, not just execute a checklist.

What You’ll Own

• Evaluation frameworks. We test on a tiered model: T1 data retrieval, T2 statistical analysis, T3 open-ended inference and root-cause, each with its own pass threshold. You will own and extend this framework, design test sets with real ground-truth rubrics (expected answer, pass criteria, known failure modes, source tables), and set the bar for what ships.
• The test harness and its runs. We run prompts against live model endpoints at scale (tens of thousands of runs), on schedules and on demand, measuring consistency across repeated iterations, ground-truth match, structural correctness of generated queries, and paraphrase robustness. You will run these executions, keep the harness healthy, and turn raw runs into a verdict.
• Model and agent monitoring systems. You will build the observability layer for a multi-step agentic system: not just “did the endpoint return 200” but did the planner route to the right tool, did the tool actually execute, did the agent loop terminate, and is the final answer grounded.
• Concrete failure modes we already fight and want caught automatically:
  • Silent empty responses — the reasoning trace renders but the answer stays empty, while every layer reports success.
  • False refusals — a tool crashes or times out, returns nothing, and the model fabricates “I don’t have that data” instead of erroring loudly.
  • Tool-routing misses — the planner should have fired a tool and didn’t, or double-counts raw identifiers instead of canonical ones.
  • Latency and non-termination — multi-tool agent loops that blow past timeout budgets.
  • Paraphrase and run-to-run instability — the same question three ways, or the same prompt three times, giving materially different answers.
• Ground truth and the feedback loop. You will work with subject-matter experts to extract vetted ground truths from deployment feedback, feed them back into the eval sets, and close the loop so every confirmed defect becomes a permanent regression guard.
• Regression discipline. After every fix ships, you devise tier-appropriate tests biased at the failure mode plus regression guards, run them against the deployment, and report pass-rate against threshold and whether the original failure recurred. You are the gate.