Inference QA Engineer

• The test harness and its runs: We run prompts against live model endpoints at scale (tens of thousands of runs), on schedules and on demand, measuring consistency across repeated iterations, ground-truth match, structural correctness of generated queries, and paraphrase robustness. You will run these executions, keep the harness healthy, and turn raw runs into a verdict.

• Model and agent monitoring systems: You will build the observability layer for a multi-step agentic system: not just “did the endpoint return 200” but did the planner route to the right tool, did the tool actually execute, did the agent loop terminate, and is the final answer grounded.

Concrete failure modes we already fight and want caught automatically:

• Silent empty responses — the reasoning trace renders but the answer stays empty, while every layer reports success.

• False refusals — a tool crashes or times out, returns nothing, and the model fabricates “I don’t have that data” instead of erroring loudly.

• Tool-routing misses — the planner should have fired a tool and didn’t, or double-counts raw identifiers instead of canonical ones.

• Latency and non-termination — multi-tool agent loops that blow past timeout budgets.

• Paraphrase and run-to-run instability — the same question three ways, or the same prompt three times, giving materially different answers.

• Ground truth and the feedback loop: You will work with subject-matter experts to extract vetted ground truths from deployment feedback, feed them back into the eval sets, and close the loop so every confirmed defect becomes a permanent regression guard.

• Regression discipline: After every fix ships, you devise tier-appropriate tests biased at the failure mode plus regression guards, run them against the deployment, and report pass-rate against threshold and whether the original failure recurred. You are the gate.

What we’re looking for

• 5+ years in software, ML, data, or QA engineering, with real ownership of a quality-critical system.
• Strong Python. Comfortable in a FastAPI + Postgres + Docker world, reading logs across services and tracing a request through a distributed pipeline.
• Fluency with LLM behaviour: prompting, tool/function calling, agentic loops, RAG, and the ways they fail (hallucination, refusal, silent truncation, non-determinism).
• Experience designing evaluation: LLM-as-judge, deterministic checks, ground-truth scoring, statistical consistency measures (e.g. coefficient of variation across repeated runs).
• SQL literacy — you can read a generated query and judge whether it answers the question and hits the right tables.
• A monitoring and observability instinct: you reach for dashboards, alerts, and trace inspection by default, and you build them when they don’t exist.
• Rigour about uncertainty. You report calibrated ranges, not overclaimed point estimates, and you say plainly when something is unverified.

Bonus

• Experience evaluating or red-teaming agentic / multi-tool LLM systems specifically.
• MLflow or similar trace and experiment tooling.
• A background talking to technical end users (engineers, analysts) and translating their “it feels off” into a reproducible test.
• Time-series, forecasting, or industrial and operational data domains.

Why it matters

Our customers are engineers who verify our output. Inference quality is the product. This role decides whether we can look a client in the eye and say the system works, and back it with numbers. You will have the mandate to build that assurance layer from the ground up.