Chaseit.
AI Solutions Engineer
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
About Chaseit
Chaseit builds AI voice agents for loan servicing and collections. Our agents make tens of thousands of human-like calls every day for lenders across Europe and beyond—automating everything from payment reminders to payment-plan negotiation, in 10+ languages, while staying compliant and genuinely empathetic.
We are a VC-backed startup with live customers, real revenue, and a lean team where every person has outsized impact.
What You Will Do
This is not a prompt-tweaking role, nor plain AI programming. As an AI Solutions Engineer, your focus is one question: Are our agents measurably better at the outcomes our customers care about?
You’ll work alongside our Deployment Strategists, who manage customer relationships and define success in production—payment rates, promise-to-pay, resolution, containment, escalations. Your responsibility is to turn these targets into a proactive, systematic improvement engine:
- Own target metrics for live deployments alongside Deployment Strategists, treating their progress as the job’s sole focus.
- Form and prioritise hypotheses about what limits agent effectiveness (e.g., conversation design, prompts, model choices, tooling, latency, handoff logic).
- Design and execute experiments and A/B tests on real call traffic:
- Define experimental shapes (treatments, controls, success metrics, guardrails, sample sizes)
- Analyze results rigorously, including failures
- Build and own automated improvement workflows:
- Evaluate every prompt, flow, and model before shipping
- Implement regression tests to catch quality drops
- Set up online monitoring to auto-surface issues and metric regressions
- Develop evaluators and evaluation datasets combining LLM-as-judge and deterministic checks to capture what “good” calls sound like in real collections scenarios.
- Mine production logs for failure clusters and high-impact backlogs, converting these into new evaluation cases and experiments.
- Close the loop by:
- Shipping winning changes
- Quantifying impact
- Documenting lessons to feed into future agent development
- Construct the tooling that accelerates agent growth with confidence for the entire team.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Your work will range from deep experiment design and iterative testing to real-time troubleshooting when production metrics degrade. You’ll thrive in both scenarios.
Why This Role Exists
The reactive approach to improving an AI agent is straight-forward: fix visible issues as they arise. This preserves quality and builds trust—but reactions to surface-level problems underestimate where actual gains lie.
High-impact opportunities sit beneath the data surface: patterns observable across tens of thousands of calls, not outliers. Critical improvements require experiments—not assumptions. That’s the window this role opens. We need someone who starts from metrics, identifies flaws, rigorously tests hypotheses, and delivers measurable victory—fresh evidence, not intuition.
Each iteration deals with same mantra: Hypothesis → Experiment → Real-World Impact.
Who You Are
Requirements
- 3+ years building with LLMs or production AI/ML systems, with a track record of measurable performance gains—not hypothetical success.
- Python fluency for:
- Coding eval pipelines
- Running experiments
- Writing scalable data pipelines (TypeScript is a plus)
- Mastery of evaluation and experiments:
- Offline vs. online evaluation
- LLM-as-judge + code-based graders
- A/B testing, experimentation platforms (e.g. Statsig)
- Technical comfort with quantitative reasoning:
- SQL proficiency
- Analyzing logs and transcripts
- Building dashboards that infer insights from raw metrics
- Product sense blended with extreme ownership:
- Articulate success criteria for indeterminate problems
- Deliver outcomes from hypothesis to deployment with autonomy
- Cool under pressure:
- Adapt alongside shifting priorities and volatile metric swings.
- Exceptional written communication: Explicit clarity for both technical and non-technical audiences.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Nice to Have
- Experience with conversational/voice AI, QA for chat/chatbots, call-center systems
- Background in lending, collections, payments, or regulated fintech
- Familiarity with:
- Experimentation platforms (e.g. Arize Phoenix, LangSmith)
- Agent orchestration frameworks
- Prompt engineering
- Competence with:
- Linear + Notion
- Advanced multilingualism (beyond English)
- Preference (or permutation tolerance) in high-growth or seed-phase startup teams
What You’ll Get
- Salary: €40,000–€70,000 gross/year, scaled to experience.
- Equity: Early team access to stock options. Your experiments build what we share.
- Collaboration: Co-create with the founding team—your work directly reshapes product and roadmap.
- Impact visibility: Daily leverage at enterprise scale, production-first deployment moments.
Location: Remote or hybrid (Vilnius, Lithuania).
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location