Software Engineering Expert — AI Model Evaluation · Contract · Remote

Role Overview

Huzzle is seeking software engineers to test how well frontier AI systems handle real engineering work, and to document where they fall short.

You will not be building products. You will be finding the point at which an AI system stops being reliable at engineering, and proving that point objectively.

Each submission has three parts: a realistic technical request, documented evidence of where the model failed it, and a scoring rubric that lets any reviewer grade a future attempt consistently.

Because this is code, most rubric criteria are verified by running it. The grading is unusually objective, which is what makes the work satisfying.

Full training and worked examples are provided. Prior AI or data-annotation experience is not required.

Key Responsibilities

• Test technical requests against a frontier AI model and review its output with the scrutiny you would apply to code you were merging.
• Document exactly where and how it failed, with the line, the failing condition, or the missing behaviour that demonstrates it. Precision matters more than volume.
• Author scoring rubrics with objective criteria another engineer could apply without repeating your work — ideally by executing the code against a named case.
• Write the underlying requests: realistic technical tasks requiring live research or real web interaction — actual API behaviour, current documentation, published licences, real datasets — plus implementation and a working file deliverable. Typically a Python script, but also CSVs, config, SQL or technical memos.
• Escalate task difficulty where the model succeeds, until a genuine gap is exposed. Note that models are strong at conventional coding problems; the gaps are in edge cases, real-world API behaviour and constraint discipline, not in algorithms.
• Log hours accurately and respond to review feedback within the working day. This is a two-week sprint.