Data Annotator

With a startup spirit and 115,000+ curious and courageous minds, we have the expertise to go deep with the world’s biggest brands—and we have fun doing it. We dream in digital, dare in reality, and reinvent the ways companies work to make an impact far bigger than just our bottom line. We’re harnessing the power of technology and humanity to create meaningful transformation that moves us forward in our pursuit of a world that works better for people. Now, we’re calling upon the thinkers and doers, those with a natural curiosity and a hunger to keep learning, keep growing. People who thrive on fearlessly experimenting, seizing opportunities, and pushing boundaries to turn our vision into reality. And as you help us create a better world, we will help you build your own intellectual firepower. Welcome to the relentless pursuit of better.

Welcome to the relentless pursuit of better.

Inviting applications for the role of Data Annotator!

In this role, you will support the development, evaluation, and improvement of AI and large language model (LLM) to help drive the required output from model-driven AI applications. This role focuses on performing complex data annotation and content evaluation tasks, designing and refining prompts, contributing to quality standards, evaluation frameworks, and process improvements. The position is suited for professionals with hands-on experience in AI content review, language analysis, or data annotation and prompt engineering.

Responsibilities

• Execute annotation, evaluation, and validation activities for AI-powered solutions using established guidelines and quality standards.
• Validate AI-generated outputs against a predefined set of golden questions and expected results.
• Create, maintain, and review annotations to support AI model evaluation and continuous improvement.
• Test and evaluate Databricks Genie responses for accuracy, completeness, relevance, and consistency.
• Write and execute SQL queries against underlying data sources to validate answers generated from golden questions and Databricks Genie outputs.
• Investigate discrepancies between expected results, source data, and AI-generated responses, and document findings.
• Lead the design, optimization, and standardization of annotation processes and prompts across multiple AI use cases.
• Collaborate with AI teams to define and govern annotation schemas, taxonomies, evaluation frameworks, and quality standards.
• Review complex or high-risk AI outputs and provide expert quality assessments and recommendations.
• Ensure consistency, accuracy, traceability, and auditability of annotation and evaluation results.
• Define reusable prompt frameworks and best practices aligned with business, compliance, and responsible AI objectives.
• Evaluate prompt performance and annotation quality metrics, identifying opportunities for improvement through data-driven insights.
• Establish quality benchmarks, calibration processes, and escalation procedures.
• Identify bias, hallucinations, and other AI risks, and recommend mitigation strategies.
• Support AI governance, compliance, and responsible AI initiatives.
• Participate in calibration sessions to align on annotation standards, quality expectations, and evaluation criteria.
• Collaborate closely with AI engineers, data scientists, product teams, and business stakeholders.
• Drive continuous process improvement through learning new AI technologies, annotation methodologies, and delivery approaches.
• Mentor and coach junior and intermediate annotators and prompt engineers.