AI Video Annotation Quality Lead

AI Data Quality Lead – Robotics & Video Annotation | Remote

About the Role

We are looking for an experienced AI Data Quality Lead to own the quality standards for egocentric training data used to develop and evaluate physical-AI and robotics models.

You will be responsible for building and maintaining the golden sets, annotation SOPs, rubrics, calibration systems, quality frameworks, and audit processes that ensure high-quality ground-truth data. You will also work closely with engineering and product teams to develop annotation workflows for hand tracking, object identification, and action labeling.

This is a ground-floor opportunity to build a 0→1 data quality system at a fast-growing, well-funded startup. You will not simply inherit an existing process—you will help establish the quality standards, workflows, and systems from the ground up.

Key Responsibilities

Own the end-to-end quality framework for egocentric training data, including

Golden and benchmark datasets

Acceptance criteria

Frame-level tolerance thresholds

Sampling strategies

Audit and quality-control loops

Author, maintain, and version annotation SOPs, rubrics, and edge-case guidelines as project specifications evolve.

Translate ambiguous requirements into precise, actionable instructions for annotators and reviewers.

Establish and manage annotator training, certification, and calibration programs.

Run calibration sessions and adjudicate disagreements between annotators.

Measure and report quality using metrics such as

Inter-Annotator Agreement (IAA)

Kappa/alpha

Tolerance-based agreement for temporal boundaries

Acceptance/rejection rates

Rework rates

Cost per accepted hour

Identify root causes of quality issues and determine whether they result from guideline ambiguity, annotation errors, or workflow issues.

Design human-in-the-loop workflows where AI/model-generated labels are reviewed, corrected, and escalated by human annotators.

Audit auto-generated labels and improve annotation throughput without compromising quality.

Partner with engineering and product teams to develop new annotation workflows for

Hand tracking

Object identification

Action labeling

Define label schemas, annotation-tool requirements, QC dashboards, and pilot-validation processes.

Validate annotation workflows through pilot programs before scaling them into production.

Work with ML teams to understand how annotation quality impacts model performance.

Contribute to tooling and workflow improvements where required.

Required Skills & Qualifications

5+ years of hands-on video annotation experience, including at least 1+ year working with egocentric/first-person video for robotics or embodied AI.

3+ years of experience in a lead or QA capacity, including annotator calibration, disagreement adjudication, and ownership of annotation guidelines.

Demonstrated experience building 0→1 annotation programs and taking them from pilot through production.

Strong command of annotation-quality methodologies, including

IAA frameworks

Kappa/alpha

Gold-set creation

Sampling strategies

Tolerance thresholds

Acceptance criteria

Ability to design quality systems and explain the practical difference between quality targets such as 95% and 99% accuracy.

Proven experience authoring, rather than simply following, annotation SOPs and rubrics.

Ability to translate ambiguous specifications into clear and actionable annotation guidelines.

Hands-on experience with at least 2 professional annotation platforms, such as

CVAT

Labelbox

Encord

V7

Label Studio

Equivalent annotation tools

Experience configuring label schemas and QA workflows within annotation platforms.

Experience auditing model-generated or automatically generated labels.

Experience designing human-in-the-loop annotation workflows.

Strong written and verbal English communication skills.

Strong analytical skills and proficiency with Excel and/or Google Sheets for quality reporting.

Strong understanding of ML concepts and how annotation quality affects model performance.

Data-driven and process-oriented mindset with strong ownership.

Comfortable working in ambiguous, fast-moving, startup environments.

Strong ability to work collaboratively with cross-functional and remote teams.

Preferred / Bonus Qualifications

Background in robotics, mechanical engineering, mechatronics, or computer vision engineering.

Experience working at a leading robotics company, AI/robotics startup, or similar project.

Experience managing distributed or global annotator workforces or external annotation vendors.

Experience with multimodal annotation, including

• 3D/depth data
• Joint pose
• Grasp outcome classification
• Experience training or fine-tuning autolabeling models.
• Experience partnering closely with ML teams responsible for automated labeling.
• Experience building annotation tooling or working closely with annotation-tooling engineering teams.
• Experience with physical AI, embodied AI, computer vision, or robotics datasets.
• Why This Role?
• The quality bar is the product.
• The ground-truth data produced through this role will become a benchmark against which physical-AI models are measured. You will have direct ownership of the systems that determine whether that data meets the required quality standards.
• This is a ground-floor opportunity at a fast-growing, well-funded startup where you will build the quality system from scratch rather than inherit an established process.
• Role Highlights
• Role: AI Data Quality Lead – Robotics & Video Annotation
• Work Type: Remote
• Engagement: [Contract / Full-Time – specify as applicable]
• Focus: AI Training Data, Robotics, Embodied AI & Video Annotation
• Experience: 5+ years relevant experience