Gmail & Google Calendar AI Assistant Evaluator

Job description

Role Title: Gmail & Google Calendar AI Assistant Evaluator

Role Type: Contractor, Remote

Location: United States only

micro1 is engaging Gmail & Google Calendar AI Assistant Evaluators to collaborate on a customer-driven project enhancing AI assistant quality through real-world task evaluation. In this role, you'll use your everyday experience managing your digital life — email, calendars, bookings, and coordination — to help train and evaluate next-generation AI assistants. Your work will shape how AI models handle the real tasks people delegate every day.

Key Responsibilities:

- Complete realistic, everyday digital tasks (emailing, scheduling, booking, coordinating) using AI assistant tools under standardized, repeatable conditions.

- Give an AI assistant clear instructions for real-world tasks, then carefully review and verify its work for accuracy and completeness.

- Apply detailed grading guidelines to evaluate, rank, and annotate assistant responses, providing structured feedback that improves model performance.

- Test scenarios involving email threads, calendar invites, online bookings, and multi-person coordination, ensuring coverage of diverse situations and edge cases.

- Utilize AI training tools and platforms to record findings and submit evaluation data in alignment with project standards and milestones.

- Collaborate with project trainers and contributors to resolve ambiguities, refine guidelines, and ensure evaluation consistency across the team.

- Participate in ongoing quality reviews, incorporating feedback to maintain rigorous evaluation standards.

Required Skills and Qualifications:

- Are based in the US, have an iPhone with iMessage, and have a Facebook or Instagram account.

- Daily, hands-on use of Gmail and Google Calendar to run your own life: sending emails, accepting invites, scheduling, and rearranging plans.

- Regular experience coordinating with others over email or text — setting up meetups, following up on threads, or confirming shared plans and payments.

- Comfort booking things online independently, such as restaurants, appointments, travel, or deliveries.

- Experience using AI tools for real, practical tasks, with comfort giving an AI assistant instructions and checking its work.

- Exceptional attention to detail and accuracy when reviewing AI-generated output.

- Strong written communication skills for clear reporting and timely updates on progress and challenges.

- Demonstrated time management and self-organization for independent, remote project participation.

Preferred Qualifications:

- Prior experience with AI evaluation, data labeling, user testing, or similar feedback-driven projects.

- Familiarity with prompt writing and a practical understanding of AI assistant behaviors and limitations.

- Experience grading or reviewing AI outputs for accuracy, helpfulness, and safety.

Originally posted on Himalayas