AI Conversation Evaluation & Context Research Specialist

About the Role

We are seeking experienced Subject Matter Experts (SMEs) from psychology, social work, counseling, qualitative research, human-computer interaction (HCI), communication, conflict resolution, coaching, and related advisory fields to support AI conversation evaluation research. You will review conversations between people and AI models, assess whether the AI gathered sufficient context before making recommendations, identify missed opportunities for clarification, and provide structured feedback to improve conversational quality. The ideal candidate has at least 3–5 years of relevant professional experience, strong analytical and interviewing skills, and the ability to distinguish essential, decision-changing information from details that are less relevant to a recommendation.

Key Responsibilities

• Review AI conversations involving interpersonal relationships, workplace situations, education, family matters, everyday planning, wellbeing, and other real-world decisions.
• Evaluate whether the AI gathered sufficient context before providing advice, recommendations, or proposed solutions.
• Identify missing information and suggest relevant follow-up questions that could improve the quality of the AI's response.
• Assess whether additional context would materially change the recommendation or decision.
• Distinguish between load-bearing information that could change the outcome and non-essential or merely descriptive details.
• Conduct qualitative reviews using structured evaluation guidelines and document findings consistently.
• Identify missed opportunities for clarification, contextual understanding, and information gathering.
• Consider relevant personal, psychological, social, financial, practical, and organizational factors when evaluating conversations.
• Collaborate with research teams to maintain consistent evaluation standards and high-quality assessments.
• Complete assigned tasks accurately within the required project timelines.

Requirements

Candidates must have at least 3–5 years of direct professional experience in one or more of the following domains.

1. Licensed Social Work

• Master's degree in Social Work (MSW) or an equivalent graduate-level qualification.
• LCSW, LICSW, LMSW, or an equivalent applicable social work credential or license.
• Experience in client assessment, case review, interviewing, counseling, or related social-work practice.

2. Counseling, Therapy, or Psychology

• Master's degree in a relevant discipline with an LMFT, LPC, LCPC, or equivalent counseling or therapy credential; or
• PhD or PsyD in Clinical, Counseling, Social, or Industrial-Organizational Psychology.
• Relevant experience in counseling, therapy, psychological assessment, interviewing, case review, or other client-facing professional work.

3. Qualitative Research, HCI, or Communication

• MA, PhD, or equivalent advanced expertise in Human-Computer Interaction, Communication, Anthropology, Sociology, Qualitative Research, or a closely related discipline.
• Experience conducting qualitative research, interviews, user research, conversation analysis, thematic analysis, qualitative coding, or similar research activities.
• Ability to analyze conversations systematically and identify missing context or patterns in human communication.

4. Mediation, Coaching, Student Services, Organizational Consulting, or Related Advisory Fields

• Relevant professional qualification, credential, or demonstrated subject-matter expertise.
• At least 5 years of direct advisory or client-facing experience for candidates qualifying through this pathway.
• Experience in coaching, mediation, student advising, organizational consulting, or comparable professional work.
• Demonstrated ability to gather context, understand individual or organizational needs, and provide informed recommendations.