AWS Details the Research Behind Amazon Connect Talent’s AI Interviews

AWS has outlined the research and design principles behind Amazon Connect Talent, an AI interview service for organizations hiring hundreds or thousands of people. The service uses structured questions, adaptive follow-ups and evidence-based scoring to collect job-related evidence while giving candidates more opportunities to explain their experience.
- Connect Talent creates structured interviews around role-specific competencies.
- Adaptive follow-ups seek additional evidence when candidate answers lack detail.
- Competency scores range from 1 to 5 and are based on a fixed formula.
- AWS reports 89.3% alignment with independent expert scoring decisions.
- The service excludes video, voice tone and physical appearance from competency scoring.
Interviews built around job-related evidence
Amazon Connect Talent creates interviews around the competencies required for a role. According to AWS, it uses job requirements and established occupational frameworks to create behaviorally anchored scorecards, with each question targeting specific behaviors and each scorecard defining strength and concern signals.
The AI interviewer reviews every answer to determine whether it contains enough evidence for the competency being assessed. If important details are missing, it asks targeted follow-up questions. AWS says this preserves the structure of the interview while allowing candidates to provide more information about their experience.
The company also says hiring teams can review, select and edit the generated competencies, scorecards and questions before using them. This human oversight gives organizations control over the interview process, according to AWS.
Interviewing and scoring are separated
Connect Talent separates the interview conversation from the evaluation process. The AI interviewer gathers evidence but does not assign scores. A separate evaluation process identifies the behaviors demonstrated by the candidate and cites supporting transcript passages.
Each competency receives a score from 1 to 5 using a fixed formula. When a transcript does not contain enough evidence, the service flags the competency instead of producing a score. AWS says missing evidence does not automatically count against the candidate, and hiring teams can review the evidence behind each rating.
The service also evaluates transcribed responses without using video, voice tone or physical appearance to calculate competency scores. Candidate identity information is excluded from scoring, and each competency is evaluated separately. AWS presents these measures as protections designed to keep assessments focused on job-related behavior.
Research findings from expert reviews
AWS engaged an independent industrial-organizational psychology firm to assess Connect Talent’s baseline interview content before human editing. The firm found that the service retained critical job information in 100% of the job descriptions reviewed. Of the generated competencies, 91% were clearly traceable to the job description, while 98% of interview questions targeted the intended competencies and 94% of rubric signals clearly measured the intended competency.
In another evaluation, the firm reviewed 36 interviews and 252 rubric signals. Connect Talent aligned with independent expert scoring decisions in 89.3% of cases. Alignment reached 97.6% when candidates provided clear evidence, and 92.9% when the review focused only on candidate concern signals. The evaluation produced a Cohen’s kappa of 0.79, which AWS describes as substantial agreement between the service and independent expert scoring.
AWS also compared evaluations based on candidates’ first responses with evaluations based on full conversations. In a preliminary evaluation, scores based only on the first response were up to 1.14 points lower on a five-point scale than scores based on the full conversation. The company says follow-up questions helped identify more strength signals, fewer concern signals and a more complete record of candidates’ experience.
Finally, AWS says Connect Talent’s structured scoring used all five points of its rating scale across hundreds of expert-rated assessments. By contrast, holistic LLM-as-judge scoring concentrated ratings within the middle three points. The findings support AWS’s claim that behaviorally anchored scoring can help hiring teams distinguish between different levels of demonstrated competency.


