AI Recruiting Vendor Reference Call Questions

Q1: Can you briefly describe your organization — company size, the ATS you run, and what types of roles you use the platform for?

Q2: How long have you been on the platform? Are you still using it actively?

Q3: How long did implementation actually take — from contract signing to first production interview?

Q4: What was the biggest source of delay during implementation?

Q5: Who owned the implementation on your side — IT, HR, or a third party? How much internal resource was required?

Q6: Did the vendor's stated implementation timeline match the actual timeline?

Q7: What would you have done differently during implementation if you knew then what you know now?

Q8: How well does the ATS integration actually work? What data is written back to the ATS, and does it land correctly?

Q9: Have you experienced any ATS integration failures or data errors after go-live? How were they handled?

Q10: Has the integration required updates or fixes since go-live? What triggered those?

Q11: Is the trigger logic — how interviews are sent — working as expected? Any issues with interviews firing incorrectly or not at all?

Q12: What completion rate are you seeing across your main role types?

Q12a: How does the platform schedule interviews with candidates — does it book them live on a call, or does it send a link for candidates to complete later? Have you seen a difference in completion rates between those two methods?

Q12b: Have you used the multilingual capability? Does it allow candidates to switch languages mid-call without it being pre-configured, or does the language need to be set ahead of time?

Q12c: Have you used the ID verification feature? What was the candidate experience like — and did you see any friction or drop-off as a result?

Q13: Have you had candidate complaints about the interview experience? What were they?

Q14: Have you had any situations where candidates could not complete the AI interview due to accessibility or technology issues?

Q15: Has the candidate experience on the platform affected your employer brand or candidate survey scores in any measurable way?

Q16: Have you experienced platform downtime or reliability issues? How frequently?

Q17: Have you needed to roll back to manual screening at any point because the platform was unavailable?

Q18: Have there been product changes or updates that broke something in your workflow?

Q19: How responsive is support when you submit a ticket? Any examples of issues that took longer than expected to resolve?

Q20: Do you have a named account manager? How would you rate that relationship?

Q21: Has the vendor proactively communicated about product changes, API updates, or ATS compatibility issues that affected you?

Q22: Have you faced any compliance questions from your legal team or HR leadership about the AI interview — bias, consent, EEOC?

Q22a: Has the vendor ever triggered a fraud alert for a candidate in your account? How was that handled — and how confident are you in how the platform handles that situation?

Q22b: Have you had to walk a candidate through opting out mid-interview? Was that process clear and easy, or did it require intervention from your team?

Q23: Has the vendor been able to support compliance documentation requests?

Q24: Have you had any candidate disputes or escalations related to the AI interview outcome?

Q25: Is the platform delivering the ROI you expected? In what way?

Q26: Where has it delivered less value than you expected?

Q27: What is the primary use case you have settled on — and has that use case changed from what you originally purchased the platform for?

Q28: If you were making the buying decision again today, would you choose the same vendor?

Q29: What would you tell someone evaluating this vendor to verify — something they probably will not be shown in a demo?

Q30: Is there anything else you think I should know before making a decision?