AI systems and evaluation interview questions
For AI rolesBuilding with models in production: retrieval, agents, guardrails, and above all how you prove the system works.
- A customer asks how you know your AI system works. Answer them.
- How do you choose a chunking strategy for contracts compared with support tickets?
- How do you make an assistant cite its sources, and how do you check the citations are real?
- Retrieval finds the right document in the top twenty but not the top three. What do you do?
- When would you fine-tune, when would you use retrieval, and when is prompting enough? Use a real example.
- A support agent reads customer emails and can issue refunds. How do you defend it against prompt injection?