Fortune Best Workplaces in Financial Services & Insurance AI Evaluation Engineer

PRIMARY PURPOSE:

The AI Evaluation Engineer designs, builds, and operates the evaluation systems that determine whether our GenAI solutions are accurate, safe, reliable, and ready for production. You will serve multiple product pods, establishing consistent evaluation standards, automated pipelines, and quality gates, and you will help teams measure and improve LLM, RAG, and agentic applications throughout their lifecycle. Fully remote role, open to U.Establishes standardized, automated evaluation frameworks for LLMs and traditional ML models, including LLM as a judge patterns, ROUGE, METEOR, and statistical validation to ensure performance, reliability, and readiness prior to production release. Deploys comprehensive observability solutions to monitor model drift, hallucination risk, latency, throughput, and token consumption, balancing model quality with infrastructure cost and operational efficiency. Define evaluation methodologies and metrics for LLM, RAG, and agent-based applications (e.g., Partner with product owners and SMEs to translate business requirements into measurable quality criteria and acceptance thresholds. Develop automated evaluation pipelines integrated into CI/CD, with regression testing and quality gates for prompt, model, and retrieval changes. Implement offline evaluation (benchmark and golden datasets) and online evaluation (production monitoring, sampling, drift and quality alerts). Support A/B and shadow testing for model, prompt, and architecture changes. Build and curate golden datasets, adversarial and edge-case suites, and synthetic test data generation approaches. Apply approved data-handling practices (de-identification, synthetic data, access controls) for any dataset derived from sensitive or regulated data. Implement red-teaming, hallucination, toxicity, PII leakage, prompt-injection, and jailbreak testing. Produce evaluation evidence and reporting that supports security, privacy, compliance, and model risk reviews. Contribute to responsible AI standards and release-readiness criteria. Build dashboards and reporting that give engineering and business leaders transparent quality and cost/latency trends. Evaluate and recommend eval tooling and platforms; document standards and best practices. Travel as required. Master’s degree in a computer science or a quantitative discipline from an accredited college or university preferred. 4+ years in software, ML, or data engineering, with at least 1-2 years working hands-on with LLM-based applications (adjust to your market). A strong understanding of statistical validation, model evaluation metrics, and the ability to partner with Data Scientists to optimize model performance Strong Python skills and software engineering fundamentals (testing, version control, CI/CD, code review). Working knowledge of statistics and experimental design: sampling, confidence intervals, significance testing, inter-rater agreement. Experience with cloud platforms (Azure, AWS, or GCP) and their managed LLM services. Clear communication skills and the ability to work with engineers, product owners, SMEs, and risk stakeholders. Ability to meet or exceed Performance Competencies Background in ML evaluation, search/relevance, NLP, or QA automation. Experience with LLMOps, monitoring and observability, and production feedback loops. Familiarity with AI risk and governance frameworks (e.g., NIST AI RMF, ISO/IEC 42001) and secure development practices in regulated environments. Experience with data annotation workflows and working with SMEs. SSO, RBAC, private networking, logging, and encryption. Automated quality gates in CI/CD for production-bound solutions. Calibrated LLM judges with documented agreement against human labels. excellent judgment, troubleshooting, problem solving, analysis, and discretion; Computer keyboarding, travel as required Auditory/Visual: Hearing, vision and talking The statements contained in this document are intended to describe the general nature and level of work being performed by a colleague assigned to this description. They are not intended to constitute a comprehensive list of functions, duties, or local variances. Sedgwick is building a diverse, equitable, and inclusive workplace and recognizes that each person possesses a unique combination of skills, knowledge, and experience.