CoreHRx Jobs

Apply in one click, save jobs and take free mock tests.

Register free Log in
Hiring? Post jobs with CoreHRx
My account
user

View & update profile

AI Evaluation Engineer

Vinylequity Verified employer

Full time On-site
Location
Mangalore
Work mode
On-site
Posted 1 month ago

Free · takes a minute · your resume fills your profile

CoreHRx never asks candidates for money. If anyone does, .

Role at a glance

You will
  • As an AI Evaluation Engineer, you'll own the quality bar for both our AI-powered features and the broader product experience they sit…
  • You'll design and run evaluation frameworks that catch regressions in model behavior (accuracy, hallucination, safety, compliance-sensitive…
  • This is a hands-on, blended role: part evaluation engineering for LLM/AI systems, part product quality ownership.
Must-have skills
Automation TestingCI/CDJavaPrompt EngineeringPythonSQL
Work mode
On-site

How well do your skills match this job?

Sign in to see your fit: the skills, experience and location this job asks for, checked against your profile.

Job description

Location: Mangaluru Office
Department: Product & Engineering

About the Role

As an AI Evaluation Engineer, you'll own the quality bar for both our AI-powered features and the broader product experience they sit inside. You'll design and run evaluation frameworks that catch regressions in model behavior (accuracy, hallucination, safety, compliance-sensitive edge cases) as well as classic product/QA issues — so that what we ship to customers handling real securities and compliance data is trustworthy every time. This is a hands-on, blended role: part evaluation engineering for LLM/AI systems, part product quality ownership.

What You'll Do

  • Design, build, and maintain evaluation frameworks and benchmark suites for AI/LLM-powered features (e.g., document extraction, compliance checks, automated workflows), covering accuracy, consistency, hallucination rate, and safety.

  • Define golden datasets, rubrics, and scoring methodologies (human-in-the-loop and automated/LLM-as-judge) to measure model and product quality objectively.

  • Build automated eval pipelines that run in CI/CD, flag regressions before release, and produce clear, trackable quality metrics over time.

  • Extend evaluation coverage beyond the model layer into full product/QA testing — functional, regression, and end-to-end testing of AI-powered features and the surrounding product.

  • Partner with product managers and engineers to translate ambiguous quality bars ("is this good enough to ship?") into measurable, repeatable evaluation criteria.

  • Investigate failures and edge cases, perform root-cause analysis across the model/product boundary, and drive fixes with engineering.

  • Maintain traceability and reporting on eval/QA results for compliance-sensitive workflows, given the regulated nature of the data we handle.

What We're Looking For

  • 5 plus years of experience in QA/test engineering, ML evaluation, or a related quality-focused engineering role.

  • Hands-on experience testing or evaluating AI/LLM-powered features — building eval sets, scoring rubrics, or benchmark harnesses (or strong adjacent automation/QA experience with a demonstrated interest in AI evaluation).

  • Solid automation/testing fundamentals: scripting (Python and/or TypeScript/Java), API testing, SQL for data validation, and CI/CD integration.

  • Comfort working with LLMs and AI tooling directly — prompt engineering, RAG pipelines, or AI-assisted development tools — either as a builder or a rigorous evaluator of them.

  • Strong analytical mindset: comfortable defining metrics for fuzzy, subjective quality questions and defending them with data.

  • Clear written communication — you'll be documenting failure modes and quality bars for both engineers and non-technical stakeholders.

  • Bonus: experience in fintech, compliance, or another regulated domain where correctness and auditability matter.

Nice to Have

  • Experience with vector databases / semantic search (FAISS, ChromaDB, or similar) and RAG evaluation.

  • Familiarity with eval tooling/frameworks (e.g., Ragas, DeepEval, promptfoo, custom LLM-as-judge pipelines).

  • Background in Agile/Scrum environments and defect-management tooling (Jira or similar).

What We Offer

  • Competitive compensation based on experience

  • Equity participation

  • Comprehensive health insurance (self + family)

  • Paid leave and wellness benefits

To apply: send your résumé to careers@vinylequity.com.

Key skills

Required

Automation TestingCI/CDJavaPrompt EngineeringPythonSQLTypeScript

Preferred

AgileJIRAScrumCompliance

Soft skills

Communication

Tags

Job market insights

12active Automation Testing jobs in India
446active CI/CD jobs in India
283active Java jobs in India

Jobs asking for Automation Testing in India also ask for

  • JIRA50%
  • Playwright50%
  • Selenium50%
  • SQL42%
  • TypeScript42%

From live job posts on CoreHRx, updated 1 Nov 2026. Skill figures count posts that list their skills; salary figures only posts that show a salary.

Questions about this job

Where is this job based?

In Mangalore (on-site).

What kind of job is this?

It is a full time job.

This guide is for Indian job seekers preparing for Java backend roles, from freshers facing campus and first-job interviews to experienced developers moving into senior positions at product companies and IT services firms. It covers what interviewers ask most in 2025 and 2026 on Java 17 and 21 and Spring Boot 3: OOP, collections, equals and hashCode, streams, exceptions, concurrency and JVM memory, then dependency injection, auto-configuration, REST controllers, validation, JPA and Hibernate, security, microservices and testing. Questions are grouped as easy, medium and hard. Read each answer, compile the code yourself and practise explaining it aloud.

Questions you may be asked

  1. What are the four pillars of OOP, with a Java example of each?
  2. What is the difference between an abstract class and an interface in Java?
  3. What is the difference between method overloading and method overriding?
  4. What is the difference between == and equals() in Java?

Get IT jobs in Mangalore by email

Create a free profile and we email you new IT jobs in Mangalore, at most once a week.

Similar jobs

Php Developer

Speedhire Consultancy
noida
AWSCI/CDPHP
Full time On-site 14 days ago

Free · takes a minute · your resume fills your profile