TixelJobs
B
Berkeley Research Group, LLCvia muse

Associate Director AI Product Engineer - Financial Institution Operations & Technology

REMOTE$180K - $250K/yrPosted 2w ago
ML EngineerExecutiveFull-time#remote

About the Role

**We Do Consulting Differently**
--------------------------------



Preferred locations: New York, Chicago, Washington, D.C., Dallas, and Boston. Other locations considered case by case.





**About the Practice**

Berkeley Research Group is building a Financial Services Operations \& Technology practice focused initially on Commercial and Consumer Banking. We combine banking expertise, technology, and AI to help clients improve operations.





**The Role**
------------

We are hiring a hands\-on AI Product Engineer focused on evaluation: proving whether AI products are accurate, reliable, and safe for banking operations. You will build automated tests, investigate failures, and provide evidence for release decisions while contributing to product development.


Working with banking experts and engineers, you will define correct results, measure performance, and turn findings into product improvements. Your core responsibility is evaluation from early development through live use. Client stakeholders retain release and risk\-acceptance authority.





**What You’ll Do**
------------------

**Own the Evaluation Approach**
-------------------------------

* Define what each product must get right, how errors could affect customers or operations, and what evidence is needed before use.
* Build reusable software that runs tests, scores results, and compares versions. Make testing part of everyday development, not a final checkpoint.
* Work with engineers to reproduce failures, identify causes, and verify fixes. Contribute product code where needed to improve accuracy and control.

**Design Tests and Measure Results**
------------------------------------

* Set pass criteria and scoring rules with business and risk teams. Distinguish minor defects from errors that must block release.
* Maintain representative test cases, including incomplete records and rare, high\-impact errors. Reserve unseen examples for final testing to avoid overstating performance.
* Have banking experts establish expected answers and resolve scoring disagreements. Measure factual accuracy, missing information, incorrect additions, and support from cited sources.
* Check automated scoring against expert review. Test whether reviewers agree and whether AI\-based scoring misses important errors.
* Compare versions on the same tests. Report results by task and error severity, with sample sizes and uncertainty, not just averages.

**Verify Accuracy and Explain Failures**
----------------------------------------

* Test whether the system finds the right information and whether its answer reflects that information. Separate source problems from answer errors.
* Check extracted facts against original records. Independently verify amounts, dates, and totals, and test whether citations actually support each important claim.
* Test when the product should answer, request clarification, or refer work to a person. Verify that reviewers can detect and correct material errors.

**Manage Risk and Monitor Performance**
---------------------------------------

* Test for invented facts, unsupported recommendations, unfair outcomes, disclosure of sensitive information, and attempts to make the system ignore its safeguards.
* Protect client data, restrict access and permitted actions, and use only approved services. Verify that safeguards work in practice.
* Document repeatable test results, unresolved failures, and limitations. Recommend release, restricted use, or further development; support independent review and required client approvals.
* Sample live outputs and track error rates and user corrections. Add confirmed failures to repeatable tests; trigger investigation or suspension when quality falls below agreed standards.
Explain findings clearly, coach colleagues, and develop reusable tests and controls. Support proposals, technical estimates, and recruiting.
*



**Qualifications**
------------------

**Required**
------------

* Minimum 5 years o
Share