Member of Technical Staff (Evals)

• We are hiring a Member of Technical Staff, Evals to help define how frontier AI systems are measured, understood, and improved
• This is a broad, high-ownership role for researchers who build
• You will partner with AI researchers, domain experts, and customers to develop new benchmarks, reward and verifier systems, agent-evaluation methodologies, and data-quality techniques
• You will work on questions at the center of frontier AI progress: what should be measured, how to design evaluations that reflect real capability, how to create high-signal feedback, and how to build the environments and data systems that make those answers actionable
• Early members of the team will have unusual influence over our technical direction, operating culture, and the open-source software, benchmarks, and research products we build
• We care more about demonstrated research capability, technical judgment, and a builder’s mindset than a specific title, degree, or career path
• Design and build evaluation frameworks, benchmarks, and methodologies for frontier LLMs, AI agents, multimodal models, and reinforcement-learning environments
• Develop reward models, programmatic verifiers, graders, and other feedback systems that make model behavior measurable and improvable
• Research what makes evaluations representative, difficult, reliable, and resistant to shortcutting or reward hacking
• Build systems for high-quality human data, including expert task design, annotation methodologies, data-quality signals, and data-attribution techniques
• Run fast, rigorous iteration loops: prototype, evaluate, interpret results, diagnose failure modes, and turn learnings into the next benchmark or system
• Publicly contribute to the field through benchmarks, open-source tools, research, and technical writing

Benefits

• Health Insurance: Comprehensive medical , dental, and vision coverage
• Fertility and family forming: Through a partnership, Handshake offers comprehensive family benefits, including infertility treatments and a $15K stipend, to its employees.
• LGTBQ+ coverage: All medical plans cover transgender services.
• Mental health: Discounts on a variety of wellness apps and an Employee Assistance Program with access to five free counseling sessions.
• Tax savings: Tax-advantaged benefits to save money on healthcare, dependent care, and other expenses.
• Life & disability insurance: Essential benefits and automatic coverage with basic life insurance, plus short-term and long-term disability coverage.
• 401k match: Dollar-for-dollar match up to 1% of pay, with a max of $1,200 per year.
• Equity: Our equity program includes grants at hire, refresher grants after two years of tenure, and grants alongside promotions.
• Financial management: Handshake partners to provide free professional financial planning services.
• Learning & development: Employees are provided with an annual stipend to use towards professional development, career development workshops, manager trainings, and 1-on-1 coaching through external partnerships
• Community leader support: Employee Resource Group leaders are compensated with a quarterly bonus for the high-impact work the role requires
• Parental leave: Leaves of 16 weeks for birth-giving parents and 10 weeks for non-birth-giving parents; the policy applies to adoptions as well. Handshake also partners to provide one-on-one coaching on preparing for leave and coming back from leave
• Flexible time off: No accrued time off or waiting period to take vacation. Two paid flexible holiday days off to observe holidays that closely align with religious, cultural, and personal needs
• Sabbatical: Paid six-week sabbatical to recharge and reset beginning at six years of continuous, full-time employment, and every four years thereafter
• Volunteer time off: Two days off for social and civil activism, volunteer work, and/or any related activities, including attending and participating in protests

• Comfort operating in an ambiguous, fast-moving environment with substantial ownership
• PhD in ML/AI, computer science, data science, or related fields (or equivalent research experience in industry)
• Publications at top AI/ML venues like NeurIPS, ICML, ICLR, COLM
• Strong Python skills, experience building scalable software, working with agents
• Builders who enjoy tinkering with agents and shipping high-quality software, benchmarks, or datasets (e.g. a strong GitHub profile / OSS contributions, or product portfolio)
• Strong knowledge of frontier AI: benchmarks, eval techniques, agent harnesses, post-training recipes, data shapes
• Join a rapidly growing company whose data business grew from zero to nearly $1B run rate in a year
• Help build an early technical organization where your work shapes the roadmap, standards, and culture
• Work at the very frontier of AI with most major AI labs, researching some of the most important problems in Data and Evaluations
• Attend (and publish at) conferences like NeurIPS, ICML, ICLR, COLM
• Publish results and work in public through open-source benchmarks and software, papers, and blogs