See how much of this job your resume covers, and what’s missing.
Want a recruiter to go through it line by line?
Get professional reviewQuestions interviewers often ask for this role, with sample answers.
One click makes a copy of your resume for this role, scores it against the posting, and proposes edits line by line. Your default resume is never changed.
Upload your resume and we draft a letter for this exact role, tailored to what it asks for.
You will build and improve AI agents and evaluation pipelines to ensure the reliability and accuracy of payroll-related workflows. This involves collaborating with domain experts to translate complex cases into testable AI behaviors and implementing improvements based on production feedback.
Get to know us
Do you want to help us shape what the future of work will look like and how it will best embrace our life's aspirations? If this sounds like a journey you want to embark on, we may have the right role for you! PayFit is an intuitive software as a service payroll and HRIS solution designed specifically for SMBs. Since 2015, we have set ourselves a mission to simplify payroll for SMBs and enable employers and employees to grow together. We are a European company operating from 3 main countries (France, Spain, and the UK) where we are supporting over 20,000 clients.
Creating a fulfilling work environment and culture is also a core mission at PayFit, and our day-to-day work philosophy is reflected in our four values:
Care: We genuinely care for others whoever they are, whatever they think.
Excellence: We aim to improve and achieve better results every day.
Humility: Staying humble and exchanging transparent feedback helps us to grow and improve.
Passion: We are the architects of PayFit's success.
A key part of our culture, and essential for our success, is also improving the diversity of our teams and building an inclusive culture where you can be yourself at work.
This is why our recruitment focuses on the skills you demonstrate, and not only on your academic background or previous professional experiences. At PayFit we understand that you can gain applicable skills through a variety of life experiences and we are interested in knowing them, too.
Location :
Flexible location: work remotely from anywhere in France or Spain
The team
You'll join the team behind Payfit AI, building AI agents that help companies prepare, check, and run payroll, so people can focus on the decisions that need their expertise.
We work closely with Customer Support and payroll experts to understand real customer situations and turn that expertise into useful product experiences. These span a wide range of payroll and HR topics, from everyday questions to complex cases in a highly regulated environment.
You'll contribute to connecting that expertise with the systems we build. Using real cases, you'll help evaluate the whole workflow: did the agent give the right answer, take the right action, or know when to ask for clarification or involve a human? Where a workflow affects payroll, evaluating its quality also means checking the resulting outcome.
This is a hands-on senior engineering role. You'll build on existing models, agents, evaluation tools, and knowledge pipelines. Depending on what you find, you might improve the knowledge base, create a specialized agent, connect a new tool, or evolve the agent harness: how agents receive context, use tools, apply guardrails, and hand off to a person. You'll use evaluations and production feedback to guide these improvements, measure their impact, and catch regressions as models, payroll rules, and workflows evolve.
Work with Customer Support and payroll experts to understand how they resolve complex cases and turn that expertise into testable AI behavior.
Build and improve evaluations using representative cases, expert reviews, and production feedback to assess answers, actions, and workflow outcomes.
Investigate failures across models, context, knowledge, retrieval, and tools, then implement and measure improvements.
Contribute to specialized agents, orchestration, and harness engineering, including context management, reusable tools, guardrails, and human handoffs.
Improve knowledge pipelines and retrieval quality, and use regression evaluations and observability to keep workflows reliable as they evolve.
7+ years of software engineering experience, including designing and operating backend services or pipelines in production.
Hands-on experience building LLM applications or agents, including tool integration, context management, and handling uncertainty.
Practical experience evaluating LLM systems using representative datasets, clear criteria, and automated or human review—and using the results to guide improvements.
Strong backend engineering skills and the ability to contribute to a TypeScript/Node.js codebase.
Comfort working directly with domain experts, translating ambiguous cases into testable behavior, and collaborating on reliable production solutions.
Care for security, permissions, observability, and maintainability.
Experience with Langfuse or similar tools for tracing, prompt management, datasets, and experiments.
Familiarity with AI SDK or comparable frameworks for building tool-using agents.
Experience with hybrid search, reranking, OpenSearch, or other search and vector technologies.
Experience measuring judge agreement, evaluation uncertainty, or agent consistency across repeated runs.
Experience building annotation tools or review workflows for domain experts.
Familiarity with AWS, data pipelines, and CI/CD for AI systems.
Experience in payroll, HR, customer support, or another domain where factual accuracy and traceability matter.
Step 1 — 30-45 min call with Talent Acquisition
Step 2 — 45-minute meeting with our Hiring Engineering Manager
Step 3 — 60-minute technical interview with engineers from the squad
Step 4 — 30-minute meeting with our Product Manager and AE Designer/CS expert
Step 5 — 45-minute final interview with the Tribe Engineering Director
Background Check (ISO27001 certification): identity, diploma, and past professional experience will be checked after the offer.
What we offer
Flexibility: We believe it is key to producing your best work and being fulfilled. We therefore offer the possibility to work away from our main offices, within France, as well as abroad for a defined period. Further requirements may apply depending on the role and your overall experience.
Learning & Development: At PayFit, we offer a comprehensive learning platform that enables you to acquire new skills daily, supported by our company.
Career Development: We want you to progress and be free to choose which direction you want to grow. There are also opportunities for internal moves.
Health insurance: Henner Mutuelle Insurance (60% covered by PayFit, free coverage for children)
Transportation: 50% of public transportation costs are covered by PayFit for those living within the Ile de France region, or assistance with sustainable mobility (a bicycle rental subscription, purchase of soft mobility equipment to get to work, etc.).
Meals: A restaurant card with our partner Swile (9€ per workday) covered at 60% by PayFit
A Work Council grant: A monthly allowance to be spent on culture, sports, personal services, etc., as well as a vacation bonus.
Work equipment: A MacBook is our standard working tool
Parental support program: Salary maintenance during the first month of additional parental leave.
Time off: 25 days of holidays + RTT days (depending on the contract).
Disability Inclusion: All of our positions are open to any person living with a disability. To guarantee equal treatment and opportunities, we will take, based on individual needs, appropriate measures to adapt the work conditions of PayFiters with disabilities, and if needed also during the recruitment process. Please let us know what you need and we will do our best to accommodate!
Stop the endless job search. Our AI finds and applies to the best jobs for you.
Featuring 218,785+ Jobs in AI Engineer
Answer easy questions
218,785+ jobs across 15+ categories
Get your best job matches
Only hand-screened, legit jobs
Find a remote job faster
No ads, scams, or junk
“I was the first applicant for a remote marketing position that got listed on the company website the same day I applied. Had an interview within 48 hours!”