Prompt Engineer: LLM Migration & Optimization
Please mention DailyRemote when applying
AI Summary
The role involves leading the end-to-end migration of template workflows into high-performance Large Language Model (LLM) autoraters, requiring the use of advanced tools and manual refinement to meet accuracy baselines. Responsibilities include supervising Automated Prompt Optimization (APO) tools, measuring quality against gold data using metrics like precision and recall, and designing manual prompts for complex edge cases.
United States • Remote • Part-Time
Are you an expert at navigating the complex logic of Large Language Models? Welo Data is seeking a technical Prompt Engineer to lead the end-to-end migration of template workflows into high-performance LLM autoraters.
This isn't just about writing prompts—it’s about engineering a technical bridge. You will use advanced APG/APO tools and manual refinement to ensure our automated systems meet (and exceed) human accuracy baselines.
The Mission: Architecting the Future of Rating
- Technical Migration: Take ownership of the workflow for transitioning templates to LLM autoraters.
- Optimization Leadership: Run and supervise Automated Prompt Optimization (APO) tools, identifying where logic plateaus and providing the manual "spark" to overcome deadlocks.
- Metrics-Driven Accuracy: Continuously measure quality against gold data, calculating critical performance metrics like precision, recall, and $F_1$ scores.
- Edge-Case Engineering: Solve complex scenarios by designing manual prompts that handle anti-patterns and broken logic in legacy architectures.
Project Details
- Schedule: Part-Time (Flexible hours within project milestones)
- Location: 100% Remote (Must be based in the United States)
- Employment Type: Freelance / Independent Contractor
Who We Are Looking For
- Linguistic & AI Mastery: 2+ years of experience as a Prompt Engineer. You must be comfortable tuning LLMs for structured outputs and complex classification tasks.
- Academic Background: BS, MS, or PhD in Computer Science, Data Science, Computational Linguistics, or a related analytical field.
- Technical Agility: Fast learner capable of mastering proprietary internal tools and interfaces (like the Goose API) with minimal supervision.
- Data Fluency: Strong ability to identify error patterns and use SQL or data analytics tools to analyze model performance.
Preferred Skills
- Familiarity with shadowbot disagreement tracking between humans and LLMs.
- Hands-on experience with Chain-of-Thought (CoT) and few-shot learning.
- Proven ability to draft high-level Launch Certification Documentation.
Automatically Apply to the Best Remote Jobs
Stop the endless job search. Our AI finds and applies to the best jobs for you.
