Software Engineering Evaluator / Code Quality Specialist

About the Role

We are seeking a detail-oriented Software Engineering Evaluator to assess and improve the quality of AI-generated software development work. In this role, you will apply your engineering expertise to design realistic coding scenarios, evaluate AI-generated solutions, and provide structured feedback on code quality, correctness, and technical execution.

This opportunity is ideal for software engineers who enjoy reviewing code, solving technical problems, and applying strong engineering judgment to ensure high-quality, production-ready outcomes.

Key Responsibilities

• Create realistic software engineering tasks based on real-world development workflows, including technical requirements, supporting documentation, test cases, and code repositories.
• Evaluate AI-generated code solutions against professional engineering standards for correctness, maintainability, performance, and reliability.
• Compare outputs from different AI systems using identical requirements and environments, documenting relative strengths and areas for improvement.
• Develop test plans, evaluation rubrics, and success criteria that clearly define expected functionality, coding standards, and edge-case handling.
• Review code changes, implementation decisions, automated tests, and design approaches to verify technical quality and adherence to requirements.
• Identify and document issues such as:
• Missed edge cases
• Incorrect or incomplete implementations
• Fragile code changes
• Ineffective testing strategies
• Requirements misinterpretation
• Unsupported assumptions or fabricated outputs
• Collaborate with engineers and reviewers to refine evaluation tasks, improve assessment methodologies, and maintain quality standards across projects.
• Contribute technical expertise across a variety of technologies, platforms, and software development environments.

Qualifications

Required

• Experience developing, testing, and maintaining software applications or systems.
• Proficiency in at least one modern programming language such as:
• Python
• JavaScript / TypeScript
• Java
• Go
• C++
• C#
• Ruby
• PHP
• Swift
• Kotlin
• Rust
• Strong understanding of software engineering principles, debugging techniques, testing methodologies, and version control systems.
• Ability to quickly understand unfamiliar codebases and evaluate implementation quality.
• Experience using tools such as Git, testing frameworks, command-line interfaces, and debugging tools.
• Excellent written communication skills with the ability to explain technical findings clearly and objectively.
• Professional-level English proficiency.

Preferred

• Two or more years of software development experience in a professional, academic, freelance, or open-source setting.
• Familiarity with containerization, CI/CD pipelines, and modern development workflows.
• Experience using AI-powered coding assistants or large language model (LLM) tools such as ChatGPT, Claude, GitHub Copilot, or similar technologies.
• Bachelor's degree (completed or in progress) in Computer Science, Software Engineering, Information Systems, or a related discipline, or equivalent practical experience.

What Success Looks Like

• Consistently identifies technical issues, risks, and implementation gaps with supporting evidence.
• Demonstrates sound engineering judgment when evaluating code quality and software behavior.
• Produces clear, actionable feedback that improves the quality of AI-generated software outputs.
• Applies best practices in testing, maintainability, performance, and software design.
• Delivers objective and well-documented evaluations across multiple technologies and domains.

Skills

• Software Development
• Code Review
• Quality Assurance
• Test Design & Validation
• Debugging & Troubleshooting
• Git & Version Control
• Software Architecture
• Technical Documentation
• Analytical Thinking
• AI-Assisted Development
• Backend, Frontend, and Full-Stack Development
• Problem Solving

This is a Permanent position based out of Tampa, FL.

The pay range for this position is $45000.00 - $60000.00/yr.

Individual compensation offered for this position within this range will depend on many factors, including qualifications, skills, relevant experience, job knowledge, geographic location, internal equity, and other pertinent job-related factors.

Independent contractor paid through paypal. No benefits offered

This is a Permanent position based out of Tampa, FL.

This position is anticipated to close on Nov 20, 2026.

This is a fully remote position.

About Aston Carter

Aston Carter provides world-class corporate talent solutions to thousands of clients across the globe. Specialized in accounting, finance, human resources, talent acquisition, procurement, supply chain and select administrative professions, we extend the capabilities of industry-leading companies. We draw on our deep recruiting expertise and expansive network to meet the evolving needs of our clients and talent community with agility and excellence. With offices across the U.S., Canada, Asia Pacific and Europe, Aston Carter serves many of the Fortune 500. We are proud to be a ClearlyRated Best of Staffing® Platinum Award winner for both client and talent service.

The company is an equal opportunity employer and will consider all applications without regard to race, sex, age, color, religion, national origin, veteran status, disability, sexual orientation, gender identity, genetic information or any characteristic protected by law.

If you would like to request a reasonable accommodation, such as the modification or adjustment of the job application process or interviewing process due to a disability, please email astoncarteraccommodation@astoncarter.com for other accommodation options.

San Francisco Fair Chance Ordinance: Pursuant to the San Francisco Fair Chance Ordinance, for all positions located in the city and county of San Francisco, we will consider for employment qualified applicants with arrest and conviction records.

Massachusetts Lie Detector: It is unlawful in Massachusetts to require or administer a lie detector test as a condition of employment or continued employment. An employer who violates this law shall be subject to criminal penalties and civil liability.

Use of Artificial Intelligence (AI): We may use Artificial Intelligence (AI) to support parts of our hiring process, including sourcing, screening, and evaluating candidates. AI helps assess applications and qualifications, but final decisions are made by our hiring team. By applying, you acknowledge and agree that your application may be reviewed using AI tools.