AI Developer

Summary

The Agentic AI Developer designs, builds, and operates agentic AI process automations and reusable LLM‑powered components on our Azure-first stack. Partnering with Business Technology Partners (BTPs) and teams across the company, you will convert business requirements and process maps into production-grade automations that are observable, governed, and reusable. A core responsibility is rigorous quality control: you will develop evaluation workflows (test prompts, regression suites), implement hallucination detection and mitigation, measure and reduce bias, enforce guardrails and schema validation, and ensure outputs are grounded, reliable, compliant, and auditable—consistent with SEC governance and U.S. legal requirements for registered financial advisors. Who you are You’re a hands-on builder fluent in modern AI tooling who can translate business intent into robust agentic workflows. You communicate clearly with technical and non-technical stakeholders, execute with precision, and bring a strong quality mindset to everything you release. You balance speed with governance, instrument your solutions for observability and cost control, and document decisions for traceability. You are adaptable, eager to learn new technologies, and able to manage multiple projects under time constraints.

What you’ll do

• Design agentic AI systems that automate multi-step workflows; orchestrate LLM tools/skills and APIs to deliver reliable outcomes.

• Build reusable components (prompt libraries, tools/functions, skills, RAG pipelines, evaluation harnesses) for enterprise reuse across teams.

• Integrate LLMs (e.g., ChatGPT, Claude) with Azure services (Azure OpenAI, Azure AI Search/Vector, Functions, Logic Apps) and Microsoft 365/Copilot Studio.

• Use n8n (and/or Power Automate) for orchestration, eventing, and integrations across line-of-business systems.

• Implement RAG (retrieval‑augmented generation) with vector search and grounding/citation policies; enforce structured outputs (JSON schemas).

• Build evaluation & QC: red-team test prompts, regression suites, groundedness checks, bias assessments, output validation, cost/latency monitoring.

• Instrument observability (logging, trace, metrics), set SLOs (e.g., first‑pass accuracy, hallucination rate, bias metrics, latency, cost‑per‑task), and implement guardrails.

• Follow intake and prioritization through BTPs/supervisor; collaborate directly with business stakeholders on requirements and acceptance tests.

• Ensure security & compliance (PII handling, RBAC, data minimization, audit trails) aligned to SEC governance and company policies (e.g., prompt/response archiving).

• Create documentation and runbooks; contribute to code reviews and best practices; support end‑users and BTPs with adoption and training.

• Troubleshoot and resolve issues related to AI automations and integrations.

• Stay current on agentic AI frameworks and Azure/M365 capabilities; propose improvements to platform, tooling, and standards.

• Other duties, responsibilities, and special projects as assigned.

Experience

• 5+ years professional programming experience; 2+ years building AI tool based solutions (LLM APIs, Copilot, ChatGPT/Claude, orchestration).

• Proven delivery of production automations on Azure and Microsoft 365 ecosystems.

• At least 2 years of experience developing solutions with AI tools (e.g., Copilot, ChatGPT, Claude, n8n, Foundry). Technical skills (core)

• Languages: Python, C#, R (plus familiarity with JavaScript/TypeScript helpful).

• Frameworks/tools: LangChain, LangGraph, LlamaIndex, AutoGen (or similar agent frameworks); Copilot Studio, GitHub, n8n.

• Azure: Azure OpenAI, Azure AI Search/Vector, Functions, Logic Apps, Key Vault, Storage, DevOps/GitHub Actions.

• RAG pipelines, vector databases (e.g., Azure AI Search; familiarity with Pinecone/Weaviate a plus).

• API design & integration, structured outputs (JSON schema), function/tool calling.

• Observability & cost control: logging, tracing, metrics, rate/usage monitoring. Quality control & evaluation (explicit)

• Ability to design and run test prompts and regression suites covering normal, edge, and adversarial scenarios.

• Skill in hallucination detection/mitigation (groundedness checks, citation requirements, retrieval confidence thresholds, fallback policies).

• Experience measuring and reducing bias (fairness metrics, scenario/corpus checks, cohort A/B testing, safe‑prompting).

• Output validation using schema enforcement, tool/function results verification, and business‑rule assertions.

• Building evaluation harnesses with clear SLOs/KPIs (first‑pass accuracy, hallucination rate, bias indicators, latency, cost‑per‑task) and reporting dashboards.

• Implementing guardrails (content filters, allow/deny lists, sensitive-topic controls) and red‑team testing before deployment.

• Documenting test plans, evaluation results, and traceability from requirements to acceptance criteria. Collaboration & compliance

• Strong communication with BTPs/end‑users; aptitude for translating requirements into technical designs and acceptance tests.

• Working knowledge of SEC governance and U.S. regulatory expectations for AI usage in financial advisory contexts; commitment to Responsible AI practices.

• Familiarity with enterprise archiving/monitoring of prompts and outputs per policy (e.g., Smarsh, Theta Lake) and change‑control practices.

Nice to have

• Experience with real‑estate AI platforms (e.g., V7, Kolena, Cadastral) or domain workflows.

• Knowledge of GenAIOps/MLOps concepts for lifecycle and evaluation (not model training).

• Containers (Docker) and CI/CD; Power Platform experience. Additional Information for Remote Role

• Work Hours: Must have at least 4 hours overlap with Mountain Time zone (Denver, CO). • Communication: Strong internet connection and reliable communication tools are essential.

• Remote Work Environment: Must have a dedicated workspace free from distractions to ensure productivity.

• Cultural Sensitivity: Ability to work effectively with a diverse team and understand cultural differences.