We're looking for a Senior Data Scientist to join our AMI group. This hands-on role combines Machine Learning, statistical modeling, Generative AI, RAG, model evaluation, and fine-tuning for enterprise AI applications. AMI supports commercial AI providers and on-premises models, with agents, MCP tools, and A2A interoperability delivered in partnership with AI Engineering.
About the Role:
You will independently design, develop, evaluate, and operationalize complex AI/ML solutions. You will contribute hands-on technical expertise, apply rigorous experimentation and MLOps/LLMOps practices, improve modeling and evaluation approaches within assigned initiatives, and mentor other Data Scientists.
You will be a part of the AI Migration team, responsible for building tools that help customers transition from competitor products to BMC solutions. You'll work at the intersection of software engineering and AI, combining code-based conversion with AI-assisted processing to tackle migration challenges. Your work will help reduce manual effort and enable services to deliver accurate, timely customer migrations
Key Responsibilities:
Machine Learning & Applied Data Science
- Apply classification, regression, ranking, forecasting, clustering, and anomaly detection to enterprise AI and agentic use cases.
- Develop and evaluate intent classification, semantic matching, agent/skill/tool ranking, and model-routing techniques; compare learned approaches with rules, embeddings, and LLM-based baselines.
- Use statistical modeling to estimate execution latency, token demand, inference costs, workload trends, and resource requirements.
- Apply clustering and anomaly detection to identify recurring intents, capability gaps, unusual agent behavior, and execution regressions.
- Conduct exploratory analysis, dataset preparation, feature engineering, hypothesis testing, and reproducible experiments.
Foundation Models, Prompting & Context Engineering
- Evaluate commercial API and self-hosted open-weight models for accuracy, reasoning, instruction adherence, structured outputs, latency, throughput, and total cost.
- Design and optimize system instructions, prompt templates, few-shot examples, tool-calling prompts, and context assembly strategies.
- Benchmark model alternatives for specific enterprise workloads and document quality/cost/latency tradeoffs.
- Collaborate with AI Engineering and Platform Engineering on inference performance, quantization tradeoffs, and GPU/VRAM capacity assumptions.
Retrieval-Augmented Generation (RAG)- Design and optimize document parsing, chunking, metadata, embedding, retrieval, reranking, and context selection strategies.
- Evaluate vector, keyword/BM25, hybrid, and semantic retrieval, including access-aware retrieval where relevant.
- Diagnose retrieval versus generation failures; measure Recall@K, nDCG, groundedness, faithfulness, and end-to-end answer quality.
- Compare embedding and reranking models using representative enterprise datasets and reproducible benchmarks.
Model Fine-Tuning & Adaptation
- Assess when prompting, RAG, supervised instruction fine-tuning, or parameter-efficient adaptation is appropriate.
- Curate instruction datasets and holdout evaluations; apply SFT, LoRA, QLoRA, or comparable methods.
- Measure gains over base-model and prompting/RAG baselines, including generalization, regressions, data leakage, and operating costs.
- Collaborate on the packaging and validation of fine-tuned model artifacts and adapters.
Agentic AI, MCP & A2A
- Apply modeling and evaluation expertise to intent routing, agent/skill discovery, MCP tool selection, workflow selection, and planning effectiveness.
- Evaluate task completion, tool-call correctness, argument extraction, delegation quality, and recovery from failures.
- Work with AI Engineering on agentic frameworks and understand MCP tool schemas and A2A Agent Cards, skills, and task delegation.
- Contribute decision models and quality signals without owning agent runtime, MCP server, or A2A protocol implementation.
AI Evaluation & Experimentation
- Design golden datasets, holdout sets, human review protocols, LLM-as-judge calibration, and statistical comparison methods.
- Automate evaluations and regression checks for prompts, foundation models, RAG, fine-tuned models, routing, and agents.
- Define measurable quality thresholds for accuracy, relevance, groundedness, instruction following, task completion, and cost/performance.
- Investigate regressions and use evidence to guide model, prompt, retrieval, and routing decisions.
- Implement and maintain reproducible experimentation, model evaluation, and lifecycle workflows for assigned AI/ML initiatives, including automated validation and production readiness checks.
MLOps, LLMOps & Model Lifecycle
- Apply reproducible experiment tracking across datasets, code, model versions, hyperparameters, prompts, retrieval configurations, metrics, and artifacts.
- Develop and maintain automated training, fine-tuning, benchmarking, and evaluation pipelines, with model registry and artifact traceability.
- Define and integrate quality gates, regression tests, and model/prompt/RAG release-readiness criteria with engineering teams.
- Analyze production quality and drift signals, agent routing accuracy, latency, throughput, and token usage; recommend remediation.
- Collaborate with engineering on packaging, deployment handoffs, observability, rollback criteria, and ongoing model lifecycle management.
Technical Leadership & Collaboration
- Independently lead complex Data Science initiatives from problem formulation through model development, evaluation, and production handoff.
- Make technical decisions within assigned initiatives, balancing model quality, performance, cost, and delivery requirements.
- Improve reusable modeling, benchmarking, experimentation, and MLOps/LLMOps practices within assigned initiatives.
- Collaborate with AI Engineering, AI Quality, Platform Engineering, Product Management, and domain experts.
- Translate business requirements into measurable AI and ML objectives.
- Mentor junior and mid-level Data Scientists and contribute to technical reviews.
- Communicate complex AI and ML concepts clearly to technical and non-technical stakeholders.
Must-Have Skills & Experience
- 8+ years of relevant Data Science, Machine Learning, or Applied AI experience.
- Bachelor’s degree in Machine Learning, Statistics, Data Science, Physics, Computer Science, or a related field, or equivalent education and industry experience.
- Advanced Python proficiency with Pandas, NumPy, scikit-learn, and PyTorch or TensorFlow; SQL and practical data preparation skills.
- Strong statistics, experimental design, hypothesis testing, ML validation, and applied classification, regression, ranking, clustering, forecasting, and anomaly detection knowledge.
- Hands-on LLM prompting, embeddings, model comparison, and RAG design/optimization, including chunking, retrieval, and reranking.
- Practical instruction fine-tuning or parameter-efficient model adaptation experience, with measurable baseline comparisons.
- Experience designing evaluation datasets, automated quality metrics, and statistical model/LLM/RAG experiments.
- Hands-on MLOps and LLMOps: experiment tracking, model/dataset/prompt versioning, reproducible pipelines, registries/artifacts, evaluation automation, quality gates, and monitoring.
- Working knowledge of agent planning, routing, tool calling, MCP, A2A, and multi-agent patterns.
- Understanding of inference latency, throughput, concurrency, token cost, and on-prem versus provider-hosted model tradeoffs.
- Familiarity with AWS and/or Kubernetes or Red Hat OpenShift (OCP), and collaboration with platform engineering.
- Clear communication, independent problem solving, cross-functional collaboration, and mentoring.
- Demonstrated ability to independently deliver complex AI/ML solutions, make sound technical decisions within assigned initiatives, and mentor other Data Scientists.
- Experience contributing to reusable modeling, evaluation, and MLOps/LLMOps practices within project teams.
- Evidence We Will Look for in Hiring
- Delivered measurable improvements in model quality, RAG retrieval, routing accuracy, fine-tuning outcomes, or inference cost against documented baselines.
- Built reproducible ML/LLM experiments, automated evaluation pipelines, tracked artifacts, and model quality gates.
- Demonstrated independent delivery of complex AI/ML solutions, rigorous model validation, collaboration on production monitoring, mentoring of other Data Scientists, and improvements to technical practices within assigned initiatives.
- Demonstrated improvements in model quality, reliability, performance, or operational efficiency within specific projects or applications.
Nice-to-Have Skills
- Experience with OpenSearch, Elasticsearch, FAISS, Milvus, pgvector, or similar retrieval platforms.
- Hands-on use of LangGraph, OpenAI Agents SDK, Google ADK, CrewAI, or other agentic frameworks.
- Familiarity with MLflow, Kubeflow, Ragas, DeepEval, OpenJudge, LangSmith, or comparable MLOps/LLMOps and evaluation tooling.
- Experience with vLLM, NVIDIA Triton, quantization, GPU memory sizing, or distributed inference.
- Python backend APIs (FastAPI, Flask, or similar); OpenTelemetry and AI observability.
- AI safety, prompt injection evaluation, privacy, responsible AI, and model governance.
- Enterprise code understanding, legacy modernization, or domain-specific model adaptation.
- Agile practices and Atlassian Jira/Confluence.
- Master’s degree in a related field.
— Sarah J.
·
Marketing Manager
★★★★★
Verified
