Operate and monitor ML/AI models and agentic systems in production. Build AI observability, logging, tracing, and evaluation pipelines. Monitor LLM outputs, detect drift and model degradation, and maintain data/feature pipelines. Develop CI/CD, model versioning, experiment tracking, and automate alerts and incident remediation while collaborating with data scientists and platform teams.
Key Responsibilities AI/ML Model Operations
- Deploy, manage, and monitor machine learning and AI models in production environments.
- Implement model performance monitoring including accuracy, latency, and inference metrics.
- Detect and mitigate concept drift, data drift, and model degradation.
AI Observability
- Design and implement AI observability frameworks to track model behavior and reliability.
- Monitor LLM outputs, hallucination rates, and response quality.
- Implement logging, tracing, and evaluation pipelines for AI systems.
Agentic Systems Monitoring
- Monitor agent-based AI workflows and autonomous systems.
- Track agent actions, tool usage, decision paths, and execution outcomes.
- Implement guardrails, safety monitoring, and failure detection for AI agents.
Data Pipeline Monitoring
- Monitor and maintain data ingestion, transformation, and feature pipelines.
- Ensure data quality, schema consistency, and pipeline reliability.
- Detect and resolve pipeline failures and anomalies.
Infrastructure & Automation
- Build and maintain CI/CD pipelines for ML models and AI systems.
- Manage model versioning, experiment tracking, and reproducibility.
- Automate monitoring alerts, incident response, and remediation.
Collaboration
- Work closely with data scientists, ML engineers, platform teams, and product teams.
- Support continuous improvement of AI system reliability and governance
-
Compensation, Benefits and Duration
Minimum Compensation: USD 50,000
Maximum Compensation: USD 177,000
Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.
Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.
This position is not available for independent contractors
No applications will be considered if received more than 120 days after the date of this post
Similar Jobs
Agency • Artificial Intelligence • Consumer Web • Digital Media • Analytics • Design
Manage end-to-end paid media campaigns across multiple clients and channels, including strategy, setup, optimization, tracking, and reporting. Analyze performance data, maintain tracking (GA4/Tag Manager), build reports and dashboards, and present insights to clients. Collaborate with account, creative, and analytics teams to drive growth and improve ROI while staying current on platform best practices.
Top Skills:
Conversions ApiFacebook/Meta AdsGoogle AdsGoogle Analytics (Ga4)Google Looker StudioGoogle SheetsGoogle Tag ManagerLinkedin AdsMicrosoft/Bing AdsPerformance MaxPinterest AdsProgrammatic AdvertisingReddit AdsTwitter/X AdsYoutube Ads
.png)
Artificial Intelligence • Big Data • Healthtech • Machine Learning • Software • Biotech
The GRAIL Galleri Consultant will drive territory growth by engaging healthcare providers, executing the sales strategy for the Galleri product, and managing customer relationships.
Top Skills:
Google Suite
eCommerce • Fashion • Retail • Sales • Wearables • Design
Lead daily store operations to drive sales, profitability, and exceptional customer service. Recruit, coach, and develop a high-performing team; manage inventory, payroll, loss prevention, visual merchandising, scheduling, and local business plans. Serve as brand ambassador and ensure inclusive, results-driven store culture.
Top Skills:
Inventory Management SystemsLabor Management SystemsMS OfficeSales Reporting ToolsSocial Media
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
