Senior MLOps Engineer

About CompScience

At CompScience, we're not just building software, we're saving lives. We're a high-growth startup on a mission to prevent 1 million workplace injuries through bold technological innovations, ensuring that everyone can go home safe at the end of the day.

Founded in 2019 and backed by investors from SpaceX, Tesla, and Anduril, we've assembled a powerhouse team that bridges two worlds:

• Cutting-Edge Technology: Our product, design, and engineering teams are composed of distinguished computer vision engineers, software architects, data scientists and product and design leaders from Amazon R&D, Meta, and the self-driving car industry. They bring unparalleled expertise in AI, machine learning, and design to the realm of workplace safety.

• Insurance Acumen: Our insurance team is made up of seasoned professionals who understand the nuances of workers' compensation policies. They work hand-in-hand with our tech experts to translate advanced analytics into tangible insurance products that truly serve our clients' needs.

Our groundbreaking perception-based risk assessment program, the first of its kind, provides the most comprehensive data stream available for risk analysis and monitoring and has proven to significantly reduce accidents in some of the world's most hazardous occupations.

About the Role

We are looking for an experienced and self-motivated Senior MLOps Engineer to join our growing team and take ownership of the infrastructure that powers our core machine learning products. As a key member of our engineering organization at a fast-growing Series B startup, you will be responsible for designing, building, and maintaining the systems that automate the entire lifecycle of our ML models—from data pipelines and training to deployment and production monitoring. This is a high-impact role where you will collaborate closely with our data science and engineering teams to ensure our cutting-edge risk assessment and underwriting models are scalable, reliable, and continuously improving.

Responsibilities

• Design and build end-to-end MLOps infrastructure on AWS, implementing high-throughput workflows with orchestration tools (Airflow, Prefect, Dagster) to support data-centric pipelines and model execution.

• Lead the evaluation of RAG and VLM systems, establishing rigorous metrics for retrieval quality, grounding checks, hallucination detection, and end-to-end performance.

• Develop production-ready web APIs and microservices, managing critical components such as instrumentation, middleware, authentication, and multi-tenant patterns.

• Establish and manage systems for experiment tracking and LLM observability to continuously monitor model health, embeddings drift, and vision-text pipeline performance.

• Orchestrate complex data pipelines for the ingestion and processing of multimodal data, including managing updates to embeddings in vector databases (e.g., Qdrant, ChromaDB).

• Develop and maintain robust CI/CD pipelines for model deployment, enforcing Git best practices and collaborating with data science teams to translate prototypes into scalable services.

Required Experience

• Bachelor's degree in Computer Science, Engineering, or a related field with 5+ years of experience in MLOps or Data Engineering, specifically focused on operationalizing ML models.

• Solid Python engineering experience (data pipelines, API integrations, evaluation scripts) coupled with expert proficiency in the AWS SDK (Boto3).

• Proficiency with SQL and at least one cloud data warehouse (Postgres, BigQuery, Redshift) for designing data-engineering-heavy pipelines.

• Hands-on experience with an LLM observability or experiment-tracking tool (Langfuse, Comet, WandB) and containerized application deployment (Docker).

• Knowledge of LLM and VLM evaluation methods, including building test sets, using LLM-as-judge, applying NLP metrics, and benchmarking vision-language grounding accuracy.

• Deep experience with core AWS services (S3, Lambda, SageMaker) and a strong understanding of networking, security, and Git workflows (branching, merging, CI/CD)

Nice-to-have

• Basic TypeScript/React skills for building lightweight internal dashboards, labeling UIs, or visualization tools.

• Familiarity with monitoring and alerting ecosystems (Grafana, Prometheus, Datadog, Sentry) and multimodal embedding models.

• Experience evaluating drift monitoring for vision/text pipelines, or an active AWS Certification (Machine Learning Specialty or DevOps Professional).

Working at CompScience

Compensation: CompScience is committed to fair and equitable compensation practices. The annual salary range for this role is $175,000 – $225,000. Compensation is determined within the range based on your qualifications and experience. Our total compensation package also includes equity and comprehensive benefits.

Benefits at CompScience:

• Fast-paced startup environment where your ideas can quickly become reality

• Opportunity to wear multiple hats and grow beyond your job description

• Remote-first culture with home office support

• Comprehensive health benefits (Medical, Dental, Vision, HSA)

• 401(k) plan and life insurance

• Flexible time off and 12 weeks parental leave

• Professional development reimbursement

Our Ideal Teammate:

• Thrives in a fast-paced startup and is comfortable navigating ambiguity

• Excited to wear multiple hats and grow rapidly

• Committed to our mission of saving lives through technology