Build and operate production machine learning training systems on Amazon SageMaker using AWS Trainium. The role focuses on optimizing PyTorch code for Trainium hardware, understanding NeuronCore compilation, diagnosing device-level issues, tuning distributed training for cost and throughput, and delivering production pipelines. The engineer will collaborate directly with client and internal teams in a forward-deployed engineering environment.
Robots & Pencils is an AWS Partner building production AI systems for enterprise clients who need real engineering, not proofs of concept that never ship. We work forward-deployed, embedded directly with client teams, solving the problems that are too new or too specialized for a typical vendor relationship to handle.
The Role
We're looking for an engineer who can operate and train models on Amazon SageMaker running on AWS Trainium, AWS's custom silicon built specifically for large-scale model training. This isn't a role where you call an API and wait. You'll be walking up the stack: understanding what a training request actually looks like at the Trainium hardware and compiler level, then carrying that understanding all the way up through PyTorch training code and into a production SageMaker pipeline.
PyTorch is the backbone of this work. If you know the framework deeply and you're comfortable reasoning about how your code actually behaves on custom accelerator hardware rather than treating it as a black box, this role is built around that skill set specifically.
What You'll Do
- Train and operate models on Amazon SageMaker with AWS Trainium as the underlying compute
- Write and optimize PyTorch training code with a real understanding of how it compiles and executes on Trainium (NeuronCore architecture, compiler behavior, memory and throughput tradeoffs)
- Diagnose training run issues that show up specifically because of the hardware, not just the model, distinguishing a data or code problem from a compiler or device-level one
- Translate a request for "a Trainium job" into an actual working, cost-aware training pipeline, end to end
- Tune distributed training runs for throughput and cost on SageMaker's training infrastructure
- Work directly with client and internal engineering teams to scope and deliver real production training workloads, not experiments that stay in a notebook
What You'll Bring
- Strong, hands-on PyTorch experience, ideally including distributed or multi-device training
- Production experience with Amazon SageMaker for training and/or inference
- Comfort working close to the hardware layer: you understand device-specific compilation and can debug issues that are actually about the accelerator, not just the model
- AWS Trainium or Inferentia (Neuron SDK) experience is a strong plus; if you don't have it yet but have deep PyTorch and a track record of picking up new hardware targets fast, we want to talk to you
- Solid Python fundamentals and comfort operating in a client-facing, production engineering environment
Similar Jobs
AdTech • Artificial Intelligence • Marketing Tech • Software • Analytics
Leads paid search strategy and client relationships while managing campaign execution across Google Ads, Microsoft Ads, and SA360. Oversees budgets, pacing, optimization, testing, reporting, and performance analysis; translates data into actionable recommendations and growth opportunities. Collaborates with internal teams and mentors junior staff to ensure high-quality execution and measurable client results.
Top Skills:
Ga4Google AdsGoogle Ads EditorMicrosoft AdsPerformance MaxSa360Shopping AdsVehicle Ads
Artificial Intelligence • Cloud • Software • Infrastructure as a Service (IaaS)
Owns technical revenue accounting for complex customer, partnership, and marketplace arrangements. Evaluates ASC 606 issues, advises commercial and product stakeholders, operationalizes new monetization models, and manages recurring accounting, reporting, controls, reconciliations, disclosures, and audit matters. Leads process, systems, automation, and AI improvements while partnering cross-functionally with Sales, Legal, Product, Billing, Engineering, and Finance.
Top Skills:
Ai-Enabled AutomationAsc 606ErpNetSuiteOracleRmcs
Artificial Intelligence • Cloud • Software • Infrastructure as a Service (IaaS)
Own technical revenue accounting for complex customer, partnership, and marketplace arrangements under ASC 606. Partner with commercial, legal, product, billing, engineering, and finance teams to assess contracts, operationalize monetization models, manage recurring accounting, support close and controls, and improve revenue operations through systems, automation, and AI-enabled processes.
Top Skills:
Artificial IntelligenceAsc 606AutomationBilling SystemsErpNetSuiteOracleRevenue PlatformsRmcs
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
