Build and operate production machine learning training systems on Amazon SageMaker using AWS Trainium. The role focuses on optimizing PyTorch code for Trainium hardware, understanding NeuronCore compilation, diagnosing device-level issues, tuning distributed training for cost and throughput, and delivering production pipelines. The engineer will collaborate directly with client and internal teams in a forward-deployed engineering environment.
Robots & Pencils is an AWS Partner building production AI systems for enterprise clients who need real engineering, not proofs of concept that never ship. We work forward-deployed, embedded directly with client teams, solving the problems that are too new or too specialized for a typical vendor relationship to handle.
The Role
We're looking for an engineer who can operate and train models on Amazon SageMaker running on AWS Trainium, AWS's custom silicon built specifically for large-scale model training. This isn't a role where you call an API and wait. You'll be walking up the stack: understanding what a training request actually looks like at the Trainium hardware and compiler level, then carrying that understanding all the way up through PyTorch training code and into a production SageMaker pipeline.
PyTorch is the backbone of this work. If you know the framework deeply and you're comfortable reasoning about how your code actually behaves on custom accelerator hardware rather than treating it as a black box, this role is built around that skill set specifically.
What You'll Do
- Train and operate models on Amazon SageMaker with AWS Trainium as the underlying compute
- Write and optimize PyTorch training code with a real understanding of how it compiles and executes on Trainium (NeuronCore architecture, compiler behavior, memory and throughput tradeoffs)
- Diagnose training run issues that show up specifically because of the hardware, not just the model, distinguishing a data or code problem from a compiler or device-level one
- Translate a request for "a Trainium job" into an actual working, cost-aware training pipeline, end to end
- Tune distributed training runs for throughput and cost on SageMaker's training infrastructure
- Work directly with client and internal engineering teams to scope and deliver real production training workloads, not experiments that stay in a notebook
What You'll Bring
- Strong, hands-on PyTorch experience, ideally including distributed or multi-device training
- Production experience with Amazon SageMaker for training and/or inference
- Comfort working close to the hardware layer: you understand device-specific compilation and can debug issues that are actually about the accelerator, not just the model
- AWS Trainium or Inferentia (Neuron SDK) experience is a strong plus; if you don't have it yet but have deep PyTorch and a track record of picking up new hardware targets fast, we want to talk to you
- Solid Python fundamentals and comfort operating in a client-facing, production engineering environment
Similar Jobs
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Hunt for historical and active threats across enterprise environments by analyzing logs, system artifacts, forensic evidence, and threat intelligence. Develop actionable remediation recommendations and communicate findings through reports and presentations to technical stakeholders, executives, and legal counsel. Help mature threat-hunting capabilities through research, methodology development, search-query creation, and scripting. The role also values incident response, cloud-platform knowledge, remediation planning, and responsible use of AI technologies.
Top Skills:
Active DirectoryAWSAzureGCPKerberosLinuxPythonWindows
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Leads internal and external compliance audits, assessments, attestations, and certifications across security, privacy, and enterprise frameworks. Develops system integrations and scalable automation to streamline audit and compliance programs. Communicates with internal and external stakeholders, evaluates risks, improves information security processes, and applies AI technologies to enhance workflows and decision-making.
Top Skills:
Artificial IntelligenceBashCloud EnvironmentsPowershellPythonWorkflow Automation And Orchestration Tools
Consumer Web • eCommerce • Marketing Tech • Payments • Software • Design • SEO
Own end-to-end product design for commercial experiences including plan selection, cart, checkout, cross-sell, pricing, and merchandising. Translate monetization strategies into scalable UX systems, partner with product, engineering, revenue systems, platform, and data science teams, and connect discovery through post-purchase expansion. Establish design standards across Commercial Growth and mentor designers while driving ambiguous, cross-functional initiatives from strategy to shipped product.
Top Skills:
A/B TestingData Science PlatformsPersonalization And Targeting SystemsRevenue Systems
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
