Design, build, and maintain cloud-native ETL/data pipelines using Databricks and AWS Glue. Ingest and transform structured and semi-structured data with PySpark, Python, and SQL. Integrate sources including MongoDB, implement data quality checks, optimize pipeline performance, and support production deployments, monitoring, and troubleshooting while collaborating with architects and analysts.
About the Role
We are looking for a Data Pipeline Engineer with strong expertise in Databricks, AWS Glue, Python, SQL, and MongoDB to design, develop, and optimize scalable data pipelines. The ideal candidate should have experience building cloud-native ETL solutions, transforming large datasets, and delivering high-quality data for analytics and business applications.
Key Responsibilities
- Develop and maintain scalable ETL/data pipelines using Databricks and AWS Glue.
- Build data ingestion and transformation workflows for structured and semi-structured data.
- Write efficient PySpark, Python, and SQL code for large-scale data processing.
- Integrate data from multiple sources, including MongoDB and relational databases.
- Implement data quality, validation, and reconciliation checks.
- Optimize pipeline performance, reliability, and scalability.
- Collaborate with architects, analysts, and application teams to deliver data solutions.
- Support production deployments, monitoring, and troubleshooting.
Required Skills
- 5+ years of experience in Data Engineering or ETL development.
- Strong hands-on experience with Databricks and PySpark.
- Experience with AWS Glue and AWS data services.
- Proficiency in Python and SQL.
- Experience working with MongoDB.
- Knowledge of ETL, data modeling, and data warehousing concepts.
- Familiarity with Delta Lake/Lakehouse architecture is preferred.
- Experience with Git and Agile development practices.
Good to Have
- Delta Lake
- Apache Spark optimization
- AWS S3
- CI/CD pipelines
- Healthcare or Benefits domain experience
Similar Jobs
.png)
Cloud • Insurance • Payments • Software • Business Intelligence • App development • Big Data Analytics
Build, model, and maintain scalable BigQuery-based data solutions on GCP. Implement performant data models, storage partitioning/clustering, ETL improvements, data validation, and documentation. Collaborate with architects, data scientists, and engineers to deliver governed, high-quality data for reporting and AI initiatives.
Top Skills:
Ansi SqlBigQueryBigquery SqlConfluenceGCPJIRAPythonSQL
Security • Cybersecurity
Lead design and modernization of scalable, secure data systems and real-time pipelines for xOT cybersecurity. Create data contracts, observability, lineage, and cross-functional solutions supporting cloud and on-prem deployments, Kubernetes, and stream processing to enable threat detection and intelligence.
Top Skills:
CloudData Observability FrameworksDockerGoJvm (Java/Scala/Kotlin)KubernetesMessage QueuesNode.jsOn-PremisesPythonRustStream Processing
Security • Cybersecurity
Design and maintain data pipelines, models, and transformation layers for analytics. Build data contracts, observability and lineage tooling, collaborate with product engineering, and optimize data structures for performance and scalability.
Top Skills:
Cloud-Native ArchitecturesData ContractsData Observability FrameworksDimensional ModelingDockerGoIncremental Data ModelsJvm (Java/Scala/Kotlin)KubernetesLineage TrackingMessage QueuingNode.jsPythonStream Processing
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
