Design and implement scalable data pipelines using Databricks, develop ETL processes, optimize workflows, ensure data security, and collaborate with stakeholders.
Data Integration Lead Responsibility Design and implement scalable data pipelines for ingesting and transforming data primarily using Databricks and leveraging PySpark notebooks, Spark SQL, and Python. Develop ETL processes to extract, transform, and load data from diverse sources into data lakehouse architecture on Databricks Analyze existing integration Terdata landscape (TPT, BTEQ, Talend, IBM Sterling) Define ingestion and integration strategy for Databricks Ensure seamless data flow from source systems to Lakehouse Lead integration design and oversee pipeline migration Optimize data processing workflows to enhance performance and efficiency using Databricks capabilities. Ensure data security and compliance with data privacy regulations throughout the data engineering process. Deliver high quality data products based on business requirements Collaborate with stakeholders to gather, understand requirements and create technical solutions using Microsoft Azure stack Create comprehensive documentation of workflows, pipelines, and architecture.
Similar Jobs
Information Technology • Consulting
The Big Data Lead will manage project progress using Burn Down Charts, lead teams with SCRUM and Agile Kanban methodologies, and resolve conflicts.
Top Skills:
Agile KanbanCrystal FrameworkScrum
Information Technology • Consulting
Responsible for designing and maintaining data infrastructure, developing data pipelines, ensuring data quality, and collaborating with various teams to enhance data utility. Also involves communication with stakeholders and innovative solutions for business success.
Top Skills:
Azure Data FactoryAzure DatabricksC#CdgcCosmosdbData QualityDihFabricHdinsightInformatica CihJavaMaster Data ManagementPower BIPythonSnowflakeSQLSQL ServerSynapse
Information Technology • Consulting
The Big Data Lead will implement ETL pipelines, ensure data integrity, troubleshoot PySpark applications, and integrate with existing frameworks while leading a team.
Top Skills:
HadoopHiveKafkaPysparkPythonSparkSQL
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
