The Data Integration Lead designs and implements scalable data pipelines using Databricks, optimizes data workflows, ensures data security, and collaborates with stakeholders to create technical solutions.
Data Integration Lead Responsibility Design and implement scalable data pipelines for ingesting and transforming data primarily using Databricks and leveraging PySpark notebooks, Spark SQL, and Python. Develop ETL processes to extract, transform, and load data from diverse sources into data lakehouse architecture on Databricks Analyze existing integration Terdata landscape (TPT, BTEQ, Talend, IBM Sterling) Define ingestion and integration strategy for Databricks Ensure seamless data flow from source systems to Lakehouse Lead integration design and oversee pipeline migration Optimize data processing workflows to enhance performance and efficiency using Databricks capabilities. Ensure data security and compliance with data privacy regulations throughout the data engineering process. Deliver high quality data products based on business requirements Collaborate with stakeholders to gather, understand requirements and create technical solutions using Microsoft Azure stack
Similar Jobs
Cloud • Software
The Forward Deployed Engineering Lead will design and implement data science and integration solutions for defense and intelligence environments, working directly at customer sites to drive mission outcomes using various technologies and platforms.
Top Skills:
Apache AirflowApigeeAzure Api ManagementBigQueryDatabricksDell BoomiDockerJavaKubernetesMulesoft Anypoint PlatformPythonRedshiftRegrelloSnowflakeSQL
Cloud • Fintech • Software • Business Intelligence • Consulting • Financial Services
The Procurement Intern supports vendor relationships and purchasing activities, managing contracts, processing purchase requests, maintaining records, and assisting in source activities within procurement operations.
Top Skills:
MS Office
Big Data • Fintech • Mobile • Payments • Financial Services
Support Affirm's revenue growth by partnering with cross-functional leaders, building financial models, analyzing performance metrics, and delivering analytics for executives.
Top Skills:
CursorExcelPythonSigmaSQL
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
