Lead the development of data pipelines and transformations in Azure Databricks, converting Scala programs to PySpark while leveraging various Azure technologies.
Job Description :
Location - US (remote) . East coast preferable
Primary Skill :
Technical Lead -Data Engneering
- Azure databricks- Build Data Pipelines using ADB-Leveraging Python and Spark
- Build Data Transformation in ADB
- Convert Scala Pgms to Py Spark Pgms
- Understand the SCALA program in DataBricks
- ADF
- Secondary Skills:
- Azure data lake gen 2
- Synapse analytics (dedicated SQL pool)
- Exp in delta Lake
- Pyspark
Nice to Have : Tidal Scheduler , Exp in Healthcare Domain
Top Skills
Adf
Azure Data Lake Gen 2
Azure Databricks
Delta Lake
Pyspark
Python
Spark
Synapse Analytics
Similar Jobs
Information Technology • Consulting
The Big Data Lead will manage database development, ETL/ELT processes, and data warehousing, optimizing performance and ensuring data pipelines work reliably.
Top Skills:
AWSAws GlueAws S3AzureAzure BlobAzure Data FactoryAzure DevopsGitJenkinsOracleSnowflakeSQLTalendTeamcity
Information Technology • Consulting
The Senior Azure Data Engineer will manage data ETL processes, mentor junior staff, and work on cloud technologies in Azure. Required experience includes Datawarehouse expertise.
Top Skills:
AdfAzureDatabricksDatawarehousePysparkPython
Information Technology • Consulting
The role involves developing data pipelines using Azure Databricks and PySpark, translating banking requirements into technical designs, and ensuring data quality and governance.
Top Skills:
Azure Data FactoryAzure DatabricksPysparkSynapse Analytics
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
