Build and maintain scalable, secure data pipelines and AI-ready datasets using Azure Databricks, Snowflake, Python, and Spark; support RAG solutions, integrate Azure AI services, implement data governance, CI/CD, and production monitoring for enterprise generative AI initiatives.
This is a remote position.
Job SummaryWe are seeking a skilled Data Engineer to support enterprise AI and Generative AI initiatives by building scalable, secure, and AI-ready data platforms. The ideal candidate will have strong expertise in Azure Databricks, Snowflake, Python, and SQL, along with experience integrating modern Azure AI services into enterprise data ecosystems.
In this role, you will partner with Data Engineers, AI Engineers, Data Scientists, and business stakeholders to develop high-quality data pipelines, enable Retrieval-Augmented Generation (RAG) solutions, and prepare enterprise data for AI-powered applications using Microsoft's AI ecosystem.
- Design, develop, and maintain scalable data pipelines using Azure Databricks, Snowflake, and Azure Data Factory.
- Build ELT/ETL pipelines to ingest data from ERP, CRM, manufacturing, supply chain, and other enterprise applications.
- Develop AI-ready datasets that support machine learning and Generative AI use cases.
- Integrate structured, semi-structured, and unstructured data into centralized data platforms.
- Support Retrieval-Augmented Generation (RAG) solutions by preparing document repositories, metadata, embeddings, and search indexes.
- Collaborate with AI Engineers to expose enterprise data securely to LLM-powered applications.
- Optimize data models, transformations, and query performance for analytics and AI workloads.
- Implement data quality, governance, lineage, monitoring, and security best practices.
- Build reusable data transformation frameworks using Python, SQL, Spark, and dbt (if applicable).
- Develop REST API integrations to support AI services and enterprise applications.
- Participate in architecture discussions, code reviews, CI/CD implementations, and Agile ceremonies.
- Support production deployments, monitoring, troubleshooting, and performance tuning.
- 6+ years of experience in Data Engineering.
- Strong hands-on experience with Azure Databricks and Snowflake.
- Proficiency in Python, SQL, and PySpark.
- Experience building scalable ELT/ETL pipelines using Azure Data Factory or similar orchestration tools.
- Strong understanding of data warehousing, dimensional modeling, and data lake architectures.
- Experience working with REST APIs and integrating cloud-based services.
- Familiarity with Git, Azure DevOps, and CI/CD pipelines.
- Experience working in Agile/Scrum environments.
- Experience supporting enterprise AI or Generative AI initiatives.
- Working knowledge of Azure Machine Learning (Azure ML).
- Experience with Microsoft AI Foundry (Azure AI Foundry) for AI solution development and orchestration.
- Experience integrating Azure OpenAI Service into enterprise applications.
- Knowledge of Azure AI Search for enterprise search and Retrieval-Augmented Generation (RAG) solutions.
- Exposure to Microsoft Copilot Studio for developing AI-powered copilots and conversational experiences.
- Understanding of vector search, embeddings, prompt engineering, and LLM integration.
- Experience preparing enterprise data for AI model training, inference, and knowledge retrieval.
Similar Jobs
Cloud • Computer Vision • Information Technology • Sales • Security • Cybersecurity
Lead design, build, and deploy of large-scale data platforms for LLMs, RAG, and agentic AI systems. Hands-on coding, architecting fault-tolerant pipelines, establishing MLOps/DataOps best practices, mentoring engineers, and operationalizing research into production across Exabyte-scale distributed systems.
Top Skills:
AirflowAWSBigQueryDaskDevsecopsDockerFlinkGCPGoJvmKafkaKubeflowKubernetesLangchainLlamaindexLlmsMlflowOciPulsarPythonRetrieval-Augmented Generation (Rag)RustSagemakerSnowflakeSparkVertex Ai
Artificial Intelligence • Big Data • Healthtech • Information Technology • Machine Learning • Software • Analytics
Design, build, and deploy AI/ML and generative AI solutions and full-stack applications. Develop frontend experiences, backend services, microservices, APIs, and cloud-native data integrations. Implement RAG, LLMs, vector DBs, dashboards, reporting, and self-service analytics to support quality, patient safety, and workflow automation. Provide technical leadership, mentor engineers, and apply CI/CD, testing, observability, security, and performance best practices.
Top Skills:
AgentsApache SupersetAutomated TestingAWSAzureAzure Ai ServicesCi/CdDatabricksEvent-Driven ArchitectureGCPGenerative AiJavaJavaScriptLangchainLlmsMicroservicesMicrosoft FabricObservabilityOpenaiPower BIPrompt EngineeringPythonRagReactRest ApisSemantic KernelSnowflakeSpring BootSQLTypescriptVector Databases
Enterprise Web • Mobile • Professional Services • Software
Design, build, and own scalable data pipelines and evaluation systems that power production AI features and internal analytics. Ensure data quality across ingestion, modeling, and reporting, collaborate with ML and analytics teams, deploy and monitor ML systems, and establish standards for data work and evaluation.
Top Skills:
AirflowAWSDagsterGCPPostgresPythonSnowflake
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
