The Data Engineer will design and maintain scalable data pipelines, ensure data quality, implement governance standards, and collaborate with scientists and engineers for AI applications.
At Agile Dream Team, we harness the power of data to drive intelligent business decisions. We are looking for a highly skilled Data Engineer to design, build, and maintain scalable data pipelines and infrastructure for AI-driven applications.
Learn more about us at: www.agiledreamteam.com
Role Overview
As a Data Engineer, you will be responsible for building and optimizing data pipelines, ensuring data quality, and enabling real-time and batch processing for analytics and AI models. You will work closely with Data Scientists, AI Engineers, and Software Developers to develop scalable data solutions that support business intelligence and machine learning applications
Key Responsibilities
- Design, develop, and maintain scalable ETL/ELT pipelines for structured and unstructured data.
- Build data architectures that support batch and real-time data processing.
- Optimize data storage, retrieval, and performance for analytics and AI applications.
- Work with big data processing frameworks such as Apache Spark, Apache Flink, or Kafka.
- Implement data governance, security, and compliance standards.
- Ensure high data quality through data validation, monitoring, and anomaly detection.
- Automate data ingestion, transformation, and processing for AI/ML workflows.
- Deploy and manage data solutions in AWS, Azure, or GCP cloud environments.
- Collaborate with Data Scientists and AI Engineers to enable data-driven AI models.
- Utilize SQL and NoSQL databases for efficient storage and retrieval of large datasets.
Required Skills & Experience
- Proficiency in data engineering frameworks: Apache Spark, Hadoop, Airflow, DBT.
- Strong knowledge of SQL, NoSQL, and data modeling techniques.
- Experience in ETL/ELT pipeline development using Python, Scala, or Java.
- Hands-on experience with cloud data platforms (AWS Redshift, BigQuery, Snowflake, Databricks).
- Expertise in real-time streaming technologies (Apache Kafka, Kinesis, Pulsar).
- Experience with data warehouse and lakehouse architectures.
- Strong understanding of data partitioning, indexing, and performance tuning.
- Familiarity with containerized deployments (Docker, Kubernetes) for data pipelines.
- Experience in CI/CD automation for data workflows.
Preferred Qualifications
- Experience with machine learning data pipelines and feature engineering.
- Hands-on knowledge of data lake technologies (Delta Lake, Iceberg, Hudi).
- Familiarity with Terraform, Ansible, or other infrastructure-as-code (IaC) tools.
- Understanding of graph databases (Neo4j, ArangoDB) and time-series databases.
- Experience in data observability, lineage tracking, and governance.
Why Join Us?
- Work with cutting-edge data technologies in a forward-thinking AI/ML company.
- 100% remote role with a flexible schedule.
- Opportunities for growth and continuous learning in Data Engineering and AI.
- Engage in high-impact data projects that power real-world AI applications.
- Competitive salary
Ready to shape the future of data engineering? Apply now!
Top Skills
Airflow
Apache Kafka
Spark
Aws Redshift
BigQuery
Ci/Cd
Databricks
Dbt
Docker
Hadoop
Java
Kinesis
Kubernetes
NoSQL
Pulsar
Python
Scala
Snowflake
SQL
Similar Jobs
Big Data • Cannabis • eCommerce • Logistics • Database • Business Intelligence • Big Data Analytics
Design and maintain a data analytics platform, build ETL pipelines, standardize and model data, optimize database performance, and ensure data quality.
Top Skills:
AirbyteAWSDbtExcelGithub ActionsGlueHevoMetabaseModeMySQLPostgresPower BIPythonSnowflakeSparkSQLTerraformThoughtspot
Aerospace • Hardware • Information Technology • Security • Software • Cybersecurity • Defense
Design and deploy BI solutions using AI techniques, develop data pipelines, ensure data quality, and collaborate with stakeholders for data analysis.
Top Skills:
AWSAzurePower BIPythonPyTorchRScikit-LearnSQLTableauTensorFlow
AdTech • Big Data • Information Technology • Marketing Tech • Sales • Software
As a Data Engineer III, you will develop and maintain high-performance ETL pipelines and scalable data processing systems on Google Cloud Platform, while collaborating with teams to deliver quality solutions.
Top Skills:
AirflowBigQueryDataflowGoogle Cloud PlatformKubernetesPythonSQL
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
