Design and build RAG-focused data pipelines for LLMs: ETL/ELT for structured and unstructured data, vector database architecture, embedding and chunking strategies, data cleaning/PII removal, metadata engineering, low-latency streaming, and evaluation/versioning to support autonomous agents.
We are seeking a Data Engineer who will be responsible for the "Ingestion-to-Insight" pipeline that allows autonomous agents to access, search, and reason over vast amounts of proprietary and public data.
Your role is critical: you will design the RAG (Retrieval-Augmented Generation) architectures and data pipelines that ensure our agents have the right context at the right time to make accurate decisions.
Key Responsibilities
- AI-Ready Data Pipelines: Design and implement scalable ETL/ELT pipelines that process both structured (SQL, logs) and unstructured (PDFs, emails, docs) data specifically for LLM consumption.
- Vector Database Management: Architect and optimize Vector Databases (e.g., Pinecone, Weaviate, Milvus, or Qdrant) to ensure high-speed, relevant similarity searches for agentic retrieval.
- Chunking & Embedding Strategies: Collaborate with AI Engineers to optimize data chunking strategies and embedding models to improve the "recall" and "precision" of the agent's knowledge retrieval.
- Data Quality for AI: Develop automated "Data Cleaning" workflows to remove noise, PII (Personally Identifiable Information), and toxicity from training/context datasets.
- Metadata Engineering: Enrich raw data with advanced metadata tagging to help agents filter and prioritize information during multi-step reasoning tasks.
- Real-time Data Streaming: Build low-latency data streams (using Kafka or Flink) to provide agents with "fresh" data, enabling them to act on real-time market or operational changes.
- Evaluation Frameworks: Construct "Gold Datasets" and versioned data snapshots to help the team benchmark agent performance over time.
Required Skills & Qualifications
- Experience: 4+ years in Data Engineering, with at least 1 year focusing on data for LLMs or AI/ML applications.
- Python Mastery: Deep expertise in Python (Pandas, Pydantic, FastAPI) for data manipulation and API integration.
- Data Tooling: Strong experience with modern data stack tools (e.g., dbt, Airflow, Dagster, Snowflake, or Databricks).
- Vector Expertise: Hands-on experience with at least one major Vector Database and knowledge of similarity search algorithms (HNSW, Cosine Similarity).
- Search Knowledge: Familiarity with hybrid search techniques (combining semantic search with traditional keyword search like Elasticsearch/BM25).
- Cloud Infrastructure: Proficiency in managing data workloads on AWS, Azure, or GCP.
Preferred Qualifications
- Experience with LlamaIndex or LangChain for data ingestion.
- Knowledge of Graph Databases (e.g., Neo4j) to help agents understand complex relationships between data points.
- Familiarity with "Data-Centric AI" principles—prioritizing data quality over model size.
Compensation, Benefits and Duration
Minimum Compensation: USD 38,000
Maximum Compensation: USD 133,000
Compensation is based on actual experience and qualifications of the candidate. The above is a reasonable and a good faith estimate for the role.
Medical, vision, and dental benefits, 401k retirement plan, variable pay/incentives, paid time off, and paid holidays are available for full time employees.
This position is not available for independent contractors
No applications will be considered if received more than 120 days after the date of this post
Similar Jobs
Agency • Information Technology
Design, build, and maintain scalable ETL/ELT pipelines into Snowflake, enforce data quality, optimize data models and query performance for Power BI reporting, produce documentation, and troubleshoot data issues while collaborating with stakeholders.
Top Skills:
AWSAzureEltETLGCPOraclePower BISnowflakeSQL
Agency • Information Technology
Design, build, modernize, and maintain operational and analytical data capabilities for Wealthscape Reporting, Analytics and Insights. Perform solution design, data analysis, ETL development, and production rollouts using Snowflake, AWS, cloud databases, and CI/CD pipelines. Collaborate across teams in a fast-paced financial services environment.
Top Skills:
AnsibleAWSAws LambdaCi/CdContainerizationDockerInformaticaJenkinsLinuxMavenOraclePostgresPythonShell ScriptingSnaplogicSnowflakeStashUnix
Big Data • Cloud • Productivity • Software • Database • Analytics • Automation
As a Senior Data Engineer at Jellyfish, you'll build and maintain data pipelines, optimize orchestration, automate CI/CD processes, and enhance data integration while ensuring high performance and reliability.
Top Skills:
AirflowBigQueryDagsterDatabricksDbtPrefectPysparkPythonRedisSnowflakeSQLTerraform
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
