Principal Data Engineer

Responsibilities

Architect and Evolve Our Core Data Platform You will own the technical vision and roadmap for our data platform, steering its evolution on our modern cloud stack and ensuring it meets the demands of a rapidly scaling business.
• Own the Architecture: Design, implement, and refine a robust data lakehouse architecture (e.g., Medallion) using Databricks and Delta Lake to ensure data reliability and performance.
• Build Scalable Ingestion Frameworks: Develop and maintain resilient, reusable patterns for ingesting data from a diverse set of sources, including our systems, transactional databases, event streams, and third-party SaaS APIs.
• Define Data Modelling Standards: Lead the implementation of our core data modelling principles (e.g., Kimball dimensional modelling) to produce curated, intuitive datasets for business intelligence and product analytics.
• Implement Robust Governance: Use tools like Unity Catalog to establish a comprehensive data governance framework, covering data lineage, fine-grained access controls, and a user-friendly data catalogue.
• Manage Platform Performance and Cost: Develop and implement strategies for monitoring, optimising, and forecasting our Databricks and cloud expenditure, ensuring the platform is both powerful and cost-effective.


Champion Engineering Excellence and Best Practice You will be the driving force for maturing our data operations, embedding a culture of quality, automation, and reliability into everything we do.
• Automate Everything with CI/CD: Implement and advocate for automated CI/CD pipelines (e.g., using GitHub Actions) for all data assets, including dbt models, infrastructure changes, and Databricks jobs.
• Embed Git-Based Workflows: Champion a Git-first culture for all data transformation code, establishing clear processes for branching, code reviews, and version control.
• Embed Automated Data Quality: Implement comprehensive, automated data quality testing at every stage of our pipelines using tools like dbt test, ensuring data is accurate and trustworthy.
• Introduce Data Observability: Establish thorough monitoring, logging, and alerting for all data pipelines to proactively detect, diagnose, and resolve issues before they impact the business


Be a Strategic Partner Across the Business You will connect the technical capabilities of the data platform to Deputy's strategic objectives, acting as a key advisor to stakeholders across the organisation.
• Translate Business Needs into Technical Solutions: Collaborate directly with leaders in Product, Engineering, Sales, finance and Customer Success to understand their challenges and design data solutions that enhance our product, improve customer outcomes, and drive business strategy.
• Guide Data Best Practices: Advise analysts, data scientists, and other stakeholders on how to best leverage the data platform for impactful analysis and data-driven decision-making.
• Act as the Technical Authority: Serve as the go-to expert on our data architecture, running workshops and design sessions to align technical direction with business needs


Lead, Mentor, and Elevate Our Data Team As a technical member of the team, you will be instrumental in upskilling your colleagues and shaping the future of the data function at Deputy.
• Mentor and Coach: Actively mentor data analysts and engineers through pair programming, constructive code reviews, and technical guidance to grow their skills in Python, SQL, and data modelling.
• Foster a Community of Practice: Lead initiatives like a 'data guild' to encourage knowledge sharing, explore new technologies, and collaboratively solve complex problems
• Shape the Team's Future: Partner with data leadership to define career progression pathways for data engineering and take a leading role in interviewing and hiring new team members.

Skills & Experience

• Mastery of data architecture principles, data modelling frameworks (e.g., dimensional modelling), and a strong understanding of data governance and security best practices.
• A strong software engineering mindset, with significant experience implementing CI/CD for data, Git-based workflows, and automated data quality testing.
• Exceptional communication and stakeholder management skills, with a proven ability to translate complex technical concepts for non-technical audiences and influence business decisions.
• A genuine passion for leadership and mentorship, with a track record of elevating the technical skills of those around you.
Tech Stack:
• Dbt
• Databricks, Unity Catalog 
• Terraform
• AWS: Redshift, Dynamo db, API gateway, Cloud Watch, Lambda, Streaming with Kenisis/Firehose, Glue, Bedrock
• Stitch & Fivetran
• Languages required include advanced SQL, python