Principal Data Platform Engineer at IHS Markit Digital
Principal Data Platform Engineer
At IHS Markit, we are building a software solution that connects data in revolutionary ways, illuminating answers that were previously impossible to find and empowering our clients to envision the future, so they can determine the best course of action in the present. We are disrupting the current digital transformation landscape with state-of-the-art AI developed by a passionate team to explore and push the boundaries of digital transformation technologies.
Our development and AI teams produce and consume a wide variety and volume of high impact data which support innovative and intelligent solutions. Experts in the subdomains of software engineering and artificial intelligence, our teams have a need for expertly crafted, scalable, and accessible data platforms from which to assemble and deliver “self-service” data pipelines. All of our new products are developed using a microservice architecture, are containerized, and are then deployed on container management systems such as Kubernetes. Our teams subscribe to a model where time-to-market functions as a vital measure of our performance, productivity, and success. We are committed to staying ahead of the curve and are always looking at new technologies and methodologies to achieve that aim.
IHS Markit is seeking an adept, architecturally minded Principal Data Platform Engineer based in Denver, CO who will report to the Executive Director of Software Development for our currently unreleased digital transformation solution. The candidate has experience translating developer, data scientist, and business needs into a vision of what should be built. This means skillfully choosing and deploying big data technologies and infrastructure (e.g. Kafka, Spark, DynamoDB, Neo4j), building frameworks atop to facilitate autonomous use for these internal customers, and abstracting production solutions so they feel like “magic” for your stakeholders such that things “just fall into place.” As Principal Data Platform Engineer, you have a diversity of software development experience rather than a single tool suite or paradigm. You are hands-on, data-driven, and highly collaborative. You should enjoy being "full stack" in the sense that you own a product from beginning to end by designing, constructing, integrating, testing, documenting, and supporting your creations. You have a passion for engineering best-practices and the ability to lucidly communicate with fellow engineers as well as non-technical colleagues. You will present and communicate team status to the Technology leaders and Executive team.
- Assume technological and administrative responsibilities for a focused team (2-3 members)
- Provide (thought) leadership in technology direction, technical services, vendor partnership, and industry standards adoption.
- Are “biased to action” and not easily blocked by problems and difficulties, instead taking ownership
- Believe in monitoring, QA, and security as a first-class citizen in any data product.
- Able to understand requirements in context to design solutions, rather than take them at face value.
- Excited to design data platforms and tools that abstract implementation details for developers, analysts, and data scientists, enabling data transit and storage “as a service.”
- Dedicated to automation, documentation, and collaboration at all stages of the engineering workflow.
- Sensitive to the balance of performance and cost with the ability to communicate and optimize for the requirements of each.
- Passionate about mentoring colleagues and educating the organization on data engineering best practices.
- Maintain an excellent understanding of the business long term goals and strategy and ensure that designs are aligned with these - able to see the forest through the trees.
Education / Experience
- Degree in Computer Science, related field or equivalent experience.
- Eight (8) or more years of increasing responsibility in technical data roles, with 3 or more years of experience as a data engineer lead.
- Expertise in various big data technologies both open source and cloud native, AWS preferred (Kafka/Kinesis, Presto/Athena, Spark/EMR, Airflow, Hive, Drill).
- Expertise in data model design with sensitivity to usage patterns and goals – schema, scalability, immutability, idempotency, etc.
- Mastery of at least two of the following languages – Scala, Java, Python, Go.
- Expertise in the suite of NoSQL models and frameworks – especially large graphs.
- Track record of choosing the right transit, storage, and analytical technology to simplify and optimize user experience.
- Real-world experience developing highly scalable solutions using micro-service architecture designed to democratize data to everyone in the organization.
- Able to function autonomously and successfully in ambiguous situations.