Lead engineering team to architect, build, and operate large-scale, distributed Node.js web scraping systems. Design data extraction pipelines, solve anti-bot challenges (CAPTCHA, fingerprinting, IP rotation), manage proxies and storage, optimize cloud-native, containerized deployments, implement monitoring and CI/CD, ensure legal compliance, and drive technology roadmap and team growth.
We are seeking a Director of Software Engineering with deep expertise in Node.js development and large-scale web scraping. This role will lead the engineering team, designing and optimizing high-performance, distributed web scraping systems. The ideal candidate has extensive experience in handling anti-bot measures, data pipeline optimization, and scalable cloud-based architectures.
Key Responsibilities- Software Engineering & Web Scraping Leadership:
- Architect, develop, and maintain scalable and distributed web scraping systems using Node.js.
- Design and implement data extraction pipelines to process large volumes of structured and unstructured data.
- Develop solutions to bypass anti-bot mechanisms, including CAPTCHA handling, session management, fingerprinting, and IP rotation.
- Optimize scraping processes for performance, reliability, and efficiency while managing proxy services(residential, datacenter, rotating).Oversee data storage and processing strategies, ensuring high availability and consistency.
- Collaborate with Product, DevOps, and Data Science teams to integrate extracted data into analytics and business applications.
- Implement best practices for microservices, API integrations, and real-time data streaming.
Key Responsibilities- Scalability, Security & DevOps:
- Lead the transition to cloud-native, containerized, and serverless architectures for web scraping.
- Ensure compliance with legal and ethical standards (robots.txt, GDPR, CCPA, etc.).Optimize cloud resources (AWS, GCP, or Azure) to support high-throughput scraping.
- Manage real-time monitoring and alerting systems to detect scraping failures, IP bans, or performance bottlenecks.
- Work closely with DevOps teams to optimize CI/CD pipelines, automated deployments, and system scalability.
Key Repsonsibilities- Engineering Team Management & Strategy:
- Lead, mentor, and grow a high-performance engineering team.
- Define and execute the technology roadmap, aligning with business objectives.
- Foster a culture of continuous learning, collaboration, and innovation.
- Implement agile development methodologies (Scrum, Kanban) to optimize project execution.
- Ensure code quality, security, and best practices across all engineering efforts.
Qualifications & Experience- Technical Expertise:
- 10+ years of experience in software engineering, with at least 5+ years in web scraping and large-scale data extraction.
- Strong hands-on expertise in Node.js, Puppeteer, Playwright, Cheerio, Selenium, and headless browser automation.
- Extensive experience in handling CAPTCHAs, IP rotation, session management, and anti-bot evasion techniques.
- Deep knowledge of proxy management (residential, datacenter, rotating, and VPNs).Experience with NoSQL/SQL databases (MongoDB, PostgreSQL, Redis, Elasticsearch, etc.).
- Familiarity with data processing frameworks (Kafka, RabbitMQ, Spark, Airflow, etc.).Strong experience with CI/CD, containerization (Docker, Kubernetes), and cloud deployment (AWS/GCP/Azure).
Qualifications & Experience- Leadership & Soft Skills:
- Proven track record of scaling engineering teams and leading complex projects.
- Strong problem-solving and debugging skills, especially for scraping challenges and performance bottlenecks.
- Excellent communication and stakeholder management skills.
- Passion for mentorship, team development, and continuous learning.
Preferred Qualifications:
- Experience with machine learning for data extraction and NLP.
- Knowledge of browser fingerprinting and bot detection mechanisms.
- Familiarity with enterprise-scale web crawling frameworks (Scrapy, Colly, Apify, etc.).
- Prior leadership experience in data-driven businesses or web scraping startups.
Similar Jobs
Cloud • eCommerce • Information Technology • Professional Services • Software
Manage multiple customer-facing technology implementation projects from kickoff through closure. Own delivery outcomes while coordinating customers, implementation engineers, executives, and cross-functional teams. Oversee scope, schedules, budgets, risks, quality, forecasting, and resource management. Lead planning sessions and delivery meetings, communicate project status, and help improve PMO best practices. The role focuses on cloud-based, integration-oriented solutions, with EDI, MFT, data movement, or file transfer experience preferred.
Top Skills:
Cloud TechnologyData MovementEdiFile Transfer TechnologiesMftProject Portfolio Management (Ppm) Software
Blockchain • Fintech • Payments • Financial Services • Cryptocurrency • Web3
Own long-term product strategy and roadmap for Circle's digital assets and capital markets platform. Lead cross-functional initiatives, manage regulatory and market risks, define forecasting and liquidity models, influence executives, and mentor product managers to scale global stablecoin and capital access infrastructure.
Top Skills:
AnalyticsBlockchainData ScienceForecastingLiquidityModelingOff-ChainOn-ChainStablecoinsTreasury
Fintech • Insurance • Financial Services
Provides advanced estate, business succession, wealth transfer, tax, and executive compensation planning guidance to high-net-worth clients and financial advisors. Evaluates legal documents and planning structures, develops actionable recommendations, leads presentations, collaborates with attorneys and tax professionals, and supports coordinated execution of client objectives. The role also manages consultation workflows, builds stakeholder relationships, and contributes thought leadership and planning education.
Top Skills:
Ai-Enabled ToolsFinra Series 6Finra Series 7MS OfficePlanning SoftwarePowerPointResearch PlatformsWorkflow Tools
What you need to know about the Colorado Tech Scene
With a business-friendly climate and research universities like CU Boulder and Colorado State, Colorado has made a name for itself as a startup ecosystem. The state boasts a skilled workforce and high quality of life thanks to its affordable housing, vibrant cultural scene and unparalleled opportunities for outdoor recreation. Colorado is also home to the National Renewable Energy Laboratory, helping cement its status as a hub for renewable energy innovation.
Key Facts About Colorado Tech
- Number of Tech Workers: 260,000; 8.5% of overall workforce (2024 CompTIA survey)
- Major Tech Employers: Lockheed Martin, Century Link, Comcast, BAE Systems, Level 3
- Key Industries: Software, artificial intelligence, aerospace, e-commerce, fintech, healthtech
- Funding Landscape: $4.9 billion in VC funding in 2024 (Pitchbook)
- Notable Investors: Access Venture Partners, Ridgeline Ventures, Techstars, Blackhorn Ventures
- Research Centers and Universities: Colorado School of Mines, University of Colorado Boulder, University of Denver, Colorado State University, Mesa Laboratory, Space Science Institute, National Center for Atmospheric Research, National Renewable Energy Laboratory, Gottlieb Institute
