Lead Data Engineer

Job Description

Who We Are:

AND Digital are a tech company focused on accelerating digital delivery and dedicated to closing the digital skills gap. We’ve been helping organisations build better digital products and stronger digital teams since 2014.

We believe our work should always leave a legacy for the client. We do this through close relationships with our offices (or ‘Clubs’) so that our partners are always prioritised by a regional team close to them.

This unique model has driven success for our clients and ourselves, evidenced by our remarkable organic growth since 2014. 

Join us - and help us fulfil our mission to close the world’s digital skills gap. This role is mostly remote, with occasional trips to the office - in either London, Birmingham, Manchester, Bristol, Reading, Edinburgh or Leeds so you would ideally be based near a local 'clubhouse.'

Lead Data Engineer

We are seeking a Lead Data Engineer to act as the technical practice lead for the Luna data team. This is an important role responsible for leading a high business priority migration from AND’s existing Redshift environment to our new AWS RDS (Postgres) architecture.

Beyond technical architecture, you will be a mentor and skills builder for the team. You will guide the technical growth of our Data Analysts, transitioning them from SQL-centric workflows to production-ready Python and DataOps practices, while collaborating closely with stakeholders to translate business needs into scalable technical roadmaps.

What you'll bring to the table:

Technical Leadership & Architecture:

• Lead new Data Pipeline Delivery: Implement the new Python-based ETL pipelines for data movement from Luna’s microservice DynamoDB tables to RDS transactional DB (Postgres OLTP) and subsequently to the RDS data warehouse (Postgres OLAP).
• Platform Optimization: Leverage expertise from previous data setups to optimize the new Postgres OLTP for transactional performance and the OLAP environment for analytics effectiveness, with both being conscious of on-going cost management. 
• Standard Setting: Establish and enforce DataOps standards, including version control (Git), automated CI / CD deployment, and schema migration management using tools like Liquibase or Drizzle ORM.

Team Enablement & Mentorship:

• Hands-on Coaching: Actively mentor team members, elevating their skills in Python, Git, and engineering workflows through code reviews and workshops.
• Code Quality: Conduct rigorous code reviews, providing detailed, educational feedback that explains best practices and clearly outlines required changes to elevate team standards.
• AI Engineering patterns: Support the definition and implementation of AI tooling and practices to augment the Data engineering team. 

Strategic Agile Delivery:

• Initiative Planning: Break down larger reporting initiatives into manageable epics and stories within our Agile framework (a mix of Scrum for larger work items, and Kanban for smaller continuous flow items).
• Stakeholder Management: Network with business domains to capture requirements and provide strategic guidance on the data platform migration plan.
• Product Partnership: Support the Data Product Owner by providing the technical context necessary to prioritize the team's backlog effectively.
  

Required Experience & Technical Skills:

• Data Engineering - 7 to 10 years experience building and supporting full-stack data pipelines from source to reporting.
• AWS Mastery - 5+ years deep experience with AWS, ideally administration and optimization of RDS / Aurora (Postgres) and Redshift.
• Python - 5+ years expert-level Python, with specific experience working on ETL pipelines, including libraries like Pandas and orchestration tools like Airflow or Prefect.
• PostgreSQL - 5+ years expert-level SQL and database architecture (partitioning, indexing) for both OLTP and OLAP.
• DataOps / Tools - 3+ years expertise in Git, with solid understanding of Git branching and workflow, and schema management (e.g., Liquibase, Flyway, or Drizzle).

Why join AND Digital?

We have three values: wonder, share, and delight. These values inform how we work with clients, and our culture: what it feels like to work for AND. We believe collaboration, ambition, curiosity and fun can drive innovation by creating a better environment for problem-solving.

By joining AND, we’ll provide:

• Opportunities to work on projects with big clients and the chance to produce meaningful work that makes a difference to people’s lives.
• A “Blended Working” model, meaning you will be able to work in a range of locations from; your home, in your clubhouse, on a client, as well as just a change of scenery.
• A dedicated career scrum team, designed to help you reach your career goals and develop the skills you need to be your best self.
• An annual budget for training and upskilling, including allocated days off so you don’t have to study in your own free time.
• Monthly and quarterly team socials - on us - ranging from after work drinks, to driving experience days with your fellow club members.
• A safe environment for you to be yourself and challenge yourself.
  

Benefits

• 25 days holiday allowance + bank holidays
• Share scheme
• A £1000 flexifund to use on a personalised list of benefits such Gym membership, Cycle to Work Scheme, Health, dental and optical cash plan
• 6% pension
• PLUS many more

For a full list of benefits -

Equal Opportunities Statement

At AND Digital we embrace diversity and are committed to equal opportunities. We are actively recruiting for a diverse and inclusive workforce so want to ensure we do everything we can to support your application.

We want you to feel safe and empowered to let us know if you require any adjustments to be made to your application or interview process so please speak to our recruitment team.