Head of Data Science - Product Experimentation & Machine Learning

Job Description

Head of/Staff Data Scientist – Product Experimentation & Machine Learning

About Checkmate
Checkmate builds the operating system for digital ordering in restaurants, powering integrations between POS systems, delivery platforms, and restaurant brands. Our products sit at the center of how millions of orders move across systems every day — which makes experimentation, automation, and ML-driven optimization core to our competitive advantage.

Role Overview
You will be the technical and strategic owner of Checkmate’s product experimentation and evaluation stack. Your mission is to design, run, and scale machine-learning-driven experiments that improve ordering accuracy, automation quality, conversion, reliability, and merchant success across the platform.

This role will require a blend of both strategic and hands on work. This person will be expected to be comfortable getting their hands in the weeds!

You will partner closely with Product, Engineering, and Data to turn ideas into controlled experiments, build predictive models to power decision-making, and translate results into product and roadmap decisions. This role blends ML, causal inference, and A/B testing in a high-volume, production environment where small improvements generate massive real-world impact.

100% Remote

Essential Job Functions

• Own end-to-end product experimentation: hypothesis generation, metric definition, experimental design (A/B, multivariate, sequential testing), analysis, and executive-level interpretation.
• Design and maintain ML-powered evaluation frameworks for product changes, automation quality, and system reliability (e.g., order accuracy, routing, error rates, conversion).
• Build and deploy predictive models, classifiers, and ranking systems that power experimentation, personalization, and product optimization.
• Partner with product and engineering to test new features, workflows, and ML models through controlled experiments and incremental rollouts.
• Lead offline and online model evaluation, comparing baselines, candidate models, and product variants using rigorous statistical methods.
• Use causal inference and quasi-experimental methods when randomized experiments are not feasible.
• Develop experiment pipelines and instrumentation: logging, dashboards, monitoring, and automated analysis to ensure measurement integrity.
• Perform failure-mode and error analysis to guide product iteration and model improvement.
• Translate experiment outcomes into clear product decisions, influencing roadmap prioritization and system design.
• Drive experimentation at scale in a fast-moving environment, balancing speed, rigor, and business impact.
• Lead and mentor data scientists and analysts, setting standards for experimentation, modeling, and evaluation across the organization.

Requirements

• 8–12+ years of experience in data science, machine learning, or applied experimentation roles.
• Demonstrated expertise in product experimentation and A/B testing, including design, execution, and statistical evaluation.
• Strong background in machine learning, statistical modeling, and causal inference applied to real-world products.
• Experience building and evaluating predictive models, classifiers, or ranking systems in production environments.
• Proven ability to operate in both startup-style experimentation and scaled product ecosystems.
• Experience leading teams, setting technical direction, and delivering cross-functional impact.
• Excellent coding skills in Python (or similar), strong SQL, and experience building data pipelines or ML systems.
• Ability to connect technical findings to product and business outcomes.
• Strong communication skills with technical and non-technical stakeholders.

Preferred Qualifications

• Experience with experiment platforms or building internal tooling for experimentation and model evaluation.
• Experience deploying ML models into high-volume transactional systems.
• Experience working with NLP, LLMs, or automation systems.
• Experience with multi-modal or operational data (e.g., orders, text, voice, or system events).

Benefits

• Health Care Plan (Medical, Dental & Vision)
• Retirement Plan (401k)
• Life Insurance (Basic, Voluntary & AD&D)
• Flexible Paid Time Off
• Family Leave (Maternity, Paternity)
• Short Term & Long Term Disability
• Training & Development
• Work From Home
• Stock Option Plan