Michael Barley
Download PDF ↓Professional Summary
Data Engineer with 6+ years of experience designing and operating high-throughput data pipelines, real-time streaming systems, and analytical data warehouses on AWS. Currently processing billions of behavioural events across 700+ ecommerce retailers at Salesfire. Skilled in Python, SQL, and TypeScript with deep expertise in ClickHouse, Kafka, Terraform, and container orchestration. Strong focus on infrastructure as code (IaC), observability, and CI/CD-driven delivery.
Experience
Data Engineer
- Designed and maintained real-time data pipelines ingesting billions of behavioural events daily across 700+ ecommerce retailers using TypeScript, SQL, and Snowplow on AWS
- Engineered and optimised a ClickHouse analytical data warehouse, reducing average dashboard query times by over 80% compared to the previous MySQL-based solution
- Built an identity resolution system on Amazon Neptune ML, matching anonymous shoppers across devices and sessions into unified customer profiles, increasing addressable audience size by approximately 35%
- Developed data infrastructure powering AI-driven product recommendations served to millions of daily active shoppers, contributing to measurable uplift in client conversion rates
- Created automated reporting pipelines for client-facing performance metrics, conversion attribution, and revenue analytics, replacing manual Excel-based processes
- Deployed and managed production AWS infrastructure (ECS, S3, Lambda, RDS, Neptune) using Terraform, Docker, and Kubernetes, achieving 99.9% pipeline uptime
- Established pipeline observability using CloudWatch, custom alerting, and structured logging, reducing mean time to detection (MTTD) for data quality issues from hours to minutes
Software Engineer
- Built and maintained a multi-tenant SaaS platform for financial advice firms using Laravel, Vue.js, MySQL, and AWS, serving 50+ active firms
- Designed and implemented RESTful APIs and database schemas supporting complex financial workflows including compliance reporting and client portfolio management
- Improved application performance through query optimisation and caching, reducing average page load times by 40%
Software Engineer
- Developed Python web scrapers processing product data from 350+ ecommerce retailer sites at scale, extracting and normalising millions of product records for feed optimisation
- Architected and built a product feed management platform on AWS handling automated feed generation, validation, and distribution for Google Shopping and other channels
- Engineered a rules-based feed optimisation engine with Google Shopping CSS integration and Google Ads API connectivity, directly improving client campaign performance
- Built internal reporting dashboards and tooling supporting 350+ client accounts and £40M+ in annual ad spend, reducing manual reporting effort by approximately 60%
- Introduced CI/CD pipelines and automated testing to the development workflow, improving deployment frequency from fortnightly to multiple times per day
Technical Skills
- Languages Python, SQL, TypeScript, JavaScript
- Cloud & Infrastructure AWS (ECS, S3, Lambda, RDS, Neptune, CloudWatch), Terraform, Docker, Kubernetes
- Databases & Warehousing ClickHouse, PostgreSQL, MySQL, Amazon Neptune
- Streaming & Ingestion Apache Kafka, Debezium (CDC), Snowplow, Amazon Kinesis
- Data Engineering Apache Spark, Delta Lake, dbt, Apache Airflow, ETL/ELT pipelines
- Observability & Quality Prometheus, Grafana, Great Expectations, CloudWatch, structured logging
- Practices CI/CD (GitHub Actions), Infrastructure as Code (IaC), DevOps, agile, code review
Education
BSc (Hons) Computing — First Class
Portfolio
Open-source portfolio projects with full documentation at michaelbarley.github.io/projects, covering real-time streaming pipelines, batch ELT orchestration, data lakehouse architecture, CDC pipelines, data quality observability, CI/CD infrastructure, and dimensional modelling.