Pipelines That Keep Running After Launch
A pipeline is easy to write and hard to keep correct. Sources change their schema, jobs fail at 3am, a dashboard shows a number nobody can explain, and people go back to spreadsheets. I build batch and streaming pipelines with Spark, Airflow and Kafka, with freshness and correctness checks, lineage and alerts built in. Today I own the pipelines that measure infrastructure cost and usage at Pinterest, a platform with 500M+ monthly users.
What Problems I Solve
Manual Data Work
Replace exports, copy-paste and hand-run scripts with scheduled pipelines that retry on failure and alert when something is wrong.
Numbers Nobody Trusts
Schemas, freshness and correctness checks, and lineage, so when a number looks wrong you can see where it came from and why.
Pipelines That Do Not Scale
Move jobs that time out or fall behind onto Spark, with partitioning and incremental processing that grow with the data.
Batch or Streaming
Choose the right mode for each use case. Nightly batch where it is enough, Kafka and Flink streaming where minutes matter.
Data Migrations
Move data between systems with different schemas, with validation and reconciliation at every phase and a safe way to re-run.
Self-Serve Analytics
Data models, a semantic layer and dashboards that let teams answer their own questions without waiting for a data engineer.
Value I Deliver
Trusted Numbers
Checks on schema, freshness and correctness run with every load, so bad data is stopped before it reaches a report.
Safe Re-Runs
Idempotent jobs that can be re-run for any date or batch without duplicates. A failed run is a retry, not a clean-up project.
Failures You Hear About
Alerts on late, failed or suspicious runs, so the data team finds out before the people reading the dashboard do.
Lineage
Every dataset shows where it came from and what depends on it, so a change upstream is not a surprise downstream.
Fewer Data Requests
Self-serve tooling and dashboards that answer the common questions, so engineers spend time on the hard ones.
Documented Models
Definitions for every metric and table, written where analysts and engineers will find them.
Real-World Implementations
Infrastructure Cost and Usage Pipelines
Challenge
Engineering teams at Pinterest need to see what their infrastructure costs and how it is used, so they can reduce their own spend. That only works if the cost and efficiency numbers are trusted across the company.
Solution
- Own the production ETL pipelines that measure infrastructure cost and usage
- Build and maintain batch and streaming pipelines on Spark and Airflow, from ingestion and data modeling to the semantic layer
- Define schemas, freshness and correctness checks, and lineage for infrastructure datasets
- Ship tooling and dashboards so infrastructure teams answer their own cost and capacity questions
Results
Infrastructure teams answer their own cost and capacity questions without a data engineer in the loop, from numbers backed by quality checks and lineage. See all case studies
Technologies Used
Credit Decisioning Engine
Challenge
Brazil's largest credit bureau needed "New Score", a credit decisioning engine that businesses across the country would rely on.
Solution
- Led a team of 6 engineers and owned the technical direction, sprint planning and code quality
- Built the data processing on PySpark to feed a neural network model
Results
New Score is used by thousands of Brazilian companies. Read the full case study
Technologies Used
Compliance Record Migration
Challenge
Compliance records had to move from Convercent to OneTrust, across two different data models, where any lost or corrupted record is a compliance incident for the customer.
Solution
- Owned the migration pipeline end to end, from design to production
- Idempotent writes, schema validation and transactional rollback across millions of records
- Phased validation across DEV, QA1, QA2 and PROD before each step went further
Results
Every client moved with zero data loss and zero production incidents. Read the full case study
Technologies Used
Technologies & Tools I Work With
Processing
Orchestration
Streaming
Storage & Query
Languages
Infrastructure
How I Work
Start From the Question
Which decisions the data supports, who reads it and how fresh it has to be. That decides batch or streaming, and how much checking is enough.
Map Sources and Contracts
Every source, its schema, its update pattern and what happens when it is late or wrong.
Build With Checks In
Schema, freshness and correctness checks are part of the pipeline from the first run, not a later project.
Run in Parallel, Then Switch
When replacing an existing pipeline, old and new run side by side until the numbers match.
Hand Over
Runbooks, metric definitions, alerts and a walkthrough, so your team can operate and extend the pipelines.
How We Can Work Together
Pipeline Review
A short, fixed-scope look at your pipelines and data quality. You get a written report: risks, what to fix first and a plan you can execute with or without me.
Fixed-Scope Project
A defined pipeline, migration or data model with a written scope, timeline and deliverables.
Monthly Retainer
Ongoing ownership of your pipelines: new sources, schema changes, quality checks and incident follow-ups, with priority response.
Not sure which fits? See how consulting engagements work or book a call.
Frequently Asked Questions
Do we need streaming, or is batch enough?
Most reporting is fine with batch, and batch is cheaper and simpler to run. Streaming is worth it when someone acts on the data within minutes. I pick per use case, and many systems use both.
How do you handle schema changes in the sources?
Schemas are checked on every load. A change that is safe passes through, a change that would break something stops the load and raises an alert before bad data reaches a report.
What happens when a job fails?
It retries, and if it still fails, someone is alerted. Jobs are idempotent, so once the cause is fixed the run is repeated for the affected dates without creating duplicates.
Can you work with our existing warehouse and tools?
Yes. I work with what you already run and change tools only when there is a clear reason, which I explain in writing.
How long does a pipeline project take?
A single pipeline for a known source is typically a few weeks. A migration or a new data platform is planned in phases. I give a written estimate after the first call, once I have seen the sources.
Is Your Data Late, Wrong or Stuck in Spreadsheets?
Tell me where the data comes from and who needs it, and I will tell you the simplest reliable way to get it there.