Work · 13 projects
Database administration first, data engineering next. Done, in progress and planned, so you can see where it's heading.
GTID replication built from scratch, a working lag alert, a failover drill run twice, and a real five-app outage from making the failover permanent: how it was diagnosed and recovered without retyping a single password.
One stable endpoint for every app, automatic promotion and fail-closed fencing across three MySQL servers. Proven under load on the live stack: 6.9 s from killing the primary to the first write (manual: 491.7 s), 0 failed writes, 0 lost acknowledged writes.
Prometheus, Grafana and Uptime Kuma watching the database host, with alerts that actually reach Discord, proved by tripping a real alert, which caused and then fixed a connection-exhaustion incident.
Debezium and Kafka stream every change from MySQL into Postgres within seconds: 8 tables, about 11,000 jobs, exact row-count match. The CDC foundation is live; the app cutover and the other 9 databases are not done yet.
Five daily Airflow DAGs pull job postings from public boards into MinIO, then clean, upsert, close, retain and quality-check them in MySQL, score new postings for scams, and stream everything on to Postgres. About 11,000 jobs across 3 search targets.
Normalize a flat, real-world jobs dataset into a 3NF MySQL schema and build analytical SQL on top.
A million-row copy of the schema, a slow query found with EXPLAIN ANALYZE and fixed with one composite index (26.8 ms → 1.6 ms), a benchmark that measures the right thing, and a backup that was actually restored.
Planned: nightly backups taken from a replica, a retention policy, and an automated restore test every night that proves each backup actually works, because replicas are not backups.
Planned: audit every database account, move to roles, mask personal data per role, and log every admin-panel change with the real person behind it.
Planned: one screen for the three-server MySQL setup: who is primary, lag, ProxySQL routing, response tests, grant comparison between servers, and a full audit trail. Read-only unless explicitly switched on.
Planned: run Postgres and SQL Server as real production systems: point-in-time recovery drills, high availability, security, and live lock/wait troubleshooting, measured side by side with the MySQL work.
Planned: rebuild the jobs data as a lakehouse on Databricks Free Edition: medallion layers in Delta, MERGE and time travel, dbt models, Airflow-triggered runs and Auto Loader streaming.
Planned: daily stock quotes from the Finnhub API through Airflow into Postgres with idempotent upserts and a quality gate, plus DuckDB for returns and moving averages.