Databricks Lakehouse Track
Planned: rebuild the jobs data as a lakehouse on Databricks Free Edition: medallion layers in Delta, MERGE and time travel, dbt models, Airflow-triggered runs and Auto Loader streaming.
Status: planned, not started. Nothing here has run yet.
The data-engineering half of the roadmap: rebuild the jobs data from the earlier projects as a lakehouse on Databricks Free Edition (free, no credit card, serverless only), then add dbt, orchestration from the existing Airflow, and streaming. Five small projects, easiest first.
Tools (planned)
| Tool | Will be used for |
|---|---|
| Databricks Free Edition | PySpark notebooks, Delta tables, Unity Catalog volumes |
| PySpark + Delta Lake | Medallion layers, MERGE INTO, time travel |
dbt Core + dbt-databricks | Tested, version-controlled SQL models |
| Apache Airflow (existing) | Triggering Databricks runs through the REST API |
| Great Expectations | Data quality checks |
| Metabase or Power BI Desktop | The final dashboard |
Plan
1. Medallion architecture
What it will do: export project1_jobs and load it into three layers: Bronze (raw), Silver (cleaned and de-duplicated), Gold (aggregates such as jobs per country and target).
bronze = spark.read.option("header", True).csv("/Volumes/main/jobs/raw/jobs.csv")
bronze.write.format("delta").mode("overwrite").saveAsTable("main.jobs.bronze_jobs")
silver = (spark.table("main.jobs.bronze_jobs")
.dropDuplicates(["dedup_key"])
.filter("title IS NOT NULL"))
silver.write.format("delta").mode("overwrite").saveAsTable("main.jobs.silver_jobs")
2. Delta MERGE INTO and time travel
What it will do: replay the MySQL pipeline's upsert ("seen again, reopen") as a Delta merge, then query the table as it was yesterday.
MERGE INTO main.jobs.silver_jobs t
USING updates s ON t.dedup_key = s.dedup_key
WHEN MATCHED THEN UPDATE SET status = 'open', last_seen_at = s.seen_at
WHEN NOT MATCHED THEN INSERT *;
SELECT COUNT(*) FROM main.jobs.silver_jobs VERSION AS OF 3;
3. dbt models
What it will do: rebuild the Gold layer as dbt models, with unique and not_null tests. dbt is a real gap: it's not used anywhere else in this portfolio yet.
4. Airflow → Databricks
What it will do: the existing Airflow triggers a Databricks notebook through the REST API, working around Free Edition's limited job scheduler.
5. Streaming with Auto Loader
What it will do: incremental ingestion from Databricks' built-in samples.bakehouse data. This is the Spark counterpart of the Kafka/Debezium CDC work.
Known constraints
- Free Edition limits: serverless-only compute, restricted Jobs, and fair-use quotas. Keep scheduled runs rare.
- Honest claims: each piece gets described as built-and-run only once it has actually been run end to end.