Save on courses or Exam now
About Instructors
-
rayanGuffy- Instructor Rating - Reviews - Students - CoursesNo biography available, please complete
🚀 Course Goals
- Read Spark execution like an engineer: jobs, stages, shuffles, partitions and caching, and use query plans to explain why something is slow instead of guessing. Run a Delta Lake lakehouse with confidence: the transaction log, MERGE and idempotent upserts, schema enforcement and evolution, time travel, restore and recovery. Ingest incrementally with Auto Loader and Structured Streaming, using checkpoints, watermarks, late-data handling and deduplication that survive a restart. Model Bronze, Silver and Gold properly, including star schemas, change data and a reprocessing strategy you can rerun without duplicating a row.
📘 About This Course
Most Databricks training stops at the notebook. This course keeps going until the
platform is something you could hand to a team: governed, orchestrated, tested, and
carrying an AI application that answers from your own data instead of guessing.
You work on one running example the whole way through. Northstar Retail
has orders, customers, products, support policies and event data, and you build their
platform twice. First as production data engineering: Spark and Delta Lake, incremental
ingestion with Auto Loader and Structured Streaming, a Bronze/Silver/Gold model, Lakeflow
jobs and declarative pipelines, Unity Catalog governance, Databricks SQL, and a reliability
gate standing between a broken run and the Gold tables. Then as generative AI on top of it:
foundation model APIs, batch LLM enrichment, embeddings and Databricks Vector Search,
grounded RAG, tool-using agents with human approval, and the evaluation, security,
monitoring and cost work that production actually demands.
Every lesson is a real session in a real workspace. You see the command, the result, and
what to do when the result is wrong — including deliberate, controlled failures: a
delete with the wrong predicate, a stream restarted on a stale checkpoint, a prompt buried
in a document that tries to trigger a refund. Each lesson ships with the slides used on
screen and the code that was actually executed.
It ends with a capstone you can show: the Northstar governed lakehouse and RAG platform,
defended with an acceptance suite and a disaster-recovery rehearsal, the way you would
defend it in a design review.
What You'll Learn
Read Spark execution like an engineer: jobs, stages, shuffles, partitions and caching, and use query plans to explain why something is slow instead of guessing.
Run a Delta Lake lakehouse with confidence: the transaction log, MERGE and idempotent upserts, schema enforcement and evolution, time travel, restore and recovery.
Ingest incrementally with Auto Loader and Structured Streaming, using checkpoints, watermarks, late-data handling and deduplication that survive a restart.
Model Bronze, Silver and Gold properly, including star schemas, change data and a reprocessing strategy you can rerun without duplicating a row.
Orchestrate with Lakeflow Jobs and declarative pipelines: task graphs, quality expectations and quarantine, scheduling, parameters, retries, alerts and backfills.
Govern with Unity Catalog: ownership and least privilege, volumes and storage boundaries, row and column access, and lineage you can hand to an auditor.
Make Databricks SQL fast and answerable: warehouses, CTEs and reusable views, query profiles, Photon, statistics, clustering and table maintenance.
Prove reliability before publication: data contracts, PySpark unit tests, run observability, freshness and SLA/SLO gates in front of the Gold layer.
Bridge to AI with MLflow tracking, feature preparation, the model registry and batch inference wired back into your pipelines.
Build generative AI on the lakehouse: foundation model APIs, structured outputs, batch LLM enrichment, serving endpoints, embeddings and Databricks Vector Search.
Ship grounded RAG and tool-using agents with governed SQL and retrieval tools, memory, and a human approval gate in front of anything that changes state.
Evaluate, secure and operate GenAI: reproducible test sets, prompt-injection and exfiltration defence, quality, latency and drift monitoring, and cost control.
🛠️ Requirements
- Comfortable with Python and SQL. Prior Spark experience helps but is not assumed: the execution model is built up from the beginning.
- An authorized Databricks workspace where you can create catalogs and schemas, run compute or a SQL warehouse, and schedule jobs.
- An Ubuntu Linux machine or VM for the engineering workstation, with Git, Python and the Databricks CLI. The course sets this up on camera in Module 0.
- Awareness that some lessons use billable Databricks features. Cost guardrails, small lab data and teardown steps are part of the course, not an afterthought.
👤 Who This Course is For
- Data engineers who can write a Spark job but want the governed, orchestrated and tested platform that has to surround it.
- Analytics engineers and BI developers moving from SQL models toward owning the pipelines, warehouses and contracts underneath them.
- ML and AI engineers who need governed data, real retrieval and honest evaluation behind an LLM application rather than a demo.
- Architects and technical leads who have to design, review or defend a lakehouse and the AI built on top of it.
53618 Students applied
100 Comments
-
-
John DoeMorbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
-
John Doe
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
-
-
John Doe
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
-
John Doe
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
-
John Doe
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet