Save on courses or Exam now
About Instructors
-
frankieblues- Instructor Rating - Reviews - Students - Courses
No biography available, please complete
🚀 Course Goals
- Build incremental ingestion that can be rerun safely: watermarks, durable checkpoints, bounded retries with backoff, and idempotent loads behind a business key. Run an Apache Iceberg lakehouse on MinIO: atomic commits, schema and partition evolution, snapshots, time travel, rollback and compaction. Transform with dbt Core and orchestrate with Airflow, including backfills and reruns that do not duplicate data. Make data quality executable: Pandera schemas, blocking dbt tests, quarantine tables, data contracts and breaking-change detection.
📘 About This Course
Most data platforms do not fail on the day they are built. They fail three months
later, when a partner re-sends yesterday's file, a backfill writes twice, a column
appears without warning, and nobody can say which version of the data trained the model
that is answering customers right now.
This course builds the platform that survives those days. You start from an empty
Ubuntu machine and finish with a running, traceable system: incremental ingestion with
checkpoints and idempotent loads, an Apache Iceberg lakehouse on MinIO with snapshots and
time travel, dbt transformations orchestrated by Airflow, executable data quality with
Pandera and data contracts, dataset versioning with DVC, experiment tracking and a model
registry in MLflow, a FastAPI inference service in a hardened container, a GitHub Actions
pipeline that blocks unsafe releases, and Prometheus and Grafana watching latency, errors
and drift.
Every lesson is a real session on a real Linux lab, not slides about one. You see the
official download, the installation, the version check, the configuration, the health
check and the reset of every tool. Each lesson also contains a controlled failure: a job
is killed mid-write, a catalog is stopped, a schema drifts, a quality gate blocks a
release. You watch the symptom, read the evidence, apply the minimal correction and rerun.
That is the part that is missing from most tutorials, and it is the part you will actually
need.
Roughly 30 % of the course explains and 70 % demonstrates. The whole stack is
free and open source, runs on one machine without a GPU and without a paid cloud account,
and every dataset is synthetic.
What You'll Learn
Build incremental ingestion that can be rerun safely: watermarks, durable checkpoints, bounded retries with backoff, and idempotent loads behind a business key.
Run an Apache Iceberg lakehouse on MinIO: atomic commits, schema and partition evolution, snapshots, time travel, rollback and compaction.
Transform with dbt Core and orchestrate with Airflow, including backfills and reruns that do not duplicate data.
Make data quality executable: Pandera schemas, blocking dbt tests, quarantine tables, data contracts and breaking-change detection.
Version datasets with DVC, track and compare experiments in MLflow, and promote a model through a registry with evidence and thresholds.
Serve a registered model from a validated FastAPI service in a non-root container, behind CI that blocks unsafe releases.
Monitor latency, errors, throughput and input drift with Prometheus and Grafana, and respond to an incident with a blameless note.
Trace any published number back to its source extract, Git commit, pipeline run and model version.
🛠️ Requirements
- Comfortable on the Linux command line and with basic SQL and Python; you do not need prior data-engineering experience.
- A machine (or VM) with 8 GB of RAM, 6 CPU cores and about 25 GB of free disk, running Ubuntu 22.04 or 24.04.
- No GPU and no paid cloud account: the whole reference stack is free, open source and runs locally.
- Everything else is installed on camera, from the official source, with a version check and an uninstall path.
👤 Who This Course is For
- Data engineers who can build a pipeline but want one that survives reruns, schema changes and incidents.
- Analytics engineers and data analysts moving from SQL models toward owning the platform underneath them.
- ML engineers and data scientists who need reproducible datasets, a model registry and a traceable path to production.
- Backend and DevOps engineers asked to run the data platform, and technical leads who have to review one.
53618 Students applied
100 Comments
-
-
John DoeMorbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
-
John Doe
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
-
-
John Doe
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
-
John Doe
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
-
John Doe
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet
Morbi nec nisi ante. Quisque lacus ligula, iaculis in elit et, interdum semper quam. Fusce in interdum tortor. Ut sollicitudin lectus dolor eget imperdiet libero pulvinar sit amet