# Path: data engineer

For work on the corpus, the database, the gates or a dataset release: from the line record to the pipeline, the editorial ledger and the pin that carries a dataset to production.

- Page: <https://docs.gurbanisoul.com/learning-paths/data-engineer/>
- Updated: 2026-09-25
- From the Sri Guru Granth Sahib Ji — Knowledge Base wiki. Sri Guru Granth Sahib Ji is quoted verbatim, cited by Ang, and never altered.

---

You will work in sggs-data, or take its releases here. Two days, in this order. The pipeline itself
runs only in the data repository; this platform consumes its output by pin.

On the rendered wiki each step has a checkbox remembered in your browser; on GitHub, tick them in
your head.

1. Read the [reverence checklist](https://docs.gurbanisoul.com/onboarding/reverence-checklist/), then all of
   [Scripture 101](https://docs.gurbanisoul.com/scripture/) — the structure page with its poster, and Gurmukhi and
   Unicode with its quiz, most carefully.
2. [Run it locally](https://docs.gurbanisoul.com/onboarding/run-it-locally/) — `make dataset` and `make dataset-check`.
3. [Data & pipeline](https://docs.gurbanisoul.com/data/): the overview, then
   [anatomy of a line record](https://docs.gurbanisoul.com/data/line-record/) with the Ang explorer.
4. [Corpus pipeline and gates](https://docs.gurbanisoul.com/data/pipeline/) — walk poster 04 step by step; read the
   `fix_text` and `reconcile.py` excerpts.
5. [The editorial ledger](https://docs.gurbanisoul.com/data/editorial-ledger/): the two kinds of transform, the eleven
   applications, how a concern travels.
6. The pinned canonical documents: [scripture integrity](https://docs.gurbanisoul.com/data/architecture/scripture-integrity/),
   the [database schema](https://docs.gurbanisoul.com/data/architecture/database-schema/), the
   [rebuild runbook](https://docs.gurbanisoul.com/data/process/runbooks/rebuild-db/).
7. [The dataset pin](https://docs.gurbanisoul.com/data/dataset-pin/) — the lock, the fetcher's guarantees, the lock-bump flow.
8. **[Exercise 03 — bump a pin in a scratch branch](https://docs.gurbanisoul.com/exercises/03-bump-a-pin/).**
9. [The Roman fold](https://docs.gurbanisoul.com/search/the-roman-fold/) — because the index-time fold is yours.
10. [Harnesses and golden vectors](https://docs.gurbanisoul.com/search/harnesses-and-golden-vectors/) — what a dataset
    change must not break, and how `make contract` proves it.
11. [The Answer Protocol for engineers](https://docs.gurbanisoul.com/scripture/answer-protocol-for-engineers/).
12. **[Exercise 04 — write an ADR](https://docs.gurbanisoul.com/exercises/04-write-an-adr/)** for a change to the data
    model you would propose.
13. [Engineering invariants](https://docs.gurbanisoul.com/engineering/invariants/), the data bullets in particular.

When you are done you can say what `reconcile.py` proves and what it does not, why a rebuild
never edits the committed corpus until two gates pass, and what a scholar sees before any repair
reaches a reader.