Skip to content

Get startedLearning paths

Path: data engineer

2 min readUpdated

Page actions

ChatGPT and Claude are third-party services. The link gives them this page's public address and a short instruction. This site records nothing.

For work on the corpus, the database, the gates or a dataset release: from the line record to the pipeline, the editorial ledger and the pin that carries a dataset to production.

You will work in sggs-data, or take its releases here. Two days, in this order. The pipeline itself runs only in the data repository; this platform consumes its output by pin.

On the rendered wiki each step has a checkbox remembered in your browser; on GitHub, tick them in your head.

  1. Read the reverence checklist, then all of Scripture 101 — the structure page with its poster, and Gurmukhi and Unicode with its quiz, most carefully.
  2. Run it locally — make dataset and make dataset-check.
  3. Data & pipeline: the overview, then anatomy of a line record with the Ang explorer.
  4. Corpus pipeline and gates — walk poster 04 step by step; read the fix_text and reconcile.py excerpts.
  5. The editorial ledger: the two kinds of transform, the eleven applications, how a concern travels.
  6. The pinned canonical documents: scripture integrity, the database schema, the rebuild runbook.
  7. The dataset pin — the lock, the fetcher’s guarantees, the lock-bump flow.
  8. Exercise 03 — bump a pin in a scratch branch.
  9. The Roman fold — because the index-time fold is yours.
  10. Harnesses and golden vectors — what a dataset change must not break, and how make contract proves it.
  11. The Answer Protocol for engineers.
  12. Exercise 04 — write an ADR for a change to the data model you would propose.
  13. Engineering invariants, the data bullets in particular.

When you are done you can say what reconcile.py proves and what it does not, why a rebuild never edits the committed corpus until two gates pass, and what a scholar sees before any repair reaches a reader.