# Data & pipeline Where the scripture data comes from, how it is proven, how it reaches this platform by pin, and which pages explain each part — with the canonical sggs-data documents published beside them. - Page: - Updated: 2026-09-27 - Verified against: `fe1ff7d` on 2026-09-25 - From the Sri Guru Granth Sahib Ji — Knowledge Base wiki. Sri Guru Granth Sahib Ji is quoted verbatim, cited by Ang, and never altered. --- Everything the API serves comes from one artifact: a SQLite database of **60,658 line records over Angs 1–1430**, built by a reproducible pipeline from the source Bir PDF and proven, character for character, to contain exactly the printed text. That artifact is built, gated and published in the data repository, [`Algorythmos-AI/sggs-data`](https://github.com/Algorythmos-AI/sggs-data); this platform never edits it and never commits a copy — it **pins** a commit and a checksum (`dataset.lock.json`) and installs exactly that object. > **Prime directive.** The Gurmukhi is sacred and verbatim. Nothing in this platform, and nothing > described on these pages, rewrites it. The two sanctioned kinds of transform — decoding the PDF > font's glyph order into logical Unicode, and a handful of scholar-reviewed repairs of impossible > sequences the font emitted — are registered in the [editorial ledger](https://docs.gurbanisoul.com/data/editorial-ledger/) and > enforced by a gate in sggs-data. ## The pages - [Anatomy of a line record](https://docs.gurbanisoul.com/data/line-record/) — The 23 columns of the `lines` table, how a page becomes rows, what search reads (Poster 03 walkthrough · a live Ang explorer against the real API) - [Corpus pipeline and gates](https://docs.gurbanisoul.com/data/pipeline/) — `rebuild_all.sh` stage by stage, and the gates that stop it (Poster 04 walkthrough · code read from sggs-data at the pinned commit) - [Data dictionary](https://docs.gurbanisoul.com/data/data-dictionary/) — Every table in the pinned database: its layer, columns, row count and the API context that reads it (Generated from the database; CI fails if it drifts) - [The dataset pin](https://docs.gurbanisoul.com/data/dataset-pin/) — How this platform obtains the database, what it verifies, how a new dataset arrives (The lock, the fetcher's guarantees, a pin bump end to end) - [The editorial ledger](https://docs.gurbanisoul.com/data/editorial-ledger/) — What may change in the text, what never does, and how a fidelity concern travels (The rules and their applications, the review flow) - [The English translation layer](https://docs.gurbanisoul.com/data/translation-layer/) — Where the English comes from, how it is matched to lines, how it stays separate, English search, and its licence status (Two sources, one translator) - [The integrity chain](https://docs.gurbanisoul.com/data/integrity-chain/) — One line of scripture from the source PDF to a screen, and the gate that proves it at each stage (Six stages, each read against the code that enforces it) ## Pinned from sggs-data The canonical data documents live in the data repository. The wiki publishes them here **by pin** (`docs-site/sources.lock.json` records the commit and the sha256 of every file), with a banner that says where the canonical file is; edit them there. | Document | Canonical file | |---|---| | Database schema and the `comp_id` model | [docs/architecture/database-schema.md](https://docs.gurbanisoul.com/data/architecture/database-schema/) | | Scripture integrity: the guarantee chain, the proof tools, the hash chain | [docs/architecture/scripture-integrity.md](https://docs.gurbanisoul.com/data/architecture/scripture-integrity/) | | Runbook: rebuild the database from the PDF | [docs/process/runbooks/rebuild-db.md](https://docs.gurbanisoul.com/data/process/runbooks/rebuild-db/) | | Runbook: restore the data from backups | [docs/process/runbooks/data-restore.md](https://docs.gurbanisoul.com/data/process/runbooks/data-restore/) | | ADR-0002 · keep-last `comp_id` regroup | [docs/adr/0002-comp-id-keep-last-regroup.md](https://docs.gurbanisoul.com/data/adr/0002-comp-id-keep-last-regroup/) | | ADR-0003 · integrity proofs | [docs/adr/0003-integrity-proofs.md](https://docs.gurbanisoul.com/data/adr/0003-integrity-proofs/) | | ADR-0004 · backups and restore | [docs/adr/0004-backups-and-restore.md](https://docs.gurbanisoul.com/data/adr/0004-backups-and-restore/) | | The Answer Protocol: how Gurbani questions are answered | [Answer-Protocol.md](https://docs.gurbanisoul.com/data/answer-protocol/) | ## The numbers that never drift | Fact | Value | Proven by | |---|---|---| | Line records | 60,658 | `data_quality.py` (sggs-data), `/api/health` | | Angs | 1–1430, gap-free | `golden_test.py`, `/api/health` | | Compositions (`comp_id`) | 4,527 distinct; `max(comp_id)` 5,376; gaps permanent (read from the pinned database) | `verify_regroup.py --invariants` | | Vaars | 22, detected by title header | `build_vaars.py` | | Editorial rules / applications | 4 / 11, all reviewed | `ledger_check.py` | | Dataset served | `dataset.lock.json` → commit, sha256, size | `make dataset-check`, the `scripture-integrity` workflow | Read next: [Anatomy of a line record](https://docs.gurbanisoul.com/data/line-record/).