Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Backing up and restoring

Everything so far lives in gonk’s store, and gonk holds the store’s only writer lock. A copy of the store’s directory taken while gonk runs is torn: its files change under the copy, and the result may not open, or may open and be quietly short. So gonk takes its own backups. It takes one every 24 hours, keeps the last five, and you can ask for one at any time over the owner’s socket. A restore always builds a new store beside the live one and checks it before it says done.

This chapter takes both kinds of backup gonk writes, a single archive and a snapshot in pieces, restores each into a directory of its own, and starts a second gonk on what it restored.

A store worth keeping

In a new empty directory, as in every chapter: if the gonk from A lifecycle for your ledger is still running, stop it first, since this one takes the same port. --no-backup turns off the schedule only. The backup resources stay bound, and the banner says so:

$ ikigai-gonk --config-home "$PWD/.config/ikigai" --data-home "$PWD/.ikigai" --port 1070 --no-quic --no-backup
ikigai-gonk 0.1.1 — holding the store at …/.ikigai/store
…
  backup  no schedule (--no-backup or gonk.backup.every = "off") — urn:iki:gonk:backup is still bound; this process is the only thing that can export the dataset
…

In another terminal, in the same directory, file three items in default and close one, and file one more in a second ledger, ops, so the store holds two ledgers’ graphs:

$ ikigai --mount "urn:gk:=$PWD/.ikigai/gonk.sock" -c 'sink urn:gk:iki:ledger:append Rotate the backup key' -c 'sink urn:gk:iki:ledger:append priority=1 Copy backups off the host' -c 'sink urn:gk:iki:ledger:append Practice a restore' -c 'sink urn:gk:iki:ledger:close item=1 reason=done'
#1 urn:iki:ledger:default:item:…
[uncacheable]
#2 urn:iki:ledger:default:item:…
[uncacheable]
#3 urn:iki:ledger:default:item:…
[uncacheable]
closed #1 (done)
[uncacheable]
— batch: 4 commands · request outcomes: 4 uncacheable
$ ikigai --mount "urn:gk:=$PWD/.ikigai/gonk.sock" -c 'sink urn:gk:iki:ledger:ops:append Check the backup disk'
ops#1 urn:iki:ledger:ops:item:…
[uncacheable]

A backup, now

urn:iki:gonk:backup takes one. It is bound behind the owner’s socket and nowhere else: the HTTP door grants it to no one, and a QUIC client’s grant cannot name it.

$ ikigai --mount "urn:gk:=$PWD/.ikigai/gonk.sock" -c 'source urn:gk:iki:gonk:backup'
backup gonk-store-….nq.gz
  43 quads in 2 graph(s), … bytes of N-Quads -> … bytes gzipped
        34  urn:iki:ledger:graph:default
         9  urn:iki:ledger:graph:ops
  sha256 (n-quads) …
  sha256 (archive) …
  scripts gonk-scripts-….json.gz: 0 script(s), 0 file(s), … bytes gzipped
  kept 5, pruned nothing
[uncacheable]
$ ls .ikigai/backups
gonk-scripts-….json.gz
gonk-store-….nq.gz
gonk-store-….nq.gz.meta.json

The archive is N-Quads, gzipped, so every quad keeps its graph. That is what keeps each ledger’s grants meaningful after a restore. The scripts from Your first saved query are files beside the store, not quads in it, so every backup writes them as a second archive under the same stamp. The sidecar .meta.json records both, with their digests and the per-graph counts. backup:status reads it back:

$ ikigai --mount "urn:gk:=$PWD/.ikigai/gonk.sock" -c 'source urn:gk:iki:gonk:backup:status'
backups in …/.ikigai/backups
  schedule  OFF (gonk.backup.every = "off") — keep 5
  newest    gonk-store-….nq.gz
    taken_at       …
    quads          43
    nquads_bytes   …
    archive_bytes  …
    nquads_sha256  …
    graphs         "urn:iki:ledger:graph:default" 34, "urn:iki:ledger:graph:ops" 9
    scripts        gonk-scripts-….json.gz: 0 script(s), 0 file(s)
  rotation  1 archive(s)
…
  timer     not scheduled in this process
[uncacheable]

A restore is a new store

Loading an archive into a store that already holds quads would be a merge: the union brings back everything either side deleted. So a restore builds a store in a directory that does not exist yet, and refuses any other. Read the archive out with backup:archive:{name} and pipe it into restore:

$ ikigai --mount "urn:gk:=$PWD/.ikigai/gonk.sock" -c "source urn:gk:iki:gonk:backup:archive:$(cd .ikigai/backups && ls gonk-store-*.nq.gz) | sink urn:gk:iki:gonk:restore into=$PWD/restored"
restored 43 quads into …/restored (gzip, sha256 …)
        34  urn:iki:ledger:graph:default
         9  urn:iki:ledger:graph:ops
  graph set and per-graph counts match the archive exactly
  this server still holds the LIVE store; to adopt this one, stop gonk and swap the directories
[2 uncacheable]

The restore counts the quads in each graph it loaded and compares them with what the archive recorded. Only a match leaves a store behind. Adopting it is your step, with gonk stopped, so the one destructive move is the one you make by hand.

Since gonk ledger-v0.22.2, every backup gonk writes can also be restored (gonk PR 374). A restore expands the whole archive in memory, and the most a run may hold is now the restore’s own bound, 1 GiB. Before that, the limit was 64 MiB, so a backup past 64 MiB of N-Quads was taken and then refused by its own restore.

A snapshot in pieces

The single archive has a ceiling: 1 GiB of N-Quads and 10 million quads. A store past it is backed up as a snapshot in pieces instead. The scheduled backup switches to pieces when it reaches the ceiling, and backup:chunks takes one directly at any size, so you can see one here:

$ ikigai --mount "urn:gk:=$PWD/.ikigai/gonk.sock" -c 'source urn:gk:iki:gonk:backup:chunks'
backup gonk-chunks-…
  43 quads in 2 graph(s), 2 bounded pieces
  manifest sha256 …
[uncacheable]
$ ls .ikigai/backups/gonk-chunks-*
00000000.nq.gz
00000001.nq.gz
manifest.json
scripts.json.gz
$ jq -c '.format, (.pieces[] | {bytes, graphs}), .graphs' .ikigai/backups/gonk-chunks-*/manifest.json
"gonk-chunks/1"
{"bytes":…,"graphs":{"urn:iki:ledger:graph:default":34}}
{"bytes":…,"graphs":{"urn:iki:ledger:graph:ops":9}}
{"urn:iki:ledger:graph:default":34,"urn:iki:ledger:graph:ops":9}

The snapshot is one directory. Each piece is gzipped N-Quads from a single graph, at most 64 MiB and 100,000 quads, so a large graph takes as many pieces as it needs. The manifest records each piece’s size, digest and per-graph counts, and the counts for the whole snapshot. The pieces are not separate backups. Copy the whole directory, manifest included, when you move one off the host.

Restore it by name, into a new directory again:

$ ikigai --mount "urn:gk:=$PWD/.ikigai/gonk.sock" -c "sink urn:gk:iki:gonk:restore:chunks content=$(cd .ikigai/backups && ls -d gonk-chunks-*) into=$PWD/restored-pieces"
restored 43 quads in 2 graph(s) into …/restored-pieces
  every piece digest and graph count verified
  graph set and per-graph counts match the snapshot exactly
  this server still holds the LIVE store; stop gonk before swapping directories
[uncacheable]
$ ikigai --mount "urn:gk:=$PWD/.ikigai/gonk.sock" -c "sink urn:gk:iki:gonk:restore:chunks content=$(cd .ikigai/backups && ls -d gonk-chunks-*) into=$PWD/restored"
error: invalid argument `into`: …/restored is not empty. Restoring into a non-empty store is a MERGE, not a restore: the union of two datasets brings back everything either of them deleted. Name a directory that does not exist yet.
       `into`: a NEW store directory, absent or empty

Each piece is expanded and checked against its digest and counts before its quads are loaded into a staged store. At the end, the restored graphs and their counts are compared with the manifest. A missing piece, a digest, length or count that does not match, bad gzip, or a quad repeated across pieces each stops the restore before it publishes anything. The target is left absent or empty, and you can try again. The second command shows the merge refusal, with the first restore’s directory as its target.

What this looks like at scale is gonk’s own drill, examples/backup-memory in its repository: 10 million quads take about 5 minutes to back up in pieces and about 4.5 minutes to restore. It has not been run for this book.

The restored store, served

A restored directory is a complete store. Start a second gonk on it, with its own data home and another port, and read it the way you read the first:

$ ikigai-gonk --config-home "$PWD/.config/ikigai" --data-home "$PWD/.ikigai-restored" --store "$PWD/restored-pieces" --port 1071 --no-quic --no-backup
ikigai-gonk 0.1.1 — holding the store at …/restored-pieces
…
$ curl -s http://127.0.0.1:1071/iki/ledger/items
   #3  open    no priority set  Practice a restore
   #2  open    priority 1  Copy backups off the host
2 item(s)
$ ikigai --mount "urn:gk:=$PWD/.ikigai-restored/gonk.sock" -c 'source urn:gk:iki:ledger:ledgers'
default                       2 open      3 total  urn:iki:ledger:graph:default
ops                           1 open      1 total  urn:iki:ledger:graph:ops
2 ledger(s)
[uncacheable]

Both ledgers came back, with the closed item still closed. Stop both gonks before the next chapter.

In resource terms: a backup is a view of the whole dataset at one snapshot, written as files. A restore is the one write that makes a new store, and it is checked against the counts that view recorded. Over the socket, the next chapter, reaches the ledger from another process, the way an agent does.