A spreadsheet, V: data that changes outside the sheet

Everything the sheet has shown so far changes for one of two reasons: somebody typed into it, or the clock moved. Real sheets read a third kind of thing, data that belongs to somebody else: a stock price, a sensor reading, a figure from another team’s system. It changes when they change it, and the sheet is not told.

This part gives the sheet a feed: a name that something outside the sheet writes, and a formula function, FEED(name), that reads it. The claim is the one Part III made about recalculation, pushed one step further. The sheet needs no machinery for outside data either. A write to a feed is a Sink on a name, the kernel cuts that name’s golden thread, and exactly the values that read the feed are computed again the next time somebody looks. What is new is the last four words. The page did not make the write, so the page does not know to look.

The model, on paper

/// The latest value written to a feed: an atom the sheet reads and never writes.
pub const FEED: &str = "urn:iki:tutorial:sheet:feed:{name}";

/// The market moves: `Sink` only. The next price, written to the feed.
pub const TICK: &str = "urn:iki:tutorial:sheet:tick:{name}";

A feed’s name is a lower-case letter, then lower-case letters and digits: acme, fx2. As with a cell, one spelling per feed, so one golden thread per feed. In a formula the name may be written any case, FEED(Acme), and compiles to the one spelling, (feed acme).

The feed: an atom somebody else writes

A feed is the input’s shape again: a map from a name to a string, in memory, Sink to write, Delete to clear, and NotFound for a feed nobody has written to:

/// `sheet-feed`: the latest value written to the feed `name`, held in memory.
///
/// The same shape as the input: `Source` answers what was written, `.cacheable()` with no
/// thread named, so the kernel hangs it from the feed's own name and cuts that on every
/// `Sink` or `Delete` here. A feed nothing has written to is `NotFound`. `Sink` stores
/// `content`, trimmed, and answers it.
///
/// Nothing in the sheet writes here. A price service, a sensor, a person at a shell: whatever
/// writes a feed is outside the sheet, and the sheet finds out the way it finds out about
/// anything, by reading a name whose thread was cut.
pub fn feed(store: Arc<FeedStore>) -> FnEndpoint {
    FnEndpoint::new("sheet-feed", move |inv: &Invocation<'_>| {
        let name = feed_binding(inv)?;
        let mut latest = store.latest.lock().expect("the feed store's lock");
        match inv.request.verb {
            Verb::Sink => {
                let written = inv.inline_str("content")?.trim();
                if written.is_empty() || written.len() > MAX_INPUT {
                    return Err(Error::InvalidArgument {
                        name: "content".to_string(),
                        detail: format!(
                            "{} bytes; a feed holds 1 to {MAX_INPUT} (to clear it, delete it)",
                            written.len()
                        ),
                    });
                }
                latest.insert(name, written.to_string());
                Ok(Representation::new(
                    text_plain_utf8(),
                    written.as_bytes().to_vec(),
                ))
            }
            Verb::Delete => {
                latest.remove(&name);
                Ok(Representation::new(text_plain_utf8(), b"ok".to_vec()))
            }
            _ => match latest.get(&name) {
                Some(written) => Ok(Representation::new(
                    text_plain_utf8(),
                    written.clone().into_bytes(),
                )
                .cacheable()),
                None => Err(Error::NotFound(format!(
                    "nothing has been written to the feed `{name}`"
                ))),
            },
        }
    })
    .with_description(
        Description::new("sheet-feed")
            .title("Feed")
            .summary("The latest value written to a feed: read it, write it, or clear it.")
            .verb(Verb::Meta)
            .action(
                ActionSpec::new(Verb::Source)
                    .summary("the latest value written; NotFound if nothing has been")
                    .input(feed_input())
                    .output(TEXT_PLAIN_UTF8),
            )
            .action(
                ActionSpec::new(Verb::Sink)
                    .summary("write the feed's latest value, and answer it")
                    .input(feed_input())
                    .input(
                        ArgSpec::new("content")
                            .summary("the value: a number, or some text")
                            .class(XSD_STRING),
                    )
                    .output(TEXT_PLAIN_UTF8),
            )
            .action(
                ActionSpec::new(Verb::Delete)
                    .summary("clear the feed")
                    .input(feed_input())
                    .output(TEXT_PLAIN_UTF8),
            ),
    )
}

The page has no price service, so it has an endpoint that plays one. tick:{name} reads the feed, works out a next price, and writes it, all through the kernel:

/// `sheet-tick`: the market moves. `Sink` only.
///
/// Reads the feed `name` through the kernel (a feed nothing has written to opens at 100),
/// writes [`next_price`] to it through the kernel, and answers the new price. The write is an
/// ordinary `Sink` on `feed:{name}`, so it cuts the feed's thread exactly as a price service
/// writing the same name would; this endpoint is here only because the page has no price
/// service. A feed holding something that is not a price is a `Conflict`: the market does not
/// move a feed somebody has set to `halted`.
pub fn tick() -> AsyncFnEndpoint {
    AsyncFnEndpoint::new("sheet-tick", |inv: &Invocation<'_>| -> InvokeFuture<'_> {
        Box::pin(async move {
            if inv.request.verb != Verb::Sink {
                return Err(Error::Endpoint(
                    "the market moves when it is sunk: `sink` it (and read the feed)".to_string(),
                ));
            }
            let name = feed_binding(inv)?;
            let target = iri(&feed_name(&name))?;
            let cents = match inv.source(&target).await {
                Ok(latest) => {
                    let text = String::from_utf8_lossy(&latest.bytes).into_owned();
                    formula::parse_number(&text)
                        .filter(|n| *n > 0.0 && *n < 1e12)
                        .map(|n| (n * 100.0).round() as u64)
                        .ok_or_else(|| {
                            Error::Conflict(format!(
                                "the feed `{name}` holds `{text}`, which is not a price"
                            ))
                        })?
                }
                Err(Error::NotFound(_)) => OPENING_CENTS,
                Err(other) => return Err(other),
            };
            let price = formula::number_text(next_price(cents) as f64 / 100.0);
            let write = Request::new(Verb::Sink, target)
                .with_arg("content", ArgRef::Inline(price.clone().into_bytes()));
            inv.issue(write).await?;
            Ok(Representation::new(text_plain_utf8(), price.into_bytes()))
        })
    })
    .with_description(
        Description::new("sheet-tick")
            .title("The market moves")
            .summary("Writes the feed's next price to it, and answers the price. Sink only.")
            .verb(Verb::Sink)
            .verb(Verb::Meta)
            .input(feed_input())
            .output(TEXT_PLAIN_UTF8),
    )
}

The next price is a pure function of the current one, so this chapter’s outputs are the same every time it is run:

/// The market, simulated: the price after `cents`, in cents, somewhere within 1.50 of it and
/// never below one cent.
///
/// A pure function of the price, so the chapter's outputs are the same on every run: a
/// generator seeded by the price, not by the time. The sheet never sees this; it sees only
/// what gets written to the feed.
///
/// ```
/// use spreadsheet::next_price;
/// assert_eq!(next_price(10_000), next_price(10_000));
/// assert!(next_price(10_000).abs_diff(10_000) <= 150);
/// assert!(next_price(1) >= 1);
/// ```
pub fn next_price(cents: u64) -> u64 {
    let mixed = cents
        .wrapping_mul(6_364_136_223_846_793_005)
        .wrapping_add(1_442_695_040_888_963_407);
    let step = (mixed >> 33) % 301;
    (cents.saturating_add(step)).saturating_sub(150).max(1)
}

None of that is the sheet. From the sheet’s side a feed is a name, and the evaluator reads it the way it reads a cell:

/// `FEED(name)`: the latest value written to the feed, read through the kernel.
///
/// A feed nothing has written to is `#N/A`. That fallback is safe to cache: the value that
/// caught the `NotFound` hangs from the feed's golden thread, so the first write to the feed
/// cuts it. It catches the ATOM's own `NotFound`, which says exactly what is missing; before
/// core 0.1.82 that was also the only safe choice, since a composite's `NotFound` hung the
/// fallback from a thread nobody cuts.
async fn read_feed(inv: &Invocation<'_>, name: &str) -> Result<Value> {
    match inv.source(&iri(&feed_name(name))?).await {
        Ok(latest) => Ok(Value::of(&String::from_utf8_lossy(&latest.bytes))),
        Err(Error::NotFound(_)) => Ok(Value::Error(NA_ERROR)),
        Err(other) => Err(other),
    }
}

A write the sheet did not make

Write a price to the feed acme, as a price service would, and build a small holding on it: the price in A1, a number of shares in A2, their value in A3. C1 is arithmetic that has nothing to do with it.

100
[uncacheable]
=FEED(acme)
[uncacheable]
10
[uncacheable]
=A1*A2
[uncacheable]
=6*7
[uncacheable]
1000
[computed]

Now the price changes. The line below writes the feed, and nothing in it mentions the sheet:

101.5
[uncacheable]
not cached
cached
not cached
cached
1015
[computed]

The write cut the feed’s thread. A1 read the feed and A3 read A1, so both went; A2 and C1 read neither and stayed. It is Part III’s edit again, with one difference that is the whole point of this part: the write came from outside the sheet, and the sheet’s code has no idea it happened. It does not need one. The dependency was recorded when A1 read the feed, and the cut found it.

The market moving is the same write, made by an endpoint rather than by hand. Let it move, and trace the holding:

100.34
[uncacheable]
trace  urn:iki:tutorial:sheet:cell:A3
  client      ikigai repl  ·  capability: root (full authority)
  transport   embedded · in-process
 
urn:iki:tutorial:sheet:cell:A3   sheet-cell · computed · ThreadId(1) · —   → 6b  1003.4
├─ urn:iki:tutorial:sheet:input:A3   sheet-input · cached · ThreadId(1) · —
├─ urn:iki:tutorial:sheet:precedents:A3   sheet-precedents · cached · ThreadId(1) · —
├─ urn:iki:tutorial:sheet:formula:A3   sheet-formula · cached · ThreadId(1) · —
├─ urn:iki:tutorial:sheet:cell:A1   sheet-cell · computed · ThreadId(1) · —
│  ├─ urn:iki:tutorial:sheet:input:A1   sheet-input · cached · ThreadId(1) · —
│  ├─ urn:iki:tutorial:sheet:precedents:A1   sheet-precedents · cached · ThreadId(1) · —
│  ├─ urn:iki:tutorial:sheet:formula:A1   sheet-formula · cached · ThreadId(1) · —
│  └─ urn:iki:tutorial:sheet:feed:acme   sheet-feed · computed · ThreadId(1) · —
└─ urn:iki:tutorial:sheet:cell:A2   sheet-cell · cached · ThreadId(1) · —

The feed was read and computed again, because the market wrote it. A1’s value read the feed, A3’s read A1’s, and A2’s was served. Nothing in the trace is about the market: the sheet sees a name whose thread was cut, and never who cut it.

A feed nobody has written to

A formula can name a feed before anything has been written to it. The feed answers NotFound, and the value is #N/A, the sheet’s word for “not available”:

=FEED(globex)
[uncacheable]
#N/A
[computed]
#N/A
[cached]
55
[uncacheable]
not cached
55
[computed]

#N/A was cached, and the first write to the feed cut it. When a request fails, the kernel records the failure as a dependency of whatever asked, so an answer built on a feed’s absence hangs from the feed’s thread, and the first write to the feed cuts it.

That has only been true all the way down since core 0.1.82, and the story of the gap is worth knowing. Before it, a failure was recorded under the thread of the name that was asked for, and nothing else. B1 asked for feed:globex itself, the atom, and the atom’s own thread is exactly what a write to it cuts, so B1 was safe. Had it asked for some composite that read the feed, and fallen back on that composite’s NotFound, the #N/A would have hung from the composite’s thread, which no write ever cuts, and stayed #N/A after the feed was written. The sheet was built around the gap: Part II’s evaluator reads a cell’s input before its compiled formula, never falling back on the formula’s NotFound. Building it is how the gap was found (ledger item 611), and 0.1.82 closed it: a failed request now carries the dependencies it recorded on its way to failing, so a fallback hangs from the atom’s thread however far down the NotFound came from.

The rule the sheet follows is still a good one, for a different reason: fall back on an atom’s own NotFound, because that is the one that says exactly what is missing. A composite’s NotFound may mean the atom is empty, or that it holds the wrong kind of thing (a cell holding a number has no compiled formula), and a fallback cannot tell which.

/// A feed nothing has written to is `#N/A`, cached — and the first write cuts it, because the
/// fallback caught the ATOM's own NotFound, so it hangs from the feed's thread.
#[test]
fn an_unwritten_feed_is_not_available_until_its_first_write() {
    let kernel = host(None);
    type_into(&kernel, "B1", "=FEED(globex)");
    assert_eq!(value(&kernel, "B1"), "#N/A");
    assert!(value_cached(&kernel, "B1"));
    sink(&kernel, "urn:iki:tutorial:sheet:feed:globex", "55").unwrap();
    assert!(!value_cached(&kernel, "B1"));
    assert_eq!(value(&kernel, "B1"), "55");
}

Bad data is a value

A feed can hold anything its writer sends, and one day that will not be a number. Say the exchange halts trading and the feed says so:

halted
[uncacheable]
#VALUE
[computed]
#VALUE
[cached]
error: conflict: the feed `acme` holds `halted`, which is not a price
100
[uncacheable]

A3 is #VALUE, and the second read is served from the cache. That is a choice, and the other choice is worse than it looks. Had the evaluator refused, the refusal would never have been cached: the kernel records every error except NotFound as already expired, since most errors are transient. So A3, and the grid that shows it, would be computed again on every read for as long as the feed said halted, which could be all afternoon. When the honest answer is “this input is wrong”, say it as a value, and it caches like any other.

The market itself does refuse, with a Conflict: it will not move a price that is not a price. That refusal is right where it is, because it answers a write, and a write is never cached. The last line puts the price back, for the page below.

Finding out: polling

The runnable cells above redraw the sheet at the bottom of the page after they run, as they did in earlier parts. That is the page reacting to something the page did. The market is different. Press Run the market below and two things start, both driven by the page’s timers and both paused until you press it: the market writes a new price to acme every three seconds, and the grid asks for itself again every two. Neither knows about the other. The market does not redraw the grid; it only writes the feed. The grid finds out the way anything that reads a name finds out about a write it did not make: it asks again.

Put =FEED(acme) somewhere, and something that reads it, and watch the count and the table under the grid. Most polls are answered entirely from the cache, and cost a lookup. The poll after each write computes exactly the values that read the feed, and serves the rest.

The market moves writes once, by hand. The grid does not change until its next poll, which is the lesson rather than a bug: the button is somebody else, and the grid is not told.

The same thing as a test, with the market writing every third second and the grid polled every second:

/// The page's poll, as a test: the grid asked every two seconds while the market writes the
/// feed every three. A poll between writes is answered from the cache; a poll after one
/// computes the feed's dependents and nothing else.
#[test]
fn a_polled_grid_computes_only_after_a_write_and_only_what_it_cut() {
    let kernel = host(None);
    type_into(&kernel, "A1", "=FEED(acme)");
    type_into(&kernel, "A2", "10");
    type_into(&kernel, "A3", "=A1*A2");
    type_into(&kernel, "B1", "=SUM(C1:C6)");
    type_into(&kernel, "C1", "=6*7");
    let mut polls = Vec::new();
    for second in 0..12 {
        if second % 3 == 0 {
            sink(&kernel, "urn:iki:tutorial:sheet:tick:acme", "").unwrap();
        }
        if second % 2 == 0 {
            let was_cached = cached(&kernel, spreadsheet::VIEW_GRID);
            let recomputed = uncached(&kernel);
            source(&kernel, spreadsheet::VIEW_GRID).unwrap();
            polls.push((second, was_cached, recomputed));
        }
    }
    let wrote = ["A1".to_string(), "A3".to_string()];
    for (second, was_cached, recomputed) in &polls[1..] {
        // The market wrote since the last poll (at 3 and 9, for the polls at 4 and 10; at 6,
        // for the poll at 6 itself) exactly when the poll computed, and then only its cells.
        let wrote_since = [4, 6, 10].contains(second);
        assert_eq!(*was_cached, !wrote_since, "the poll at {second}s");
        if wrote_since {
            assert_eq!(recomputed, &wrote, "the poll at {second}s");
        } else {
            assert!(recomputed.is_empty(), "the poll at {second}s");
        }
    }
}

A poll answered from the cache is cheap in the page, and a host serving the sheet over HTTP makes it cheaper still. ikigai’s web edge gives a cached answer a strong ETag, and answers a poll that sends it back (If-None-Match) with 304 Not Modified and no body: the grid is not even sent again. That is the whole cost of polling a resource whose answer has not changed: a lookup, and a few bytes of headers. It is why polling is a reasonable first answer, and the time chapter’s clock, polled every second and computed once a minute, was the same argument.

Its cost is latency and waste at the edges. A write is seen up to one polling interval late, and every poll between writes is a request that learns nothing. For a sheet that must show a price the moment it moves, and for a thousand readers of one feed, the answer is the other direction: the kernel tells the page.

Next: push

This part stops at polling on purpose. Push needs pieces this book does not have yet, and it is worth saying exactly what they are, and which of them the kernel already provides.

From the kernel: somebody to tell. A cut moves a counter on, and by default tells nobody; that laziness is what makes an edit cost nothing until somebody reads (Part III). Push needs a host to be able to say “tell me when this thread is cut”, for an exact name or for a prefix like urn:iki:tutorial:sheet:feed:, and to be told, with the thread’s name, when it is. This is NetKernel’s golden-thread listener, and since core 0.1.82 the kernel has one, Kernel::listen, built to the constraints a kernel with no runtime puts on it. It cannot call anyone back on a task of its own, and a cut runs inside the write that caused it, so it calls nobody: it appends the cut to a queue the host drains when it chooses, and issues no request from inside a cut. Listening is an authority, so it is a capability, and a listener hears only the names its own reads were already given. And the queue is bounded: a cut that finds it full is counted as dropped, so a slow host learns that it missed something rather than acting on half a history.

From the host: what to redraw. Hearing that feed:acme was cut is not yet knowing that the grid changed, but the listener says that too: each cut comes with the cached answers it invalidated, the grid among them when the grid read the feed. So a host that hears a cut can push the views it has handed out that are on that list, or read them again straight away, so the recomputation is done before the reader asks: a recalculating golden thread, which NetKernel also had. The host half is Part VII: there the page registers a listener with its kernel and drains it.

From the page: a channel. In this book the host is the page, so being told is a function call. A sheet served by a real host needs a connection that the server can write to: server-sent events or a WebSocket, which htmx can already consume. The markup would change from “ask every two seconds” to “listen for grid”, and nothing else in the sheet would.

Time does not cut. A value that goes because its deadline passed was never cut, so a listener never hears about it. A NOW() cell pushed to a page needs the host to set a timer for the earliest deadline among the answers it has handed out; the kernel knows each deadline, and would have to say what it is.

And the same answer twice. A cut is by name, not by content, so the market writing the price the feed already holds cuts it anyway, and push would send a grid identical to the last. Stopping there needs the recomputed answer compared with the old one before anything downstream is told: early cutoff, which Part III noted the kernel does not do.

What was built

The sheet gained an atom and learned to read it, and nothing about recalculation changed. The market is the only new code that does anything, and it is not the sheet’s: in a real deployment it is somebody else’s system, writing a name. A cell’s value is still a plain name, cell:{ref}, and so are an input and a feed, which is what the next part needs. A personal scenario, my own numbers laid over the shared sheet’s, is a corridor in front of input:{ref}, and the sheet reads the names it always read.