Transreption
Resolution said a resource does not have a format: it has whatever formats the kernel can reach from what it has. This chapter is where you find out what “reach” means, by using a transreptor and then writing one.
The code is crates/building-endpoints, this part’s crate. Its host chains onto Part I’s
space, so everything you built there is bound here too, plus four endpoints: banner, which
Configuration borrowed from this part, and the three these
chapters add:
/// This part's space: Part I's bindings, the published SPARQL face, and the four
/// endpoints these chapters build. `home` is where [`config`] looks for its file — a
/// test hands it a directory of its own; [`kernel`] hands it this machine's.
pub fn space(home: Option<PathBuf>) -> Fallback {
let ours: EndpointSpace = hello_camel::space()
.bind(
Exact::new("urn:iki:tutorial:markdown"),
transreption::markdown(),
)
.bind(
Exact::new("urn:iki:tutorial:counter"),
multi_verb::counter(),
)
.bind(Exact::new("urn:iki:tutorial:stamp"), hermetic::stamp())
.bind(
Exact::new("urn:iki:tutorial:banner"),
config::banner(home, Some("yours")),
);
// Two spaces, first match wins: ours, then `urn:sparql:*` from the published crate.
Fallback::new(vec![
Arc::new(ours) as Arc<dyn Space>,
Arc::new(ikigai_sparql::space()) as Arc<dyn Space>,
])
}Using one
The ikigai CLI links a transreptor between RDF syntaxes, urn:rdf:transrept. It
declares what it converts — describe it and look for the transrepts: line — and it is
driven the way every transreptor is driven: content in through the pipe, the target type
in as:
ikigai --plain -c 'source urn:kernel:catalog | urn:rdf:transrept as=application/ld+json'
The catalog came out as Turtle and went in as Turtle; what came back is the same graph as
JSON-LD. Nothing about urn:kernel:catalog knows JSON-LD exists. That is the shape:
the format is a property of the route, not of the resource.
What a transreptor declares
An ordinary endpoint’s description says what it does. A transreptor’s says, in addition,
what it converts between — Description::transreptor(from, to) — and that declaration
is what the kernel selects on. Two rules follow from how selection works, and they are
worth knowing before you write one:
- A transreptor is auto-invocable when its only required inputs are
contentandas. Those are exactly the two the kernel supplies when it drives a step. Declare a third required input — a stylesheet, say — and it is still a transreptor, still callable by hand, and never chosen automatically. - Routing is one hop, or two through Turtle. If no transreptor goes directly from
what you have to what you want, the kernel looks for one that reaches
text/turtleand one that leaves it. Turtle is the pivot, which is why the vocabulary renders to Turtle first and everything else second.
Writing one
The endpoint below turns the ikigai description vocabulary — the Turtle every Meta
answers with — into Markdown. It parses real triples rather than lines, so it works on
one endpoint’s description or a whole catalog, and it knows nothing about which. It is
also, deliberately, not a transreption in the strict sense: it keeps the title, the
summary and the inputs and drops the rest, so the Turtle cannot be recovered from it. That
makes it a rendering — a projection — and the description says so:
/// Render the ikigai description vocabulary as Markdown: one section per endpoint, its
/// summary, and a bullet per declared input.
///
/// The input is parsed as RDF — real triples, not lines — so this works on any Turtle
/// carrying the vocabulary, whether it came from one `Meta` or a whole catalog. The
/// `content` argument is the document; a transreptor reads its input from there because
/// that is where the kernel puts it when it drives a transreption step.
pub fn markdown_impl(inv: &Invocation<'_>) -> Result<Representation> {
let turtle = inv.inline_arg("content")?;
let triples: Vec<Triple> = oxttl::TurtleParser::new()
.for_slice(turtle)
.collect::<std::result::Result<_, _>>()
.map_err(|e| Error::Endpoint(format!("not Turtle: {e}")))?;
// Every triple, grouped by subject and then by predicate — stable order, so the
// output is diffable and a test can assert on it.
let mut by_subject: BTreeMap<String, BTreeMap<String, Vec<Term>>> = BTreeMap::new();
for triple in triples {
let subject = match triple.subject {
NamedOrBlankNode::NamedNode(node) => node.as_str().to_string(),
other => other.to_string(),
};
by_subject
.entry(subject)
.or_default()
.entry(triple.predicate.as_str().to_string())
.or_default()
.push(triple.object);
}
let literal = |node: &BTreeMap<String, Vec<Term>>, property: &str| -> Option<String> {
node.get(&format!("{IK}{property}"))
.and_then(|terms| terms.first())
.and_then(|term| match term {
Term::Literal(literal) => Some(literal.value().to_string()),
_ => None,
})
};
let mut out = String::new();
for (iri, node) in &by_subject {
let is_endpoint = node
.get(RDF_TYPE)
.is_some_and(|types| types.iter().any(|t| t.to_string().contains("ns#Endpoint")));
if !is_endpoint {
continue;
}
let id = literal(node, "id").unwrap_or_else(|| iri.clone());
let title = literal(node, "title").unwrap_or_else(|| id.clone());
out.push_str(&format!("## {title} (`{id}`)\n\n"));
if let Some(summary) = literal(node, "summary") {
out.push_str(&format!("{summary}\n\n"));
}
// Inputs are nodes of their own; follow the `ik:input` edges to them.
let inputs = node.get(&format!("{IK}input")).into_iter().flatten();
let mut listed = false;
for input in inputs {
let Term::NamedNode(input_iri) = input else {
continue;
};
let Some(input_node) = by_subject.get(input_iri.as_str()) else {
continue;
};
let name = literal(input_node, "inputName").unwrap_or_default();
let required = literal(input_node, "required").as_deref() == Some("true");
let summary = literal(input_node, "summary").unwrap_or_default();
out.push_str(&format!(
"- `{name}`{} — {summary}\n",
if required { " (required)" } else { "" }
));
listed = true;
}
if listed {
out.push('\n');
}
}
Ok(Representation::new(text_markdown_utf8(), out.into_bytes()).cacheable())
}And its description, which is the half that makes it reachable:
/// `markdown`: a transreptor from `text/turtle` to `text/markdown` — declared **lossy**.
///
/// The description is the interesting half. `.transreptor(from, to)` is what makes the
/// kernel *select* this endpoint when something asks for Markdown and has Turtle; the
/// two inputs are `content` and `as`, and both are what the kernel supplies when it drives
/// a step — an auto-invocable transreptor is one whose required inputs are exactly
/// those. Declare a third required input and it is still a transreptor, but the kernel
/// will not pick it up on its own.
///
/// `.lossy()` says what kind of conversion this is. A transreption proper changes form
/// without changing what is represented — the Turtle could be recovered from the output.
/// This one cannot be undone: it keeps a title, a summary and the inputs and drops the
/// verbs, the capabilities, the output type and everything else, so it is a *rendering*
/// of the description, a projection. Saying so is what lets the kernel refuse to route a
/// `Meta` through it silently — the caller has to consent with `lossy=allow` — while a
/// transreptor that says nothing claims, by default, to be lossless.
pub fn markdown() -> FnEndpoint {
FnEndpoint::new("markdown", markdown_impl).with_description(
Description::new("markdown")
.title("Turtle to Markdown")
.summary("Renders the ikigai description vocabulary as Markdown.")
.verb(Verb::Source)
.verb(Verb::Meta)
.input(ArgSpec::new("content").summary("the Turtle document to render"))
.input(
ArgSpec::new("as")
.summary("the target type; only text/markdown")
.optional(),
)
.output("text/markdown;charset=utf-8")
.transreptor(["text/turtle"], ["text/markdown"])
.lossy(),
)
}The kernel finds it
Bound at urn:iki:tutorial:markdown, the transreptor is found by what it declared, not by
its name:
/// The kernel finds the transreptor by its declaration, not by its name — and reads
/// the whole declaration: a lossy edge is planned only with consent.
#[test]
fn the_kernel_selects_the_transreptor_from_its_declaration() {
let kernel = kernel_in(None);
// The default policy plans through lossless edges only, and this one is lossy.
assert!(kernel
.select_transreptor("text/turtle", "text/markdown")
.is_none());
// With consent the route is one hop, and the step reports what it declared.
let plan = kernel
.select_transreptor_with(
"text/turtle",
"text/markdown",
&TransreptionPolicy::allow_lossy(),
)
.expect("a route from Turtle to Markdown exists, with consent");
assert_eq!(plan.len(), 1, "one hop");
assert_eq!(plan[0].endpoint, "urn:iki:tutorial:markdown");
assert_eq!(plan[0].to, "text/markdown");
assert!(!plan[0].lossless);
// ...and no route to a type nothing produces, consent or not.
assert!(kernel
.select_transreptor_with(
"text/turtle",
"application/pdf",
&TransreptionPolicy::allow_lossy()
)
.is_none());
}And then the payoff. camel-case has never heard of Markdown. Ask for its description
as Markdown — with consent to the projection, the next section — and the kernel renders
Turtle, finds the route, and runs it:
/// `Meta … as=text/markdown lossy=allow` on an endpoint that knows nothing about
/// Markdown. The renderer emits Turtle; the kernel finds the route; the answer is
/// Markdown. The consent is there because the route is a projection — see the
/// refusal test below for what happens without it.
#[test]
fn meta_reaches_markdown_through_the_transreptor() {
let kernel = kernel_in(None);
let request = Request::new(
Verb::Meta,
Iri::parse("urn:iki:tutorial:camel-case").expect("iri"),
)
.with_arg("as", ArgRef::Inline(b"text/markdown".to_vec()))
.with_arg("lossy", ArgRef::Inline(b"allow".to_vec()));
let repr = block_on(kernel.issue(request, &Capability::root())).expect("resolves");
let markdown = String::from_utf8(repr.bytes).expect("utf-8");
assert_eq!(repr.repr_type.media_type, "text/markdown");
assert!(
markdown.starts_with("## Camel-case (`camel-case`)\n"),
"{markdown}"
);
assert!(
markdown.contains("- `in` (required) — the text to camel-case"),
"{markdown}"
);
}cargo run -p building-endpoints -- --markdown
## Camel-case (`camel-case`)
Camel-cases the UTF-8 text supplied in the `in` argument.
- `in` (required) — the text to camel-case
Look at what was not written: no as=text/markdown branch in camel-case, no registry of
formats, no call from the requester to the transreptor. A new format became reachable
from every endpoint in the host by binding one endpoint that declared what it converts.
⚠ What the kernel routes today is
Meta. ASourcewithas=reaches the endpoint withasas an ordinary argument — an endpoint that offers faces (text and Turtle, say) reads it and answers, and one that does not ignores it. Routing aSourcethrough a transreptor chain is what the pipe does explicitly:source X | urn:rdf:transrept as=…. If you expectedsource urn:iki:tutorial:camel-case as=text/markdownto produce Markdown, that is the chapter’s one disappointment, and it is stated here so it is not discovered at a keyboard.
Declared, and the planner honors it
A transreptor claims to change form without changing what is represented: the function
is injective, and the caller can treat the output as the same resource in another
syntax. A summary, a thumbnail, an embedding, this chapter’s Markdown are not injective.
They are projections — different resources that have lost information — and a
projection registered as a transreptor would become a silent hop in a route, handing the
caller less than it asked for under exactly the type it asked for, which no reader of
the type can detect. So the declaration carries the distinction, since core 0.1.77:
.transreptor(from, to) claims lossless by default, and .lossy() says this edge is
not. It is a declaration, not a checked property — core cannot decide injectivity — but
the planner reads it. Selection plans through lossless edges only, unless the caller
consents; consent widens the search without reordering it, so a lossless route still
wins wherever one exists, and every step of a plan reports its own declaration.
What that looks like at the Meta face. Ask for Markdown and the kernel finds the route,
sees that its one hop is a projection, and refuses — typed, naming the step and the
consent, never a silent projection and never a substitution that would hide that a route
exists. Ask again with lossy=allow and the same route runs:
#![allow(unused)]
fn main() {
extern crate building_endpoints;
extern crate ikigai_core;
extern crate futures;
use futures::executor::block_on;
use ikigai_core::{ArgRef, Capability, Error, Iri, Request, Verb};
let kernel = building_endpoints::kernel_in(None);
let root = Capability::root();
let meta_as_markdown = || {
Request::new(Verb::Meta, Iri::parse("urn:iki:tutorial:camel-case").unwrap())
.with_arg("as", ArgRef::Inline(b"text/markdown".to_vec()))
};
// Without consent: refused, and the refusal says which step is lossy and what consents.
let refused = block_on(kernel.issue(meta_as_markdown(), &root)).unwrap_err();
assert!(matches!(refused, Error::Endpoint(_)));
assert_eq!(
refused.to_string(),
"endpoint error: meta face `text/markdown` is reachable only through a lossy \
transreptor (`urn:iki:tutorial:markdown`): a projection loses information and \
would not describe the endpoint; pass `lossy=allow` to consent"
);
// With consent: the route runs, and what comes back is Markdown.
let consented = meta_as_markdown().with_arg("lossy", ArgRef::Inline(b"allow".to_vec()));
let repr = block_on(kernel.issue(consented, &root)).unwrap();
assert_eq!(repr.repr_type.media_type, "text/markdown");
assert!(String::from_utf8_lossy(&repr.bytes).starts_with("## Camel-case (`camel-case`)\n"));
}The error is an Endpoint error on purpose. The face is not absent, so it is not
NotFound; no capability is at issue, so it is not Denied — consent is per request,
not per holder; and the arguments are well-formed, so it is not InvalidArgument. A
mistyped consent (lossy=alow) is an InvalidArgument, rather than being read as the
default and blamed on the face. At the REPL the same refusal is what
describe urn:iki:tutorial:camel-case text/markdown prints.
⚠ One thing the declaration still does not do. When no route at all reaches the type a
Metarequest asked for, the kernel substitutes the canonical Turtle rather than failing: a description in a form you did not ask for, over no description. The type on what comes back saystext/turtle, so a caller that reads it knows. The refusal above is only for a route that exists and is lossy; a route that does not exist is a substitution, and that is unchanged.
A format that is not RDF
Everything above went into Turtle or came out of it. Turtle is the pivot of a two-hop route, but it is not the only thing a transreptor converts. Any representation to any representation is the same mechanism: a declaration, and a plan the kernel makes from it.
ikigai-json (0.1) is a module with three resources:
urn:json:queryruns a jq program over any resource;urn:json:from-csvreads CSV as JSON;urn:json:to-csvgoes the other way.
from-csv declares .transreptor(["text/csv"], ["application/json"]), so it is also the
kernel’s route from CSV to JSON. Nothing about it involves Turtle.
The data is a small shop’s stock, a CSV table the way a spreadsheet exports it. It lives in
the crate crates/shop, which a later chapter, Short names in a space,
builds on. The table is an atom: Source answers it as text/csv, and Sink replaces it.
item,shelf,count
flour,A,12
sugar,A,0
rice,B,30
lentils,B,7
urn:json:query reads its src through the kernel, under the caller’s capability. When the
answer is not JSON, it asks the kernel for a transreptor chain from what was served to
application/json (Invocation::select_transreptor, the kernel’s planner, by the rules above), runs it, and gives the program the result. jq never sees CSV. Each record becomes an
object keyed by the header row, and every value is a string, because CSV has no types and
the converter does not guess.
The page’s kernel links ikigai-json and the shop, so these cells run here. Elements from
the collection, one record, and a value from that record:
item,shelf,count
flour,A,12
sugar,A,0
rice,B,30
lentils,B,7
[computed]
["flour","sugar","rice","lentils"]
[computed]
{"item":"rice","shelf":"B","count":"30"}
[computed]
30
[computed]
A program that yields one result answers that value, and as=text/plain answers a single
scalar as bare text, with no quotes. That is the form a template splices, which
Tic-tac-toe, V comes back to.
⚠ A
Sourceof the stock withas=application/jsonstill answers CSV. As the note above says,as=on aSourceis the endpoint’s own argument, and the stock has one face. The route through the transreptor is planned by whoever asks for JSON. Here that isurn:json:query, and in the REPL it is a pipe:source urn:iki:tutorial:shop:stock | urn:json:from-csv.
Cached, and cut by a write
A query’s answer is cached exactly when its read of src is, and it hangs from the stock’s
thread. The conversion is a step inside it, so a second query over the same table finds the
answer cached. A query that uses a number converts the string itself (tonumber):
["flour","rice"] [computed] ["flour","rice"] [cached]
Now change the table. The line below reads the stock as JSON and keeps only what is on a
shelf. Then it writes that back as CSV through urn:json:to-csv and sinks it to the stock.
The round trip is three resources and a pipe. The Sink cuts the stock’s thread, as every
Sink does (Golden threads in practice), and both
cached selections above go with it. The first recomputes to the same answer, since flour and
rice are still on their shelves. The list of items, read in the first cell, recomputes without
sugar:
ok [2 computed · 1 uncacheable] item,shelf,count flour,A,12 rice,B,30 lentils,B,7 [computed] ["flour","rice"] [computed] ["flour","rice","lentils"] [computed]
The pipe line counts what it ran: the query and to-csv computed, and the Sink, which is
never cached. The table came back with CRLF line ends, which is what RFC 4180 writes, and
from-csv reads it the same as before. to-csv is declared lossy: 36 and "36" write the same. The
kernel would not plan it into a route without consent, and it did not have to. The pipe
called it by name.
The same listing in Rust, against the shop’s own host:
#![allow(unused)]
fn main() {
extern crate shop;
use shop::{source, Host, STOCK};
let mut host = Host::start("shop");
let shelved = || {
source(
"urn:json:query",
&[("src", STOCK), ("q", "[.[] | select(.count | tonumber > 10) | .item]")],
)
};
assert_eq!(host.read(shelved()).unwrap(), r#"["flour","rice"]"#);
assert!(host.is_cached(&shelved()));
// A write to the CSV cuts its thread, and the selection recomputes from the new table.
host.write(STOCK, "item,shelf,count\nflour,A,9\nrice,B,30\n");
assert!(!host.is_cached(&shelved()));
assert_eq!(host.read(shelved()).unwrap(), r#"["rice"]"#);
}jq here is a sandbox: ikigai-json rewrites the program so every step is metered, and bounds
its size, its input, its work, its depth and its output. On a native host a bound reached is
a typed refusal. In this page it is an abort, because WebAssembly cannot unwind, and it stops
the page’s kernel until you reload. Edit these cells freely, but keep programs small: a
recursion or a string built a million times over is the way to find that out.
Try it
#![allow(unused)]
fn main() {
extern crate building_endpoints;
extern crate ikigai_core;
extern crate futures;
use futures::executor::block_on;
use ikigai_core::{ArgRef, Capability, Iri, Request, Verb};
let kernel = building_endpoints::kernel_in(None);
// The transreptor, driven by hand, exactly as the kernel drives it: content plus as.
let turtle = "@prefix ik: <https://ikigai-rs.dev/ns#> .
<urn:x> a ik:Endpoint ; ik:id \"x\" ; ik:title \"Example\" ; ik:summary \"Made up.\" .";
let request = Request::new(Verb::Source, Iri::parse("urn:iki:tutorial:markdown").unwrap())
.with_arg("content", ArgRef::Inline(turtle.as_bytes().to_vec()))
.with_arg("as", ArgRef::Inline(b"text/markdown".to_vec()));
let repr = block_on(kernel.issue(request, &Capability::root())).unwrap();
assert_eq!(String::from_utf8_lossy(&repr.bytes), "## Example (`x`)\n\nMade up.\n\n");
}