Provenance and archiving: an agent run as evidence

One object, five views

Anything you run, whether a single chat, a deliberation, or an experiment, converts to one object through as_agent_run(). That object answers five questions, and you ask each by naming a level:

panel <- list(
  agent("Optimist", cfg, persona = "You see opportunity first."),
  agent("Skeptic",  cfg, persona = "You probe for hidden costs.")
)
debate <- deliberate(panel, "Should the department adopt a four-day week?",
                     rounds = 1)

run <- as_agent_run(debate)

The levels are deliberately plain tibbles, so the rest of your analysis is ordinary data work:

tibble::as_tibble(run, level = "utterance")  # who said what, in order
tibble::as_tibble(run, level = "call")       # one row per model call
tibble::as_tibble(run, level = "event")      # the full event graph
tibble::as_tibble(run, level = "tool")       # every tool invocation
tibble::as_tibble(run, level = "state")      # each agent's memory at the end

The utterance level is the transcript: the thing you usually read. The call level is the instrument log: every model call as a row, with the served model version, the token counts, a success flag, and a request_hash that uniquely identifies the exact request that produced it. The agent recorded all of this while it answered.

Diagnostics for a run

diagnostics() reads the call level and returns a one-row summary. The numbers come from the same place the wider ‘LLMR’ ecosystem reads them, so they agree across tools rather than reporting two different figures:

diagnostics(run)

A high failure count, an unexpected model version, a token bill larger than the work warrants – these surface here before they surface in your conclusions.

Reporting within scope

report() writes a short methods paragraph and, importantly, refuses to overstate. An agent run is model-conditioned output, not a sample from a human population, and report() says so. Claim labels can describe an instrument pilot, theory probe, or coding exercise, but none removes that scope condition.

report(run)

The manifest: identity without the transcript

Two runs are “the same study” when their design is the same: same personas, same tools, same turn policy, same model and settings. They are emphatically not the same when any of those change, even if the transcripts happen to read alike. agent_manifest() captures the declared design as a set of content hashes, keeping outcome (the transcript) separate from recorded apparatus:

m <- agent_manifest(run)
m$manifest_hash    # changes when a hashed declared component changes

A reworded persona, a changed tool body, or a different temperature moves the hash. Values captured in a tool closure are not part of that identity; record such state in the design when it matters. This is the same split ‘LLMR’ makes between a request’s identity and its content, lifted to the level of a whole study.

Archiving: seal the evidence

archive_agent_study() writes a self-describing directory. It contains the transcript, event graph, manifest, artifacts, drafted methods note, a data-only RDS projection, and a checksum file that seals the whole thing. The centerpiece is calls.jsonl, which is a genuine ‘LLMR’ audit log that ‘LLMR’ can read.

dir <- file.path(tempdir(), "four_day_week_study")
archive_agent_study(run, dir)
list.files(dir)

The archive never serializes live agents, caller closures, tool functions, or model configurations. The archive keeps the original hashes. They are computed over the original text, so they match the live run even if you redact the stored copy for privacy. And the request hash in calls.jsonl joins to the request_hash you saw at the call level. That join is the integrity check: it lets a skeptic confirm that the archived log describes the run it claims to describe.

calls   <- tibble::as_tibble(run, level = "call")
archived <- LLMR::llm_log_read(file.path(dir, "calls.jsonl"))

# every archived call is one of the calls the run actually made
all(archived$manifest$request_hash %in% calls$request_hash)

If you keep a live audit log during the run (call LLMR::llm_log_enable("calls.jsonl") before you start), the archive copies that real file, filtered to this run’s calls, rather than reconstructing it, and so preserves the log’s own record hashes.

Redaction without lying

Survey responses and interview text often cannot be stored verbatim. redact scrubs the free-text columns of the stored copy, while the hashes stay computed over the original. The archive remains internally consistent: it does not claim a request body it did not send, and the join still holds.

archive_agent_study(run, file.path(tempdir(), "redacted_study"),
                    redact = c("four-day", "salary"))

For privacy that must omit the request entirely, pass include_messages = FALSE: the archive then carries metadata and hashes but no message text at all, including in run.rds. A nonempty destination is refused unless overwrite = TRUE.

Why archive

A reviewer asks which model you used and whether you cherry-picked. A replication team, a year on, has only your files. The archive preserves the recorded design, transcript, calls, and hashes that join them. It makes a model-generated result something one can inspect, verify, and defend; it is not an executable checkpoint.