LLMRpanel administers survey and experimental instruments to panels of language model personas. Use it to pretest instruments, pilot experimental designs, or measure how a configured model responds under specified personas. Without comparison against a supplied human benchmark, its response shares describe the configured model and personas, not a human population.
To work without writing code, run_panel_studio() opens a point-and-click
interface that builds a panel, administers an instrument, and compares the
result with a human benchmark. It has an offline demonstration mode, so it can
be explored without a provider key. See "Graphical interface" below.
install.packages("LLMR")
# remotes::install_github("asanaei/LLMRpanel")
install.packages("LLMRpanel", repos = c('https://asanaei.r-universe.dev', 'https://cloud.r-project.org'))
cfg <- LLMR::llm_config(
"groq",
"openai/gpt-oss-20b",
temperature = 0.8
)
Administration requires an explicit LLMR::llm_config(). Under the hood,
LLMRpanel calls language models through the LLMR package, which reads your API
key from an environment variable such as GROQ_API_KEY. Set it once in your
~/.Renviron file, a plain text file in your home directory.
The workflow builds a persona panel and an instrument. Administration records the displayed option order, and benchmark comparisons use supplied human response shares.
panel_from_margins() samples each attribute independently from supplied
population margins. This route is appropriate when the available population
information consists of marginal tables.
library(LLMRpanel)
set.seed(110)
panel <- panel_from_margins(
list(
cohort = c(young = .30, middle = .45, older = .25),
party = c(left = .45, right = .45, independent = .10)
),
n = 60,
persona_template = "A {cohort} voter who leans {party}."
)
When microdata are available, panel_from_data() samples complete rows and can
use a sampling weight. It therefore retains relationships among the selected
attributes. as_persona_frame() attaches question wording and distinguishes
demographic fields from stated answers before those rows are rendered.
source_rows <- data.frame(
age = c("18 to 34", "35 to 64", "65 plus"),
party = c("left", "independent", "right"),
survey_weight = c(1.2, 0.9, 1.4)
)
persona_rows <- as_persona_frame(
source_rows,
questions = c(party = "Party identification"),
demographics = "age",
answers = "party"
)
microdata_panel <- panel_from_data(
persona_rows,
n = 60,
columns = c("age", "party"),
weights = "survey_weight"
)
anes_panel <- panel_from_personas(LLMR::anes_2024_personas, n = 60)
panel_from_personas() uses prepared persona rows, including the question
wording and demographic metadata in LLMR::anes_2024_personas. Reports identify
whether a panel came from margins, microdata rows, or prepared personas.
An instrument can combine Likert, choice, and open response items.
instrument <- panel_instrument(list(
item_likert(
"wk4",
"A four-day work week would benefit society."
),
item_choice(
"fund",
"Which investment should be funded first?",
c("public transit", "road repair")
),
item_open(
"reason",
"What should the city consider when choosing between public transit and road repair?"
)
))
panel_administer() sends a separate request for each persona-item pair.
A request contains the persona and one item. It does not contain earlier items
or their responses.
With the default instrument settings, choice options are permuted for each
request. A Likert scale is displayed in its original or reversed order, each
with probability one-half. Set randomize = character(0) in
panel_instrument() to retain the supplied option order.
The response data record the displayed options in option_order.
item_position is the item's fixed, 1-based position in the instrument.
It does not record an order of exposure.
resp <- panel_administer(panel, instrument, cfg)
resp
resp$data
panel_benchmark() compares valid model response shares with supplied human
shares for matching item and response values. It records benchmark coverage,
deviations, and nonresponse. Printed results distinguish unbenchmarked,
partially benchmarked, and benchmarked studies.
bench <- data.frame(
item_id = "fund",
response = c("public transit", "road repair"),
share = c(.41, .59)
)
resp <- panel_benchmark(resp, bench, "city survey, 2025")
resp
resp$benchmark$table
resp$benchmark$nonresponse
plot(resp)
LLMR::report(resp)
Coverage of a closed item permits comparison for that item. It does not turn uncovered items into estimates of a human population.
panel_bias_audit() counts execution and parsing failures by item. For closed
items with randomized options, it tests whether the selected response is
associated with the option shown first.
panel_bias_audit(resp)
LLMR::diagnostics(resp)
The test concerns the first option shown, not the full option permutation.
conjoint_design() defines the attribute universe and number of tasks.
Administration draws profiles for each respondent and records those profiles
with the response. conjoint_amce() estimates average marginal component
effects from the recorded assignments, with standard errors clustered by
persona.
set.seed(110)
design <- conjoint_design(
list(
price = c("low", "high"),
origin = c("domestic", "imported")
),
n_tasks = 4
)
cj_instrument <- conjoint_instrument(
design,
"Which product would you buy?"
)
cj_resp <- panel_administer(panel, cj_instrument, cfg)
conjoint_amce(cj_resp)
run_panel_studio() provides a Shiny interface for panel construction,
instrument administration, benchmark comparison, and study artifacts.
run_panel_studio()
For large studies, panel_batch_submit() sends the administration to a
provider's asynchronous batch API. panel_batch_status() checks the job and
panel_batch_fetch() retrieves completed results. A state_path stores the job
for later status or fetch calls.
job <- panel_batch_submit(
panel,
instrument,
cfg,
state_path = "panel-job.rds"
)
panel_batch_status(job)
batch_resp <- panel_batch_fetch(job)
Synchronous and batch administration return the same response fields and
attached study information. Both require an explicit configuration and stop
above max_calls unless confirm = TRUE.
A panel_responses object stores response rows in $data and the panel,
instrument, benchmark, and usage records as separate components. Response rows
retain response_text, response_id, success, model, and provider;
finish_reason is present when the response function supplies it. This keeps
an unmatched reply available for inspection. panel_usage() summarizes the
usage component and retains model and provider, which permits a supplied price
table to match the model that incurred the usage.
panel_usage(batch_resp)
tibble::as_tibble(batch_resp)
FocusGroup::create_agents_from_data() accepts a silicon_panel, so panel
personas can define participants in a moderated group discussion.
LLMR::anes_2024_personas supplies prepared persona data used by both packages.
LLMRpanel uses LLMR, the common provider interface on CRAN. LLMRcontent codes source text with codebooks, validates labels against text coded by humans, and builds replication archives. FocusGroup runs moderated group discussions. LLMRagent provides tools for agent experiments. The ecosystem page describes the package boundaries.
Report bugs and feature requests in the GitHub repository. Pull requests may be submitted there.
This project uses the MIT License; see LICENSE.