AGENTIC-SYSTEMSJuly 7, 20261 min read

Silicon Sampling via Cross-Survey Transfer

A new study proposes 'cross-survey transfer' to rigorously evaluate how well LLMs can simulate human survey respondents, revealing that zero-shot LLMs can predict individual responses with surprising accuracy.

Computer Science > Artificial Intelligence

Title:Silicon Sampling via Cross-Survey Transfer

View PDFAbstract:Silicon sampling-using large language models (LLMs) to simulate human survey respondents-has emerged as a promising approach for augmenting traditional survey research. However, most evaluations rely on distributional comparisons rather than individual-level prediction, which risks conflating pattern matching with coherent respondent-level prediction. We propose cross-survey transfer, a more rigorous evaluation framework in which an LLM is given a respondent's answers to one set of questions and must predict their answers to entirely different questions from the same survey. Using data from the Taiwan Election and Democratization Study (TEDS) 2024, three open-weight LLMs (27B-120B parameters), and supervised machine learning baselines, we find that: (1) zero-shot LLMs achieve 52% accuracy on genuinely unseen items, closing to within 6 percentage points (pp) of a supervised random forest trained on same-population data; (2) a stable construct predictability hierarchy emerges, from 67% for partisan attitudes to 23% for sovereignty; and (3) variance collapse and safety alignment effects-two commonly cited LLM limitations-turn out to be more nuanced than previously reported, with variance collapse affecting supervised models as well and alignment effects varying dramatically across model families. These findings clarify both the promise and boundaries of silicon sampling.

Current browse context:

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Source: arXiv cs.AI Recent

Stay in the loop

Get the best AI & tech insights in your inbox — no spam, unsubscribe anytime.

More in this category

agentic-systems

Automated Data Readiness for Scientific AI

Researchers have introduced REDI, an open-source framework that automates the transformation of raw scientific datasets into AI-ready data. Tested on the Frontier supercomputer, REDI addresses the data preparation bottleneck in scientific AI domains.

agentic-systems

ASK in the Dark: Uncertainty-Gated LLM Assistance under Partial Observability

Researchers have introduced ASK+, a novel framework that significantly enhances small language model (SLM) assistance for reinforcement learning agents under partial observability. By providing trajectory-aware context and structured reasoning, ASK+ demonstrates that prompt design and selective gating can outperform sheer model scale.

agentic-systems

When Aggregate Alignment Misleads: Auditing Policy Repair Without Per-State Expert Actions

Evaluating agentic policy repair without per-state expert labels is challenging, as aggregate behavioral alignment metrics can be highly misleading. A new study using a hotel-pricing simulator demonstrates that LLM-based editors can achieve near-benchmark revenue by leveraging indirect diagnostic feedback, highlighting the need for closed-loop evaluation.

agentic-systems

Embodied Operators and Benchmarking: Toward Reusable and Deployable Embodied Intelligence Systems

A new study proposes 'embodied operators' to modularize embodied intelligence systems, enabling reusable and deployable robot pipelines beyond end-to-end models, supported by a multi-dimensional benchmarking framework.

agentic-systems

Evaluating Generative Agents with Actions Grounded in Socially Distributed Task Environments using Incognita

Researchers introduce Incognita, a framework to evaluate generative agents in socially distributed task environments where knowledge is partitioned across multiple entities. Testing shows that while advanced models perform better at gathering hidden information, overall reliability in executing grounded actions remains low.

agentic-systems

Object-Centric Environment Modeling for Agentic Tasks

Researchers propose Object-Centric Environment Modeling (OCM) to help AI agents organize accumulated experiences into an executable programmatic model, addressing the limitations of traditional free-form textual memories.