AGENTIC-SYSTEMSApril 3, 20261 min read13 views

A Reliability Evaluation of Hybrid Deterministic-LLM Based Approaches for Academic Course Registration PDF Information Extraction

NOW LET US Article – A Reliability Evaluation of Hybrid Deterministic-LLM Based Approaches for Academic Course Registration PDF Information Extraction

This study evaluates hybrid approaches combining LLMs with deterministic methods for extracting information from academic PDFs, finding that a Camelot-based pipeline with LLM fallback offers the best balance of accuracy and speed on consumer hardware.

Computer Science > Computation and Language

Title:A Reliability Evaluation of Hybrid Deterministic-LLM Based Approaches for Academic Course Registration PDF Information Extraction

Abstract:This study evaluates the reliability of information extraction approaches from KRS documents using three strategies: LLM only, Hybrid Deterministic - LLM (regex + LLM), and a Camelot based pipeline with LLM fallback. Experiments were conducted on 140 documents for the LLM based test and 860 documents for the Camelot based pipeline evaluation, covering four study programs with varying data in tables and metadata. Three 12 - 14B LLM models (Gemma 3, Phi 4, and Qwen 2.5) were run locally using Ollama and a consumer grade CPU without a GPU. Evaluations used exact match (EM) and Levenshtein similarity (LS) metrics with a threshold of 0.7. Although not applicable to all models, the results show that the hybrid approach can improve efficiency compared to LLM only, especially for deterministic metadata. The Camelot based pipeline with LLM fallback produced the best combination of accuracy (EM and LS up to 0.99 - 1.00) and computational efficiency (less than 1 second per PDF in most cases). The Qwen 2.5:14b model demonstrated the most consistent performance across all scenarios. These findings confirm that integrating deterministic and LLM methods is increasingly reliable and efficient for information extraction from text based academic documents in computationally constrained environments.

Source: arXiv cs.AI Recent

More in this category

agentic-systems

VeriSimpl: Robust Optimization Modeling from Natural Language using Simplification-based Verification

Researchers have introduced VeriSimpl, a novel framework that leverages LLMs and simplification-based verification to accurately translate natural language descriptions into executable optimization formulations.

agentic-systems

SonicSampler: Unified Tile-Aware Kernels for LLM Sampling and Speculative Verification

Researchers introduced SonicSampler, a unified suite of tile-aware Triton kernels designed to accelerate LLM sampling and speculative verification. By fusing the entire pipeline into a single CUDA Graph-compatible kernel, SonicSampler achieves up to 16x speedups over state-of-the-art baselines.

agentic-systems

PersonaTrail: Benchmarking Personalized Web Agents through Browsing Trails

Researchers have introduced PersonaTrail, a novel benchmark evaluating personalized web agents through realistic browsing histories. Alongside it, the PACMem memory framework structures raw browsing data to significantly boost agent performance in inferring user preferences.

agentic-systems

Benchmarking the Personalization Capabilities of Large Language Models

A new study introduces SDR-Bench and SDR-Arena to benchmark the personalization capabilities of LLMs in two-party persuasion scenarios, revealing a personalization plateau among frontier models.

agentic-systems

Incomplete Prompt Jailbreaks in Large Language Models

Researchers have conceptualized Incomplete Prompt Jailbreaks (IPJ), a vulnerability where incomplete harmful prompts bypass LLM safeguards due to delayed refusal behaviors. The discovery of functional neurons offers a novel path for precise neuron-level defense interventions.

agentic-systems

Benchmarking Large Language Models on Multi-Sensor Physical Hazard Assessment

A new benchmark evaluates five leading large language models on multi-sensor physical hazard assessments, revealing a critical vulnerability when evaluating combined sensor risks below individual thresholds. Despite near-perfect accuracy on single-sensor breaches, all models failed to issue precautionary signals for multi-sensor hazards.