NOW LET US – AI RAG SaaS Studio TP.HCM
NOW LET US
Digital Product Studio
Back to news
AGENTIC-SYSTEMS...1 min read

Toten: Knowledge-Based Ontological Tokenization Of Physical Quantities And Technical Notation In Brazilian Portuguese

Share
NOW LET US Article – Toten: Knowledge-Based Ontological Tokenization Of Physical Quantities And Technical Notation In Brazilian Portuguese

Traditional tokenization methods like BPE often fragment physical quantities and technical notations, degrading AI performance in specialized domains. To address this, TOTEN introduces a knowledge-based ontological tokenization framework that preserves the semantic integrity of engineering entities.

Computer Science > Artificial Intelligence

Title:Toten: Knowledge-Based Ontological Tokenization Of Physical Quantities And Technical Notation In Brazilian Portuguese

View PDF HTML (experimental)Abstract:Byte-Pair Encoding tokenization is statistically efficient for vocabulary compression, but semantically blind to structured technical entities, fragmenting physical quantities, numbers, units, and symbolic expressions into lexically arbitrary subwords. We present TOTEN, a knowledge-based ontological tokenization framework that replaces statistical derivation with declarative classification grounded in a formal ontology of engineering entities (OEE). We formalize TOTEN as the triple <O, classify, {inst_tau}>: the ontology gathers types, structural principles, composition relations, and preservable invariants; the classification function maps raw text into typed regions; and the instantiator family yields a self-descriptive structured representation. Robustness derives from deterministic coupling with three external oracles: Pint (dimensional), Unicode Character Database (typographic), and RSLP (Portuguese morphology). Intrinsic evaluation covers four properties verifiable by construction -- ontological atomicity, dimensional equivalence, typographic robustness, and numerical reconstruction -- over an internal, physically validated benchmark (EngQuant, N=800) and four Brazilian Portuguese external corpora (N=1771 eligible cases). We also report detection recall, distinguishing coverage from conditional atomicity. Against eight state-of-the-art baselines, TOTEN achieves unit ontological atomicity in all contrasts and numerical reconstruction of 0.775-0.904 on external corpora, vs. 0.627-0.703 for the best baseline (Quantulum3); on EngQuant, 0.780 vs. 0.340. Differences are statistically significant (McNemar with Holm correction). Spearman correlation between internal and external rankings confirms concurrent validity of the control benchmark. Dimensional equivalence shows statistical parity with Pint, the oracle from which the system inherits dimensional authority.

© 2026 Now Let Us. All rights reserved.

Source: arXiv cs.AI Recent

Advertisement
Ad slot ready: 5887729102

More in this category

NOW LET US Related – Analyzing the Narration Gap in LLM-Solver Loops

agentic-systems

Analyzing the Narration Gap in LLM-Solver Loops

A new study highlights the 'narration gap' in hybrid LLM-solver systems, revealing that while the formal solver produces sound results, adversaries can still manipulate the LLM to invert the final answer via prompt injection.

NOW LET US Related – Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why

agentic-systems

Configurable Clinical Information Extraction with Agentic RAG: What Works, What Breaks, and Why

Researchers at University Medicine Essen have deployed ACIE, an on-premise agentic RAG pipeline that extracts complex clinical information with a 96.5% clinician acceptance rate, overcoming the limitations of standard RAG in handling unstructured medical data.

NOW LET US Related – Uncertainty Decomposition for Clarification Seeking in LLM Agents

agentic-systems

Uncertainty Decomposition for Clarification Seeking in LLM Agents

A new study proposes a prompt-based uncertainty decomposition method that enables LLM agents to detect ambiguous user requests and proactively seek clarification. This approach significantly outperforms existing methods across multiple large language models, including GPT-5.1 and DeepSeek-v3.2.

NOW LET US Related – ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

agentic-systems

ITNet: A Learnable Integral Transform That Subsumes Convolution, Attention, and Recurrence

Researchers have introduced ITNet, a unified neural network architecture that mathematically subsumes convolution, self-attention, and recurrence under a single learnable integral transform, matching or exceeding specialized baselines across multiple modalities.

NOW LET US Related – LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

agentic-systems

LLM Doesn't Know What It Doesn't Know: Detecting Epistemic Blind Spots via Cross-Model Attribution Divergence on Clinical Tabular Data

A recent study reveals that Large Language Models (LLMs) struggle to recognize their own knowledge limits when processing structured clinical tabular data. By comparing Qwen 2.5 7B with XGBoost, researchers identified critical epistemic blind spots and proposed a cross-model calibration method to address this limitation.

NOW LET US Related – Deontic Policies for Runtime Governance of Agentic AI Systems

agentic-systems

Deontic Policies for Runtime Governance of Agentic AI Systems

Autonomous agentic AI systems introduce novel security and compliance challenges that exceed the capabilities of current policy engines. To address this, researchers propose AgenticRei, a runtime governance framework utilizing deontic policies to strictly control AI behavior outside the LLM.

EXPLORE TOPICS

Discover All Categories

Deep dive into the specific technology sectors that matter most to you.

AI Frontier

AI Frontier

Foundation models, multimodal systems, and enterprise use cases.

Agentic Systems

Agentic Systems

Task-oriented automation workflows for real operations.

SaaS Growth

SaaS Growth

CAC, LTV, retention, and growth optimization.

Dev Tools

Dev Tools

Codegen tools, next-gen IDEs, and modern dev workflows.

Cybersecurity

Cybersecurity

AI security, Zero Trust, and proactive defense.

Cloud & Infra

Cloud & Infra

Cloud Native, Edge Computing, and GPU Clusters.

Robotics

Robotics

Humanoid robots, autonomous vehicles, and smart hardware.

Startups & VC

Startups & VC

Funding trends, M&A, and the startup ecosystem.