Quantitative Benchmark of Vision-Language Models for 3D Physical Property Inference: Structured Output, Content Fidelity, and Operational Efficiency

Read full text Summarise with VIRAT
Authors: Min-gyu Kim, Hoan-Suk Choi, Minsu Jo, Jinhong Yang
Cited by: 0

Summary

No summary yet. VIRAT can read this paper and write a plain-language summary of its aim, methods, key findings and limitations.

Summarise with VIRAT Free to try with your Science Catalogs account.

Abstract

Manufacturing robotics and simulation require structured physical attributes—material, density, and movement type—for each part of 3D assets. We systematically evaluate whether open-source 8B parameter vision language models (VLMs) can infer these properties from multi-view rendered images and output them in a prescribed JSON schema. Using 2,000 samples from PhysXNet as ground truth, we benchmark Qwen3-VL-8B, MiniCPM-V4.5-8B, and InternVL3.5-8B under identical one-shot prompts and deterministic decoding, with Gemini 3 Flash Preview as a frontier reference. Evaluation metrics are separated into three axes: structured output capability (JSON validity, schema conformity), content fidelity (part-count match, material accuracy, density error, movement-type accuracy, rank correlation), and operational cost (latency, output tokens, OOM rate). Results reveal that 8B models exhibit distinct strength–weakness profiles: InternVL3.5 achieves the highest JSON validity (97.2%) but the lowest material accuracy (41.8%), while Qwen3 shows relatively lower validity (75.3%) yet the best movement-type accuracy (73.1%) among 8B models. All 8B models display output collapse, predicting only 2–4 parts regardless of input complexity. Notably, even the frontier model struggles with density estimation (MAPE 101.8%), suggesting that certain physical quantities are inherently difficult to infer from visual information alone. We provide deployment-oriented selection guidelines based on task priority and resource constraints.

Fields of Study & Subjects

Multimodal Machine Learning Applications Advanced Neural Network Applications Robot Manipulation and Learning

Publication details

Journal / source
Journal of Korea Multimedia Society
Volume
29
Issue
2
Pages
506–514
Publication date
28 February 2026
Language
EN