ROMA: LLM System for Real-World Object-Centric
Multi-Sensory Active Perception

I saw, I touched, I understood.

Ruoxuan Feng^*, Yutong Chen^*, Ruihua Song, Huan Yang, Zhongyuan Wang, Guocai Yao, Di Hu^✉
*Equal contribution   ^✉Corresponding author

---

News

• [2026/10/20] We released the ROMA Bench QA tasks and the evaluation code.
• [2026/10/16] We built an online GUI demo. Try it now!
• [2026/10/07] We released the ROMA-7B checkpoint and the inference demo code.

Introduction

Humans build an understanding of the physical world through an active process: when sensory evidence is insufficient, we decide what information is missing, how to acquire it, and when enough evidence has been obtained. Existing multi-sensory robot systems, in contrast, mostly integrate whatever sensory inputs they are given.

ROMA is an LLM-based system for Real-World Object-Centric Multi-Sensory Active Perception. It integrates vision, audio, touch, and force into a reasoning-interaction-feedback loop: the LLM identifies the missing evidence and selects the target object, the interaction (lift, press, collide, shake, rotate, squeeze), and the sensory modalities, while a physical interface executes the interaction and returns the multi-sensory feedback.

This repository accompanies the paper arXiv:2610.06955 and contains:

• ROMA-7B (checkpoint), a multi-sensory LLM built on Qwen2.5-Omni with action / modality tokens, an audio branch, and an AnyTouch 2 tactile branch, trained with multi-sensory alignment followed by active-perception SFT.
• ROMI-2K (dataset), a real-world multi-sensory object interaction dataset covering nearly 2,000 objects and 6 atomic interactions with synchronized visual, audio, tactile, and force feedback.
• ROMA Bench (dataset), 2,100 scene-level tasks (single-chain, multi-chain, and intent-driven) for evaluating active perception.
• A local web demo that streams ROMA's active-perception process on a real tabletop scene.
• The ROMA Bench evaluation code (multi-GPU batched inference and result statistics).

Highlights

Model	ROMA Bench (total)	Real-world execution (total)
GPT-5.4	43.5	40.2
Gemini 3.5 Flash	53.0	41.7
Qwen 2.5-Omni	22.0	23.5
Qwen 3-Omni	19.7	23.5
ROMA-7B	72.9	61.4

Task success rate (%). See the paper for the full results.

Demo

The demo runs locally. Given the initial scene image and a question, ROMA reasons step by step; whenever it emits an action token, the corresponding recorded feedback (wrist image, tactile images, contact sound, gravity force) is loaded from resources/example_data/1 and fed back to the model, and generation continues until it answers. Everything is streamed to the page:

• every grasp is drawn on the scene image as a numbered, colored box;
• each interaction is shown as an Interaction & Feedback card with the returned sensory evidence;
• the final answer is highlighted.

See Usage to launch it.

Installation

conda create -n roma python=3.10.20
conda activate roma

pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
pip install -r requirements.txt

(Highly Recommended) Install FlashAttention for faster inference. When it is installed, demo.py and eval_roma_7b.py use it automatically (attn_implementation="flash_attention_2"); otherwise they fall back to the default attention.

Usage

1. Download the checkpoint

Before running, pull the GeWu-Lab/ROMA-7B checkpoint from Hugging Face into the resources directory (the demo only needs the checkpoint; the datasets are released separately in GeWu-Lab/ROMA):

hf download GeWu-Lab/ROMA-7B --repo-type model --local-dir resources/models/ROMA-7B

The ROMA-7B checkpoint directory (ROMA-7B) is expected to contain:

ROMA-7B/
├── model-0000*-of-00005.safetensors  # base Qwen2.5-Omni-7B model
├── model.safetensors.index.json
├── anytouch2.pth        # tactile encoder
├── audio.bin            # fine-tuned audio adapter and encoder
├── tactile.bin          # fine-tuned tactile adapter
├── ROMA-LLM.bin         # ROMA LLM weights
└── config / tokenizer files

2. Check the checkpoint path

The demo reads the checkpoint directory from MODEL_PATH_2_5 in config/global_name.py, which defaults to resources/models/ROMA-7B. If you downloaded the checkpoint elsewhere, edit this path accordingly.

3. Run the web demo

python demo.py --gpu 0 --port 7860

Then open http://localhost:7860, type a question (or pick an example), and press Ask ROMA. The base model and all adapters are loaded from the checkpoint directory in the same order as during training (base Qwen2.5-Omni, tactile encoder, new tokens, audio adapter, tactile adapter, ROMA LLM). Options: --gpu, --host, --port, --share (create a public Gradio link).

Evaluation on ROMA Bench

1. Download the dataset

Download the GeWu-Lab/ROMA dataset into resources/datasets:

hf download GeWu-Lab/ROMA --repo-type dataset --local-dir resources/datasets

The evaluation uses:

resources/datasets/
├── ROMA-Bench/              # scenes: initial image, save.json, and the recorded feedback of each object
└── roma_bench_tasks.json    # 2,100 QA tasks and labels

The paths are set in config/global_name.py (ROMA_BENCH_DIR, ROMA_BENCH_QA_PATH, and ROMA_BENCH_RESULTS_DIR for the outputs, default results/roma_bench); edit them if you put the data elsewhere.

2. Run inference

bash run_eval.sh "0,1,2,3" 8 50   # [gpus] [batch_size] [save_every]

run_eval.sh starts one eval_roma_7b.py process per GPU. Each process handles its own shard of the tasks and runs batch_size tasks in parallel with continuous batching. For every task, ROMA generates until it emits an action token, the recorded feedback of the grasped object is loaded from ROMA-Bench, and generation continues until the model answers (at most 20 rounds). Each GPU saves its results to eval_roma_bench_<shard>.json / error_roma_bench_<shard>.json every save_every finished tasks and once more at the end; logs are written to logs/.

To run a single process directly: python eval_roma_7b.py --gpu 0 --batch_size 8 --save_every 10 (see --help for the path options).

3. Compute the results

python check_eval_results.py

It reads every eval_roma_bench_*.json in the results directory (or the files matched by --results "<glob>") and reports the overall accuracy, the accuracy by level / category / task type, interaction statistics, and a final Level × Category summary table.

Note: The Correct / Error counts printed at the end of eval_roma_7b.py are only a rough exact match, since the model output is not always just the option letter(s). Use the results of check_eval_results.py.

Repository Structure

ROMA
├── demo.py                # local web demo (Gradio)
├── eval_roma_7b.py        # ROMA Bench inference (batched, one process per GPU)
├── run_eval.sh            # multi-GPU launcher for eval_roma_7b.py
├── check_eval_results.py  # ROMA Bench result statistics
├── qwen25omni/            # Qwen2.5-Omni modified with the tactile modality
├── anytouch2/             # tactile encoder
├── my_qwen_omni_utils/    # audio / vision / tactile input processing
├── config/, utils/        # arguments, DeepSpeed and checkpoint utilities
└── resources/             # checkpoints, datasets, and the demo example scene

TODO

• Release the local web demo
• Release ROMA-7B checkpoint and the ROMI-2K dataset
• Release the ROMA Bench evaluation code
• Release the training code (multi-sensory alignment and active-perception SFT)
• Release the physical interface (grasp interface and interaction execution)

Citation

If you find this work useful, please consider citing:

@article{feng2026roma,
  title   = {ROMA: LLM System for Real-World Object-Centric Multi-Sensory Active Perception},
  author  = {Feng, Ruoxuan and Chen, Yutong and Song, Ruihua and Yang, Huan and
             Wang, Zhongyuan and Yao, Guocai and Hu, Di},
  journal = {arXiv preprint arXiv:2610.06955},
  url     = {https://arxiv.org/abs/2610.06955},
  year    = {2026}
}

Acknowledgements

ROMA builds on Qwen2.5-Omni, AnyTouch 2, and AnyGrasp. We thank the authors for their open-source contributions.