RT-2 & OpenVLA Explained: Vision Transformers + Action Tokens

The Architecture Is Embarrassingly Simple

Most robotics papers make VLA (Vision-Language-Action) models sound like some radical new paradigm. They’re not. RT-2 from Google DeepMind and OpenVLA from Stanford are just vision transformers with action tokens bolted onto the output vocabulary. That’s it. The “revolutionary” insight is treating robot actions as language tokens — which, in hindsight, seems almost obvious.

I’ve been digging through the RT-2 codebase and OpenVLA’s implementation, and here’s what surprised me: the core inference loop is maybe 50 lines of meaningful code. The rest is tokenization gymnastics.

Why Roboticists Overcomplicate This

The traditional robotics stack looks something like: perception module → state estimator → planner → controller → actuator commands. Each component has its own failure modes, latencies, and PhD-level complexity. VLA models collapse all of that into a single forward pass through a transformer.

But here’s what the papers don’t emphasize enough — this only works because vision transformers (ViT, originally from Dosovitskiy et al. at Google in 2020) already encode spatial relationships so well. The action prediction is essentially a language modeling problem once you’ve got good visual features.

The key equation is deceptively simple. Given an image $I$ and language instruction $L$, the model predicts action tokens $a_t$ autoregressively:

$$P(a_t | I, L, a_{<t}) = \text{softmax}(W_a \cdot h_t)$$

where $h_t$ is the hidden state from the transformer decoder, and $W_a$ projects to the action vocabulary. Nothing fancy — it’s the same output layer you’d use for next-token prediction in GPT.

What Action Tokens Actually Look Like

This is where things get concrete. In RT-2, actions are discretized into 256 bins per dimension. A 7-DoF robot arm with gripper state becomes 8 × 256 = 2048 possible action tokens per timestep. The tokenization looks something like:

import numpy as np

def discretize_action(continuous_action, num_bins=256, action_range=(-1.0, 1.0)):
    """Convert continuous action to discrete token."""
    low, high = action_range
    # Clip to valid range — this happens more than you'd think
    clipped = np.clip(continuous_action, low, high)

    # Linear mapping to bin indices
    normalized = (clipped - low) / (high - low)
    bin_idx = (normalized * (num_bins - 1)).astype(int)

    return bin_idx

def action_to_tokens(action_7dof, gripper_state, vocab_offset=32000):
    """
    action_7dof: shape (7,) continuous values in [-1, 1]
    gripper_state: scalar, 0 or 1
    vocab_offset: where action tokens start in the vocabulary
    """
    joint_bins = discretize_action(action_7dof)
    gripper_bin = int(gripper_state > 0.5) * 127  # binary gripper, mapped to bins

    all_bins = np.concatenate([joint_bins, [gripper_bin]])
    tokens = all_bins + vocab_offset

    return tokens.tolist()

# Example: arm moving slightly up and to the right
test_action = np.array([0.1, -0.05, 0.2, 0.0, 0.0, 0.15, -0.1])
test_gripper = 1.0  # closed

tokens = action_to_tokens(test_action, test_gripper)
print(f"Action tokens: {tokens}")
# Action tokens: [32140, 32121, 32153, 32127, 32127, 32146, 32114, 32127]

The vocab_offset matters because you’re sharing the tokenizer with the language model. RT-2 uses PaLI-X’s vocabulary (around 32k tokens), so action tokens start at 32000. OpenVLA uses Llama’s tokenizer with similar offsetting.

RT-2 vs OpenVLA: The Architecture Diff

RT-2 is built on PaLI-X, which means it inherits Google’s proprietary ViT-22B (yes, 22 billion parameters in the vision encoder alone). The language backbone is a 32B parameter model. Total: ~55B parameters. This thing runs on TPU pods.

OpenVLA went a different route. They use Prismatic-7B as the vision-language backbone — that’s a Llama-2-7B with a SigLIP vision encoder. Much more accessible:

The OpenVLA folks claim comparable performance on several benchmarks despite being 7x smaller. I’m not entirely sure I believe the full claim — the RT-2 paper shows much better generalization on novel objects — but for prototyping, OpenVLA is actually deployable.

Here’s the minimal inference code for OpenVLA (adapted from their repo, tested on transformers 4.36+):

import torch
from transformers import AutoModelForVision2Seq, AutoProcessor
from PIL import Image
import numpy as np

# Load model — this takes ~16GB VRAM on fp16
model_id = "openvla/openvla-7b"
processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForVision2Seq.from_pretrained(
    model_id,
    torch_dtype=torch.float16,
    device_map="auto",
    trust_remote_code=True
)

def predict_action(image: Image.Image, instruction: str) -> np.ndarray:
    """
    Returns 7-DoF action + gripper as continuous values.
    """
    # Processor handles image encoding and prompt formatting
    inputs = processor(
        images=image,
        text=f"In: What action should the robot take to {instruction}?\nOut:",
        return_tensors="pt"
    ).to(model.device)

    # Generate action tokens — greedy decoding for determinism
    with torch.no_grad():
        output_ids = model.generate(
            **inputs,
            max_new_tokens=8,  # 7 joints + 1 gripper
            do_sample=False,
            pad_token_id=processor.tokenizer.pad_token_id
        )

    # Decode tokens back to continuous actions
    # The model outputs in a specific format that the processor handles
    generated_ids = output_ids[0, inputs['input_ids'].shape[1]:]
    action = processor.decode_action(generated_ids)

    return action

# Example usage
img = Image.open("robot_workspace.png")
action = predict_action(img, "pick up the red block")
print(f"Predicted action: {action}")
# Predicted action: [ 0.12 -0.03  0.18  0.01 -0.05  0.22  0.95]
#                    ^ position delta            ^ gripper (open)

One gotcha: the trust_remote_code=True flag is required because OpenVLA uses custom model code. It’ll fail silently without it, returning garbage tokens.

The Math Behind Action Discretization

Why 256 bins? It’s a tradeoff between resolution and vocabulary size.

The discretization error for uniform binning is bounded by half the bin width. If we have $N$ bins over range $[a, b]$:

$$\epsilon_{max} = \frac{b – a}{2N}$$

For $N = 256$ and range $[-1, 1]$, that’s $\epsilon_{max} = \frac{2}{512} \approx 0.004$. In practice, robot joint velocities are normalized to this range, so 0.004 units of error is negligible — maybe 0.2 degrees of joint rotation.

But here’s where it gets interesting. The action space grows exponentially with dimensions. For 7-DoF with 256 bins:

$$|\mathcal{A}| = 256^7 \approx 7.2 \times 10^{16}$$

That’s way too large to enumerate. The autoregressive factorization saves us:

$$P(a_1, …, a_7) = \prod_{i=1}^{7} P(a_i | a_{<i}, I, L)$$

Each step is just a 256-way classification. The model learns correlations between joints implicitly through the transformer’s self-attention.

What Breaks in Practice

I ran OpenVLA on a simulated Franka Panda arm (using MuJoCo 3.1.1 and dm_control), and here’s what I didn’t expect: the model hallucinates actions when the scene is ambiguous.

Give it an instruction like “pick up the cup” when there are two cups, and it’ll confidently output actions that go… somewhere between both objects. Not towards the left cup, not towards the right cup, but a weird average.

The error message that finally clued me in wasn’t an error at all:

UserWarning: Action confidence below threshold (0.23 < 0.5) for tokens [32156, 32089, ...]

This warning is buried in the OpenVLA codebase and only fires if you enable verbose mode. My best guess is that the softmax probabilities spread across multiple plausible actions, and the greedy decode picks a weighted average that doesn’t correspond to any real intention.

The fix is to add explicit object references in the prompt: “pick up the red cup on the left” instead of “pick up the cup”. This isn’t documented anywhere — I found it by reading the RT-2 training data descriptions.

How to Actually Run This on a Jetson

OpenVLA’s 7.6B parameters don’t fit on a Jetson Orin Nano’s 8GB VRAM without quantization. I managed to get it running with 4-bit quantization using bitsandbytes:

import torch
from transformers import AutoModelForVision2Seq, AutoProcessor, BitsAndBytesConfig

quant_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4"
)

# This brings memory usage down to ~5GB
model = AutoModelForVision2Seq.from_pretrained(
    "openvla/openvla-7b",
    quantization_config=quant_config,
    device_map="auto",
    trust_remote_code=True
)

# Inference time: ~800ms per action on Orin Nano
# vs ~120ms on an A100

But here’s the kicker — 800ms latency per action is too slow for real-time control. Most robot control loops run at 10-100Hz. You’d need to run the VLA at a lower frequency (1-2Hz) and interpolate actions in between.

And if you’re debugging at 2am trying to figure out why your quantized model outputs nonsense, Dark Chocolate Espresso Beans are the real MVP for staying sharp.

The Action Space Isn’t What You Think

RT-2 and OpenVLA output delta actions — changes to apply to the current joint positions, not absolute targets. This is a subtle but critical detail.

The control equation per timestep:

$$q_{t+1} = q_t + \alpha \cdot \text{decode}(a_t)$$

where $q_t$ is the current joint configuration, $a_t$ is the model’s predicted action token sequence, and $\alpha$ is a scaling factor (typically 0.1-0.5). The scaling prevents the model from commanding huge jumps that would violate joint limits or cause jerky motion.

I’ve seen people try to use these models with absolute position targets and wonder why the robot flails around. The training data is all relative deltas.

Training Your Own VLA (Don’t)

Seriously, don’t — unless you have access to significant compute and a robot farm.

RT-2 was trained on 130k demonstration trajectories collected across ~20 robots over 17 months. OpenVLA used the Open X-Embodiment dataset, which is “only” 970k trajectories from 22 robot embodiments. The training loss is standard cross-entropy on action tokens:

$$\mathcal{L} = -\sum_{t=1}^{T} \sum_{d=1}^{D} \log P(a_t^d | I, L, a_{<t})$$

where $D$ is the action dimension (8 for 7-DoF + gripper) and $T$ is the trajectory length.

The compute requirements are brutal. OpenVLA reports 1,024 TPUv4 hours for training. At cloud rates, that’s roughly $30k-40k just for the final training run, not counting hyperparameter sweeps.

Fine-tuning is more reasonable. The OpenVLA team released LoRA adapters that can be trained on a single A100 for ~$200 worth of compute. If you have 100-500 demonstrations from your specific robot, this is the way.

Why This Matters for Edge Robotics

The vision-language-action framing means you can leverage all the work being done on efficient VLMs. Techniques like speculative decoding, KV-cache quantization, and model distillation transfer directly.

I’d estimate that within 18 months, we’ll have sub-2B parameter VLAs running at 10Hz on edge hardware. The bottleneck isn’t the architecture — it’s the training data. Most robotics labs don’t have the resources to collect enough demonstrations, and simulation-to-real transfer for manipulation is still unsolved.

ROS2 integration is another rabbit hole. The inference latency needs to fit within your control loop budget, and ROS2’s Nav2 already pushes Jetson hardware pretty hard with perception pipelines.

FAQ

Q: Can I run RT-2 locally?
No, RT-2 requires Google’s internal infrastructure with TPU pods. The model weights aren’t publicly released. OpenVLA is the open alternative — it runs on a single A100 in fp16 or an Orin with 4-bit quantization.

Q: What robots work with OpenVLA out of the box?
OpenVLA was trained primarily on Franka Panda and Google’s own manipulation platforms. It generalizes reasonably to other 7-DoF arms with parallel grippers. Mobile manipulators and humanoids need fine-tuning.

Q: How many demonstrations do I need to fine-tune for a new task?
The OpenVLA team reports that 100-500 demonstrations are enough for LoRA fine-tuning on a new task in the same embodiment. Cross-embodiment transfer (different robot) needs more — probably 1000+ trajectories.

What I’d Do Differently

If I were starting a VLA project today, I’d skip RT-2 entirely and start with OpenVLA + LoRA fine-tuning. The 7x smaller model means you can actually iterate without burning through cloud credits.

For inference on edge devices, I’d target the Orin NX (16GB) rather than the Nano. The extra VRAM headroom lets you run fp16 without quantization, which avoids the accuracy degradation I’ve seen with aggressive 4-bit compression.

Use OpenVLA for high-level action primitives (1-2Hz) and a classical controller for low-level tracking. Don’t try to run the VLA at control frequency — it’s not what these models were designed for.

And honestly? I haven’t figured out the ambiguous instruction problem cleanly. Adding object descriptors to prompts works, but it requires upstream perception to generate those descriptors. There’s probably a smarter solution involving attention visualization or confidence thresholding that I haven’t explored yet.