Create an action chunk

curl --request POST \
  --url https://api.abstract.inc/v1/chat/completions \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "model": "allenai/molmoact2:yam",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "put the red cup in the bin"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,<top JPEG>"
          }
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,<left JPEG>"
          }
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,<right JPEG>"
          }
        }
      ]
    }
  ],
  "robot_state": {
    "joint_position": [
      0,
      -0.4,
      0.6,
      0,
      0.2,
      0,
      0,
      -0.4,
      0.6,
      0,
      0.2,
      0
    ],
    "gripper_position": [
      1,
      0.5
    ]
  }
}
'

{
  "id": "gen-3f9a1c2b7d4e",
  "object": "chat.completion",
  "created": 1790000000,
  "model": "allenai/molmoact2:yam",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\"session_id\":\"anon-8c1d2e3f4a5b\",\"step\":0,\"dt\":0.03333333333333333,\"action_dim\":14,\"action_space\":\"joint_position\",\"actions\":[[0.01,-0.39,0.61,0.0,0.2,0.0,1.0,0.0,-0.4,0.6,0.0,0.2,0.0,0.5]],\"next_frames_per_camera\":1,\"timing_ms\":{\"decode\":4.1,\"queue\":0.0,\"prefill\":0.0,\"denoise\":212.5,\"encode\":0.3}}"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 650,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "completion_tokens": 30,
    "total_tokens": 680
  }
}

{
  "error": {
    "message": "A new session needs 3 image parts (one per camera); got 12.",
    "type": "invalid_request_error",
    "param": "messages",
    "code": "image_count_mismatch"
  }
}

{
  "error": {
    "message": "Invalid API key.",
    "type": "invalid_request_error",
    "code": "invalid_api_key"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

POST

chat

completions

curl --request POST \
  --url https://api.abstract.inc/v1/chat/completions \
  --header 'Authorization: Bearer <token>' \
  --header 'Content-Type: application/json' \
  --data '
{
  "model": "allenai/molmoact2:yam",
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "put the red cup in the bin"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,<top JPEG>"
          }
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,<left JPEG>"
          }
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/jpeg;base64,<right JPEG>"
          }
        }
      ]
    }
  ],
  "robot_state": {
    "joint_position": [
      0,
      -0.4,
      0.6,
      0,
      0.2,
      0,
      0,
      -0.4,
      0.6,
      0,
      0.2,
      0
    ],
    "gripper_position": [
      1,
      0.5
    ]
  }
}
'

{
  "id": "gen-3f9a1c2b7d4e",
  "object": "chat.completion",
  "created": 1790000000,
  "model": "allenai/molmoact2:yam",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\"session_id\":\"anon-8c1d2e3f4a5b\",\"step\":0,\"dt\":0.03333333333333333,\"action_dim\":14,\"action_space\":\"joint_position\",\"actions\":[[0.01,-0.39,0.61,0.0,0.2,0.0,1.0,0.0,-0.4,0.6,0.0,0.2,0.0,0.5]],\"next_frames_per_camera\":1,\"timing_ms\":{\"decode\":4.1,\"queue\":0.0,\"prefill\":0.0,\"denoise\":212.5,\"encode\":0.3}}"
      },
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 650,
    "prompt_tokens_details": {
      "cached_tokens": 0
    },
    "completion_tokens": 30,
    "total_tokens": 680
  }
}

{
  "error": {
    "message": "A new session needs 3 image parts (one per camera); got 12.",
    "type": "invalid_request_error",
    "param": "messages",
    "code": "image_count_mismatch"
  }
}

{
  "error": {
    "message": "Invalid API key.",
    "type": "invalid_request_error",
    "code": "invalid_api_key"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

{
  "error": {
    "message": "<string>",
    "type": "<string>",
    "code": "<string>",
    "param": "<string>"
  }
}

The last user message carries the task instruction as a text part and the camera frames as JPEG image parts, camera-major and oldest first within a camera. robot_state, session_id and the controls are top-level fields. The action chunk comes back as a JSON string in choices[0].message.content. With stream: true the answer is text/event-stream: : processing comments every 250 ms, one event with the whole chunk, one event with finish_reason and usage, then data: [DONE]. A failure after the stream starts arrives as an event with an error object and finish_reason: "error"; the HTTP status stays 200.

Authorizations

Authorization

string

header

required

Your Abstract API key. Get it with abstract login, pass it as $(abstract token).

Body

application/json

model

enum<string>

required

A robot model id.

Available options:

abstract/dreamzero-astra:yam,

stealth/picasso:yam,

allenai/molmoact2:yam,

microsoft/rho:yam,

perceptron/isaac-0.5:yam,

physical-intelligence/pi0.5:yam

messages

object[]

required

The server reads the last user message only. Its text parts are the task instruction; its image parts are the camera frames.

Minimum array length: 1

Show child attributes

robot_state

YAM joint state · object

The robot's state at the time of the latest frame. Required with image parts. The model picks the shape.

YAM joint state
Rho Cartesian state

Show child attributes

session_id

string

The episode id you choose. Required with image parts on abstract/dreamzero-astra:yam and stealth/picasso:yam; optional on the others. A new value starts a new episode.

Maximum string length: 128

observation_id

string

Logical observation identity. Reuse with an identical body to replay the original result without another execution or charge.

Pattern: ^[A-Za-z0-9._:-]{1,128}$

expected_step

integer

Expected next session step. A mismatch returns 409 without inference.

Required range: x >= 0

stream

boolean

default:false

true returns server-sent events.

action_horizon

integer

Action rows to return, up to the model's rows per chunk (24, 30 or 25). Default all rows. The model still computes and bills the whole chunk.

Required range: 1 <= x <= 30

num_inference_steps

integer

Accepted for compatibility. No served model reads it.

Required range: 1 <= x <= 4

handoff

boolean

default:false

true is refused with handoff_unavailable until an operator can take over. Send false or omit it.

Response

The action chunk, or a text description of the model for a request without images.

string

required

object

string

required

Allowed value: "chat.completion"

created

integer

required

Unix seconds.

model

string

required

choices

object[]

required

Show child attributes

usage

object

required

Show child attributes

Overview

Robot models

Robot runs

LLM models

Create an action chunk

Authorizations

Body

Response