Language Models

ComputeText

Compute text using a language model.

prompt
string

Input prompt.

image_uris
array[string]
Optional

Image prompts.

temperature
float[0..1]
Optional

Sampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.

Default: 0.4
max_tokens
integer
Optional

Maximum number of tokens to generate.

model
string
Optional

Selected model. Firellava13B is automatically selected when image_uris is provided.

Options: Mixtral8x7BInstructLlama3Instruct8BLlama3Instruct70BLlama3Instruct405BFirellava13Bgpt-4ogpt-4o-miniclaude-3-5-sonnet-20240620
Default: Llama3Instruct8B
Python
TypeScript

ComputeText(
prompt="Who is Don Quixote?",
temperature=0.4,
max_tokens=800,
)

Output

{
"text": "Don Quixote is a fictional character in the novel of the same name by Miguel de Cervantes."
}

MultiComputeText

Generate multiple text choices using a language model.

prompt
string

Input prompt.

num_choices
integer[1..8]

Number of choices to generate.

Default: 1
temperature
float[0..1]
Optional

Sampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.

Default: 0.4
max_tokens
integer
Optional

Maximum number of tokens to generate.

model
string
Optional

Selected model.

Options: Mixtral8x7BInstructLlama3Instruct8BLlama3Instruct70B
Default: Llama3Instruct8B
Python
TypeScript

MultiComputeText(
prompt="Who is Don Quixote?",
num_choices=2,
max_tokens=800,
)

Output

{
"choices": [
{
"text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..."
},
{
"text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..."
}
]
}

BatchComputeText

Compute text for multiple prompts in batch using a language model.

prompts
array[string]

Batch input prompts.

temperature
float[0..1]
Optional

Sampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.

Default: 0.4
max_tokens
integer
Optional

Maximum number of tokens to generate.

model
string
Optional

Selected model.

Default: Llama3Instruct8B
Python
TypeScript

BatchComputeText(
prompts=[
"Who is Don Quixote?",
"Who is Sancho Panza?",
],
max_tokens=800,
)

Output

{
"outputs": [
{
"text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..."
},
{
"text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..."
}
]
}

ComputeJSON

Compute JSON using a language model.

prompt
string

Input prompt.

json_schema
object

JSON schema to guide json_object response.

temperature
float[0..1]
Optional

Sampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.

Default: 0.4
max_tokens
integer
Optional

Maximum number of tokens to generate.

model
string
Optional

Selected model.

Options: Mixtral8x7BInstructLlama3Instruct8BLlama3Instruct70Bgpt-4o
Default: Llama3Instruct8B
Python
TypeScript

ComputeJSON(
prompt="Who wrote Don Quixote?",
json_schema={
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The name of the author.",
},
"bio": {
"type": "string",
"description": "Concise biography of the author.",
},
},
},
temperature=0.4,
max_tokens=800,
)

Output

{
"json_object": {}
}

MultiComputeJSON

Compute multiple JSON choices using a language model.

prompt
string

Input prompt.

json_schema
object

JSON schema to guide json_object response.

num_choices
integer[1..8]

Number of choices to generate.

Default: 2
temperature
float[0..1]
Optional

Sampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.

Default: 0.4
max_tokens
integer
Optional

Maximum number of tokens to generate.

model
string
Optional

Selected model.

Options: Mixtral8x7BInstructLlama3Instruct8B
Default: Llama3Instruct8B
Python
TypeScript

MultiComputeJSON(
prompt="Who wrote Don Quixote?",
json_schema={
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The name of the author.",
},
"bio": {
"type": "string",
"description": "Concise biography of the author.",
},
},
},
num_choices=2,
temperature=0.4,
max_tokens=800,
)

Output

{
"choices": [
{
"json_object": {}
},
{
"json_object": {}
}
]
}

BatchComputeJSON

Compute JSON for multiple prompts in batch using a language model.

prompts
array[string]

Batch input prompts.

json_schema
object

JSON schema to guide json_object response.

temperature
float[0..1]
Optional

Sampling temperature to use. Higher values make the output more random, lower values make the output more deterministic.

Default: 0.4
max_tokens
integer
Optional

Maximum number of tokens to generate.

model
string
Optional

Selected model.

Default: Llama3Instruct8B
Python
TypeScript

BatchComputeJSON(
prompts=[
"Who is Don Quixote?",
"Who is Sancho Panza?",
],
max_tokens=800,
json_schema={
"type": "object",
"properties": {
"name": {
"type": "string",
"description": "The name of the character.",
},
"bio": {
"type": "string",
"description": "Concise biography of the character.",
},
},
},
)

Output

{
"outputs": [
{
"json_object": {}
},
{
"json_object": {}
}
]
}

Mistral7BInstruct

Compute text using Mistral 7B Instruct.

prompt
string

Input prompt.

system_prompt
string
Optional

System prompt.

num_choices
integer[1..8]
Optional

Number of choices to generate.

Default: 1
json_schema
object
Optional

JSON schema to guide response.

temperature
float[0..1]
Optional

Higher values make the output more random, lower values make the output more deterministic.

frequency_penalty
float[-2..2]
Optional

Higher values decrease the likelihood of repeating previous tokens.

Default: 0
repetition_penalty
float[-2..2]
Optional

Higher values decrease the likelihood of repeated sequences.

Default: 1
presence_penalty
float[-2..2]
Optional

Higher values increase the likelihood of new topics appearing.

Default: 1.1
top_p
float[0..1]
Optional

Probability below which less likely tokens are filtered out.

Default: 0.95
max_tokens
integer
Optional

Maximum number of tokens to generate.

Python
TypeScript

Mistral7BInstruct(
prompt="Who is Don Quixote?",
num_choices=2,
temperature=0.4,
max_tokens=800,
)

Output

{
"choices": [
{
"text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..."
},
{
"text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..."
}
]
}

Mixtral8x7BInstruct

Compute text using instruct-tuned Mixtral 8x7B.

prompt
string

Input prompt.

system_prompt
string
Optional

System prompt.

num_choices
integer[1..8]
Optional

Number of choices to generate.

Default: 1
json_schema
object
Optional

JSON schema to guide response.

temperature
float[0..1]
Optional

Higher values make the output more random, lower values make the output more deterministic.

frequency_penalty
float[-2..2]
Optional

Higher values decrease the likelihood of repeating previous tokens.

Default: 0
repetition_penalty
float[-2..2]
Optional

Higher values decrease the likelihood of repeated sequences.

Default: 1
presence_penalty
float[-2..2]
Optional

Higher values increase the likelihood of new topics appearing.

Default: 1.1
top_p
float[0..1]
Optional

Probability below which less likely tokens are filtered out.

Default: 0.95
max_tokens
integer
Optional

Maximum number of tokens to generate.

Python
TypeScript

Mixtral8x7BInstruct(
prompt="Who is Don Quixote?",
num_choices=2,
temperature=0.4,
max_tokens=800,
)

Output

{
"choices": [
{
"text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..."
},
{
"text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..."
}
]
}

Llama3Instruct8B

Compute text using instruct-tuned Llama 3 8B.

prompt
string

Input prompt.

system_prompt
string
Optional

System prompt.

num_choices
integer[1..8]
Optional

Number of choices to generate.

Default: 1
temperature
float[0..1]
Optional

Higher values make the output more random, lower values make the output more deterministic.

frequency_penalty
float[-2..2]
Optional

Higher values decrease the likelihood of repeating previous tokens.

Default: 0
repetition_penalty
float[-2..2]
Optional

Higher values decrease the likelihood of repeated sequences.

Default: 1
presence_penalty
float[-2..2]
Optional

Higher values increase the likelihood of new topics appearing.

Default: 1.1
top_p
float[0..1]
Optional

Probability below which less likely tokens are filtered out.

Default: 0.95
max_tokens
integer
Optional

Maximum number of tokens to generate.

json_schema
object
Optional

JSON schema to guide response.

Python
TypeScript

Llama3Instruct8B(
prompt="Who is Don Quixote?",
num_choices=2,
temperature=0.4,
max_tokens=800,
)

Output

{
"choices": [
{
"text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..."
},
{
"text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..."
}
]
}

Llama3Instruct70B

Compute text using instruct-tuned Llama 3 70B.

prompt
string

Input prompt.

system_prompt
string
Optional

System prompt.

num_choices
integer[1..8]
Optional

Number of choices to generate.

Default: 1
temperature
float[0..1]
Optional

Higher values make the output more random, lower values make the output more deterministic.

frequency_penalty
float[-2..2]
Optional

Higher values decrease the likelihood of repeating previous tokens.

Default: 0
repetition_penalty
float[-2..2]
Optional

Higher values decrease the likelihood of repeated sequences.

Default: 1
presence_penalty
float[-2..2]
Optional

Higher values increase the likelihood of new topics appearing.

Default: 1.1
top_p
float[0..1]
Optional

Probability below which less likely tokens are filtered out.

Default: 0.95
max_tokens
integer
Optional

Maximum number of tokens to generate.

Python
TypeScript

Llama3Instruct70B(
prompt="Who is Don Quixote?",
num_choices=2,
temperature=0.4,
max_tokens=800,
)

Output

{
"choices": [
{
"text": "Don Quixote is a fictional character and the protagonist of the novel Don Quixote by Miguel..."
},
{
"text": "Don Quixote is a fictional character created by the Spanish author Miguel de Cervantes..."
}
]
}

Firellava13B

Compute text with image input using FireLLaVA 13B.

prompt
string

Text prompt.

image_uris
array[string]

Image prompts.

max_tokens
integer
Optional

Maximum number of tokens to generate.

Python
TypeScript

Firellava13B(
prompt="what are these paintings of and who made them?",
image_uris=[
"https://media.substrate.run/docs-fuji-red.jpg",
"https://media.substrate.run/docs-fuji-blue.jpg",
],
)

Output

{
"text": "The artist who created these paintings is Hokusai Katsushika, a renowned Japanese artist known for his woodblock prints and paintings."
}

Image Models

GenerateImage

Generate an image.

prompt
string

Text prompt.

store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

Python
TypeScript

GenerateImage(
prompt="hokusai futuristic supercell spiral cloud with glowing core over turbulent ocean",
store="hosted",
)

Output

{
"image_uri": "https://assets.substrate.run/84848484.jpg"
}

MultiGenerateImage

Generate multiple images.

prompt
string

Text prompt.

num_images
integer[1..8]

Number of images to generate.

Default: 2
store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

Python
TypeScript

MultiGenerateImage(
prompt="hokusai futuristic supercell spiral cloud with glowing core over turbulent ocean",
num_images=2,
store="hosted",
)

Output

{
"outputs": [
{
"image_uri": "https://assets.substrate.run/84848484.jpg"
},
{
"image_uri": "https://assets.substrate.run/48484848.jpg"
}
]
}

InpaintImage

Edit an image using image generation inside part of the image or the full image.

image_uri
string

Original image.

prompt
string

Text prompt.

mask_image_uri
string
Optional

Mask image that controls which pixels are inpainted. If unset, the entire image is edited (image-to-image).

store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

Python
TypeScript

InpaintImage(
image_uri="https://media.substrate.run/docs-klimt-park.jpg",
mask_image_uri="https://media.substrate.run/spiral-logo.jpeg",
prompt="large tropical colorful bright anime birds in a dark jungle full of vines, high resolution",
store="hosted",
)

Output

{
"image_uri": "https://assets.substrate.run/84848484.jpg"
}

MultiInpaintImage

Edit multiple images using image generation.

image_uri
string

Original image.

prompt
string

Text prompt.

mask_image_uri
string
Optional

Mask image that controls which pixels are edited (inpainting). If unset, the entire image is edited (image-to-image).

num_images
integer[1..8]

Number of images to generate.

Default: 2
store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

Python
TypeScript

MultiInpaintImage(
image_uri="https://media.substrate.run/docs-klimt-park.jpg",
mask_image_uri="https://media.substrate.run/spiral-logo.jpeg",
prompt="large tropical colorful bright anime birds in a dark jungle full of vines, high resolution",
num_images=2,
store="hosted",
)

Output

{
"outputs": [
{
"image_uri": "https://assets.substrate.run/84848484.jpg"
},
{
"image_uri": "https://assets.substrate.run/48484848.jpg"
}
]
}

UpscaleImage

Upscale an image using image generation.

prompt
string
Optional

Prompt to guide model on the content of image to upscale.

image_uri
string

Input image.

output_resolution
integer[512..2048]
Optional

Resolution of the output image, in pixels.

Default: 1024
store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

Python
TypeScript

UpscaleImage(
prompt="high resolution detailed spiral shell",
image_uri="https://media.substrate.run/docs-shell-emoji.jpg",
store="hosted",
)

Output

{
"image_uri": "https://assets.substrate.run/84848484.jpg"
}

EraseImage

Erase the masked part of an image, e.g. to remove an object by inpainting.

image_uri
string

Input image.

mask_image_uri
string

Mask image that controls which pixels are inpainted.

store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

Python
TypeScript

EraseImage(
image_uri="https://media.substrate.run/apple-forest.jpeg",
mask_image_uri="https://media.substrate.run/apple-forest-mask.jpeg",
store="hosted",
)

Output

{
"image_uri": "https://assets.substrate.run/84848484.jpg"
}

InterpolateFrames

Generates a interpolation frames between each adjacent frames.

frame_uris
array[string]

Frames.

store
string
Optional

Use "hosted" to return a video URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the video data will be returned as a base64-encoded string.

output_format
string
Optional

Output video format.

Options: gifwebpmp4frames
Default: gif
fps
integer[1..]
Optional

Frames per second of the generated video. Ignored if output format is frames.

Default: 7
num_steps
integer[1..]
Optional

Number of interpolation steps. Each step adds an interpolated frame between adjacent frames. For example, 2 steps over 2 frames produces 5 frames.

Default: 2
Python
TypeScript

InterpolateFrames(
frame_uris=[
"https://media.substrate.run/apple-forest2.jpeg",
"https://media.substrate.run/apple-forest3.jpeg",
],
store="hosted",
)

Output

{
"video_uri": "https://assets.substrate.run/84848484.mp4"
}

StableDiffusionXLLightning

Generate an image using Stable Diffusion XL Lightning.

prompt
string

Text prompt.

negative_prompt
string
Optional

Negative input prompt.

num_images
integer[1..8]
Optional

Number of images to generate.

Default: 1
store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

height
integer[256..1536]
Optional

Height of output image, in pixels.

Default: 1024
width
integer[256..1536]
Optional

Width of output image, in pixels.

Default: 1024
seeds
array[integer]
Optional

Seeds for deterministic generation. Default is a random seed.

Python
TypeScript

StableDiffusionXLLightning(
prompt="hokusai futuristic supercell spiral cloud with glowing core over turbulent ocean",
negative_prompt="night, moon",
num_images=2,
seeds=[
330699,
136464,
],
store="hosted",
)

Output

{
"outputs": [
{
"image_uri": "https://assets.substrate.run/84848484.jpg",
"seed": 330418
},
{
"image_uri": "https://assets.substrate.run/48484848.jpg",
"seed": 1364164
}
]
}

StableDiffusionXLInpaint

Edit an image using Stable Diffusion XL. Supports inpainting (edit part of the image with a mask) and image-to-image (edit the full image).

image_uri
string

Original image.

prompt
string

Text prompt.

mask_image_uri
string
Optional

Mask image that controls which pixels are edited (inpainting). If unset, the entire image is edited (image-to-image).

num_images
integer[1..8]

Number of images to generate.

Default: 1
output_resolution
integer[512..2048]
Optional

Resolution of the output image, in pixels.

Default: 1024
negative_prompt
string
Optional

Negative input prompt.

store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

strength
float[0..1]
Optional

Controls the strength of the generation process.

Default: 0.8
seeds
array[integer]
Optional

Random noise seeds. Default is random seeds for each generation.

Python
TypeScript

StableDiffusionXLInpaint(
image_uri="https://media.substrate.run/docs-klimt-park.jpg",
mask_image_uri="https://media.substrate.run/spiral-logo.jpeg",
prompt="large tropical colorful bright birds in a jungle, high resolution oil painting",
negative_prompt="dark, cartoon, anime",
strength=0.8,
num_images=2,
store="hosted",
seeds=[
1607280,
1720395,
],
)

Output

{
"outputs": [
{
"image_uri": "https://assets.substrate.run/84848484.jpg",
"seed": 1607326
},
{
"image_uri": "https://assets.substrate.run/48484848.jpg",
"seed": 1720398
}
]
}

StableDiffusionXLControlNet

Generate an image with generation structured by an input image, using Stable Diffusion XL with ControlNet.

image_uri
string

Input image.

control_method
string

Strategy to control generation using the input image.

Options: edgedepthillusiontile
prompt
string

Text prompt.

num_images
integer[1..8]

Number of images to generate.

Default: 1
output_resolution
integer[512..2048]
Optional

Resolution of the output image, in pixels.

Default: 1024
negative_prompt
string
Optional

Negative input prompt.

store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

conditioning_scale
float[0..1]
Optional

Controls the influence of the input image on the generated output.

Default: 0.5
strength
float[0..1]
Optional

Controls how much to transform the input image.

Default: 0.5
seeds
array[integer]
Optional

Random noise seeds. Default is random seeds for each generation.

Python
TypeScript

StableDiffusionXLControlNet(
image_uri="https://media.substrate.run/spiral-logo.jpeg",
prompt="the futuristic solarpunk city of atlantis at sunset, cinematic bokeh HD",
control_method="illusion",
conditioning_scale=1.0,
strength=1.0,
store="hosted",
num_images=2,
seeds=[
1607226,
1720395,
],
)

Output

{
"outputs": [
{
"image_uri": "https://assets.substrate.run/84848484.jpg",
"seed": 1607266
},
{
"image_uri": "https://assets.substrate.run/48484848.jpg",
"seed": 1720398
}
]
}

StableVideoDiffusion

Generates a video using a still image as conditioning frame.

image_uri
string

Original image.

store
string
Optional

Use "hosted" to return a video URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the video data will be returned as a base64-encoded string.

output_format
string
Optional

Output video format.

Options: gifwebpmp4frames
Default: gif
seed
integer
Optional

Seed for deterministic generation. Default is a random seed.

fps
integer[1..]
Optional

Frames per second of the generated video. Ignored if output format is frames.

Default: 7
motion_bucket_id
integer
Optional

The motion bucket id to use for the generated video. This can be used to control the motion of the generated video. Increasing the motion bucket id increases the motion of the generated video.

Default: 180
noise
float
Optional

The amount of noise added to the conditioning image. The higher the values the less the video resembles the conditioning image. Increasing this value also increases the motion of the generated video.

Default: 0.1
Python
TypeScript

StableVideoDiffusion(
image_uri="https://media.substrate.run/apple-forest.jpeg",
store="hosted",
)

Output

{
"video_uri": "https://assets.substrate.run/84848484.mp4"
}

Segmentation Models

RemoveBackground

Remove the background from an image and return the foreground segment as a cut-out or a mask.

image_uri
string

Input image.

return_mask
boolean
Optional

Return a mask image instead of the original content.

Default: false
invert_mask
boolean
Optional

Invert the mask image. Only takes effect if return_mask is true.

Default: false
background_color
string
Optional

Hex value background color. Transparent if unset.

store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

Python
TypeScript

RemoveBackground(
image_uri="https://media.substrate.run/apple-forest.jpeg",
store="hosted",
)

Output

{
"image_uri": "https://assets.substrate.run/84848484.jpg"
}

SegmentUnderPoint

Segment an image under a point and return the segment.

image_uri
string

Input image.

point
Point

Point prompt.

x
integer

X position.

y
integer

Y position.

store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

Python
TypeScript

SegmentUnderPoint(
image_uri="https://media.substrate.run/docs-vg-bedroom.jpg",
point={
"x": 189,
"y": 537,
},
store="hosted",
)

Output

{
"mask_image_uri": "https://assets.substrate.run/84848484.jpg"
}

SegmentAnything

Segment an image using SegmentAnything.

image_uri
string

Input image.

point_prompts
array[Point]
Optional

Point prompts, to detect a segment under the point. One of point_prompts or box_prompts must be set.

x
integer

X position.

y
integer

Y position.

box_prompts
array[BoundingBox]
Optional

Box prompts, to detect a segment within the bounding box. One of point_prompts or box_prompts must be set.

x1
float

Top left corner x.

y1
float

Top left corner y.

x2
float

Bottom right corner x.

y2
float

Bottom right corner y.

store
string
Optional

Use "hosted" to return an image URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the image data will be returned as a base64-encoded string.

Python
TypeScript

SegmentAnything(
image_uri="https://media.substrate.run/docs-vg-bedroom.jpg",
point_prompts=[
{
"x": 189,
"y": 537,
},
],
store="hosted",
)

Output

{
"mask_image_uri": "https://assets.substrate.run/84848484.jpg"
}

Vector Embeddings

SplitDocument

Split document into text segments.

uri
string

URI of the document.

doc_id
string
Optional

Document ID.

metadata
object
Optional

Document metadata.

chunk_size
integer[1..]
Optional

Maximum number of units per chunk. Defaults to 1024 tokens for text or 40 lines for code.

chunk_overlap
integer
Optional

Number of units to overlap between chunks. Defaults to 200 tokens for text or 15 lines for code.

Python
TypeScript

SplitDocument(
doc_id="example_pdf",
uri="https://arxiv.org/pdf/2405.07945",
metadata={
"title": "GRASS II: Simulations of Potential Granulation Noise Mitigation Methods",
},
)

Output

{
"items": [
{
"text": "This is the first chunk of the pdf",
"metadata": {
"title": "GRASS II: Simulations of Potential Granulation Noise Mitigation Methods",
"chunk_id": "chk_asd897asdhnad0j8qd8qnd98"
},
"doc_id": "example_pdf"
},
{
"text": "This is the second chunk of the pdf",
"metadata": {
"title": "GRASS II: Simulations of Potential Granulation Noise Mitigation Methods",
"chunk_id": "chk_nvsiusd89adsy89dahd9abs8"
},
"doc_id": "example_pdf"
}
]
}

EmbedText

Generate embedding for a text document.

text
string

Text to embed.

collection_name
string
Optional

Vector store name.

metadata
object
Optional

Metadata that can be used to query the vector store. Ignored if collection_name is unset.

embedded_metadata_keys
array[string]
Optional

Choose keys from metadata to embed with text.

doc_id
string
Optional

Vector store document ID. Ignored if store is unset.

model
string
Optional

Selected embedding model.

Options: jina-v2clip
Default: jina-v2
Python
TypeScript

EmbedText(
text="Argon is the third most abundant gas in Earth's atmosphere, at 0.934% (9340 ppmv). It is more than twice as abundant as water vapor.",
model="jina-v2",
collection_name="smoke_tests",
metadata={
"group": "18",
},
embedded_metadata_keys=[
"group",
],
)

Output

{
"embedding": {
"vector": [
-0.035030052065849304,
-0.04128379374742508,
0.05782046541571617
],
"doc_id": "c9de81fb98804ce0afb2b8ac17c0799b",
"metadata": {
"group": "18",
"doc_id": "c9de81fb98804ce0afb2b8ac17c0799b",
"doc": "group: 18\n\nArgon is the third most abundant gas in Earth's atmosphere, at 0.934% (9340 ppmv). It is more than twice as abundant as water vapor."
}
}
}

MultiEmbedText

Generate embeddings for multiple text documents.

items
array[EmbedTextItem]

Items to embed.

text
string

Text to embed.

metadata
object
Optional

Metadata that can be used to query the vector store. Ignored if collection_name is unset.

doc_id
string
Optional

Vector store document ID. Ignored if collection_name is unset.

collection_name
string
Optional

Vector store name.

embedded_metadata_keys
array[string]
Optional

Choose keys from metadata to embed with text.

model
string
Optional

Selected embedding model.

Options: jina-v2clip
Default: jina-v2
Python
TypeScript

MultiEmbedText(
model="jina-v2",
items=[
{
"text": "Osmium is the densest naturally occurring element. When experimentally measured using X-ray crystallography, it has a density of 22.59 g/cm3. Manufacturers use its alloys with platinum, iridium, and other platinum-group metals to make fountain pen nib tipping, electrical contacts, and in other applications that require extreme durability and hardness.",
"metadata": {
"group": "8",
},
},
{
"text": "Despite its abundant presence in the universe and Solar System—ranking fifth in cosmic abundance following hydrogen, helium, oxygen, and carbon—neon is comparatively scarce on Earth.",
"metadata": {
"group": "18",
},
},
],
collection_name="smoke_tests",
embedded_metadata_keys=[
"group",
],
)

Output

{
"embeddings": [
{
"vector": [
-0.035030052065849304,
-0.04128379374742508,
0.05782046541571617
],
"doc_id": "c9de81fb98804ce0afb2b8ac17c0799b",
"metadata": {
"group": "8",
"doc_id": "c9de81fb98804ce0afb2b8ac17c0799b",
"doc": "group: 8\n\nOsmium is the densest naturally occurring element. When experimentally measured using X-ray crystallography, it has a density of 22.59 g/cm3. Manufacturers use its alloys with platinum, iridium, and other platinum-group metals to make fountain pen nib tipping, electrical contacts, and in other applications that require extreme durability and hardness."
}
},
{
"vector": [
0.0003024724137503654,
-0.025219274684786797,
-0.009984994307160378
],
"doc_id": "c4464f69c93946a896925589681d38b4",
"metadata": {
"group": "18",
"doc_id": "c4464f69c93946a896925589681d38b4",
"doc": "group: 18\n\nDespite its abundant presence in the universe and Solar System\u2014ranking fifth in cosmic abundance following hydrogen, helium, oxygen, and carbon\u2014neon is comparatively scarce on Earth."
}
}
]
}

EmbedImage

Generate embedding for an image.

image_uri
string

Image to embed.

collection_name
string
Optional

Vector store name.

doc_id
string
Optional

Vector store document ID. Ignored if collection_name is unset.

model
string
Optional

Selected embedding model.

Default: clip
Python
TypeScript

EmbedImage(
image_uri="https://media.substrate.run/docs-fuji-red.jpg",
collection_name="smoke_tests",
)

Output

{
"embedding": {
"vector": [
0.0003024724137503654,
-0.025219274684786797,
-0.009984994307160378
],
"doc_id": "c4464f69c93946a896925589681d38b4"
}
}

MultiEmbedImage

Generate embeddings for multiple images.

items
array[EmbedImageItem]

Items to embed.

image_uri
string

Image to embed.

doc_id
string
Optional

Vector store document ID. Ignored if collection_name is unset.

collection_name
string
Optional

Vector store name.

model
string
Optional

Selected embedding model.

Default: clip
Python
TypeScript

MultiEmbedImage(
items=[
{
"image_uri": "https://media.substrate.run/docs-fuji-red.jpg",
},
{
"image_uri": "https://media.substrate.run/docs-fuji-blue.jpg",
},
],
collection_name="smoke_tests",
)

Output

{
"embeddings": [
{
"vector": [
-0.035030052065849304,
-0.04128379374742508,
0.05782046541571617
],
"doc_id": "c9de81fb98804ce0afb2b8ac17c0799b"
},
{
"vector": [
0.0003024724137503654,
-0.025219274684786797,
-0.009984994307160378
],
"doc_id": "c4464f69c93946a896925589681d38b4"
}
]
}

JinaV2

Generate embeddings for multiple text documents using Jina Embeddings 2.

items
array[EmbedTextItem]

Items to embed.

text
string

Text to embed.

metadata
object
Optional

Metadata that can be used to query the vector store. Ignored if collection_name is unset.

doc_id
string
Optional

Vector store document ID. Ignored if collection_name is unset.

collection_name
string
Optional

Vector store name.

embedded_metadata_keys
array[string]
Optional

Choose keys from metadata to embed with text.

Python
TypeScript

JinaV2(
items=[
{
"text": "Hassium is a superheavy element; it has been produced in a laboratory only in very small quantities by fusing heavy nuclei with lighter ones. Natural occurrences of the element have been hypothesised but never found.",
"metadata": {
"group": "8",
},
},
{
"text": "Xenon is also used to search for hypothetical weakly interacting massive particles and as a propellant for ion thrusters in spacecraft.",
"metadata": {
"group": "18",
},
},
],
collection_name="smoke_tests",
embedded_metadata_keys=[
"group",
],
)

Output

{
"embeddings": [
{
"vector": [
-0.035030052065849304,
-0.04128379374742508,
0.05782046541571617
],
"doc_id": "c9de81fb98804ce0afb2b8ac17c0799b",
"metadata": {
"group": "8",
"doc_id": "c9de81fb98804ce0afb2b8ac17c0799b",
"doc": "group: 8\n\nHassium is a superheavy element; it has been produced in a laboratory only in very small quantities by fusing heavy nuclei with lighter ones. Natural occurrences of the element have been hypothesised but never found."
}
},
{
"vector": [
0.0003024724137503654,
-0.025219274684786797,
-0.009984994307160378
],
"doc_id": "c4464f69c93946a896925589681d38b4",
"metadata": {
"group": "18",
"doc_id": "c4464f69c93946a896925589681d38b4",
"doc": "group: 18\n\nXenon is also used to search for hypothetical weakly interacting massive particles and as a propellant for ion thrusters in spacecraft."
}
}
]
}

CLIP

Generate embeddings for text or images using CLIP.

items
array[EmbedTextOrImageItem]

Items to embed.

image_uri
string
Optional

Image to embed.

text
string
Optional

Text to embed.

metadata
object
Optional

Metadata that can be used to query the vector store. Ignored if collection_name is unset.

doc_id
string
Optional

Vector store document ID. Ignored if collection_name is unset.

collection_name
string
Optional

Vector store name.

embedded_metadata_keys
array[string]
Optional

Choose keys from metadata to embed with text. Only applies to text items.

Python
TypeScript

CLIP(
items=[
{
"image_uri": "https://media.substrate.run/docs-fuji-red.jpg",
},
{
"image_uri": "https://media.substrate.run/docs-fuji-blue.jpg",
},
],
collection_name="smoke_tests",
)

Output

{
"embeddings": [
{
"vector": [
-0.035030052065849304,
-0.04128379374742508,
0.05782046541571617
],
"doc_id": "c9de81fb98804ce0afb2b8ac17c0799b"
},
{
"vector": [
0.0003024724137503654,
-0.025219274684786797,
-0.009984994307160378
],
"doc_id": "c4464f69c93946a896925589681d38b4"
}
]
}

Vector Stores

FindOrCreateVectorStore

Find a vector store matching the given collection name, or create a new vector store.

collection_name
string

Vector store name.

model
string

Selected embedding model.

Options: jina-v2clip
Python
TypeScript

FindOrCreateVectorStore(
collection_name="smoke_tests",
model="jina-v2",
)

Output

{
"collection_name": "smoke_tests",
"model": "jina-v2"
}

ListVectorStores

List all vector stores.

Python
TypeScript

ListVectorStores()

Output

{
"items": [
{
"collection_name": "comments",
"model": "jina-v2"
},
{
"collection_name": "images",
"model": "jina-v2"
}
]
}

DeleteVectorStore

Delete a vector store.

collection_name
string

Vector store name.

model
string

Selected embedding model.

Options: jina-v2clip
Python
TypeScript

DeleteVectorStore(
collection_name="fake_store",
model="jina-v2",
)

Output

{
"collection_name": "comments",
"model": "jina-v2"
}

QueryVectorStore

Query a vector store for similar vectors.

collection_name
string

Vector store to query against.

model
string

Selected embedding model.

Options: jina-v2clip
query_strings
array[string]
Optional

Texts to embed and use for the query.

query_image_uris
array[string]
Optional

Image URIs to embed and use for the query.

query_vectors
array[array]
Optional

Vectors to use for the query.

query_ids
array[string]
Optional

Document IDs to use for the query.

top_k
integer[1..1000]
Optional

Number of results to return.

Default: 10
ef_search
integer[1..1000]
Optional

The size of the dynamic candidate list for searching the index graph.

Default: 40
num_leaves_to_search
integer[1..1000]
Optional

The number of leaves in the index tree to search.

Default: 40
include_values
boolean
Optional

Include the values of the vectors in the response.

Default: false
include_metadata
boolean
Optional

Include the metadata of the vectors in the response.

Default: false
filters
object
Optional

Filter metadata by key-value pairs.

Python
TypeScript

QueryVectorStore(
collection_name="smoke_tests",
model="jina-v2",
query_strings=[
"gas",
"metal",
],
top_k=1,
include_metadata=True,
)

Output

{
"results": [
[
{
"id": "483e75021c9d4ad69c3d78ace76da2ea",
"distance": -0.78324556350708,
"metadata": {
"doc": "group: 18\n\nArgon is the third most abundant gas in Earth's atmosphere, at 0.934% (9340 ppmv). It is more than twice as abundant as water vapor.",
"group": "18",
"doc_id": "483e75021c9d4ad69c3d78ace76da2ea"
}
}
],
[
{
"id": "dd8f3774e05d42caa53cfbaa7389c08f",
"distance": -0.74278724193573,
"metadata": {
"doc": "group: 8\n\nOsmium is the densest naturally occurring element. When experimentally measured using X-ray crystallography, it has a density of 22.59 g/cm3. Manufacturers use its alloys with platinum, iridium, and other platinum-group metals to make fountain pen nib tipping, electrical contacts, and in other applications that require extreme durability and hardness.",
"group": "8",
"doc_id": "dd8f3774e05d42caa53cfbaa7389c08f"
}
}
]
],
"collection_name": "comments",
"model": "jina-v2"
}

FetchVectors

Fetch vectors from a vector store.

collection_name
string

Vector store name.

model
string

Selected embedding model.

Options: jina-v2clip
ids
array[string]

Document IDs to retrieve.

Python
TypeScript

FetchVectors(
collection_name="smoke_tests",
model="jina-v2",
ids=[
"dd8f3774e05d42caa53cfbaa7389c08f",
],
)

Output

{
"vectors": [
{
"id": "dd8f3774e05d42caa53cfbaa7389c08f",
"vector": [
0.036658343,
-0.0066040196,
0.028221145
],
"metadata": {
"doc": "group: 8\n\nOsmium is the densest naturally occurring element. When experimentally measured using X-ray crystallography, it has a density of 22.59 g/cm3. Manufacturers use its alloys with platinum, iridium, and other platinum-group metals to make fountain pen nib tipping, electrical contacts, and in other applications that require extreme durability and hardness.",
"group": "8",
"doc_id": "dd8f3774e05d42caa53cfbaa7389c08f"
}
}
]
}

UpdateVectors

Update vectors in a vector store.

collection_name
string

Vector store name.

model
string

Selected embedding model.

Options: jina-v2clip
vectors
array[UpdateVectorParams]

Vectors to upsert.

id
string

Document ID.

vector
array[number]
Optional

Embedding vector.

metadata
object
Optional

Document metadata.

Python
TypeScript

UpdateVectors(
collection_name="smoke_tests",
model="jina-v2",
vectors=[
{
"id": "dd8f3774e05d42caa53cfbaa7389c08f",
"metadata": {
"appearance": "silvery, blue cast",
},
},
],
)

Output

{
"count": 1
}

DeleteVectors

Delete vectors in a vector store.

collection_name
string

Vector store name.

model
string

Selected embedding model.

Options: jina-v2clip
ids
array[string]

Document IDs to delete.

Python
TypeScript

DeleteVectors(
collection_name="smoke_tests",
model="jina-v2",
ids=[
"ac32b9a133dd4e3689004f6e8f0fd6cd",
"629df177c7644062a68bceeff223cefa",
],
)

Output

{
"count": 2
}

Audio Models

TranscribeSpeech

Transcribe speech in an audio or video file.

audio_uri
string

Input audio.

prompt
string
Optional

Prompt to guide model on the content and context of input audio.

language
string
Optional

Language of input audio in ISO-639-1 format. Use auto to automatically detect the language.

Default: en
segment
boolean
Optional

(Deprecated) Segment the text into sentences with approximate timestamps.

Default: false
align
boolean
Optional

Align transcription to produce more accurate sentence-level timestamps and word-level timestamps. An array of word segments will be included in each sentence segment.

Default: false
diarize
boolean
Optional

Identify speakers for each segment. Speaker IDs will be included in each segment.

Default: false
suggest_chapters
boolean
Optional

Suggest automatic chapter markers.

Default: false
Python
TypeScript

TranscribeSpeech(
audio_uri="https://media.substrate.run/dfw-clip.m4a",
prompt="David Foster Wallace interviewed about US culture, and Infinite Jest",
segment=True,
align=True,
diarize=True,
suggest_chapters=True,
)

Output

{
"text": "language like that, the wounded inner child, the inner pain, is part of a kind of pop psychological movement in the United States that is a sort of popular Freudianism that ...",
"segments": [
{
"start": 0.874,
"end": 15.353,
"speaker": "SPEAKER_00",
"text": "language like that, the wounded inner child, the inner pain, is part of a kind of pop psychological movement in the United States that is a sort of popular Freudianism that",
"words": [
{
"word": "language",
"start": 0.874,
"end": 1.275,
"speaker": "SPEAKER_00"
},
{
"word": "like",
"start": 1.295,
"end": 1.455,
"speaker": "SPEAKER_00"
}
]
}
],
"chapters": [
{
"title": "Introduction to the Wounded Inner Child and Popular Psychology in US",
"start": 0.794
},
{
"title": "The Paradox of Popular Psychology and Anger in America",
"start": 16.186
}
]
}

GenerateSpeech

Generate speech from text.

text
string

Input text.

store
string
Optional

Use "hosted" to return an audio URL hosted on Substrate. You can also provide a URL to a registered file store. If unset, the audio data will be returned as a base64-encoded string.

Python
TypeScript

GenerateSpeech(
text="Substrate: an underlying substance or layer.",
store="hosted",
)

Output

{
"audio_uri": "https://assets.substrate.run/84848484.wav"
}

Utility Nodes

Box

Combine multiple values into a single output.

value
Any

Values to box.

Python
TypeScript

Box(
value={
"a": "b",
"c": {
"d": [
1,
2,
3,
],
},
},
)

Output

{
"value": {
"a": "b",
"c": {
"d": [
1,
2,
3
]
}
}
}

If

Return one of two options based on a condition.

condition
boolean

Condition.

value_if_true
Any

Result when condition is true.

value_if_false
Any
Optional

Result when condition is false.

Python
TypeScript

If(
condition=True,
value_if_true="yes",
value_if_false="no",
)

Output

{
"result": "yes"
}