Batches

Perform batch inference with Vertex AI

With Portkey, you can perform batch inference operations with Vertex AI models. This is the most efficient way to:

• Process large volumes of data with Vertex AI models
• Test your data with different foundation models
• Perform A/B testing with different foundation models

Before You Start

1. Portkey API key and a Vertex AI provider configured in Model Catalog.
2. A GCS bucket in the same region as your model + aiplatform-service-agent permission on the file.
3. (Only for Portkey-native batching) A Portkey File (input_file_id).
4. Familiarity with the Create Batch OpenAPI spec.

Portkey supports two modes on Vertex:

• Provider Batch API (cheapest, completion_window: 24h, 48h, etc.)
• Portkey Batch API (fast, provider-agnostic, completion_window: immediate)

Using Vertex Batch API through Portkey

Upload a file for batch inference

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
    provider="@VERTEX_PROVIDER", 
    vertex_storage_bucket_name="your_bucket_name", # Specify the GCS bucket name
    provider_file_name="your_file_name.jsonl", # Specify the file name in GCS
    provider_model="gemini-1.5-flash-001" # Specify the model to use
)

# Upload a file for batch inference
file = portkey.files.create(
    file=open("dataset.jsonl", "rb"),
    purpose="batch"
)

print(file)

Create a batch job

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
    provider="@VERTEX_PROVIDER" 
)

# Create a batch inference job
batch_job = portkey.batches.create(
    input_file_id="<file_id>", # File ID from the upload step
    endpoint="/v1/chat/completions", # API endpoint to use
    completion_window="24h", # Time window for completion
    model="gemini-1.5-flash-001"
)

print(batch_job)

List batch jobs

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
    provider="@VERTEX_PROVIDER" 
)

# List all batch jobs
jobs = portkey.batches.list(
    limit=10  # Optional: Number of jobs to retrieve (default: 20)
)

print(jobs)

Get a batch job

Python

from portkey_ai import Portkey

# Initialize the Portkey client
portkey = Portkey(
    api_key="PORTKEY_API_KEY", # Replace with your Portkey API key
    provider="@VERTEX_PROVIDER" 
)

# Retrieve a specific batch job
job = portkey.batches.retrieve(
    "job_id"  # The ID of the batch job to retrieve
)

print(job)

Get batch job output

curl

curl -X GET --header 'x-portkey-api-key: <portkey_api_key>' \
--header 'x-portkey-provider: @your-vertex-provider' \
'https://api.portkey.ai/v1/batches/<job_id>/output'