​

AI Agent Capabilities

The AI agent is a full tool-calling system. It receives context about your current workspace (active images, channels, zoom, metadata) and dispatches actions both in the browser and on the server.

​

Available tools

​

Image analysis

​

Cell counting

Ask the agent to count cells and it will use Cellpose segmentation to detect and count cells in your image. Example prompts:

• “Count all cells in this image”
• “How many blue cells are there?”
• “Count cells and show me a chart”

Blue cell counting uses HSV color analysis to classify cells by color after Cellpose detection.

​

Cell segmentation

The agent can run SAM3 (Segment Anything Model 3) for interactive segmentation:

• Click points to segment specific objects
• Draw bounding boxes for region-based segmentation
• Use text prompts like “segment red neurons” or “find cell nuclei”

​

Image adjustments

Ask the agent to adjust images using natural language:

• “Make it darker” — adjusts brightness
• “Increase contrast” — adjusts contrast
• “Apply grayscale” — converts to grayscale
• “Remove the background” — background removal with threshold
• “Invert the colors” — color inversion
• “Add some blur” — applies blur filter

​

Object transforms

Manipulate objects on the canvas via natural language:

• “Rotate it 90 degrees”
• “Flip horizontally”
• “Scale it to 50%“

​

Image generation

Generate images from text descriptions using Google Imagen:

• “Generate a diagram of a neuron”
• “Create a schematic of a microscope setup”

​

Scientific figures

Generate publication-ready 3x3 figure grids:

• Original images in row 1
• Size distribution charts in row 2
• Segmentation results in row 3
• Includes scale bars, statistics overlay, and cell count labels

Example prompt: “Create a scientific figure from these three images”

​

Edge detection & background removal

• Request edge detection using Sobel or Canny methods
• Ask for background removal with configurable thresholds

​

Context awareness

The agent automatically receives context about your workspace on every request:

• Active pane state — file name, type, dimensions, channels, Z-slice, time point
• Scientific metadata — pixel scale, channel wavelengths, acquisition date, magnification, objective
• Selected region — if you’ve selected a region, the agent knows its bounds
• Available models — which models can process the current file type
• Current pipeline — if you have a draft pipeline open

This means the agent gives contextually relevant responses — it knows what you’re looking at.

​

Code execution

The AI agent can write and execute custom Python code in a sandboxed Docker environment.

​

How it works

1. The agent generates Python code based on your request
2. Code is validated against security rules (no network access, no system commands)
3. A Docker container runs the code with your input files mounted
4. Output files (images, CSV, numpy arrays) are saved and displayed

​

Available packages

The sandbox includes 21 pre-approved scientific Python packages:

​

Execution limits

​

Security

Code execution is fully sandboxed:

• No network access — outbound connections are disabled
• No system commands — subprocess, os.system, eval, exec are blocked
• Read-only code — the /code directory is read-only
• Non-root user — runs as uid=1000
• Restricted filesystem — only /input and /output directories are accessible
• Package allowlist — only pre-approved packages can be installed

​

File organization

The AI agent can intelligently organize your files into folder structures: Example prompt: “Organize my files by experiment” Available organization strategies:

• Auto — AI determines the best grouping
• By date — group files by creation date
• By type — group files by format/modality
• By experiment — detect experiment names from file naming patterns (BIDS, dated formats, mouse01_trial03 patterns)
• Flatten — flatten deeply nested folder structures

The agent proposes a folder structure and file moves (up to 5,000 files, max 15 folders). You preview the changes before applying them. Files are never renamed — only moved.

​

Pipeline generation & management

The agent has deep pipeline integration:

• Generate — describe a goal and the agent creates a multi-step pipeline
• Explain — ask the agent to explain why each step is included
• Preview — run the pipeline on your current image to preview results
• Batch process — run the pipeline across multiple images with progress tracking
• Cost estimate — get a cost breakdown before running (GPU steps vs. free client-side steps)
• Edit — ask the agent to add, remove, or reorder pipeline steps

Example prompts:

• “Create a pipeline to segment cells, count them, and export the results”
• “Explain this pipeline step by step”
• “Run this pipeline on all images in my folder”
• “How much will this pipeline cost to run?”

See Building Pipelines for more details.

​

Session limits