> ## Documentation Index
> Fetch the complete documentation index at: https://docs.biom.science/llms.txt
> Use this file to discover all available pages before exploring further.

# AI Agent Capabilities

> Everything the AI agent can do — from data analysis to code execution.

# AI Agent Capabilities

The AI agent is a full tool-calling system. It receives context about your current workspace (active images, channels, zoom, metadata) and dispatches actions both in the browser and on the server.

## Available tools

| Tool                      | Description                                                                                                |
| ------------------------- | ---------------------------------------------------------------------------------------------------------- |
| **analyze\_image**        | Run Cellpose segmentation to count and analyze cells                                                       |
| **run\_model**            | Launch any scientific model (SAM3, DeepLabCut, Suite2p, etc.)                                              |
| **adjust\_image**         | Precise adjustments: brightness, contrast, gamma, saturation, hue, blur, color balance, background removal |
| **adjust\_image\_nl**     | Natural language adjustments (*"make it brighter"*, *"increase contrast"*)                                 |
| **transform\_object**     | Rotate, scale, flip, skew, reposition objects                                                              |
| **generate\_image**       | Generate images from text prompts via Google Imagen                                                        |
| **generate\_pipeline**    | AI-generate a multi-step processing pipeline                                                               |
| **pipeline\_manage**      | Create, edit, reorder, and save pipeline steps                                                             |
| **pipeline\_explain**     | Explain each pipeline step's scientific rationale                                                          |
| **pipeline\_preview**     | Run a draft pipeline on the current image                                                                  |
| **pipeline\_batch**       | Batch-process multiple images through a pipeline                                                           |
| **model\_estimate\_cost** | Get cost breakdown for a pipeline (GPU vs. free steps)                                                     |
| **search\_web**           | Search the web and return a synthesized answer                                                             |
| **get\_job\_status**      | Poll a running model job's status and results                                                              |
| **ask\_followup**         | Present structured follow-up questions as interactive chips                                                |

## Image analysis

### Cell counting

Ask the agent to count cells and it will use Cellpose segmentation to detect and count cells in your image.

**Example prompts:**

* *"Count all cells in this image"*
* *"How many blue cells are there?"*
* *"Count cells and show me a chart"*

Blue cell counting uses HSV color analysis to classify cells by color after Cellpose detection.

### Cell segmentation

The agent can run SAM3 (Segment Anything Model 3) for interactive segmentation:

* Click points to segment specific objects
* Draw bounding boxes for region-based segmentation
* Use text prompts like *"segment red neurons"* or *"find cell nuclei"*

### Image adjustments

Ask the agent to adjust images using natural language:

* *"Make it darker"* — adjusts brightness
* *"Increase contrast"* — adjusts contrast
* *"Apply grayscale"* — converts to grayscale
* *"Remove the background"* — background removal with threshold
* *"Invert the colors"* — color inversion
* *"Add some blur"* — applies blur filter

### Object transforms

Manipulate objects on the canvas via natural language:

* *"Rotate it 90 degrees"*
* *"Flip horizontally"*
* *"Scale it to 50%"*

### Image generation

Generate images from text descriptions using Google Imagen:

* *"Generate a diagram of a neuron"*
* *"Create a schematic of a microscope setup"*

### Scientific figures

Generate publication-ready 3x3 figure grids:

* Original images in row 1
* Size distribution charts in row 2
* Segmentation results in row 3
* Includes scale bars, statistics overlay, and cell count labels

**Example prompt:** *"Create a scientific figure from these three images"*

### Edge detection & background removal

* Request edge detection using Sobel or Canny methods
* Ask for background removal with configurable thresholds

## Context awareness

The agent automatically receives context about your workspace on every request:

* **Active pane state** — file name, type, dimensions, channels, Z-slice, time point
* **Scientific metadata** — pixel scale, channel wavelengths, acquisition date, magnification, objective
* **Selected region** — if you've selected a region, the agent knows its bounds
* **Available models** — which models can process the current file type
* **Current pipeline** — if you have a draft pipeline open

This means the agent gives contextually relevant responses — it knows what you're looking at.

## Code execution

The AI agent can write and execute custom Python code in a sandboxed Docker environment.

### How it works

1. The agent generates Python code based on your request
2. Code is validated against security rules (no network access, no system commands)
3. A Docker container runs the code with your input files mounted
4. Output files (images, CSV, numpy arrays) are saved and displayed

### Available packages

The sandbox includes 21 pre-approved scientific Python packages:

| Category          | Packages                               |
| ----------------- | -------------------------------------- |
| **Core data**     | NumPy, Pandas, SciPy, xarray           |
| **Imaging**       | scikit-image, OpenCV, Pillow, tifffile |
| **ML**            | PyTorch, scikit-learn                  |
| **Neuroscience**  | PyNWB, Neo, Elephant                   |
| **Visualization** | Matplotlib, Seaborn                    |
| **I/O**           | h5py, zarr                             |

### Execution limits

| Limit            | Value    |
| ---------------- | -------- |
| Max code size    | 1 MB     |
| Max output size  | 5 GB     |
| Max runtime      | 1 hour   |
| Max memory       | 32 GB    |
| Max output files | 100      |
| Network access   | Disabled |

### Security

Code execution is fully sandboxed:

* **No network access** — outbound connections are disabled
* **No system commands** — subprocess, os.system, eval, exec are blocked
* **Read-only code** — the `/code` directory is read-only
* **Non-root user** — runs as `uid=1000`
* **Restricted filesystem** — only `/input` and `/output` directories are accessible
* **Package allowlist** — only pre-approved packages can be installed

## File organization

The AI agent can intelligently organize your files into folder structures:

**Example prompt:** *"Organize my files by experiment"*

Available organization strategies:

* **Auto** — AI determines the best grouping
* **By date** — group files by creation date
* **By type** — group files by format/modality
* **By experiment** — detect experiment names from file naming patterns (BIDS, dated formats, mouse01\_trial03 patterns)
* **Flatten** — flatten deeply nested folder structures

The agent proposes a folder structure and file moves (up to 5,000 files, max 15 folders). You preview the changes before applying them. Files are never renamed — only moved.

## Pipeline generation & management

The agent has deep pipeline integration:

* **Generate** — describe a goal and the agent creates a multi-step pipeline
* **Explain** — ask the agent to explain why each step is included
* **Preview** — run the pipeline on your current image to preview results
* **Batch process** — run the pipeline across multiple images with progress tracking
* **Cost estimate** — get a cost breakdown before running (GPU steps vs. free client-side steps)
* **Edit** — ask the agent to add, remove, or reorder pipeline steps

**Example prompts:**

* *"Create a pipeline to segment cells, count them, and export the results"*
* *"Explain this pipeline step by step"*
* *"Run this pipeline on all images in my folder"*
* *"How much will this pipeline cost to run?"*

See [Building Pipelines](/pipelines/building-pipelines) for more details.

## Session limits

| Limit                | Value            |
| -------------------- | ---------------- |
| Requests per session | 50               |
| Max input length     | 1,200 characters |