AI Agent Capabilities
The AI agent is a full tool-calling system. It receives context about your current workspace (active images, channels, zoom, metadata) and dispatches actions both in the browser and on the server.Available tools
| Tool | Description |
|---|---|
| analyze_image | Run Cellpose segmentation to count and analyze cells |
| run_model | Launch any scientific model (SAM3, DeepLabCut, Suite2p, etc.) |
| adjust_image | Precise adjustments: brightness, contrast, gamma, saturation, hue, blur, color balance, background removal |
| adjust_image_nl | Natural language adjustments (“make it brighter”, “increase contrast”) |
| transform_object | Rotate, scale, flip, skew, reposition objects |
| generate_image | Generate images from text prompts via Google Imagen |
| generate_pipeline | AI-generate a multi-step processing pipeline |
| pipeline_manage | Create, edit, reorder, and save pipeline steps |
| pipeline_explain | Explain each pipeline step’s scientific rationale |
| pipeline_preview | Run a draft pipeline on the current image |
| pipeline_batch | Batch-process multiple images through a pipeline |
| model_estimate_cost | Get cost breakdown for a pipeline (GPU vs. free steps) |
| search_web | Search the web and return a synthesized answer |
| get_job_status | Poll a running model job’s status and results |
| ask_followup | Present structured follow-up questions as interactive chips |
Image analysis
Cell counting
Ask the agent to count cells and it will use Cellpose segmentation to detect and count cells in your image. Example prompts:- “Count all cells in this image”
- “How many blue cells are there?”
- “Count cells and show me a chart”
Cell segmentation
The agent can run SAM3 (Segment Anything Model 3) for interactive segmentation:- Click points to segment specific objects
- Draw bounding boxes for region-based segmentation
- Use text prompts like “segment red neurons” or “find cell nuclei”
Image adjustments
Ask the agent to adjust images using natural language:- “Make it darker” — adjusts brightness
- “Increase contrast” — adjusts contrast
- “Apply grayscale” — converts to grayscale
- “Remove the background” — background removal with threshold
- “Invert the colors” — color inversion
- “Add some blur” — applies blur filter
Object transforms
Manipulate objects on the canvas via natural language:- “Rotate it 90 degrees”
- “Flip horizontally”
- “Scale it to 50%“
Image generation
Generate images from text descriptions using Google Imagen:- “Generate a diagram of a neuron”
- “Create a schematic of a microscope setup”
Scientific figures
Generate publication-ready 3x3 figure grids:- Original images in row 1
- Size distribution charts in row 2
- Segmentation results in row 3
- Includes scale bars, statistics overlay, and cell count labels
Edge detection & background removal
- Request edge detection using Sobel or Canny methods
- Ask for background removal with configurable thresholds
Context awareness
The agent automatically receives context about your workspace on every request:- Active pane state — file name, type, dimensions, channels, Z-slice, time point
- Scientific metadata — pixel scale, channel wavelengths, acquisition date, magnification, objective
- Selected region — if you’ve selected a region, the agent knows its bounds
- Available models — which models can process the current file type
- Current pipeline — if you have a draft pipeline open
Code execution
The AI agent can write and execute custom Python code in a sandboxed Docker environment.How it works
- The agent generates Python code based on your request
- Code is validated against security rules (no network access, no system commands)
- A Docker container runs the code with your input files mounted
- Output files (images, CSV, numpy arrays) are saved and displayed
Available packages
The sandbox includes 21 pre-approved scientific Python packages:| Category | Packages |
|---|---|
| Core data | NumPy, Pandas, SciPy, xarray |
| Imaging | scikit-image, OpenCV, Pillow, tifffile |
| ML | PyTorch, scikit-learn |
| Neuroscience | PyNWB, Neo, Elephant |
| Visualization | Matplotlib, Seaborn |
| I/O | h5py, zarr |
Execution limits
| Limit | Value |
|---|---|
| Max code size | 1 MB |
| Max output size | 5 GB |
| Max runtime | 1 hour |
| Max memory | 32 GB |
| Max output files | 100 |
| Network access | Disabled |
Security
Code execution is fully sandboxed:- No network access — outbound connections are disabled
- No system commands — subprocess, os.system, eval, exec are blocked
- Read-only code — the
/codedirectory is read-only - Non-root user — runs as
uid=1000 - Restricted filesystem — only
/inputand/outputdirectories are accessible - Package allowlist — only pre-approved packages can be installed
File organization
The AI agent can intelligently organize your files into folder structures: Example prompt: “Organize my files by experiment” Available organization strategies:- Auto — AI determines the best grouping
- By date — group files by creation date
- By type — group files by format/modality
- By experiment — detect experiment names from file naming patterns (BIDS, dated formats, mouse01_trial03 patterns)
- Flatten — flatten deeply nested folder structures
Pipeline generation & management
The agent has deep pipeline integration:- Generate — describe a goal and the agent creates a multi-step pipeline
- Explain — ask the agent to explain why each step is included
- Preview — run the pipeline on your current image to preview results
- Batch process — run the pipeline across multiple images with progress tracking
- Cost estimate — get a cost breakdown before running (GPU steps vs. free client-side steps)
- Edit — ask the agent to add, remove, or reorder pipeline steps
- “Create a pipeline to segment cells, count them, and export the results”
- “Explain this pipeline step by step”
- “Run this pipeline on all images in my folder”
- “How much will this pipeline cost to run?”
Session limits
| Limit | Value |
|---|---|
| Requests per session | 50 |
| Max input length | 1,200 characters |