feat: chat desktop backend by hugodutka · Pull Request #23005 · coder/coder

hugodutka · 2026-03-12T17:36:40Z

Implement the backend for the desktop feature for agents.

Adds a new /api/experimental/chats/$id/desktop endpoint to coderd which exposes a VNC stream from a portabledesktop process running inside the workspace
Adds a new spawn_computer_use_agent tool to chatd, which spawns a subagent that has access to the computer tool which lets it interact with the portabledesktop process running inside the workspace
Adds the plumbing to make the above possible

There's a follow up frontend PR here: #23006

coder-tasks · 2026-03-12T17:39:22Z

Documentation Check

New Documentation Needed

docs/ai-coder/computer-use.md (or additions to docs/ai-coder/tasks.md) — Document the new computer use capability: AI agents can now spawn a dedicated spawn_computer_use_agent subagent that takes screenshots, moves the mouse, and types text in the workspace desktop. Include requirements (e.g. VNC/desktop environment in the workspace), how it integrates with Tasks, and the model used (claude-opus-4-6).
docs/manifest.json — Navigation entry added for the new Chats API reference page.

Context

This PR adds:

A new agentdesktop package providing VNC streaming, screenshots, and computer actions (key, type, click, scroll) via the agent API
A new spawn_computer_use_agent tool for chat subagents that delegates to Anthropic's computer use model
A new GET /chats/{chat}/desktop WebSocket endpoint for VNC desktop streaming

The feature is marked EXPERIMENTAL in code. Documentation may be intentionally deferred, but the spawn_computer_use_agent tool is user-visible behavior worth capturing under docs/ai-coder/.

Automated review via Coder Tasks

coderd/chatd/chattool/computeruse.go

coderd/chatd/subagent.go

codersdk/workspacesdk/agentconn.go

kylecarbs · 2026-03-13T15:39:05Z

agent/agentdesktop/portabledesktop.go

+				slog.Error(err),
+			)
+			if attempt < downloadRetries-1 {
+				time.Sleep(downloadRetryDelay)


Should ensureBinary have a context? It seems weird that it doesn't. I kinda expected this to be piped through everything for a graceful shutdown.

kylecarbs · 2026-03-13T15:40:07Z

agent/agentdesktop/portabledesktop.go

+// platformBinaries maps GOARCH to download URL and expected SHA-256
+// digest for each supported platform.
+var platformBinaries = map[string]struct {
+	URL    string
+	SHA256 string
+}{
+	"amd64": {
+		URL:    "https://github.com/coder/portabledesktop/releases/download/" + portableDesktopVersion + "/portabledesktop-linux-x64",
+		SHA256: "a04e05e6c7d6f2e6b3acbf1729a7b21271276300b4fee321f4ffee6136538317",
+	},
+	"arm64": {
+		URL:    "https://github.com/coder/portabledesktop/releases/download/" + portableDesktopVersion + "/portabledesktop-linux-arm64",
+		SHA256: "b8cb9142dc32d46a608f25229cbe8168ff2a3aadc54253c74ff54cd347e16ca6",
+	},
+}


I'm somewhat skeptical that we should be doing this downloading at all... since customers will likely have to download a browser too, it's maybe best for this change if we just require it on $PATH and customers will need to adjust their workspace images?

I'd rather pin a specific version of portabledesktop and auto-download it for now in case we need to make breaking changes to the portabledesktop cli.

We DM'd about it: I'm going to leave the current approach as is in this PR. I'll do a follow up PR next week to only get portabledesktop from PATH and publish a template module to set it up.

kylecarbs · 2026-03-13T15:40:32Z

coderd/database/migrations/000434_add_chat_type.up.sql

@@ -0,0 +1,3 @@
+CREATE TYPE chat_type AS ENUM ('computer_use');
+
+ALTER TABLE chats ADD COLUMN chat_type chat_type;


Should we call this mode instead? Just because chat.chat_type is janky.

kylecarbs · 2026-03-13T15:40:47Z

coderd/chatd/subagent.go

 			},
 		),
+		fantasy.NewAgentTool(
+			"spawn_computer_use_agent",


It's great that this is a separate tool 👍

kylecarbs · 2026-03-13T15:42:09Z

coderd/chatd/subagent.go

+				// Check that the Anthropic provider is configured.
+				// Computer use requires an Anthropic model; verify
+				// the key is available before creating the child chat.
+				anthropicAvailable := p.providerAPIKeys.APIKey("anthropic") != ""
+				if !anthropicAvailable {
+					dbProviders, err := p.db.GetEnabledChatProviders(ctx)
+					if err == nil {
+						for _, prov := range dbProviders {
+							if chatprovider.NormalizeProvider(prov.Provider) == "anthropic" && strings.TrimSpace(prov.APIKey) != "" {
+								anthropicAvailable = true
+								break
+							}
+						}
+					}
+				}
+				if !anthropicAvailable {
+					return fantasy.NewTextErrorResponse("Computer use requires an Anthropic API key."), nil
+				}


Would it be better if we just omitted this tool unless Anthropic is configured?

I think the ideal would be that we only inject this if Anthropic is configured and portabledesktop is in the path of the corresponding workspace, but we don't have to do that immediately.

kylecarbs · 2026-03-13T15:43:15Z

coderd/chatd/chatloop/chatloop.go

+// ToolDefiner is an optional interface that AgentTools can
+// implement to control how they appear in Call.Tools. When
+// present, buildToolDefinitions uses the returned Tool instead
+// of constructing a FunctionTool from Info().
+type ToolDefiner interface {
+	ToolDefinition() fantasy.Tool
+}


Why is this necessary? Isn't it just a normal tool definition that we return anyways?

It's a provider tool like web_search. ToolDefiner was old code that remained from before you added support for provider tools in chatd. I got rid of it now.

github-actions bot assigned hugodutka Mar 12, 2026

hugodutka marked this pull request as draft March 12, 2026 17:37

hugodutka force-pushed the hugodutka/agents-desktop branch from 08a8db5 to 5660da1 Compare March 12, 2026 17:37

kylecarbs reviewed Mar 12, 2026

View reviewed changes

coderd/chatd/chattool/computeruse.go Show resolved Hide resolved

coderd/chatd/subagent.go Outdated Show resolved Hide resolved

codersdk/workspacesdk/agentconn.go Outdated Show resolved Hide resolved

codersdk/workspacesdk/agentconn.go Outdated Show resolved Hide resolved

hugodutka force-pushed the hugodutka/agents-desktop branch 5 times, most recently from cb5a27d to eab5e1d Compare March 13, 2026 15:31

kylecarbs reviewed Mar 13, 2026

View reviewed changes

hugodutka force-pushed the hugodutka/agents-desktop branch 9 times, most recently from 564f60c to ea9f7e2 Compare March 13, 2026 17:58

hugodutka marked this pull request as ready for review March 13, 2026 18:10

hugodutka requested a review from mafredri March 13, 2026 18:10

desktop backend

0b00ce3

hugodutka force-pushed the hugodutka/agents-desktop branch from ea9f7e2 to 0b00ce3 Compare March 13, 2026 18:21

kylecarbs approved these changes Mar 13, 2026

View reviewed changes

hugodutka merged commit 8452739 into main Mar 13, 2026
24 checks passed

hugodutka deleted the hugodutka/agents-desktop branch March 13, 2026 18:49

github-actions bot locked and limited conversation to collaborators Mar 13, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: chat desktop backend#23005

feat: chat desktop backend#23005
hugodutka merged 1 commit intomainfrom
hugodutka/agents-desktop

hugodutka commented Mar 12, 2026 •

edited

Loading

Uh oh!

coder-tasks bot commented Mar 12, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kylecarbs Mar 13, 2026

Uh oh!

kylecarbs Mar 13, 2026

Uh oh!

hugodutka Mar 13, 2026

Uh oh!

hugodutka Mar 13, 2026

Uh oh!

kylecarbs Mar 13, 2026

Uh oh!

kylecarbs Mar 13, 2026

Uh oh!

kylecarbs Mar 13, 2026

Uh oh!

kylecarbs Mar 13, 2026

Uh oh!

hugodutka Mar 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -0,0 +1,3 @@
		CREATE TYPE chat_type AS ENUM ('computer_use');

		ALTER TABLE chats ADD COLUMN chat_type chat_type;

Conversation

hugodutka commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

coder-tasks bot commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Documentation Check

New Documentation Needed

Context

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hugodutka commented Mar 12, 2026 •

edited

Loading

coder-tasks bot commented Mar 12, 2026 •

edited

Loading