Skip to content

Add ConptyReparentPseudoConsole support - #995

Merged
Anthony Kim (anthonykim1) merged 12 commits into
mainfrom
anthonykim1/conpty-parent-window
Oct 22, 2026
Merged

Anthony Kim (anthonykim1) merged 12 commits into
mainfrom
anthonykim1/conpty-parent-window

Conversation

@anthonykim1

@anthonykim1 Anthony Kim (anthonykim1) commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor
  • Add IPty.setParentWindow(handle), backed by ConptyReparentPseudoConsole from the bundled conpty.dll. The window becomes the owner of the console window, so dialogs that console apps parent to the console window or its owner, like the Connect-AzAccount sign-in picker, open in front of the host instead of behind it.
  • Don't defer it until the pty is ready, since the pseudoconsole already exists. Called right after spawn(), the request is sent before the shell launches, because connect() waits for the conout worker. Hosts call it again when the terminal moves to another window.
  • Return whether the request was sent; ConPTY applies it asynchronously. It's false once the pseudoconsole is closed, and always false unless on Windows with useConptyDll, since in-box ConPTY has no reparent API.
  • Take the handle as a Buffer the size of an HWND, the way Electron's BrowserWindow.getNativeWindowHandle() returns it, so hosts can pass that Buffer directly. Throw for anything else.
  • Null the HPCON in PtyKill so it's only closed once, and skip resize, clear, and reparent after that, since the baton outlives the pseudoconsole until the exit thread erases it.
  • Add Windows tests for the owner when set right after spawn() and while the shell runs, the no-op once the pseudoconsole is closed, and invalid handles. Extend the Windows ConPTY backend data race in ptyHandles can crash in remove_pty_baton #921 race test to call setParentWindow.
  • Keep setParentWindow optional in the public typings, so existing IPty implementations keep compiling. VS Code adopts this in Parent the ConPTY console window to the VS Code window vscode#342784.
  • Re-enabling the host window when a modal dialog's process dies is tracked in Re-enable the parent window when a modal dialog's process dies #996.

Inspirations from:

Add a conptyParentWindow fork option and IPty.setParentWindow(), backed by
ConptyReparentPseudoConsole from conpty.dll. The host window becomes the
owner of the console window returned by GetConsoleWindow(), so windows that
console apps parent to it (for example the WAM sign-in picker) open above the
host instead of behind it.

@anthonykim1 Anthony Kim (anthonykim1) left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Findings are inline: one p1 on the native guard, the rest p2.

  • PtyReparent relies on JS-side bookkeeping (_isKilled) to avoid writing through a freed PseudoConsole*; the native entry should be safe on its own.
  • The new export swallows every failure (HRESULT, missing export, lossy BigInt), so callers can't tell whether the owner was applied.
  • The #921 race extension never reaches PtyReparent.
  • IPty.setParentWindow as a required member breaks IPty implementers in a 1.x minor, and the @param example doesn't match the signature.
  • conptyParentWindow is applied from the WindowsTerminal constructor after the agent already started the pty; a throw there leaks the agent.

Comment thread src/win/conpty.cc Outdated
Comment thread src/win/conpty.cc Outdated
Comment thread src/windowsTerminal.test.ts Outdated
Comment thread typings/node-pty.d.ts Outdated
Comment thread typings/node-pty.d.ts Outdated
Comment thread src/windowsTerminal.ts Outdated
- Null the HPCON in PtyKill and skip resize/clear/reparent once it is
  closed, instead of relying on JS-side bookkeeping.
- Return whether ConptyReparentPseudoConsole succeeded, and reject BigInt
  handles that don't fit in 64 bits.
- Apply conptyParentWindow inside WindowsPtyAgent, right after the
  pseudoconsole is created, and clean up if that throws.
- Re-enable the parent window when the shell exits if a modal dialog that
  died with it left the window disabled.
- Make IPty.setParentWindow optional in the public typings and show how to
  convert Electron's native window handle.
@anthonykim1

Anthony Kim (anthonykim1) commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Pushed 909b13e for the review. All six threads are addressed and resolved.

It also included a fix from the VS Code review (microsoft/vscode#342784 (comment)) that re-enabled the parent window when a modal dialog died with the shell. That has since moved out of this PR to #996.

Verified on Windows 11:

@anthonykim1

Copy link
Copy Markdown
Contributor Author

@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as ready for review October 20, 2026 17:44
@DHowett

Copy link
Copy Markdown
Member

/cc Leonard Hecker (@lhecker) as well

@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as draft October 20, 2026 17:45
conpty.dll is no longer experimental, so the APIs added on top of it are
documented as regular APIs.
@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as ready for review October 20, 2026 18:06
@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as draft October 20, 2026 18:08
- Add conptyReenableParentWindowOnExit (defaults to false). Re-enabling the
  window on exit is a heuristic, so hosts have to opt in.
- Look for visible owned windows through the whole owner chain, and compare
  window handles by their low 32 bits.
- Run the recovery off the exit thread, since EnableWindow waits for the
  window's thread.
- Validate handles before creating the pseudoconsole: numbers must be safe
  integers, and bigints must fit in 64 bits.
- Only close the pseudoconsole once in kill().
- setParentWindow returns whether the request was sent; ConPTY applies it
  asynchronously.
- The parent window tests wait for every pty to exit, since exiting while a
  conout worker still flushes its pipe can hang mocha.
@anthonykim1

Anthony Kim (anthonykim1) commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor Author

Pushed f164f25 after another pass over the review, then 95ed9a4 to keep this PR to one new API, cd33f21 for the handle review, and f699175 to take the handle as a Buffer:

  • setParentWindow() is the only new API. conptyParentWindow is gone: hosts call setParentWindow() right after spawn(). It isn't deferred, and connect() waits for the conout worker, so the request is sent before the shell launches.
  • Re-enabling the parent window when a modal dialog's process dies moved to Re-enable the parent window when a modal dialog's process dies #996, with the implementation from f164f25.
  • The handle is a Buffer the size of an HWND, the way Electron's getNativeWindowHandle() returns it; anything else throws.
  • kill() never closes the pseudoconsole twice, and setParentWindow returns whether the request was sent.
  • The parent window tests kill and wait for every pty they start, so a failing test can't keep mocha from exiting.

Verified locally at f699175:

(generated by copilot)

It's an exact check: the window is still disabled and no visible window is
owned by it. What it can't tell is why the window is disabled, which is why
hosts opt in.
Remove conptyParentWindow. setParentWindow() isn't deferred, and the
shell is only launched once the conout worker is ready, so calling it
right after spawn() sends the request before the shell starts.

Remove conptyReenableParentWindowOnExit and re-enabling the parent
window on exit. That moves to a follow-up issue, so this PR stays
focused on the parenting change.
Anthony Kim (anthonykim1) added a commit to microsoft/vscode that referenced this pull request Oct 20, 2026
microsoft/node-pty#995 no longer has conptyParentWindow or
conptyReenableParentWindowOnExit, so TerminalProcess calls
setParentWindow right after spawn, which is before node-pty launches
the shell, and no longer opts in to the re-enable.
@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as ready for review October 21, 2026 21:22
Comment thread src/windowsPtyAgent.ts Outdated
Comment thread src/win/conpty.cc Outdated
Comment thread src/win/conpty.cc Outdated
Comment thread src/win/conpty.cc Outdated
Comment thread src/win/conpty.cc Outdated
Comment thread src/win/conpty.cc Outdated
@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as draft October 21, 2026 21:49
HWNDs are pointers, so negative handles are now rejected and the handle
is converted to an HWND through uintptr_t in C++, which also covers
32-bit builds. Numbers must be non-negative safe integers, since
Node-API's integer getters silently truncate fractions, map NaN and
Infinity to 0 and lose precision above 2^53.

Remove the JS check, so the native binding is the only place that
validates the handle.
Only Int64Value() loses precision past Number.MAX_SAFE_INTEGER; the 32-bit
getters wrap instead. The invalid handle test also rejects unsafe integers,
so say safe integers in its name.
@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as ready for review October 22, 2026 01:58
@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as draft October 22, 2026 01:59
@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as ready for review October 22, 2026 04:23
roblourens
roblourens previously approved these changes Oct 22, 2026
@deepak1556
Robo (deepak1556) self-requested a review October 22, 2026 06:28
Comment thread src/win/conpty.cc Outdated
@anthonykim1
Anthony Kim (anthonykim1) marked this pull request as draft October 22, 2026 14:30
setParentWindow() now takes the HWND as a Buffer the size of an HWND, in
native byte order, which is how Electron's
BrowserWindow.getNativeWindowHandle() returns it, so hosts can pass that
Buffer directly. The native side copies it into an HWND, which replaces
the number and bigint parsing.
@anthonykim1
Anthony Kim (anthonykim1) merged commit 1012192 into main Oct 22, 2026
9 checks passed
@anthonykim1
Anthony Kim (anthonykim1) deleted the anthonykim1/conpty-parent-window branch October 22, 2026 15:26
Anthony Kim (anthonykim1) added a commit to microsoft/vscode that referenced this pull request Oct 23, 2026
Adds IPty.setParentWindow() from microsoft/node-pty#995, which this PR uses.
Anthony Kim (anthonykim1) added a commit to microsoft/vscode that referenced this pull request Oct 23, 2026
* Parent the ConPTY console window to the VS Code window

Pass the native handle of the window hosting each terminal down to the pty
host and node-pty, which calls ConptyReparentPseudoConsole. Windows that
console apps parent to GetConsoleWindow(), such as the WAM sign-in picker
used by Connect-AzAccount, then open above VS Code instead of behind it.

* Address review: wait for the handle in createProcess, add an off switch

- Start the window handle lookup without awaiting it in TerminalInstance;
  createProcess now waits for the handle of the latest window before
  start(), so a move during startup can't let the shell start unowned.
- Drop the second per-window cache; NativeHostService already memoizes the
  lookup.
- Sign-extend the low 32 bits of the HWND, like Windows does for handles.
- Add terminal.integrated.windowsReparentConsoleWindow (experimental,
  ExP-controllable) as an off switch that keeps conpty.dll.
- Trace-log node-pty's setParentWindow and whether it applied.
- Drop the agent host TestPty stub now that IPty.setParentWindow is optional.

* Don't tag windowsReparentConsoleWindow as experimental

The setting stays ExP-controllable through `experiment`, but it is not an
experimental setting.

* Opt in to re-enabling the window when a terminal's dialog dies

node-pty only re-enables the parent window on exit when asked to
(conptyReenableParentWindowOnExit). VS Code only disables its windows
while showing a window they own, so it opts in. Also log setParentWindow's
result as "sent", since ConPTY applies the new owner asynchronously.

* Send the parent window right after spawn

microsoft/node-pty#995 no longer has conptyParentWindow or
conptyReenableParentWindowOnExit, so TerminalProcess calls
setParentWindow right after spawn, which is before node-pty launches
the shell, and no longer opts in to the re-enable.

* Pass the parent window to node-pty as an unsigned handle

node-pty now rejects negative window handles, since HWNDs are pointers.
Sign-extend the signed 32-bit handle to an unsigned 64-bit value, the
same bits node-pty used before, and never let setting the parent window
fail the terminal launch.

* Pass the window handle bytes through to node-pty

node-pty now takes the HWND as the Buffer that Electron's
getNativeWindowHandle() returns, so send the VSBuffer from
IHostService.getNativeWindowHandle to the pty host unchanged instead of
converting it to a number and back.

* chore: bump node-pty@1.2.0-beta.17

Adds IPty.setParentWindow() from microsoft/node-pty#995, which this PR uses.

* chore: bump node-pty@1.2.0-beta.18

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants