Stability Matrix Docs

Appearance

v2.16.4 Technical Notes ​

Extended technical notes for the v2.16.4 release — the deeper "how and why" behind each change. For the user-facing summary, see the CHANGELOG.

Home

Table of Contents ​

Guarding User Data During Install and Uninstall ​

The headline of this patch release is a data-loss fix (#1733). A user moved a 1.2 TB standalone ComfyUI install into Data\Packages\ComfyUI intending to import it, and the folder was recursively deleted. The reporter's code-path analysis was substantially correct, and their logs gave a clean timeline: toggling shared model folders tried to delete a real models\checkpoints directory (throwing an unobserved exception), the uninstall flow then wiped the tree behind a confirmation dialog that omitted models from its warning list, and the first-run one-click installer finished off the remainder with no confirmation at all.

The root of the problem is that Data\Packages\{Name} is overloaded: it is both the "place your folder here to import" path and the path the install flows recursively delete before installing. Four changes close the paths:

  • Install flows confirm before deleting. Both the first-run one-click installer and the Package Manager install now check whether the install location exists and is non-empty. If it is, a confirmation dialog shows the path plus a live-computed total size and file count (junction contents excluded), defaults to cancel, and points at the Import option instead. In the first-run flow the picker closes before the confirmation appears, so there is exactly one dialog on screen.
  • Link teardown never deletes real directories. Turning off shared model folders previously called a plain directory delete on each link location with no reparse-point check. Non-empty real directories threw unobserved exceptions, and empty real directories were silently deleted. Teardown now removes only actual links and skips real directories with a warning. This covers symlink-mode model folders, output-folder links, and the remove-links-on-shutdown path.
  • The uninstall dialog tells the truth about models. It only listed models/checkpoints as deleted when the sharing method was None. Configuration mode (ComfyUI's default) leaves models as real files inside the package folder, so the dialog implied they were safe right before deleting them. It now warns for every non-symlink sharing mode and shows the package folder's total size, so a 1.37 TB uninstall no longer reads like a 5 GB one.
  • Toggle failures surface. The shared-model toggle handlers dropped the link-removal task on the floor, which is where the unobserved-exception crash dialog came from. They now share one helper that logs and raises an error notification on failure.

New tests pin the teardown invariant: junctions are removed, target files are kept, and real directories, empty or not, are never deleted. All five scenarios were also verified live on Windows before merge.

Not in this release, and candidates for follow-up: routing large deletes through the recycle bin, and un-overloading Data\Packages\{Name} as both import staging and install target.

A supporter replaced Data\Workflows with a link to Packages\ComfyUI\user\default\workflows, after which the Workflows page loaded forever and eventually froze the app. The cause is the workflow link added in v2.16.3: at ComfyUI launch, Stability Matrix creates a Stability Matrix link inside ComfyUI's workflows folder pointing back at Data\Workflows. With the library itself linked to that folder, the library now contained a link to itself. .NET's recursive file enumeration follows junctions, and a probe reproduced the failure exactly: roughly 1,900 levels of Stability Matrix\Stability Matrix\... until a path-too-long exception escaped the enumerator outside the per-file error handling.

The fix has two layers so neither has to be perfect on its own:

  • Don't create the loop. Link creation now refuses a link whose parent physically is, or lies within, the target, and removes an existing looping link at that spot. This covers the reported layout (library linked to ComfyUI's folder), the reverse (ComfyUI's folder linked to the library), and model folder links.
  • Survive a loop anyway. A new link-safe file system helper resolves links along the whole path, detects would-be cycles, and enumerates files following links at most once per physical directory with a depth cap and per-directory error isolation. Both workflow scans and both model index scans use it, since Data\Models has the same hazard.

Affected users self-heal: the Workflows page works immediately even with the looping link still present, and the stale link is deleted at the next ComfyUI launch. Linked subfolders inside the library and model trees are still followed (once), and yielded paths stay rooted at the scanned path so relative folders keep working. Fourteen new cycle tests cover the reported layout, the reverse setup, depth capping, and the non-cycle case of a library sitting below the link's parent.

ComfyUI Cross Attention Launch Options ​

Contributed by @e-nord.

Comfy Kitchen Attention. --use-ck-attention joins the Cross Attention Method launch options for ComfyUI and ComfyUI-Zluda.

Mutual exclusivity. The Cross Attention Method flags are mutually exclusive in ComfyUI's argument parser, but the launch options dialog rendered every boolean option as an independent checkbox. Selecting two (for example --use-ck-attention alongside --use-sage-attention) made ComfyUI refuse to start. The launch option definition already had an unused maximum-selected-options field; setting it to 1 now renders that group as radio buttons instead of checkboxes.

Windows ROCm Package Repositories ​

Contributed by @NeuralFault.

Users on Windows ROCm reported CUDA error: invalid argument and hipErrorInvalidValue failures in workflows after the ROCm 7.14 to 7.14.1 patch bump, which coincided with the release of ROCm 10. The same breakage was observed outside Stability Matrix. Windows ROCm PyTorch installs now come from AMD's new permanent repositories: the stable index moves from the multi-arch 7.14.1 repository to stable.repo.amd.com/rocm/whl-next (ROCm 10.0), and the nightly index moves to nightly.repo.amd.com/rocm/whl-next.

Packages already installed against 7.14.1 are not migrated automatically. They need either a fresh install of the package, or a package update whose upgrade path also reinstalls PyTorch (ComfyUI does this when configured to). Further ROCm handling changes are in progress; this one was fast-tracked to unblock affected users.

AI-Toolkit on Pre-2.16.3 Installs ​

Contributed by @NeuralFault.

v2.16.3 moved AI-Toolkit to Python 3.12 and set SETUPTOOLS_USE_DISTUTILS=local so setuptools uses its bundled distutils. Installs from before v2.16.3 still run Python 3.11, where the standard-library distutils is present and the local setting re-arms setuptools' distutils shim, crashing jobs with a distutils error (#1725). The environment setup now reads the venv's real Python version and uses stdlib below 3.12 and local on 3.12 and up, so older installs keep working without a reinstall.

Wan2GP Deepy and the Gradio Logging Wrapper ​

Wan2GP v13's Deepy panel repeatedly showed Connection to server lost and logged issubclass() arg 1 must be a class (Wan2GP #2298). Deepy registers gradio.Error as a FastAPI exception handler. Stability Matrix's console logging wrapper, which mirrors Gradio error messages to stderr so they appear in the package console, had replaced that class with a plain function, so the mounted app failed to build its middleware stack. Diagnosis was confirmed from NeuralFault's comment on the upstream issue.

The wrapper now wraps the original class's constructor instead of replacing the class, preserving identity across gradio.Error, gradio.exceptions.Error, and any previously imported aliases. Error messages still reach stderr, and constructor defaults, exception catching, subclassing, and signature introspection all keep working. The exact Starlette failure was reproduced against Gradio 5.29.0 (Wan2GP's pinned version), and a standalone regression suite passes on Python 3.10 and 3.11, including a mounted FastAPI app returning 400 through the registered Gradio handler. Because the wrapper is rewritten on every launch, existing installs pick up the fix after updating Stability Matrix and relaunching Wan2GP, with no reinstall.