Directory groups → Generic Project ("dirproject") #9738

this was generated during implementation of #9737. Removed about 30% of the text. Might be still a little too verbose / redundant but the gist is in it (in case anyone wants to read it - i have ;)).

I used exclusively small/local models for this (27b, 8b and 4b) do evaluate if such thing is possible.

Status: implemented (build + unit tests green); the document
describes the initial draft implementation.

Idea: when a directory becomes a group in the Projects view, promote
the directory itself into a GenericProject — so it appears as a real
project node (logical view = all files, plus the full stock project menu).
The group becomes the stock SubprojectsGroup for the promoted project:
nested projects are still listed as project nodes, but discovery is
delegated to a ProjectContainerProvider in the project's lookup (see
below) instead of a hard-coded scan in projectui.

What DirectoryGroup is today

ide/projectui/src/org/netbeans/modules/project/ui/groups/DirectoryGroup.java:

• Created by the "New Group" dialog: NewGroupPanel.java:221
  (DirectoryGroup.create(name, f)).
• Persisted in NbPreferences as name / kind (KIND = "directory") /
  path (directory URL) under the group node; re-instantiated by the
  KIND switch in Group.load.
• findProjects scans the directory for projects (with a
  SharabilityQuery guard) — the group node lists nested project groups.
  Files are never shown.
• A group is per-user view state: everything lives in preferences, the
  file tree is never touched.

The relationship reversal (at a glance)

Today the view knows the discovery policy — DirectoryGroup walks
the directory itself and hard-codes how projects are found. After the
change the project knows — the stock group simply asks the
project's lookup, and the policy lives in the project type. Push
becomes pull:

BEFORE — the view reaches into the directory (push)

  projectui (view)                        filesystem
  ┌──────────────────────────────────┐
  │ DirectoryGroup                   │
  │   findProjects()                 │── directory scan ──▶  X/
  │     processFolderChildren()      │                      ├── a/          (project)
  │       walk folders,              │                      ├── b/
  │       findProject(),             │                      │    └── c/      (project)
  │       sharability guard,         │                      └── ...
  │       convertor filter           │
  └──────────────────────────────────┘
      ↑ discovery policy hard-coded here
        (view module knows how to find
         projects in a directory)
        projects inside are passive —
        they know nothing about being listed


AFTER — the view asks the project (pull)

  projectui (view)                project type (generic.project)
  ┌────────────────────────────┐
  │ SubprojectsGroup (stock,   │
  │ unchanged)                 │
  │   findProjects()           │
  │     visitSubprojects(p)    │
  │       p.getLookup()        │◄─────────────────────────┐
  │        .lookup(            │  "which projects do       │
  │  ProjectContainerProvider) │   you contain?"           │
  │                            │                           │
  └────────────────────────────┘                           │
                                                           │
                                       ┌───────────────────┴───────────────┐
                                       │ GpProjectContainerProvider        │
                                       │ (in p's lookup,                  │
                                       │  base-lookup inner class         │
                                       │   walks X/ + sharability +       │
                                       │   convertor filters              │
                                       └──────────────────────────────────┘
                                             ↑ discovery policy lives here
                                               (in the project type)

The "New Group" flow through the promotion seam:

  "New Group" → directory X
        │
        ▼
  NewGroupPanel (projectui)
        │  Lookup.lookup(DirectoryProjectProvider.class)
        ▼
  GpDirectoryProjectProvider (generic.project)
        │
        ├── findProject(X) != null? ─── yes ──▶ return the existing project
        ├── X NOT_SHARABLE? ─────────── yes ──▶ null (declined)
        └── set MARKER_ATTRIBUTE, clearNonProjectCache(), findProject(X)
        │
        ▼
  OpenProjectList.open({p}, …, asynchronously=true)
  +  SubprojectsGroup.create(name, p)

The open uses the stock new-project-wizard pattern
OpenProjectList.open(projects, /*prime=*/false, /*openSubprojects=*/false, /*asynchronously=*/true, /*mainProject=*/null) — OpenProjects.requestOpen does not exist in
this tree — and runs off the EDT: GroupsMenu posts create() to a
request processor. A null result (decline), an absent provider, or an
IOException is a hard error at the seam (IllegalStateException /
AssertionError) — fail loud rather than fall back to a passive
directory group.

Design

Semantics of the promotion

• Private (attribute-based) promotion, always for the group flow:
  set GenericProject.MARKER_ATTRIBUTE on the directory's FileObject
  (stored in userdir's attributes.xml: per-user, file tree untouched,
  reversible). Rationale: a group is per-user state — a per-user UI
  action must not mutate the file tree or make the folder a project for
  every user on the machine.
• Canonical instance: set attribute → ProjectManager.findProject(dir).
  Never new GenericProject directly (two identities for one directory
  would break FileOwnerQuery, open lists, ...).
  Gotcha (found by the tests): NbProjectManager negatively
  caches directories that were not projects — the idempotency
  findProject(dir) call at the top of the promotion stores a
  NO_SUCH_PROJECT entry that the second findProject(dir) would
  honor. The stock wizard flow hits the same wall and solves it with
  ProjectManager.clearNonProjectCache() ("useful after creating
  project metadata in a folder") — the promotion does the same between
  setting the attribute and the final lookup.
• Sharability guard: do not promote a NOT_SHARABLE directory —
  createProject declines with null and the seam reports the failure
  (there is no passive fallback group anymore). Note this is the
  flip side of the SharabilityQuery check in findProjects (which is
  about reading projects out of non-sharable dirs) — keep the two
  decisions distinct in comments.
• Idempotent: if the directory already is a project (of any type),
  return/open it instead of promoting.
• Group removal does not clear the attribute — the folder remains
  your project; it stays re-openable and is still deletable via the
  stock "Delete" menu entry.

The seam: a lookup service, not a hard dependency

projectui must not depend on generic.project (a common project-UI
module depending on one specific project type is the wrong direction).
With zero new module dependencies (both sides already depend on
projectuiapi):

1. New interface in projectuiapi (package e.g.
   org.netbeans.spi.project.ui.support):

   public interface DirectoryProjectProvider {
       /**
        * @return a project for the directory — the existing project if
        *         the directory already is one, a newly promoted one if
        *         this provider can make it so — or null if it cannot
        *         handle it.
        */
       Project createProject(FileObject dir) throws IOException;
   }

2. generic.project registers the implementation via
   @ServiceProvider (no new dependency):
   findProject(dir) != null → return it; else sharability check, set
   MARKER_ATTRIBUTE, return findProject(dir). This is the same
   primitive as GenericProjectAPI.markAsLocalProject(dir) from the
   API-module design — the two conversations unify on it.
3. projectui changes the call site (NewGroupPanel:221): ask the
   Lookup for DirectoryProjectProvider; non-null result → open the
   project and create a stock SubprojectsGroup.create(name, p)
   (KIND = "subprojects" — already in Group.load's switch); null
   (decline), absent provider, or IOException → hard error
   (IllegalStateException / AssertionError) — fail loud rather than
   fall back to a passive directory group.

No dependency cycle: generic.project → projectuiapi and
projectui → projectuiapi both already exist; projectui never imports
our classes.

Why there is no shared boolean anywhere near the seam

• The group flow is essentially per-user, so it only ever wants the
  private variant; a shared flag on createProject would be dead
  weight in the SPI until some hypothetical "New Group" checkbox asks
  for it.
• The shared/private axis is expressed at the API level as two named
  methods, not a boolean:

  GenericProjectAPI.markAsLocalProject(dir);    // attribute, per-user
  GenericProjectAPI.markAsGenericProject(dir);  // marker file, shared

• The one place a creation-time choice makes sense is the explicit
  "Convert to Generic Project" command on a never-project folder — the
  wizard's job, which already offers it. That command calls the API
  directly and never goes through the SPI.

Private by default, share on demand

The upgrade path is already implemented and cheap ("two clicks"):
GenericProject.setMarkerState(dir, true) is a symmetric toggle that
creates the marker before clearing the redundant attribute (and sets
the attribute before deleting the marker on the way down), so the
directory is never in an unrecognized state. Recognition is the OR of
both mechanisms, so the project's identity never changes — same
GenericProject instance, no reload, no IDE restart.

• Default to the reversible side: private→shared is lossless (nothing
  in the tree changed).
• Follow-up (separate, trivial): a "Make Shared / Make Private" menu
  entry — one line over setMarkerState.

Nested projects: the stock ProjectContainerProvider

ProjectContainerProvider (projectapi SPI, since 1.56 — "a less vague
variant of SubprojectProvider for code that wants to access projects
that the current project serves as container for") is exactly the seam
to delegate discovery to the project type. And the stock discovery
built on it already exists — no rewrite needed:
SubprojectsGroup.visitSubprojects (the group's findProjects) does,
per project:

1. look up ProjectContainerProvider in the project's lookup;
2. if present: add getContainedProjects().getProjects(); if
   !res.isRecursive(), re-visit the same project (see caveat below);
3. else fall back to the semi-deprecated SubprojectProvider and
   visit each returned project the same way.

So the promotion flow just creates a stock SubprojectsGroup for the
promoted project, and all discovery runs through our provider.
SubprojectsGroup persists the project directory URL (KIND
"subprojects", already handled by Group.load), falls back to the
project's display name (so our editable name shows), and resolves to an
empty group if the directory ever stops being a project.

Our provider — new generic.project class
GpProjectContainerProvider implements ProjectContainerProvider, SubprojectProvider:

• Registered as a base-lookup inner class of GenericProject
  (added to the project's base Lookup in the constructor) — not the
  maven @ProjectServiceProvider layer pattern: every other service in
  this module is a base-lookup inner class, it replaces the previous
  direct-children-only GpSubprojectProvider in place, and no layer
  entry is needed for a class that exists for the life of the project
  instance.
• Implements both interfaces over the same set: the stock
  OpenSubprojects ("Open Required Projects" menu item) enables itself
  on the presence of the old SubprojectProvider only.
• getContainedProjects() = a single pass over the project
  directory's folders: getFolders(false), findProject per folder,
  skip ProjectConvertors.isConvertorProject, skip the children of a
  NOT_SHARABLE folder (stop re-checking once the folder is proven
  SHARABLE), exclude the project itself, Thread.interrupted()
  checks (callers include the open-project dialog — maven does the
  same).
• Returns the projects directly contained in the directory,
  reported as complete (recursive=false). A single pass over the
  directory's folders: the deeper containment scan proved too slow
  for large trees, so the exposed set is what the old
  GpSubprojectProvider reported — projects nested deeper are not
  exposed through this provider.
  Maven already trips over this (returns recursive=false): a 3-level
  maven tree shows only the first module level in a stock subprojects
  group. (OpenProjectList.open handles recursive=false correctly
  via its toHandle queue — the no-op is specific to
  SubprojectsGroup.)
• ChangeSupport is present but never fires in v1 — no
  FileChangeListener is registered on the project directory. Rationale
  (revisited at implementation time): no stock consumer registers a
  listener today (maven's own TODO: "is someone actually currently
  listening on SubprojectProvider changes?"), and a listener on the
  project directory would pin the project instance as long as the
  listener lives. The group re-queries whenever projects are opened or
  closed; a silently added nested project may not show until the next
  refresh — acceptable for v1.

Blast radius — every in-tree consumer of
ProjectContainerProvider, checked:

Resulting user experience

• "New Group → directory group" for folder X → X is promoted and
  opened as a project (logical view = all files, full stock project
  menu: New File, Find, Rename/Move/Copy/Delete, versioning, Customize).
  The group is a stock SubprojectsGroup for X, so the projects
  contained in X appear as project nodes under it — the old
  DirectoryGroup UX, minus the hard-coded scan.
• Contained projects are visible in two places, as in any container setup:
  as project nodes in the group, and as folders in X's file logical
  view. Optional polish: a ProjectNodeFactory in the generic logical
  view to render them as project nodes there too.

Files touched

Note: this deliberately breaks the "projectui / projectuiapi are
read-only reference" rule from the generic.project work — by a small,
well-justified amount (one interface, one call-site rework).

Pre-existing groups

DirectoryGroup (class, edit panel, form, bundle keys) is removed;
legacy nodes are migrated on load. Group.load's KIND switch accepts
the legacy kind="directory" and runs a one-shot migration before
returning a stock SubprojectsGroup:

• the stored directory is resolved and passed to
  ProjectManager.findProject; if that finds no project, a
  DirectoryProjectProvider is looked up and createProject
  (promotion) is attempted — so a legacy group on an already-project
  directory is salvaged even in a distribution without
  generic.project;
• the node's kind is rewritten to "subprojects" in all cases —
  success and failure alike — so the migration runs once; a transient
  failure (e.g. a one-time IO error) is not retried, the group stays
  empty until the user removes it or creates a new one on the directory;
• any failure (unresolvable path, no provider registered, provider
  declined, IO error) degrades to an empty SubprojectsGroup —
  visible in Manage Groups (re-pointable to any project, or deletable),
  with one warning logged per group.