Feature: Add data source folder management#15663
Open
akshaysasidrn wants to merge 27 commits intomainfrom
Open
Feature: Add data source folder management#15663akshaysasidrn wants to merge 27 commits intomainfrom
akshaysasidrn wants to merge 27 commits intomainfrom
Conversation
* feat: Folder permission system * fix(group-permissions): resolve custom group validation, folder edit check, and UI inconsistencie * edit folder container && no folder in custom resource * fix the ui for custom in empty state * fix: coercion logic for folder permissions * feat: enhance folder permissions handling in app components * feat: add folder granular permissions handling in user apps permissions * feat: implement granular folder permissions in ability guard and service * feat: improve error handling for folder permissions with specific messages * feat: enhance EnvironmentSelect component to handle disabled state and improve display logic * chore: bump ee submodules * feat: Update permission prop to isEditable in BaseManageGranularAccess component * chore: bump ee server submodule * fix: refine folder visibility logic based on user permissions * feat: enhance MultiValue rendering and styling for "All environments" option * feat: implement folder ownership checks and enhance app permissions handling * feat: enhance folder permissions handling for app ownership and actions * chore: clarify folder creation and deletion permissions in workspace context * fix: update folder permission labels * fixed folder permission cases * fixed css class issue * minor fix * feat: streamline folder permissions handling by removing redundant checks and simplifying access logic * refactor: made error message consistent * fix: add missing permission message for folder deletion action * refactor: consolidate forbidden messages for folder actions and maintain consistency * feat: streamline permission handling and improve app visibility logic * fix: remove default access denial message in AbilityGuard * fixed all user page functionality falky case * Fixed profile flaky case * fixed granular access flaky case * chore: revert package-lock.json * chore: revert frontend/package-lock.json to main * refactor: revert folder operations * fix: remove unused AppEnvironmentsModule and AppsUtilService exports --------- Co-authored-by: gsmithun4 <gsmithun4@gmail.com> Co-authored-by: Pratush <pratush@Pratushs-MBP.lan> Co-authored-by: Shantanu Mane <maneshantanu.20@gmail.com> Co-authored-by: Yukti Goyal <yuktigoyal02@gmail.com>
Co-authored-by: gsmithun4 <3417097+gsmithun4@users.noreply.github.com>
…se 1) Cherry-picked from feature/ds-folders (c7d4184) onto main. Resolved EE submodule conflicts: kept main's folderCreate/folderDelete alongside new dataSourceFolderCRUD.
Add endpoints for moving data sources between folders: - POST /:id/data-sources — add DS to folder (auto-removes from previous) - DELETE /:id/data-sources/:dsId — remove DS from folder - PUT /:id/data-sources — atomic bulk move with transaction - Validates only global-scope DS can be added (rejects app-scoped with 400) - 12 new e2e tests covering add, move, remove, bulk, permissions Closes ToolJet/tj-ee#4853 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add GET endpoints for folder-aware data source browsing: - GET /data-source-folders — list folders alphabetically with DS count, optional ?search= filter - GET /data-source-folders/:id/data-sources — list data sources within a folder - Util service uses QueryBuilder with LEFT JOIN + GROUP BY for counts - 7 new e2e tests covering listing, search, empty states Closes ToolJet/tj-ee#4854 Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…Phase 5) New entities (DsFoldersGroupPermissions, GroupDsFolders), migration, controller routes, and service logic for folder-level granular permissions. Hierarchy: edit folder > configure DS > build with DS. End-user groups restricted to restrictQueryRun only. Admin/Builder get full access defaults. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Restores FOLDER enum, entities (FoldersGroupPermissions, GroupFolders), type definitions, FEATURE_KEY entries, and relation mappings that were dropped when PR #15028 was merged to main. Also fixes EE constants to use folderCRUD (matching actual DB schema) instead of folderCreate/ folderDelete.
- Add dataSourceFolderCRUD ability constants and type definitions - Add Phase 5 granular permissions migration (1774255111000) - Wire DS folder permission DTOs and type exports Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add .send({ dataSourceIds: [] }) to PUT auth test to avoid 400
from ValidationPipe running before JwtAuthGuard
- Wire folder type awareness into folders service for DS folders
- Update app service for DS folder support
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Point EE submodule to DS folder branch with Phase 1-6 commits - Update plugin dependencies Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
akshaysasidrn
requested review from
ganesh8056,
gsmithun4,
johnsoncherian,
kavinvenkatachalam and
shah21
as code owners
- Test query blocked (403) when restrictQueryRun=true on DS folder - Test query allowed when restrictQueryRun=false - Test query allowed when DS is not in any folder Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
akshaysasidrn
changed the title
Resolves ~19 merge conflicts from main's PR #15028 (app-folder permission system). Key changes: - Replace folderCRUD with folderCreate/folderDelete (main's split) - Keep dataSourceFolderCRUD for DS folder feature - Accept main's folder entity (createdBy), ability refactor, folder permission guards, and CASL rules - Add DS folder types/endpoints alongside main's folder types - Delete redundant migration 1774255111000 (superseded by main's 1766500000000-AddFolderPermissionSystem) - Fix EE external-apis and ability services for new column names - Update all test helpers to use new column names Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Remove duplicate [ResourceType.FOLDER] entries, duplicate FEATURE_KEY entries, and duplicate interface declarations from the auto-merge. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…onService Fixes pre-existing rename from main merge in workflows test helper. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
CE base method now returns FoldersGroupPermissions to match EE override, fixing class hierarchy type error. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… visibility - Replace monolithic dataSourceFolderCRUD with granular dataSourceFolderCreate/dataSourceFolderDelete across types, entity, migration, constants, DTOs, ability services, and tests - Implement permission-based folder listing in EE: non-admin users only see folders they have granular access to - Add owner bypass: folder creators see their own folders when they hold the dataSourceFolderCreate master permission - Convert 3 todo tests into passing implementations (50/50 green) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
All other permission columns use the can* prefix. Rename restrictQueryRun (default false, true=blocked) to canRunQuery (default true, false=blocked) to match the convention. Same enforcement semantics, consistent naming. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…ard DI - Rename all ds_folder/DsFolder/DS_FOLDER abbreviations to fully spelled data_source_folder/DataSourceFolder/DATA_SOURCE_FOLDER per PRD convention - Rename tables: folder_data_sources → data_source_folders, ds_folders_group_permissions → data_source_folders_group_permissions, group_ds_folders → group_data_source_folders - Rename feature keys: CREATE_DS_FOLDER → CREATE_DATA_SOURCE_FOLDER etc. - Fix GroupExistenceGuard NestJS DI: register guard as provider and alias CE GroupPermissionsUtilService to dynamically loaded EE instance Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Support fetching folders with nested data source arrays in a single API call, eliminating N+1 requests from the frontend. Includes search filtering by folder name and DS name, ungrouped DS bucket, and EE permission filtering. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…→ folder_data_source - Add error message verification to all error-path tests (400, 401, 403, 409) - Convert individual key assertions to structural toMatchObject checks - Rename permission tests to intent-based names (e.g., 'canEditFolder=true auto-elevates...') - Remove defensive if(404/405) early-exits from granular permission CRUD tests - Rename data_source_folder → folder_data_source across entities, constants, and types Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…ments - Standardize on TIMESTAMPTZ NOT NULL DEFAULT now() across all DS folder migrations - Rename migration file to drop Ds abbreviation - Remove restating-the-code comments from migrations and services - Remove unused DTO aliases - Update submodule pointer for server/ee
Revert plugins/package.json (mariadb addition), lock files, and useAppData.js changes that were merge noise.
# Conflicts: # .version # frontend/.version # server/.version # server/ee
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
📝 What this does
Adds full backend for data source folders — CRUD, move, listing, granular permissions, and runtime query-run enforcement. Organizes data sources into folders with fine-grained access control at the folder level, gated behind the
folderDataSourceCreate/folderDataSourceDeletelicense permissions.Relates to: tj-ee #4814, tj-ee #4815
Sub-issues:
🏗️ Architecture
Entity Model
erDiagram Folder { uuid id PK string name string type "front_end | data_source" uuid organizationId FK uuid createdBy FK } FolderDataSource { uuid id PK uuid folderId FK uuid dataSourceId FK } GranularPermissions { uuid id PK uuid groupId FK string type "folder_data_source" } FolderDataSourcesGroupPermissions { uuid id PK uuid granularPermissionId FK boolean canEditFolder boolean canConfigureDs boolean canUseDs boolean canRunQuery } GroupFolderDataSources { uuid id PK uuid folderDataSourcesGroupPermissionsId FK uuid folderId FK } Folder ||--o{ FolderDataSource : contains FolderDataSource }o--|| DataSource : references GranularPermissions ||--|| FolderDataSourcesGroupPermissions : has FolderDataSourcesGroupPermissions ||--o{ GroupFolderDataSources : targets GroupFolderDataSources }o--|| Folder : referencesPermission Hierarchy
graph TD A[canEditFolder] -->|auto-sets| B[canConfigureDs] B -->|auto-sets| C[canUseDs] D[canRunQuery] -.->|independent| C style A fill:#4f46e5,color:#fff style B fill:#7c3aed,color:#fff style C fill:#9333ea,color:#fff style D fill:#dc2626,color:#fffWhen a data source (e.g. pgsql1) has BOTH a folder-level permission AND a singular DS permission:
Key rules:
Query Run Enforcement Flow
sequenceDiagram participant Client participant DataQueriesService participant DB Client->>DataQueriesService: runAndGetResult(user, dataQuery) DataQueriesService->>DB: find FolderDataSource where dataSourceId = X alt DS not in any folder DataQueriesService->>Client: execute query normally else DS in folder(s) DataQueriesService->>DB: find user's groupIds via GroupUsers DataQueriesService->>DB: join GroupFolderDataSources → FolderDataSourcesGroupPermissions WHERE canRunQuery = false alt match found DataQueriesService->>Client: 403 ForbiddenException else no match DataQueriesService->>Client: execute query normally end endFolder Visibility
Non-admin users see only folders they have access to:
FolderDataSourcesGroupPermissions(specific orisAll)folderDataSourceCreatemaster permission🔌 API Reference
Folder CRUD & Data Source Management
/api/data-source-foldersGET_FOLDER_DATA_SOURCESsearch(optional),include_data_sources(optional,true)[{ id, name, type, organizationId, count }]— withinclude_data_sources=true:[{ id, name, count, data_sources: [{ id, name, kind, ... }] }](includes ungrouped bucket withid: null)/api/data-source-foldersCREATE_FOLDER_DATA_SOURCE{ name }(1-50 chars,[a-zA-Z0-9_ -]){ id, name, ... }/api/data-source-folders/:idUPDATE_FOLDER_DATA_SOURCE{ name }/api/data-source-folders/:idDELETE_FOLDER_DATA_SOURCE/api/data-source-folders/:id/data-sourcesGET_DATA_SOURCES_IN_FOLDERpage(default 1),per_page(default 25){ data_sources: [DataSource], meta: { total_count, total_pages, current_page, per_page } }/api/data-source-folders/:id/data-sourcesADD_DATA_SOURCE_TO_FOLDER{ dataSourceId: UUID }/api/data-source-folders/:id/data-sources/:dsIdREMOVE_DATA_SOURCE_FROM_FOLDER/api/data-source-folders/:id/data-sourcesBULK_MOVE_DATA_SOURCES{ dataSourceIds: UUID[] }Granular Folder Permissions (EE)
/api/v2/group-permissions/:groupId/granular-permissions/data-source-folderCREATE_GRANULAR_FOLDER_DATA_SOURCE_PERMISSIONS{ name, isAll, type, createResourcePermissionObject }{ id, name, type, isAll, canEditFolder, canConfigureDs, canUseDs, canRunQuery }/api/v2/group-permissions/:groupId/granular-permissions/data-source-folder/:permIdUPDATE_GRANULAR_FOLDER_DATA_SOURCE_PERMISSIONS{ actions, resourcesToAdd, resourcesToDelete }/api/v2/group-permissions/:groupId/granular-permissions/data-source-folder/:permIdDELETE_GRANULAR_FOLDER_DATA_SOURCE_PERMISSIONS/api/v2/group-permissions/granular-permissions/addable-foldersGET_ADDABLE_FOLDER_DATA_SOURCES[{ id, name }]🔀 Changes
FolderDataSourceentity, junction table, and 4 migrations for DS folder schemaFolderDataSourcesGroupPermissionsentity and enforcementfolderDataSourceCRUDintofolderDataSourceCreate/folderDataSourceDelete(entity, migration, types, DTOs, constants, ability services)restrictQueryRuntocanRunQueryfor consistency with thecan*naming conventionds_folder/DsFolder(nowfolder_data_source/FolderDataSource) abbreviations to fully spelledfolder_data_source/FolderDataSourcecanRunQueryenforcement at query execution time in EE data-queries serviceGroupExistenceGuardNestJS DI resolution for EE editionorigin/main(PR Feat: Folder permission system #15028 app-folder permission system) into feature branch🧪 How to test
POST /api/data-source-foldersGET /api/data-source-folderscanEditFolder=trueauto-setscanConfigureDsandcanUseDscanRunQuery=falseon a group and verify queries on that DS return 403npx jest --config test/jest-e2e.json --testPathPatterns=folder-data-sources(50/50 pass)