feat: base64 import - #2469
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The new embedded-attachment processing has confirmed runtime/type-safety issues (e.g., undefined src/href trimming and null caching) and transaction usage concerns that should be addressed before merging.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR addresses issue #2458 by adding support for importing embedded Base64 (data URI) assets in standalone HTML/Markdown page imports, materializing them as stored attachments so the imported content can reference /api/files/... paths.
Changes:
- Wrap standalone page import creation in a DB transaction and generate a page ID upfront for attachment linking.
- Add
processEmbeddedAttachments()to detect Base64data:URIs in HTML and upload them as attachments, rewritingsrc/href. - Add MIME-type → extension resolution and enforce upload size limits for embedded payloads.
File summaries
| File | Description |
|---|---|
| apps/server/src/integrations/import/services/import.service.ts | Refactors standalone import flow to pre-generate pageId, process embedded attachments for md/html, and insert the page within a transaction. |
| apps/server/src/integrations/import/services/import-attachment.service.ts | Adds embedded Base64 data URI detection/upload + HTML rewrite logic, plus MIME/size handling for created attachments. |
Review details
Suppressed comments (4)
apps/server/src/integrations/import/services/import-attachment.service.ts:116
hrefcan be undefined, so callingtrim()can throw at runtime. This loop should follow the same safe normalization/caching logic as thesrcloop (including treating “not processed yet” differently from a cachednullresult).
const $element = $(element);
const href = $element.attr('href');
const normalized = href.trim();
let apiFilePath = processed.get(normalized);
apps/server/src/integrations/import/services/import.service.ts:149
- The outer catch block wraps all errors as "Failed to create imported page", which hides more specific
BadRequestExceptions thrown inside the transaction (e.g., unsupported file types or ProseMirror conversion failures). Re-throwBadRequestExceptionas-is and only wrap unexpected errors.
} catch (err) {
const message = 'Failed to create imported page';
this.logger.error(message, err);
throw new BadRequestException(message);
}
apps/server/src/integrations/import/services/import.service.ts:121
- Inside the transaction,
getNewPagePosition()queries viathis.db(outside the passedtrx), so the “read last position + insert” sequence is not atomic even though the page insert uses the transaction. Consider updatinggetNewPagePositionto accept an optional transaction and usedbOrTx(this.db, trx)(or inline the query here) to avoid inconsistent positions under concurrency.
const pageTitle = title || fileName;
const pagePosition = await this.getNewPagePosition(spaceId);
apps/server/src/integrations/import/services/import.service.ts:75
executeTx()starts the DB transaction before running potentially expensive, non-DB work (markdown->HTML conversion, Cheerio normalization, ProseMirror/Ydoc generation). This keeps a transaction/connection open longer than necessary and can reduce throughput under load. Consider doing the pure transformation steps outside the transaction and only wrapping the DB writes (attachment rows + page insert) inexecuteTx.
try {
createdPage = await executeTx(this.db, async (trx) => {
if (fileExtension.endsWith('.md') || fileExtension.endsWith('.html')) {
const rawHtml = fileExtension.endsWith('.md')
- Files reviewed: 2/2 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| import { isBase64 } from "class-validator"; | ||
| import { EnvironmentService } from "src/integrations/environment/environment.service"; |
| const processed = new Map<string, string>(); | ||
|
|
||
| for (const element of $( | ||
| 'img[src], video[src], audio[src], source[src]', | ||
| ).toArray()) { | ||
| const $element = $(element); | ||
| const src = $element.attr('src'); | ||
| const normalized = src.trim(); | ||
|
|
||
| let apiFilePath = processed.get(normalized); | ||
| if (!apiFilePath) { | ||
| apiFilePath = await this.uploadDataUri({ uri: normalized, ...rest }); | ||
| processed.set(normalized, apiFilePath); | ||
| } | ||
| if (!apiFilePath) continue; |
| import { load } from 'cheerio'; | ||
| import { normalizeImportHtml } from '../utils/import-formatter'; | ||
| import { ImportAttachmentService } from './import-attachment.service'; | ||
| import { executeTx } from "@docmost/db/utils"; |
closes: #2458
ee: https://github.com/docmost/ee/pull/66/changes