Skip to content

feat: base64 import - #2469

Open
salihudickson wants to merge 3 commits into
mainfrom
feat/base64-import
Open

feat: base64 import#2469
salihudickson wants to merge 3 commits into
mainfrom
feat/base64-import

Conversation

@salihudickson

@salihudickson salihudickson commented Sep 2, 2026

Copy link
Copy Markdown
Collaborator

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The new embedded-attachment processing has confirmed runtime/type-safety issues (e.g., undefined src/href trimming and null caching) and transaction usage concerns that should be addressed before merging.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR addresses issue #2458 by adding support for importing embedded Base64 (data URI) assets in standalone HTML/Markdown page imports, materializing them as stored attachments so the imported content can reference /api/files/... paths.

Changes:

  • Wrap standalone page import creation in a DB transaction and generate a page ID upfront for attachment linking.
  • Add processEmbeddedAttachments() to detect Base64 data: URIs in HTML and upload them as attachments, rewriting src/href.
  • Add MIME-type → extension resolution and enforce upload size limits for embedded payloads.
File summaries
File Description
apps/server/src/integrations/import/services/import.service.ts Refactors standalone import flow to pre-generate pageId, process embedded attachments for md/html, and insert the page within a transaction.
apps/server/src/integrations/import/services/import-attachment.service.ts Adds embedded Base64 data URI detection/upload + HTML rewrite logic, plus MIME/size handling for created attachments.
Review details

Suppressed comments (4)

apps/server/src/integrations/import/services/import-attachment.service.ts:116

  • href can be undefined, so calling trim() can throw at runtime. This loop should follow the same safe normalization/caching logic as the src loop (including treating “not processed yet” differently from a cached null result).
      const $element = $(element);
      const href = $element.attr('href');
      const normalized = href.trim();

      let apiFilePath = processed.get(normalized);

apps/server/src/integrations/import/services/import.service.ts:149

  • The outer catch block wraps all errors as "Failed to create imported page", which hides more specific BadRequestExceptions thrown inside the transaction (e.g., unsupported file types or ProseMirror conversion failures). Re-throw BadRequestException as-is and only wrap unexpected errors.
    } catch (err) {
      const message = 'Failed to create imported page';
      this.logger.error(message, err);
      throw new BadRequestException(message);
    }

apps/server/src/integrations/import/services/import.service.ts:121

  • Inside the transaction, getNewPagePosition() queries via this.db (outside the passed trx), so the “read last position + insert” sequence is not atomic even though the page insert uses the transaction. Consider updating getNewPagePosition to accept an optional transaction and use dbOrTx(this.db, trx) (or inline the query here) to avoid inconsistent positions under concurrency.
        const pageTitle = title || fileName;
        const pagePosition = await this.getNewPagePosition(spaceId);

apps/server/src/integrations/import/services/import.service.ts:75

  • executeTx() starts the DB transaction before running potentially expensive, non-DB work (markdown->HTML conversion, Cheerio normalization, ProseMirror/Ydoc generation). This keeps a transaction/connection open longer than necessary and can reduce throughput under load. Consider doing the pure transformation steps outside the transaction and only wrapping the DB writes (attachment rows + page insert) in executeTx.
    try {
      createdPage = await executeTx(this.db, async (trx) => {

        if (fileExtension.endsWith('.md') || fileExtension.endsWith('.html')) {
          const rawHtml = fileExtension.endsWith('.md')
  • Files reviewed: 2/2 changed files
  • Comments generated: 3
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +23 to +24
import { isBase64 } from "class-validator";
import { EnvironmentService } from "src/integrations/environment/environment.service";
Comment on lines +87 to +101
const processed = new Map<string, string>();

for (const element of $(
'img[src], video[src], audio[src], source[src]',
).toArray()) {
const $element = $(element);
const src = $element.attr('src');
const normalized = src.trim();

let apiFilePath = processed.get(normalized);
if (!apiFilePath) {
apiFilePath = await this.uploadDataUri({ uri: normalized, ...rest });
processed.set(normalized, apiFilePath);
}
if (!apiFilePath) continue;
import { load } from 'cheerio';
import { normalizeImportHtml } from '../utils/import-formatter';
import { ImportAttachmentService } from './import-attachment.service';
import { executeTx } from "@docmost/db/utils";
@salihudickson salihudickson changed the title base64 import init feat: base64 import Sep 2, 2026
@salihudickson
salihudickson marked this pull request as ready for review September 3, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

HTML base64 images are not getting imported

2 participants