Skip to content

Add an option to disable or rename special fields in output connectors #162

Open

Description

@zxqfd555

Is your feature request related to a problem? Please describe.
Currently, output connectors always emit two additional metadata columns alongside user data:

  • time, which contains the number of computational minibatch in Pathway;
  • diff, which contains either +1 or -1, indicating whether the change corresponds to a row insertion or deletion.

In many use cases, this metadata is not required by the user and only adds noise to the output. At the moment, there is no way to disable these fields, even when they are irrelevant for downstream consumers.

Describe the solution you'd like
Introduce an additional configuration field for every output connector, and add an extra optional argument to each emitting method: special_fields_config.

This config should:

  • Allow renaming each of the special metadata fields (time, diff) to custom names;
  • Allow disabling the emission of each of these fields entirely.

This would give users fine-grained control over what exactly is included in the output.

Describe alternatives you've considered
Keeping the current behavior as-is, with special fields always included.

Additional context
Given the large number of output connectors in Pathway, this feature may be introduced gradually, prioritizing the most commonly used connectors first if necessary.

The change must be fully backward-compatible: the default value of the new entity should reproduce the current behavior exactly (i.e., both fields enabled with their existing names).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions