Process Pipelines

Every migration has a single source and a single destination. Most migrations have several fields in the process section, and each field may involve several steps. Typically, the process section is the most complex part of the migration.

Each step uses a single process plugin, and a chain of process plugins is called a pipeline. Usually, the first step in a process pipeline has an explicit source, and then the output of each step is passed as the input for the next step. A common mistake is to add a source property to a process plugin, which overrides the value from the previous step in the pipeline. Sometimes this is the right thing to do: see the Skip Process if Not Set section below for an example.

The core Migrate module and the contributed Migrate Plus module provide many process plugins. See the "Related content" section below for a list. The API documentation for each process plugin includes one or more examples of how to use the plugin, but it takes some practice and creativity to combine several process plugins into a useful pipeline. This page has examples of such pipelines.

There is an open issue #3123534: Process plugin: snippet to re-use YAML config to make it easier to reuse process pipelines.

The contributed Migrate Sandbox module offers a UI for experimenting with process plugins and the process pipeline without running real migrations.

Clean up white space in an HTML string

Various characters and strings can be considered "whitespace" in an HTML string. Sometimes it is right to preserve them, but sometimes you want to replace them all with simple space characters:

process:
  field_formatted_text:
    -
      plugin: callback
      source: html_string
      callable: htmlentities
    -
      plugin: str_replace
      search:
        - ' '
        - ' '
      replace: ' '
    -
      plugin: str_replace
      regex: true
      search: '@\s+@'
      replace: ' '
    -
      plugin: callback
      callable: trim

This snippet will

• replace actual non-breaking spaces (and other special characters) with HTML entities using the PHP htmlentities() function;
• replace the numeric or named HTML entities for non-breaking spaces with actual space characters;
• replace one or more whitespace characters (including newlines and tabs) with a single space, like preg_replace('@\s+@', ' ', $string);
• remove leading and trailing spaces with the PHP trim() function.

Combine boolean fields into a List(text) field

This example relies on Extended callback process plugin to call functions with multiple parameters, which adds the unpack_source option to the callback process plugin and the Migrate Source CSV module, which provides the csv source plugin.

The field field_list_text is a List(text) field with allowed values key_1, key_2, key_3. Multiple values are allowed.

The source is a CSV file. Instead of a single column for field_list_text, it has a column for each allowed value. The rows use 0/1 or ''/'X' to indicate whether to include each allowed value.

In the source section of the migration, add a constant with the allowed values, as an array:

source:
  plugin: csv
  # other configuration ...
  constants:
    list_text_values:
      - key_1
      - key_2
      - key_3

In the process section of the migration, combine keys and values, then filter out the empty keys:

  list_text_filter:
    plugin: get
    source:
      - key_1
      - key_2
      - key_3
  field_list_text:
    - plugin: callback
      callable: array_combine
      unpack_source: true
      source:
        - constants/list_text_values
        - '@list_text_filter'
    - plugin: callback
      callable: array_filter
    - plugin: callback
      callable: array_keys

Combine 2 link fields into 1 multi-value link field

In this example, the source content type has 2 single-value link fields e.g. field_link_1 and field_link_2. In the destination content type we would like to combine those 2 fields into just 1 field. Here's what the incoming arrays may look like (you can output this to the terminal by using the debug plugin which comes with the migrate_devel module).

Array (
    [0] => Array
        (
            [link_1_url] => /node/123
            [link_1_title] => Node 123
            [link_1_attributes] => a:0:{}
        )
)

Array (
    [0] => Array
        (
            [link_2_url] => /node/456
            [link_2_title] => Node 456
            [link_2_attributes] => a:0:{}
        )
)

But in order to achieve our objective we need to combine those 2 arrays as follows.

Array (
    [0] => Array
        (
            [uri] => /node/123
            [title] => Node 123
            [options] => a:0:{}
        )
    [1] => Array
        (
            [uri] => /node/456
            [title] => Node 456
            [options] => a:0:{}
        )
)

Note that, unlike in the individual field arrays, each inner array above uses the same keys (uri, title, options).

We can create the above array by preparing the data of each field using pseudo-fields (see Migrate API overview), prefixed with psf for clarity, before using the Migrate Plus merge plugin to combine the individual field arrays. And the code in your YML file would look like this.

process:
  # Pre-process Link-1.
  psf_link_1:
    - plugin: sub_process
      source: link_1
      process:
        uri: field_link_1_url
        title: field_link_1_title
        options: field_link_1_attributes

  # Pre-process Link-2.
  psf_link_2:
    - plugin: sub_process
      source: link_2
      process:
        uri: field_link_2_url
        title: field_link_2_title
        options: field_link_2_attributes

The 2 pseudo-fields become the source in the final step. At last we can condense the code as follows.

process:
  psf_link_1:
    - plugin: sub_process
      source: link_1
      process:
        uri: field_link_1_url
        title: field_link_1_title
        options: field_link_1_attributes
  psf_link_2:
    - plugin: sub_process
      source: link_2
      process:
        uri: field_link_2_url
        title: field_link_2_title
        options: field_link_2_attributes
  field_links:
    - plugin: merge
      source:
        - '@psf_link_1'
        - '@psf_link_2'

Note how we reference pseudo-fields using an @ and single quotes.

Combine formats, values, teasers into a multi-valued body field

Let's say this is your source:

Array
(
    [format] => format 1|format 2|format 3
    [value] => This is the first value|Here is the second value|Third value now
    [teaser] => Teaser 1|Teaser 2|Teaser 3
)

The process pipeline below uses the transpose plugin from migrate_plus.

format_array:
  plugin: explode
  delimiter: "|"
  source: format
value_array: 
  plugin: explode
  delimiter: "|"
  source: value
teaser_array: 
  plugin: explode
  delimiter: "|"
  source: teaser
body_not_associative:
  plugin: transpose
  source:
    - '@format_array'
    - '@value_array'
    - '@teaser_array'
body:
  plugin: sub_process
  source: '@body_not_associative'
  process:
    format:
      plugin: extract
      source:
        - 0
      index:
        - 0
    value:
      plugin: extract
      source:
        - 1
      index:
        - 0
    teaser:
      plugin: extract
      source:
        - 2
      index:
        - 0

This results in

Array
(
  [body] => Array
        (
            [0] => Array
                (
                    [format] => format 1
                    [value] => This is the first value
                    [teaser] => Teaser 1
                )

            [1] => Array
                (
                    [format] => format 2
                    [value] => Here is the second value
                    [teaser] => Teaser 2
                )

            [2] => Array
                (
                    [format] => format 3
                    [value] => Third value now
                    [teaser] => Teaser 3
                )

        )
)

if (x) { do 1 } elseif { do 2 }

The Migrate Conditions contrib module has a dedicated process plugin for this scenario, but it can be accomplished with core plugins and a bit of cleverness.

process:
  _temp1:
    - plugin: skip_on_value
      source: thing1
      value: testvalue
    - plugin: get
      source: value1
  _temp2: value2
  value:
    plugin: null_coalesce
    source:
      - @_temp1
      - @_temp2

 Readable Label for Stubbed Entity

When an entity stub is created as a result of using the migrate_lookup process plugin, the label of the stub is a randomly generated hash that is very long and break layouts and can't be pronounced. You can create a more useful label for your stub using the null_coalesce process plugin to create a fallback label. When a stub is being generated, the only source values available in the migration will be the source IDs. So we use a source ID as the fallback label.

Let's say the following snippet is in a migration called my_node_migration.

process:
  title:
    plugin: null_coalesce
    source:
      - the_real_title
      - whatever_source_property_is_the_source_id

In this case, if a node is stubbed out during a migrate_lookup against my_node_migration, the stub's label will be the source ID. This is because when a stub is being created, the only source property (or properties) available will be the source ID(s). Whatever the_real_title is will be null, but we fall back to the source ID.

You could do fancier things as well by concatenating the source ID with a constant that you define.

source:
  ...
  constants:
    my_stub_prefix: '(Stub) '
process:
  _fallback_title:
    plugin: concat
    source:
      - 'constants/my_stub_suffix'
      - whatever_source_property_is_the_source_id
  title:
    plugin: null_coalesce
    source:
      - the_real_title
      - '@_fallback_title'

Skip Process if Not Set

The core Migrate module provides a skip_row_if_not_set process plugin. However, there is no dedicated plugin to skip a process rather than the row if the source if not set. This can be easily accomplished using the Migrate Conditions contributed module as described in these docs.

That said, the equivalent of a skip_process_if_not_set can be achieved with existing core plugins. Including the following in a process pipeline will skip the process if my_source_value is not set.

-
  plugin: callback
  callable: is_null
  source: my_source_value
-
  plugin: static_map
  map:
    0: 1
    1: 0
-
  plugin: skip_on_empty
  method: process
-
  plugin: get
  source: my_source_value
-
  ...the rest of the pipeline

Skip On Not Empty 

Migrate provides a skip_on_empty process plugin. Migrate Plus provides a skip_on_value process plugin. The equivalent of a skip_on_not_empty plugin can be created configuring the skip_on_value plugin as follows.

plugin: skip_on_value
not_equals: true
source: my_value
method: row/process
value:
 - null
 - false
 - ''

This can be also accomplished using the contributed Migrate Conditions module as described here.