sed regexp strangeness

Question

trying to replace a string in sed with embedded { and $ and with -E or -r, the { is problematic for me.

according to the docs and many examples I have read, I should only have to escape the { and $ if testing for them. I have whittled it down the simplest case (below)

Probably something I do not understand

I can use my workaround, but this is something that should work with the \{ ?

Given string:

: "${TARGET_PART:=${MMC_PART2}}"

and part=5

I want to replace the ${MMC_PART2}} with ${MMC_PART5}}

The following does not work:

echo ': "${TARGET_PART:=${MMC_PART2}}"'|  sed -r "s/:=\{(MMC_PART).}}/\1$part}}/g"

but the following does work - (replaced "{" with ".")

echo ': "${TARGET_PART:=${MMC_PART2}}"'|  sed -r "s/:=.(MMC_PART).}}/\1$part}}/g"

What am I missing?

See here for the difference between Basic Regular Expressions (BRE)and Extended Regular Expressions(ERE). — potong
– potong, Commented Aug 30, 2024 at 23:37
Aside: -r is the argument to enable EREs in old versions of GNU sed only. Modern versions of GNU and BSD sed both use -E to enable EREs, just like grep does. — Ed Morton
– Ed Morton, Commented Aug 31, 2024 at 12:32
@EdMorton -E has also made it into the latest standard - pubs.opengroup.org/onlinepubs/9799919799/utilities/sed.html — jhnc
– jhnc, Commented Aug 31, 2024 at 23:13

jhnc · Accepted Answer · 2024-08-30 23:25:20Z

4

Your sed command is syntactically correct but fails to account for the $ in:

: "${TARGET_PART:=${MMC_PART2}}"

The replacement seems wrong too.

The first command should be:

part=5
echo ': "${TARGET_PART:=${MMC_PART2}}"'|  sed -r "s/(:=\\\$\{MMC_PART).(}})/\1$part\2/g"

to output:

: "${TARGET_PART:=${MMC_PART5}}"

Note that inside double-quotes, backslash and dollar should be escaped to avoid alteration by the shell. Inside single-quotes, this is not necessary:

echo ': "${TARGET_PART:=${MMC_PART2}}"'|  sed -r 's/(:=\$\{MMC_PART).(}})/\1'"$part"'\2/g'

Your second command does not perform any substitution for me. Perhaps it was transcribed incorrectly into the question.

edited Aug 30, 2024 at 23:25

answered Aug 30, 2024 at 22:35

jhnc

18.8k2 gold badges14 silver badges33 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Osman Tunahan ARIKAN · Accepted Answer · 2024-08-30 17:29:21Z

2

The issue you’re encountering is related to how sed interprets special characters like { and } in extended regular expressions (with -E or -r). In sed, { and } are used for interval expressions, and they need to be escaped if you want to match them literally, even when you’re not using them in a pattern repetition context.

In your first sed command, the { in the pattern s/:={(MMC_PART).}}/ is being interpreted as the start of an interval expression rather than a literal brace.

Here’s the correct way to handle it:

1. Escape the { and } with a backslash to match them literally.

So, your sed command should look like this:

echo ': "${TARGET_PART:=${MMC_PART2}}"' | sed -r "s/:=\\{(MMC_PART)\\}\\}/:\\{\1$part\\}\\}/g"

Alternatively, if you want to avoid escaping:

2. Use sed without -E or -r and rely on backslashes for all special characters:

echo ': "${TARGET_PART:=${MMC_PART2}}"' | sed "s/:=\{MMC_PART2\}/:=\{MMC_PART$part\}/g"

This ensures that sed treats the { and } as literal characters, allowing you to perform the substitution as intended.

answered Aug 30, 2024 at 17:29

Osman Tunahan ARIKAN

1105 bronze badges

4 Comments

Floyd Brown Over a year ago

the "{" has 2 backslashes? //{ I definitely missed something. Must be a bash thing.

jhnc Over a year ago

@FloydBrown it is a precaution to ensure bash passes a literal backslash to the command. It is not necessary in this case, but avoids needing to remember when it is needed. bash treats dollar, backtick, backslash, and sometimes exclamation as metacharacters inside double-quotes.

The backslash retains its special meaning only when followed by one of the following characters: $, `, ", \, or <newline>.

. POSIX sh does same: pubs.opengroup.org/onlinepubs/9699919799/utilities/…

jhnc Over a year ago

for 2., the string to sed should be "s/:={MMC..." since \{ will enable treatment as a range, which is not wanted here

Benjamin W. Over a year ago

To expand on what @jhnc says, this answer has it exactly backwards: -E/-r doesn't require escapes for {, [, + etc.; sed without parameters does. It's the difference between "basic" and "extended" POSIX regular expressions. In GNU sed, they only differ in what needs to be escaped and what doesn't.

Collectives™ on Stack Overflow

sed regexp strangeness

2 Answers 2

Comments

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related