Next: , Previous: , Up: Portable Shell Programming   [Contents][Index]

11.9 Shell Substitutions

Contrary to a persistent urban legend, the Bourne shell does not systematically split variables and back-quoted expressions, in particular on the right-hand side of assignments and in the argument of case. For instance, the following code: 

case "$given_srcdir" in
.)  top_srcdir="`printf '%s\n' "$dots" | sed 's|/$||'`" ;;
*)  top_srcdir="$dots$given_srcdir" ;;
esac

is more readable when written as: 

case $given_srcdir in
.)  top_srcdir=`printf '%s\n' "$dots" | sed 's|/$||'` ;;
*)  top_srcdir=$dots$given_srcdir ;;
esac

and in fact it is even more portable: in the first case of the first attempt, the computation of top_srcdir is not portable, since not all shells properly understand "`…"…"…`", for example Solaris 10 ksh: 

$ foo="`echo " bar" | sed 's, ,,'`"
ksh: : cannot execute
ksh: bar | sed 's, ,,': cannot execute

POSIX does not specify behavior for this sequence. On the other hand, behavior for "`…\"…\"…`" is specified by POSIX, but in practice, not all shells understand it the same way: pdksh 5.2.14 prints spurious quotes when in POSIX mode: 

$ echo "`echo \"hello\"`"
hello
$ set -o posix
$ echo "`echo \"hello\"`"
"hello"

There is just no portable way to use double-quoted strings inside double-quoted back-quoted expressions (pfew!).

Bash 4.1 has a bug where quoted empty strings adjacent to unquoted parameter expansions are elided during word splitting. Meanwhile, zsh does not perform word splitting except when in Bourne compatibility mode. In the example below, the correct behavior is to have five arguments to the function, and exactly two spaces on either side of the middle ‘-’, since word splitting collapses multiple spaces in ‘$f’ but leaves empty arguments intact. 

$ bash -c 'n() { echo "$#$@"; }; f="  -  "; n - ""$f"" -'
3- - -
$ ksh -c 'n() { echo "$#$@"; }; f="  -  "; n - ""$f"" -'
5-  -  -
$ zsh -c 'n() { echo "$#$@"; }; f="  -  "; n - ""$f"" -'
3-   -   -
$ zsh -c 'emulate sh;
> n() { echo "$#$@"; }; f="  -  "; n - ""$f"" -'
5-  -  -

You can work around this by doing manual word splitting, such as using ‘"$str" $list’ rather than ‘"$str"$list’.

There are also portability pitfalls with particular expansions:

$@

Autoconf macros often use the set command to update ‘$@’, so if you are writing shell code intended for configure you should not assume that the value of ‘$@’ persists for any length of time.

You may see usages like ‘${1+"$@"}’ in older shell scripts designed to work around a portability problem in ancient shells. Unfortunately this runs afoul of bugs in more-recent shells, and nowadays it is better to use plain ‘"$@"’ instead.

The portability problem with ancient shells was significant. When there are no positional arguments ‘"$@"’ should be discarded, but the original Unix version 7 Bourne shell mistakenly treated it as equivalent to ‘""’ instead, and many ancient shells followed its lead.

For many years shell scripts worked around this portability problem by using ‘${1+"$@"}’ instead of ‘"$@"’, and you may see this usage in older scripts. Unfortunately, ‘${1+"$@"}’ does not work with ksh93 M 93t+ (2009) as shipped in AIX 7.2 (2015), as this shell drops a trailing empty argument: 

$ set a b c ""
$ set ${1+"$@"}
$ echo $#
3

Also, ‘${1+"$@"}’ does not work with Zsh 4.2.6 (2005) and earlier, as shipped in Mac OS X releases before 10.5, as this old Zsh incorrectly word splits the result: 

zsh $ emulate sh
zsh $ for i in "$@"; do echo $i; done
Hello World
!
zsh $ for i in ${1+"$@"}; do echo $i; done
Hello
World
!

To work around these problems Autoconf does two things. First, in the shell code that it generates Autoconf avoids ‘"$@"’ if it is possible that there may be no positional arguments. You can use this workaround in your own code, too, if you want it to be portable to ancient shells. For example, instead of: 

cat conftest.c "$@"

you can use this: 

case $# in
  0) cat conftest.c;;
  *) cat conftest.c "$@";;
esac

Second, Autoconf-generated configure scripts work around most of the old Zsh problem by using Zsh’s “global aliases” to convert ‘${1+"$@"}’ into ‘"$@"’ by itself: 

test ${ZSH_VERSION+y} && alias -g '${1+"$@"}'='"$@"'

This workaround is for the benefit of any instances of ‘${1+"$@"}’ in user-written code appearing in configure scripts. However, it is not a complete solution, as Zsh recognizes the alias only when a shell word matches it exactly, which means older Zsh still mishandles more-complicated cases like ‘"foo"${1+"$@"}’.

${10}

The 10th, 11th, … positional parameters can be accessed only after a shift. The 7th Edition shell reported an error if given ${10}, and Solaris 10 /bin/sh still acts that way: 

$ set 1 2 3 4 5 6 7 8 9 10
$ echo ${10}
bad substitution

Conversely, not all shells obey the POSIX rule that when braces are omitted, multiple digits beyond a ‘$’ imply the single-digit positional parameter expansion concatenated with the remaining literal digits. To work around the issue, you must use braces. 

$ bash -c 'set a b c d e f g h i j; echo $10 ${1}0'
a0 a0
$ dash -c 'set a b c d e f g h i j; echo $10 ${1}0'
j a0
${var-value}
${var:-value}
${var=value}
${var:=value}
${var?value}
${var:?value}
${var+value}
${var:+value}

When using ‘${var-value}’ or similar notations that modify a parameter expansion, POSIX requires that value must be a single shell word, which can contain quoted strings but cannot contain unquoted spaces. If this requirement is not met Solaris 10 /bin/sh sometimes complains, and anyway the behavior is not portable. 

$ /bin/sh -c 'echo ${a-b c}'
/bin/sh: bad substitution
$ /bin/sh -c 'echo ${a-'\''b c'\''}'
b c
$ /bin/sh -c 'echo "${a-b c}"'
b c
$ /bin/sh -c 'cat <<EOF
${a-b c}
EOF
b c

Most shells treat the special parameters * and @ as being unset if there are no positional parameters. However, some shells treat them as being set to the empty string. POSIX does not clearly specify either behavior. 

$ bash -c 'echo "* is ${*-unset}."'
* is unset.
$ dash -c 'echo "* is ${*-unset}."'
* is .

According to POSIX, if an expansion occurs inside double quotes, then the use of unquoted double quotes within value is unspecified, and any single quotes become literal characters; in that case, escaping must be done with backslash. Likewise, the use of unquoted here-documents is a case where double quotes have unspecified results: 

$ /bin/sh -c 'echo "${a-"b  c"}"'
/bin/sh: bad substitution
$ ksh -c 'echo "${a-"b  c"}"'
b c
$ bash -c 'echo "${a-"b  c"}"'
b  c
$ /bin/sh -c 'a=; echo ${a+'\''b  c'\''}'
b  c
$ /bin/sh -c 'a=; echo "${a+'\''b  c'\''}"'
'b  c'
$ /bin/sh -c 'a=; echo "${a+\"b  c\"}"'
"b  c"
$ /bin/sh -c 'a=; echo "${a+b  c}"'
b  c
$ /bin/sh -c 'cat <<EOF
${a-"b  c"}
EOF'
"b  c"
$ /bin/sh -c 'cat <<EOF
${a-'b  c'}
EOF'
'b  c'
$ bash -c 'cat <<EOF
${a-"b  c"}
EOF'
b  c
$ bash -c 'cat <<EOF
${a-'b  c'}
EOF'
'b  c'

Perhaps the easiest way to work around quoting issues in a manner portable to all shells is to place the results in a temporary variable, then use ‘$t’ as the value, rather than trying to inline the expression needing quoting. 

$ /bin/sh -c 't="b  c\"'\''}\\"; echo "${a-$t}"'
b  c"'}\
$ ksh -c 't="b  c\"'\''}\\"; echo "${a-$t}"'
b  c"'}\
$ bash -c 't="b  c\"'\''}\\"; echo "${a-$t}"'
b  c"'}\
${var=value}

When using ‘${var=value}’ to assign a default value to var, remember that even though the assignment to var does not undergo file name expansion, the result of the variable expansion does unless the expansion occurred within double quotes. In particular, when using : followed by unquoted variable expansion for the side effect of setting a default value, if the final value of ‘$var’ contains any globbing characters (either from value or from prior contents), the shell has to spend time performing file name expansion and field splitting even though those results will not be used. Therefore, it is a good idea to consider double quotes when performing default initialization; while remembering how this impacts any quoting characters appearing in value. 

$ time bash -c ': "${a=/usr/bin/*}"; echo "$a"'
/usr/bin/*

real	0m0.005s
user	0m0.002s
sys	0m0.003s
$ time bash -c ': ${a=/usr/bin/*}; echo "$a"'
/usr/bin/*

real	0m0.039s
user	0m0.026s
sys	0m0.009s
$ time bash -c 'a=/usr/bin/*; : ${a=noglob}; echo "$a"'
/usr/bin/*

real	0m0.031s
user	0m0.020s
sys	0m0.010s

$ time bash -c 'a=/usr/bin/*; : "${a=noglob}"; echo "$a"'
/usr/bin/*

real	0m0.006s
user	0m0.002s
sys	0m0.003s

As with ‘+’ and ‘-’, value must be a single shell word, otherwise some shells, such as Solaris 10 /bin/sh or on Digital Unix V 5.0, die because of a “bad substitution”. Meanwhile, POSIX requires that with ‘=’, quote removal happens prior to the assignment, and the expansion be the final contents of var without quoting (and thus subject to field splitting), in contrast to the behavior with ‘-’ passing the quoting through to the final expansion. However, bash 4.1 does not obey this rule. 

$ ksh -c 'echo ${var-a\ \ b}'
a  b
$ ksh -c 'echo ${var=a\ \ b}'
a b
$ bash -c 'echo ${var=a\ \ b}'
a  b

Finally, POSIX states that when mixing ‘${a=b}’ with regular commands, it is unspecified whether the assignments affect the parent shell environment. It is best to perform assignments independently from commands, to avoid the problems demonstrated in this example running on Solaris 10: 

$ cmd='x= y=${x:=b} sh -c "echo +\$x+\$y+";printf "%s\\n" -$x-'
$ bash -c "$cmd"
+b+b+
-b-
$ /bin/sh -c "$cmd"
++b+
--
$ ksh -c "$cmd"
+b+b+
--
${var=value}

Solaris 10 /bin/sh has a frightening bug in its handling of literal assignments. Imagine you need set a variable to a string containing ‘}’. This ‘}’ character confuses Solaris 10 /bin/sh when the affected variable was already set. This bug can be exercised by running: 

$ unset foo
$ foo=${foo='}'}
$ echo $foo
}
$ foo=${foo='}'   # no error; this hints to what the bug is
$ echo $foo
}
$ foo=${foo='}'}
$ echo $foo
}}
 ^ ugh!

It seems that ‘}’ is interpreted as matching ‘${’, even though it is enclosed in single quotes. The problem doesn’t happen using double quotes, or when using a temporary variable holding the problematic string.

${var=expanded-value}

On shells so old that they are no longer relevant, the command 

# Set the shell variable to a default value
# if it is not already set.
: ${var="$default"}

misbehaved badly in some cases. Older scripts worked around the bugs by using one of following two lines, the latter of which was more portable: 

var=${var="$default"}
test ${var+y} || var=$default

However, these workarounds are no longer needed.

${#var}
${var%word}
${var%%word}
${var#word}
${var##word}

POSIX requires support for these usages, but they do not work with many traditional shells, e.g., Solaris 10 /bin/sh.

Also, pdksh 5.2.14 mishandles some word forms. For example if ‘$1’ is ‘a/b’ and ‘$2’ is ‘a’, then ‘${1#$2}’ should yield ‘/b’, but with pdksh it yields the empty string.

`commands`

POSIX requires shells to trim all trailing newlines from command output before substituting it, so assignments like ‘dir=`printf '%s\n' "$file" | tr a A`’ do not work as expected if ‘$file’ ends in a newline.

While in general it makes no sense, do not substitute a single builtin with side effects, because Ash 0.2, trying to optimize, does not fork a subshell to perform the command. For instance, if you wanted to check that cd is silent, do not use ‘test -z "`cd /`"’ because the following can happen: 

$ pwd
/tmp
$ test -z "`cd /`" && pwd
/

The result of ‘foo=`exit 1`’ is left as an exercise to the reader.

The MSYS shell leaves a stray byte in the expansion of a double-quoted command substitution of a native program, if the end of the substitution is not aligned with the end of the double quote. This may be worked around by inserting another pair of quotes: 

$ echo "`printf 'foo\r\n'` bar" > broken
$ echo "`printf 'foo\r\n'`"" bar" | cmp - broken
- broken differ: char 4, line 1

Upon interrupt or SIGTERM, some shells may abort a command substitution, replace it with a null string, and wrongly evaluate the enclosing command before entering the trap or ending the script. This can lead to spurious errors: 

$ sh -c 'if test `sleep 5; echo hi` = hi; then echo yes; fi'
$ ^C
sh: test: hi: unexpected operator/operand

You can avoid this by assigning the command substitution to a temporary variable: 

$ sh -c 'res=`sleep 5; echo hi`
         if test "x$res" = xhi; then echo yes; fi'
$ ^C
$(commands)

This construct is meant to replace ‘`commands`’, and it has most of the problems listed under `commands`.

This construct can be nested while this is impossible to do portably with back quotes. Although it is almost universally supported, unfortunately Solaris 10 and earlier releases lack it: 

$ showrev -c /bin/sh | grep version
Command version: SunOS 5.10 Generic 142251-02 Sep 2010
$ echo $(echo blah)
syntax error: `(' unexpected

If you do use ‘$(commands)’, make sure that the commands do not start with a parenthesis, as that would cause confusion with a different notation ‘$((expression))’ that in modern shells is an arithmetic expression not a command. To avoid the confusion, insert a space between the two opening parentheses.

Avoid commands that contain unbalanced parentheses in here-documents, comments, or case statement patterns, as many shells mishandle them. For example, Bash 3.1, ‘ksh88’, pdksh 5.2.14, and Zsh 4.2.6 all mishandle the following valid command: 

echo $(case x in x) echo hello;; esac)
$((expression))

Arithmetic expansion is not portable as some shells (most notably Solaris 10 /bin/sh) don’t support it.

Among shells that do support ‘$(( ))’, not all of them obey the POSIX rule that octal and hexadecimal constants must be recognized: 

$ bash -c 'echo $(( 010 + 0x10 ))'
24
$ zsh -c 'echo $(( 010 + 0x10 ))'
26
$ zsh -c 'emulate sh; echo $(( 010 + 0x10 ))'
24
$ pdksh -c 'echo $(( 010 + 0x10 ))'
pdksh:  010 + 0x10 : bad number `0x10'
$ pdksh -c 'echo $(( 010 ))'
10

When it is available, using arithmetic expansion provides a noticeable speedup in script execution; but testing for support requires eval to avoid syntax errors. The following construct is used by AS_VAR_ARITH to provide arithmetic computation when all arguments are decimal integers without leading zeros, and all operators are properly quoted and appear as distinct arguments: 

if ( eval 'test $(( 1 + 1 )) = 2' ) 2>/dev/null; then
  eval 'func_arith ()
  {
    func_arith_result=$(( $* ))
  }'
else
  func_arith ()
  {
    func_arith_result=`expr "$@"`
  }
fi
func_arith 1 + 1
foo=$func_arith_result
^

Always quote ‘^’, otherwise traditional shells such as /bin/sh on Solaris 10 treat this like ‘|’.

Next: Assignments, Previous: Shell Pattern Matching, Up: Portable Shell Programming   [Contents][Index]