No, no, we are serious: some shells do have limitations! :)
You should always keep in mind that any builtin or command may support
options, and therefore differ in behavior with arguments
starting with a dash. For instance, even the innocent ‘echo "$word"’
can give unexpected results when word starts with a dash. It is
often possible to avoid this problem using ‘echo "x$word"’, taking
the ‘x’ into account later in the pipe. Many of these limitations
can be worked around using M4sh (see Programming in M4sh).
Not all shells gracefully handle syntax errors within a sourced file. On one extreme, some non-interactive shells abort the entire script. On the other, zsh 4.3.10 has a bug where it fails to react to the syntax error.
$ echo 'fi' > syntax
$ bash -c '. ./syntax; echo $?'
./syntax: line 1: syntax error near unexpected token `fi'
./syntax: line 1: `fi'
1
$ ash -c '. ./syntax; echo $?'
./syntax: 1: Syntax error: "fi" unexpected
$ zsh -c '. ./syntax; echo $?'
./syntax:1: parse error near `fi'
0
$ sh -c '! : | :'; echo $?
1
$ ash -c '! : | :'; echo $?
0
$ sh -c '! { :; }'; echo $?
1
$ ash -c '! { :; }'; echo $?
{: not found
Syntax error: "}" unexpected
2
Shell code like this:
if ! cmp file1 file2 >/dev/null 2>&1; then
echo files differ or trouble
fi
is therefore not portable in practice. Typically it is easy to rewrite such code, e.g.:
cmp file1 file2 >/dev/null 2>&1 ||
echo files differ or trouble
More generally, one can always rewrite ‘! command’ as:
if command; then (exit 1); else :; fi
$ bash -c '{ echo foo; } >/bad; echo $?'
bash: line 1: /bad: Permission denied
0
$ bash -c 'while :; do echo; done >/bad; echo $?'
bash: line 1: /bad: Permission denied
0
To work around the bug, prepend ‘:;’:
$ bash -c ':;{ echo foo; } >/bad; echo $?'
bash: line 1: /bad: Permission denied
1
Posix requires a syntax error if a brace list has no contents. However, not all shells obey this rule; and on shells where empty lists are permitted, the effect on ‘$?’ is inconsistent. To avoid problems, ensure that a brace list is never empty.
$ bash -c 'false; { }; echo $?' || echo $?
bash: line 1: syntax error near unexpected token `}'
bash: line 1: `false; { }; echo $?'
2
$ zsh -c 'false; { }; echo $?' || echo $?
1
$ pdksh -c 'false; { }; echo $?' || echo $?
0
You don't need the final ‘;;’, but you should use it.
Posix requires support for case patterns with opening
parentheses like this:
case $file_name in
(*.c) echo "C source code";;
esac
but the ( in this example is not portable to many Bourne
shell implementations, which is a pity for those of us using tools that
rely on balanced parentheses. For instance, with Solaris
/bin/sh:
$ case foo in (foo) echo foo;; esac
error-->syntax error: `(' unexpected
The leading ‘(’ can be omitted safely. Unfortunately, there are contexts where unbalanced parentheses cause other problems, such as when using a syntax-highlighting editor that searches for the balancing counterpart, or more importantly, when using a case statement as an underquoted argument to an Autoconf macro. See Balancing Parentheses, for tradeoffs involved in various styles of dealing with unbalanced ‘)’.
Zsh handles pattern fragments derived from parameter expansions or command substitutions as though quoted:
$ pat=\?; case aa in ?$pat) echo match;; esac
$ pat=\?; case a? in ?$pat) echo match;; esac
match
Because of a bug in its fnmatch, Bash fails to properly
handle backslashes in character classes:
bash-2.02$ case /tmp in [/\\]*) echo OK;; esac
bash-2.02$
This is extremely unfortunate, since you are likely to use this code to handle Posix or ms-dos absolute file names. To work around this bug, always put the backslash first:
bash-2.02$ case '\TMP' in [\\/]*) echo OK;; esac
OK
bash-2.02$ case /tmp in [\\/]*) echo OK;; esac
OK
Many Bourne shells cannot handle closing brackets in character classes correctly.
Some shells also have problems with backslash escaping in case you do not want to match the backslash: both a backslash and the escaped character match this pattern. To work around this, specify the character class in a variable, so that quote removal does not apply afterwards, and the special characters don't have to be backslash-escaped:
$ case '\' in [\<]) echo OK;; esac
OK
$ scanset='[<]'; case '\' in $scanset) echo OK;; esac
$
Even with this, Solaris ksh matches a backslash if the set contains any of the characters ‘|’, ‘&’, ‘(’, or ‘)’.
Conversely, Tru64 ksh (circa 2003) erroneously always matches a closing parenthesis if not specified in a character class:
$ case foo in *\)*) echo fail ;; esac
fail
$ case foo in *')'*) echo fail ;; esac
fail
Some shells, such as Ash 0.3.8, are confused by an empty
case/esac:
ash-0.3.8 $ case foo in esac;
error-->Syntax error: ";" unexpected (expecting ")")
Posix requires case to give an exit status of 0 if no cases
match. However, /bin/sh in Solaris 10 does not obey this
rule. Meanwhile, it is unclear whether a case that matches, but
contains no statements, must also change the exit status to 0. The M4sh
macro AS_CASE works around these inconsistencies.
$ bash -c 'case `false` in ?) ;; esac; echo $?'
0
$ /bin/sh -c 'case `false` in ?) ;; esac; echo $?'
255
Portable scripts should assume neither option is supported, and should
assume neither behavior is the default. This can be a bit tricky,
since the Posix default behavior means that, for example,
‘ls ..’ and ‘cd ..’ may refer to different directories if
the current logical directory is a symbolic link. It is safe to use
cd dir if dir contains no .. components.
Also, Autoconf-generated scripts check for this problem when computing
variables like ac_top_srcdir (see Configuration Actions),
so it is safe to cd to these variables.
See See Special Shell Variables, for portability problems involving cd and the CDPATH environment variable. Also please see the discussion of the pwd command.
Do not use backslashes in the arguments, as there is no consensus on their handling. For ‘echo '\n' | wc -l’, the sh of Solaris outputs 2, but Bash and Zsh (in sh emulation mode) output 1. The problem is truly echo: all the shells understand ‘'\n'’ as the string composed of a backslash and an ‘n’. Within a command substitution, ‘echo 'string\c'’ will mess up the internal state of ksh88 on AIX 6.1 so that it will print the first character ‘s’ only, followed by a newline, and then entirely drop the output of the next echo in a command substitution.
Because of these problems, do not pass a string containing arbitrary characters to echo. For example, ‘echo "$foo"’ is safe only if you know that foo's value cannot contain backslashes and cannot start with ‘-’.
If this may not be true, printf is in general safer and easier to use than echo and echo -n. Thus, scripts where portability is not a major concern should use printf '%s\n' whenever echo could fail, and similarly use printf %s instead of echo -n. For portable shell scripts, instead, it is suggested to use a here-document like this:
cat <<EOF
$foo
EOF
Alternatively, M4sh provides AS_ECHO and AS_ECHO_N macros
which choose between various portable implementations: ‘echo’
or ‘print’ where they work, printf if it is available,
or else other creative tricks in order to work around the above problems.
You should also be wary of common bugs in eval implementations. In some shell implementations (e.g., older ash, OpenBSD 3.8 sh, pdksh v5.2.14 99/07/13.2, and zsh 4.2.5), the arguments of ‘eval’ are evaluated in a context where ‘$?’ is 0, so they exhibit behavior like this:
$ false; eval 'echo $?'
0
The correct behavior here is to output a nonzero value, but portable scripts should not rely on this.
You should not rely on LINENO within eval.
See Special Shell Variables.
Note that, even though these bugs are easily avoided, eval is tricky to use on arbitrary arguments. It is obviously unwise to use ‘eval $cmd’ if the string value of ‘cmd’ was derived from an untrustworthy source. But even if the string value is valid, ‘eval $cmd’ might not work as intended, since it causes field splitting and file name expansion to occur twice, once for the eval and once for the command itself. It is therefore safer to use ‘eval "$cmd"’. For example, if cmd has the value ‘cat test?.c’, ‘eval $cmd’ might expand to the equivalent of ‘cat test;.c’ if there happens to be a file named test;.c in the current directory; and this in turn mistakenly attempts to invoke cat on the file test and then execute the command .c. To avoid this problem, use ‘eval "$cmd"’ rather than ‘eval $cmd’.
However, suppose that you want to output the text of the evaluated
command just before executing it. Assuming the previous example,
‘echo "Executing: $cmd"’ outputs ‘Executing: cat test?.c’, but
this output doesn't show the user that ‘test;.c’ is the actual name
of the copied file. Conversely, ‘eval "echo Executing: $cmd"’
works on this example, but it fails with ‘cmd='cat foo >bar'’,
since it mistakenly replaces the contents of bar by the
string ‘cat foo’. No simple, general, and portable solution to
this problem is known.
$ sh -c 'exec cd /tmp'
sh: line 0: exec: cd: not found
All other built-ins that provide utilities specified by Posix must have a counterpart executable that exists on PATH, although Posix allows exec to use the built-in instead of the executable. For example, contrast bash 3.2 and pdksh 5.2.14:
$ bash -c 'pwd --version' | head -n1
bash: line 0: pwd: --: invalid option
pwd: usage: pwd [-LP]
$ bash -c 'exec pwd --version' | head -n1
pwd (GNU coreutils) 6.10
$ pdksh -c 'exec pwd --version' | head -n1
pdksh: pwd: --: unknown option
When it is desired to avoid a regular shell built-in, the workaround is to use some other forwarding command, such as env or nice, that will ensure a path search:
$ pdksh -c 'exec true --version' | head -n1
$ pdksh -c 'nice true --version' | head -n1
true (GNU coreutils) 6.10
$ pdksh -c 'env true --version' | head -n1
true (GNU coreutils) 6.10
$?;
unfortunately, some shells, such as the DJGPP port of Bash 2.04, just
perform ‘exit 0’.
bash-2.04$ foo=`exit 1` || echo fail
fail
bash-2.04$ foo=`(exit 1)` || echo fail
fail
bash-2.04$ foo=`(exit 1); exit` || echo fail
bash-2.04$
Using ‘exit $?’ restores the expected behavior.
Some shell scripts, such as those generated by autoconf, use a trap to clean up before exiting. If the last shell command exited with nonzero status, the trap also exits with nonzero status so that the invoker can tell that an error occurred.
Unfortunately, in some shells, such as Solaris /bin/sh, an exit
trap ignores the exit command's argument. In these shells, a trap
cannot determine whether it was invoked by plain exit or by
exit 1. Instead of calling exit directly, use the
AC_MSG_ERROR macro that has a workaround for this problem.
Alas, many shells, such as Solaris /bin/sh, irix 6.3, irix 5.2, AIX 4.1.5, and Digital Unix 4.0, forget to export the environment variables they receive. As a result, two variables coexist: the environment variable and the shell variable. The following code demonstrates this failure:
#!/bin/sh
echo $FOO
FOO=bar
echo $FOO
exec /bin/sh $0
when run with ‘FOO=foo’ in the environment, these shells print alternately ‘foo’ and ‘bar’, although they should print only ‘foo’ and then a sequence of ‘bar’s.
Therefore you should export again each environment variable that you update; the export can occur before or after the assignment.
Posix is not clear on whether the export of an undefined variable causes the variable to be defined with the value of an empty string, or merely marks any future definition of a variable by that name for export. Various shells behave differently in this regard:
$ sh -c 'export foo; env | grep foo'
$ ash -c 'export foo; env | grep foo'
foo=
for arg
do
echo "$arg"
done
You may not leave the do on the same line as for,
since some shells improperly grok:
for arg; do
echo "$arg"
done
If you want to explicitly refer to the positional arguments, given the ‘$@’ bug (see Shell Substitutions), use:
for arg in ${1+"$@"}; do
echo "$arg"
done
But keep in mind that Zsh, even in Bourne shell emulation mode, performs word splitting on ‘${1+"$@"}’; see Shell Substitutions, item ‘$@’, for more.
if ! cmp -s file file.new; then
mv file.new file
fi
use:
if cmp -s file file.new; then :; else
mv file.new file
fi
Or, especially if the else branch is short, you can use ||.
In M4sh, the AS_IF macro provides an easy way to write these kinds
of conditionals:
AS_IF([cmp -s file file.new], [], [mv file.new file])
This is especially useful in other M4 macros, where the then and else branches might be macro arguments.
Some very old shells did not reset the exit status from an if with no else:
$ if (exit 42); then true; fi; echo $?
42
whereas a proper shell should have printed ‘0’. But this is no longer a portability problem; any shell that supports functions gets it correct. However, it explains why some makefiles have lengthy constructs:
if test -f "$file"; then
install "$file" "$dest"
else
:
fi
printf %s -foo
Bash 2.03 mishandles an escape sequence that happens to evaluate to ‘%’:
$ printf '\045'
bash: printf: `%': missing format character
Large outputs may cause trouble. On Solaris 2.5.1 through 10, for example, /usr/bin/printf is buggy, so when using /bin/sh the command ‘printf %010000x 123’ normally dumps core.
Since printf is not always a shell builtin, there is a
potential speed penalty for using printf '%s\n' as a replacement
for an echo that does not interpret ‘\’ or leading
‘-’. With Solaris ksh, it is possible to use print
-r -- for this role instead.
For a discussion of portable alternatives to both printf
and echo, See Limitations of Shell Builtins.
Posix 1003.1-2001 requires that pwd must support the -L (“logical”) and -P (“physical”) options, with -L being the default. However, traditional shells do not support these options, and their pwd command has the -P behavior.
Portable scripts should assume neither option is supported, and should assume neither behavior is the default. Also, on many hosts ‘/bin/pwd’ is equivalent to ‘pwd -P’, but Posix does not require this behavior and portable scripts should not rely on it.
Typically it's best to use plain pwd. On modern hosts this outputs logical directory names, which have the following advantages:
Also please see the discussion of the cd command.
The set builtin faces the usual problem with arguments starting with a dash. Modern shells such as Bash or Zsh understand -- to specify the end of the options (any argument after -- is a parameter, even ‘-x’ for instance), but many traditional shells (e.g., Solaris 10 /bin/sh) simply stop option processing as soon as a non-option argument is found. Therefore, use ‘dummy’ or simply ‘x’ to end the option processing, and use shift to pop it out:
set x $my_list; shift
Avoid ‘set -’, e.g., ‘set - $my_list’. Posix no longer requires support for this command, and in traditional shells ‘set - $my_list’ resets the -v and -x options, which makes scripts harder to debug.
Some nonstandard shells do not recognize more than one option (e.g., ‘set -e -x’ assigns ‘-x’ to the command line). It is better to combine them:
set -ex
The option -e has historically been underspecified, with enough
ambiguities to cause numerous differences across various shell
implementations. Perhaps the best reference is
this link, recommending a change to Posix 2008 to match ksh88
behavior. Note that mixing set -e and shell functions is asking
for surprises:
set -e
doit()
{
rm file
echo one
}
doit || echo two
According to the recommendation, ‘one’ should always be output regardless of whether the rm failed, because it occurs within the body of the shell function ‘doit’ invoked on the left side of ‘||’, where the effects of ‘set -e’ are not enforced. Likewise, ‘two’ should never be printed, since the failure of rm does not abort the function, such that the status of ‘doit’ is 0.
The BSD shell has had several problems with the -e option. Older versions of the BSD shell (circa 1990) mishandled ‘&&’, ‘||’, ‘if’, and ‘case’ when -e was in effect, causing the shell to exit unexpectedly in some cases. This was particularly a problem with makefiles, and led to circumlocutions like ‘sh -c 'test -f file || touch file'’, where the seemingly-unnecessary ‘sh -c '...'’ wrapper works around the bug (see Failure in Make Rules).
Even relatively-recent versions of the BSD shell (e.g., OpenBSD 3.4) wrongly exit with -e if a command within ‘&&’ fails inside a compound statement. For example:
#! /bin/sh
set -e
foo=''
test -n "$foo" && exit 1
echo one
if :; then
test -n "$foo" && exit 1
fi
echo two
does not print ‘two’. One workaround is to use ‘if test -n "$foo"; then exit 1; fi’ rather than ‘test -n "$foo" && exit 1’. Another possibility is to warn BSD users not to use ‘sh -e’.
When ‘set -e’ is in effect, a failed command substitution in Solaris /bin/sh cannot be ignored, even with ‘||’.
$ /bin/sh -c 'set -d; foo=`false` || echo foo; echo bar'
$ bash -c 'set -d; foo=`false` || echo foo; echo bar'
foo
bar
Portable scripts should not use ‘set -e’ if trap is used to install an exit handler. This is because Tru64/OSF 5.1 sh sometimes enters the trap handler with the exit status of the command prior to the one that triggered the errexit handler:
$ sh -ec 'trap '\''echo $?'\'' 0; false'
0
$ sh -c 'set -e; trap '\''echo $?'\'' 0; false'
1
Thus, when writing a script in M4sh, rather than trying to rely on ‘set -e’, it is better to append ‘|| AS_EXIT’ to any statement where it is desirable to abort on failure.
Job control is not provided by all shells, so the use of ‘set -m’ or ‘set -b’ must be done with care. When using zsh in native mode, asynchronous notification (‘set -b’) is enabled by default, and using ‘emulate sh’ to switch to Posix mode does not clear this setting (although asynchronous notification has no impact unless job monitoring is also enabled). Also, zsh 4.3.10 and earlier have a bug where job control can be manipulated in interactive shells, but not in subshells or scripts. Furthermore, some shells, like pdksh, fail to treat subshells as interactive, even though the parent shell was.
$ echo $ZSH_VERSION
4.3.10
$ set -m; echo $?
0
$ zsh -c 'set -m; echo $?'
set: can't change option: -m
$ (set -m); echo $?
set: can't change option: -m
1
$ pdksh -ci 'echo $-; (echo $-)'
cim
c
Don't use ‘shift 2’ etc.; while it in the SVR1 shell (1983),
it is also absent in many pre-Posix shells.
test program is the way to perform many file and string
tests. It is often invoked by the alternate name ‘[’, but using
that name in Autoconf code is asking for trouble since it is an M4 quote
character.
The -a, -o, ‘(’, and ‘)’ operands are not present in all implementations, and have been marked obsolete by Posix 2008. This is because there are inherent ambiguities in using them. For example, ‘test "$1" -a "$2"’ looks like a binary operator to check whether two strings are both non-empty, but if ‘$1’ is the literal ‘!’, then some implementations of test treat it as a negation of the unary operator -a.
Thus, portable uses of test should never have more than four arguments, and scripts should use shell constructs like ‘&&’ and ‘||’ instead. If you combine ‘&&’ and ‘||’ in the same statement, keep in mind that they have equal precedence, so it is often better to parenthesize even when this is redundant. For example:
# Not portable:
test "X$a" = "X$b" -a \
'(' "X$c" != "X$d" -o "X$e" = "X$f" ')'
# Portable:
test "X$a" = "X$b" &&
{ test "X$c" != "X$d" || test "X$e" = "X$f"; }
test does not process options like most other commands do; for example, it does not recognize the -- argument as marking the end of options.
It is safe to use ‘!’ as a test operator. For example,
‘if test ! -d foo; ...’ is portable even though ‘if ! test
-d foo; ...’ is not.
/bin/sh support only -h.
Posix also says that ‘test ! "string"’, ‘test -n "string"’ and ‘test -z "string"’ work with any string, but many shells (such as Solaris, AIX 3.2, unicos 10.0.0.6, Digital Unix 4, etc.) get confused if string looks like an operator:
$ test -n =
test: argument expected
$ test ! -n
test: argument expected
Similarly, Posix says that both ‘test "string1" = "string2"’ and ‘test "string1" != "string2"’ work for any pairs of strings, but in practice this is not true for troublesome strings that look like operators or parentheses, or that begin with ‘-’.
It is best to protect such strings with a leading ‘X’, e.g., ‘test "Xstring" != X’ rather than ‘test -n "string"’ or ‘test ! "string"’.
It is common to find variations of the following idiom:
test -n "`echo $ac_feature | sed 's/[-a-zA-Z0-9_]//g'`" &&
action
to take an action when a token matches a given pattern. Such constructs should be avoided by using:
case $ac_feature in
*[!-a-zA-Z0-9_]*) action;;
esac
If the pattern is a complicated regular expression that cannot be expressed as a shell pattern, use something like this instead:
expr "X$ac_feature" : 'X.*[^-a-zA-Z0-9_]' >/dev/null &&
action
‘expr "Xfoo" : "Xbar"’ is more robust than ‘echo "Xfoo" | grep "^Xbar"’, because it avoids problems when ‘foo’ contains backslashes.
Posix says that ‘trap - 1 2 13 15’ resets the traps for the specified signals to their default values, but many common shells (e.g., Solaris /bin/sh) misinterpret this and attempt to execute a “command” named - when the specified conditions arise. Posix 2008 also added a requirement to support ‘trap 1 2 13 15’ to reset traps, as this is supported by a larger set of shells, but there are still shells like dash that mistakenly try to execute 1 instead of resetting the traps. Therefore, there is no portable workaround, except for ‘trap - 0’, for which ‘trap '' 0’ is a portable substitute.
Although Posix is not absolutely clear on this point, it is widely admitted that when entering the trap ‘$?’ should be set to the exit status of the last command run before the trap. The ambiguity can be summarized as: “when the trap is launched by an exit, what is the last command run: that before exit, or exit itself?”
Bash considers exit to be the last command, while Zsh and Solaris /bin/sh consider that when the trap is run it is still in the exit, hence it is the previous exit status that the trap receives:
$ cat trap.sh
trap 'echo $?' 0
(exit 42); exit 0
$ zsh trap.sh
42
$ bash trap.sh
0
The portable solution is then simple: when you want to ‘exit 42’,
run ‘(exit 42); exit 42’, the first exit being used to
set the exit status to 42 for Zsh, and the second to trigger the trap
and pass 42 as exit status for Bash. In M4sh, this is covered by using
AS_EXIT.
The shell in FreeBSD 4.0 has the following bug: ‘$?’ is reset to 0 by empty lines if the code is inside trap.
$ trap 'false
echo $?' 0
$ exit
0
Fortunately, this bug only affects trap.
Several shells fail to execute an exit trap that is defined inside a subshell, when the last command of that subshell is not a builtin. A workaround is to use ‘exit $?’ as the shell builtin.
$ bash -c '(trap "echo hi" 0; /bin/true)'
hi
$ /bin/sh -c '(trap "echo hi" 0; /bin/true)'
$ /bin/sh -c '(trap "echo hi" 0; /bin/true; exit $?)'
hi
Likewise, older implementations of bash failed to preserve ‘$?’ across an exit trap consisting of a single cleanup command.
$ bash -c 'trap "/bin/true" 0; exit 2'; echo $?
2
$ bash-2.05b -c 'trap "/bin/true" 0; exit 2'; echo $?
0
$ bash-2.05b -c 'trap ":; /bin/true" 0; exit 2'; echo $?
2
In a sense, yes, because if it doesn't exist, the shell will produce an exit status of failure, which is correct for false, but not for true.
unset FOO fails
when FOO is not set. You can use
FOO=; unset FOO
if you are not sure that FOO is set.
A few ancient shells lack unset entirely. For some variables
such as PS1, you can use a neutralizing value instead:
PS1='$ '
Usually, shells that do not support unset need less effort to
make the environment sane, so for example is not a problem if you cannot
unset CDPATH on those shells. However, Bash 2.01 mishandles
unset MAIL in some cases and dumps core. So, you should do
something like
( (unset MAIL) || exit 1) >/dev/null 2>&1 && unset MAIL || :
See Special Shell Variables, for some neutralizing values. Also, see
Limitations of Builtins, for
the case of environment variables.