GNU Findutils

2.3.1 Base Name Patterns

Test: -name pattern ¶

Test: -iname pattern ¶

True if the base of the file name (the path with the leading directories removed) matches shell pattern pattern. For ‘-iname’, the match is case-insensitive.¹ To ignore a whole directory tree, use ‘-prune’ (see Directories). As an example, to find Texinfo source files in /usr/local/doc:

find /usr/local/doc -name '*.texi'

Notice that the wildcard must be enclosed in quotes in order to protect it from expansion by the shell.

As of findutils version 4.2.2, patterns for ‘-name’ and ‘-iname’ match a file name with a leading ‘.’. For example the command ‘find /tmp -name \*bar’ match the file /tmp/.foobar. Braces within the pattern (‘{}’) are not considered to be special (that is, find . -name 'foo{1,2}' matches a file named foo{1,2}, not the files foo1 and foo2.

Because the leading directories are removed, the file names considered for a match with ‘-name’ will never include a slash, so ‘-name a/b’ will never match anything (you probably need to use ‘-path’ instead).

2.3.2 Full Name Patterns

Test: -path pattern ¶

Test: -wholename pattern ¶

True if the entire file name, starting with the command line argument under which the file was found, matches shell pattern pattern. To ignore a whole directory tree, use ‘-prune’ rather than checking every file in the tree (see Directories). The “entire file name” as used by find starts with the starting-point specified on the command line, and is not converted to an absolute pathname, so for example cd /; find tmp -wholename /tmp will never match anything.

Find compares the ‘-path’ argument with the concatenation of a directory name and the base name of the file it’s considering. Since the concatenation will never end with a slash, ‘-path’ arguments ending in ‘/’ will match nothing (except perhaps a start point specified on the command line).

The name ‘-wholename’ is GNU-specific, but ‘-path’ is more portable; it is supported by HP-UX find and is part of the POSIX 2008 standard.

Test: -ipath pattern ¶

Test: -iwholename pattern ¶

These tests are like ‘-wholename’ and ‘-path’, but the match is case-insensitive.

In the context of the tests ‘-path’, ‘-wholename’, ‘-ipath’ and ‘-iwholename’, a “full path” is the name of all the directories traversed from find’s start point to the file being tested, followed by the base name of the file itself. These paths are often not absolute paths; for example

$ cd /tmp
$ mkdir -p foo/bar/baz
$ find foo -path foo/bar -print
foo/bar
$ find foo -path /tmp/foo/bar -print
$ find /tmp/foo -path /tmp/foo/bar -print
/tmp/foo/bar

Notice that the second find command prints nothing, even though /tmp/foo/bar exists and was examined by find.

Unlike file name expansion on the command line, a ‘*’ in the pattern will match both ‘/’ and leading dots in file names:

$ find .  -path '*f'
./quux/bar/baz/f
$ find .  -path '*/*config'
./quux/bar/baz/.config

Test: -regex expr ¶

Test: -iregex expr ¶

True if the entire file name matches regular expression expr. This is a match on the whole path, not a search. For example, to match a file named ./fubar3, you can use the regular expression ‘.*bar.’ or ‘.*b.*3’, but not ‘f.*r3’. For ‘-iregex’, the match is case-insensitive.

As for ‘-path’, the candidate file name never ends with a slash, so regular expressions which only match something that ends in slash will always fail.

There are several varieties of regular expressions; by default this test uses GNU Emacs regular expressions, but this can be changed with the option ‘-regextype’.

Option: -regextype name ¶

This option controls the variety of regular expression syntax understood by the ‘-regex’ and ‘-iregex’ tests. This option is positional; that is, it only affects regular expressions which occur later in the command line. If this option is not given, GNU Emacs regular expressions are assumed. Currently-implemented types are

‘emacs’

Regular expressions compatible with GNU Emacs; this is also the default behaviour if this option is not used.

‘posix-awk’

Regular expressions compatible with the POSIX awk command (not GNU awk)

‘posix-basic’

POSIX Basic Regular Expressions.

‘posix-egrep’

Regular expressions compatible with the POSIX egrep command

‘posix-extended’

POSIX Extended Regular Expressions

Regular Expressions for more information on the regular expression dialects understood by GNU findutils.

2.3.3 Fast Full Name Search

To search for files by name without having to actually scan the directories on the disk (which can be slow), you can use the locate program. For each shell pattern you give it, locate searches one or more databases of file names and displays the file names that contain the pattern. See Shell Pattern Matching, for details about shell patterns.

If a pattern is a plain string – it contains no metacharacters – locate displays all file names in the database that contain that string. If a pattern contains metacharacters, locate only displays file names that match the pattern exactly. As a result, patterns that contain metacharacters should usually begin with a ‘*’, and will most often end with one as well. The exceptions are patterns that are intended to explicitly match the beginning or end of a file name.

If you only want locate to match against the last component of the file names (the “base name” of the files) you can use the ‘--basename’ option. The opposite behaviour is the default, but can be selected explicitly by using the option ‘--wholename’.

The command

locate pattern

is almost equivalent to

find directories -name pattern

where directories are the directories for which the file name databases contain information. The differences are that the locate information might be out of date, and that by default locate matches wildcards against the whole file name (not just its base name) (see Shell Pattern Matching).

The file name databases contain lists of files that were on the system when the databases were last updated. The system administrator can choose the file name of the default database, the frequency with which the databases are updated, and the directories for which they contain entries.

Here is how to select which file name databases locate searches. The default is system-dependent. At the time this document was generated, the default was /usr/local/var/locatedb.

--database=path

-d path

Instead of searching the default file name database, search the file name databases in path, which is a colon-separated list of database file names. You can also use the environment variable LOCATE_PATH to set the list of database files to search. The option overrides the environment variable if both are used.

GNU locate can read file name databases generated by the slocate package. However, these generally contain a list of all the files on the system, and so when using this database, locate will produce output only for files which are accessible to you. See Invoking locate, for a description of the ‘--existing’ option which is used to do this.

The updatedb program can also generate database in a format compatible with slocate. See Invoking updatedb, for a description of its ‘--dbformat’ and ‘--output’ options.

2.3.4 Shell Pattern Matching

find and locate can compare file names, or parts of file names, to shell patterns. A shell pattern is a string that may contain the following special characters, which are known as wildcards or metacharacters.

You must quote patterns that contain metacharacters to prevent the shell from expanding them itself. Double and single quotes both work; so does escaping with a backslash.

*

Matches any zero or more characters.

?

Matches any one character.

[string]

Matches exactly one character that is a member of the string string. This is called a character class. As a shorthand, string may contain ranges, which consist of two characters with a dash between them. For example, the class ‘[a-z0-9_]’ matches a lowercase letter, a number, or an underscore. You can negate a class by placing a ‘!’ or ‘^’ immediately after the opening bracket. Thus, ‘[^A-Z@]’ matches any character except an uppercase letter or an at sign.

\

Removes the special meaning of the character that follows it. This works even in character classes.

In the find tests that do shell pattern matching (‘-name’, ‘-wholename’, etc.), wildcards in the pattern will match a ‘.’ at the beginning of a file name. This is also the case for locate. Thus, ‘find -name '*macs'’ will match a file named .emacs, as will ‘locate '*macs'’.

Slash characters have no special significance in the shell pattern matching that find and locate do, unlike in the shell, in which wildcards do not match them. Therefore, a pattern ‘foo*bar’ can match a file name ‘foo3/bar’, and a pattern ‘./sr*sc’ can match a file name ‘./src/misc’.

If you want to locate some files with the ‘locate’ command but don’t need to see the full list you can use the ‘--limit’ option to see just a small number of results, or the ‘--count’ option to display only the total number of matches.

2.4.1 Symbolic Links

Symbolic links are names that reference other files. GNU find will handle symbolic links in one of two ways; firstly, it can dereference the links for you - this means that if it comes across a symbolic link, it examines the file that the link points to, in order to see if it matches the criteria you have specified. Secondly, it can check the link itself in case you might be looking for the actual link. If the file that the symbolic link points to is also within the directory hierarchy you are searching with the find command, you may not see a great deal of difference between these two alternatives.

By default, find examines symbolic links themselves when it finds them (and, if it later comes across the linked-to file, it will examine that, too). If you would prefer find to dereference the links and examine the file that each link points to, specify the ‘-L’ option to find. You can explicitly specify the default behaviour by using the ‘-P’ option. The ‘-H’ option is a half-way-between option which ensures that any symbolic links listed on the command line are dereferenced, but other symbolic links are not.

Symbolic links are different from “hard links” in the sense that you need permission to search the directories in the linked-to file name to dereference the link. This can mean that even if you specify the ‘-L’ option, find may not be able to determine the properties of the file that the link points to (because you don’t have sufficient permission). In this situation, find uses the properties of the link itself. This also occurs if a symbolic link exists but points to a file that is missing, or where the symbolic link points to itself (directly or indirectly).

The options controlling the behaviour of find with respect to links are as follows:

‘-P’

find does not dereference symbolic links at all. This is the default behaviour. This option must be specified before any of the file names on the command line.

‘-H’

find does not dereference symbolic links (except in the case of file names on the command line, which are dereferenced). If a symbolic link cannot be dereferenced, the information for the symbolic link itself is used. This option must be specified before any of the file names on the command line.

‘-L’

find dereferences symbolic links where possible, and where this is not possible it uses the properties of the symbolic link itself. This option must be specified before any of the file names on the command line. Use of this option also implies the same behaviour as the ‘-noleaf’ option. If you later use the ‘-H’ or ‘-P’ options, this does not turn off ‘-noleaf’.

Actions that can cause symbolic links to become broken while ‘find’ is executing (for example ‘-delete’) can give rise to confusing behaviour. Take for example the command line ‘find -L . -type d -delete’. This will delete empty directories. If a subtree includes only directories and symbolic links to directoires, this command may still not successfully delete it, since deletion of the target of the symbolic link will cause the symbolic link to become broken and ‘-type d’ is false for broken symbolic links.

‘-follow’

This option forms part of the “expression” and must be specified after the file names, but it is otherwise equivalent to ‘-L’. The ‘-follow’ option affects only those tests which appear after it on the command line. This option is deprecated. Where possible, you should use ‘-L’ instead.

The following differences in behaviour occur when the ‘-L’ option is used:

• find follows symbolic links to directories when searching directory trees.
• ‘-lname’ and ‘-ilname’ always return false (unless they happen to match broken symbolic links).
• ‘-type’ reports the types of the files that symbolic links point to. This means that in combination with ‘-L’, ‘-type l’ will be true only for broken symbolic links. To check for symbolic links when ‘-L’ has been specified, use ‘-xtype l’.
• Implies ‘-noleaf’ (see Directories).

If the ‘-L’ option or the ‘-H’ option is used, the file names used as arguments to ‘-newer’, ‘-anewer’, and ‘-cnewer’ are dereferenced and the timestamp from the pointed-to file is used instead (if possible – otherwise the timestamp from the symbolic link is used).

Test: -lname pattern ¶

Test: -ilname pattern ¶

True if the file is a symbolic link whose contents match shell pattern pattern. For ‘-ilname’, the match is case-insensitive. See Shell Pattern Matching, for details about the pattern argument. If the ‘-L’ option is in effect, this test will always return false for symbolic links unless they are broken. So, to list any symbolic links to sysdep.c in the current directory and its subdirectories, you can do:

find . -lname '*sysdep.c'

2.4.2 Hard Links

Hard links allow more than one name to refer to the same file on a file system, i.e., to the same inode. To find all the names which refer to the same file as name, use ‘-samefile name’.

Test: -samefile name ¶

True if the file is a hard link to the same inode as name. This implies that name and the file reside on the same file system, i.e., they have the same device number.

Unless the ‘-L’ option is also given to follow symbolic links, one may confine the search to one file system by using the ‘-xdev’ option. This is useful because hard links cannot point outside a single file system, so this can cut down on needless searching.

If the ‘-L’ option is in effect, then dereferencing of symbolic links applies both to the name argument of the ‘-samefile’ primary and to each file examined during the traversal of the directory hierarchy. Therefore, ‘find -L -samefile name’ will find both hard links and symbolic links pointing to the file referenced by name.

find also allows searching for files by inode number.

This can occasionally be useful in diagnosing problems with file systems; for example, fsck and lsof tend to print inode numbers. Inode numbers also occasionally turn up in log messages for some types of software.

You can learn a file’s inode number and the number of links to it by running ‘ls -li’, ‘stat’ or ‘find -ls’.

You can search for hard links to inode number NUM by using ‘-inum NUM’. If there are any file system mount points below the directory where you are starting the search, use the ‘-xdev’ option unless you are also using the ‘-L’ option. Using ‘-xdev’ saves needless searching, since hard links to a file must be on the same file system. See Filesystems.

Test: -inum n ¶

True if the file has inode number n. The ‘+’ and ‘-’ qualifiers also work, though these are rarely useful.

Please note that the ‘-inum’ primary simply compares the inode number against the given n. This means that a search for a certain inode number in several file systems may return several files with that inode number, but as each file system has its own device number, those files are not necessarily hard links to the same file.

Therefore, it is much of the time easier to use ‘-samefile’ rather than this option.

find also allows searching for files that have a certain number of links, with ‘-links’.

A directory normally has at least two hard links: the entry named in its parent directory, and the . entry inside of the directory. If a directory has subdirectories, each of those also has a hard link called .. to its parent directory.

The . and .. directory entries are not normally searched unless they are mentioned on the find command line.

Test: -links n ¶

File has n hard links.

Test: -links +n ¶

File has more than n hard links.

Test: -links -n ¶

File has fewer than n hard links.

2.5.1 Age Ranges

These tests are mainly useful with ranges (‘+n’ and ‘-n’).

Test: -atime n ¶

Test: -ctime n ¶

Test: -mtime n ¶

True if the file was last accessed (or its status changed, or it was modified) n*24 hours ago. The number of 24-hour periods since the file’s timestamp is always rounded down; therefore 0 means “less than 24 hours ago”, 1 means “between 24 and 48 hours ago”, and so forth. Fractional values are supported but this only really makes sense for the case where ranges (‘+n’ and ‘-n’) are used.

Test: -amin n ¶

Test: -cmin n ¶

Test: -mmin n ¶

True if the file was last accessed (or its status changed, or it was modified) n minutes ago. These tests provide finer granularity of measurement than ‘-atime’ et al., but rounding is done in a similar way (again, fractions are supported). For example, to list files in /u/bill that were last read from 2 to 6 minutes ago:

find /u/bill -amin +2 -amin -6

Option: -daystart ¶

Measure times from the beginning of today rather than from 24 hours ago. So, to list the regular files in your home directory that were modified yesterday, do

find ~/ -daystart -type f -mtime 1

The ‘-daystart’ option is unlike most other options in that it has an effect on the way that other tests are performed. The affected tests are ‘-amin’, ‘-cmin’, ‘-mmin’, ‘-atime’, ‘-ctime’ and ‘-mtime’. The ‘-daystart’ option only affects the behaviour of any tests which appear after it on the command line.

2.5.2 Comparing Timestamps

Test: -newerXY reference ¶

Succeeds if timestamp ‘X’ of the file being considered is newer than timestamp ‘Y’ of the file reference. The letters ‘X’ and ‘Y’ can be any of the following letters:

‘a’

Last-access time of reference

‘B’

Birth time of reference (when this is not known, the test cannot succeed)

‘c’

Last-change time of reference

‘m’

Last-modification time of reference

‘t’

The reference argument is interpreted as a literal time, rather than the name of a file. See Date input formats, for a description of how the timestamp is understood. Tests of the form ‘-newerXt’ are valid but tests of the form ‘-newertY’ are not.

For example the test -newerac /tmp/foo succeeds for all files which have been accessed more recently than /tmp/foo was changed. Here ‘X’ is ‘a’ and ‘Y’ is ‘c’.

Not all files have a known birth time. If ‘Y’ is ‘B’ and the birth time of reference is not available, find exits with an explanatory error message. If ‘X’ is ‘B’ and we do not know the birth time the file currently being considered, the test simply fails (that is, it behaves like -false does).

Some operating systems (for example, most implementations of Unix) do not support file birth times. Some others, for example NetBSD-3.1, do. Even on operating systems which support file birth times, the information may not be available for specific files. For example, under NetBSD, file birth times are supported on UFS2 file systems, but not UFS1 file systems.

There are two ways to list files in /usr modified after February 1 of the current year. One uses ‘-newermt’:

find /usr -newermt "Feb 1"

The other way of doing this works on the versions of find before 4.3.3:

touch -t 02010000 /tmp/stamp$$
find /usr -newer /tmp/stamp$$
rm -f /tmp/stamp$$

Test: -anewer reference ¶

Test: -cnewer reference ¶

Test: -newer reference ¶

True if the time of the last access (or status change or data modification) of the current file is more recent than that of the last data modification of the reference file. As such, ‘-anewer’ is equivalent to ‘-neweram’, ‘-cnewer’ to ‘-newercm’, and ‘-newer’ to ‘-newermm’.

If reference is a symbolic link and the ‘-H’ option or the ‘-L’ option is in effect, then the time of the last data modification of the file it points to is always used.

These tests are affected by ‘-follow’ only if ‘-follow’ comes before them on the command line. See Symbolic Links, for more information on ‘-follow’.

As an example, to list any files modified since /bin/sh was last modified:

find . -newer /bin/sh

Test: -used n ¶

True if the file was last accessed n days after its status was last changed. Useful for finding files that are not being used, and could perhaps be archived or removed to save disk space.

3.2.1 Escapes

The escapes that ‘-printf’ and ‘-fprintf’ recognise are:

\a

Alarm bell.

\b

Backspace.

\c

Stop printing from this format immediately and flush the output.

\f

Form feed.

\n

Newline.

\r

Carriage return.

\t

Horizontal tab.

\v

Vertical tab.

\\

A literal backslash (‘\’).

\0

ASCII NUL.

\NNN

The character whose ASCII code is NNN (octal).

A ‘\’ character followed by any other character is treated as an ordinary character, so they both are printed, and a warning message is printed to the standard error output (because it was probably a typo).

3.2.2 Format Directives

‘-printf’ and ‘-fprintf’ support the following format directives to print information about the file being processed. The C printf function, field width and precision specifiers are supported, as applied to string (%s) types. That is, you can specify "minimum field width"."maximum field width" for each directive. Format flags (like ‘#’ for example) may not work as you expect because many of the fields, even numeric ones, are printed with %s. The format flag ‘-’ does work; it forces left-alignment of the field.

‘%%’ is a literal percent sign. See Reserved and Unknown Directives, for a description of how format directives not mentioned below are handled.

A ‘%’ at the end of the format argument causes undefined behaviour since there is no following character. In some locales, it may hide your door keys, while in others it may remove the final page from the novel you are reading.

• Name Directives
• Ownership Directives
• Size Directives
• Location Directives
• Time Directives
• Other Directives
• Reserved and Unknown Directives

$ find \
  . .. / /tmp /tmp/TRACE compile compile/64/tests/find \
  -maxdepth 0 -printf '%p: [%h][%f]\n'
.: [.][.]
..: [.][..]
/: [][/]
/tmp: [][tmp]
/tmp/TRACE: [/tmp][TRACE]
compile: [.][compile]
compile/64/tests/find: [compile/64/tests][find]

Wed Nov  2 00:42:36 1994

3.2.3 Time Formats

Below is an incomplete list of formats for the directives ‘%A’, ‘%B’, ‘%C’, and ‘%T’, which print the file’s timestamps. Please refer to the documentation of strftime for the full list. Some of these formats might not be available on all systems, due to differences in the implementation of the C strftime function.

• Time Components
• Date Components
• Combined Time Formats

3.2.4 Formatting Flags

The ‘%m’ and ‘%d’ directives support the ‘#’, ‘0’ and ‘+’ flags, but the other directives do not, even if they print numbers. Numeric directives that do not support these flags include

‘G’, ‘U’, ‘b’, ‘D’, ‘k’ and ‘n’.

All fields support the format flag ‘-’, which makes fields left-aligned. That is, if the field width is greater than the actual contents of the field, the requisite number of spaces are printed after the field content instead of before it.

3.3.1 Single File

Here is how to run a command on one file at a time.

Action: -execdir command ; ¶

Execute command; true if command returns zero. find takes all arguments after ‘-execdir’ to be part of the command until an argument consisting of ‘;’ is reached. It replaces the string ‘{}’ by the current file name being processed everywhere it occurs in the command. Both of these constructions need to be escaped (with a ‘\’) or quoted to protect them from expansion by the shell. The command is executed in the directory which find was searching at the time the action was executed (that is, {} will expand to a file in the local directory).

For example, to compare each C header file in or below the current directory with the file /tmp/master:

find . -name '*.h' -execdir diff -u '{}' /tmp/master ';'

If you use ‘-execdir’, you must ensure that the PATH variable contains only absolute directory names. Having an empty element in PATH or explicitly including ‘.’ (or any other non-absolute name) is insecure. GNU find will refuse to run if you use ‘-execdir’ and it thinks your PATH setting is insecure. For example:

‘/bin:/usr/bin:’

Insecure; empty path element (at the end)

‘:/bin:/usr/bin:/usr/local/bin’

Insecure; empty path element (at the start)

‘/bin:/usr/bin::/usr/local/bin’

Insecure; empty path element (two colons in a row)

‘/bin:/usr/bin:.:/usr/local/bin’

Insecure; ‘.’ is a path element (. is not an absolute file name)

‘/bin:/usr/bin:sbin:/usr/local/bin’

Insecure; ‘sbin’ is not an absolute file name

‘/bin:/usr/bin:/sbin:/usr/local/bin’

Secure (if you control the contents of those directories and any access to them)

Another similar option, ‘-exec’ is supported, but is less secure. See Security Considerations, for a discussion of the security problems surrounding ‘-exec’.

Action: -exec command ; ¶

This insecure variant of the ‘-execdir’ action is specified by POSIX. Like ‘-execdir command ;’ it is true if zero is returned by command. The main difference is that the command is executed in the directory from which find was invoked, meaning that ‘{}’ is expanded to a relative path starting with the name of one of the starting directories, rather than just the basename of the matched file.

While some implementations of find replace the ‘{}’ only where it appears on its own in an argument, GNU find replaces ‘{}’ wherever it appears.

3.3.2 Multiple Files

Sometimes you need to process files one at a time. But usually this is not necessary, and, it is faster to run a command on as many files as possible at a time, rather than once per file. Doing this saves on the time it takes to start up the command each time.

The ‘-execdir’ and ‘-exec’ actions have variants that build command lines containing as many matched files as possible.

Action: -execdir command {} + ¶

This works as for ‘-execdir command ;’, except that the result is always true, and the ‘{}’ at the end of the command is expanded to a list of names of matching files. This expansion is done in such a way as to avoid exceeding the maximum command line length available on the system. Only one ‘{}’ is allowed within the command, and it must appear at the end, immediately before the ‘+’. A ‘+’ appearing in any position other than immediately after ‘{}’ is not considered to be special (that is, it does not terminate the command).

Action: -exec command {} + ¶

This insecure variant of the ‘-execdir’ action is specified by POSIX. The main difference is that the command is executed in the directory from which find was invoked, meaning that ‘{}’ is expanded to a relative path starting with the name of one of the starting directories, rather than just the basename of the matched file. The result is always true.

Before find exits, any partially-built command lines are executed. This happens even if the exit was caused by the ‘-quit’ action. However, some types of error (for example not being able to invoke stat() on the current directory) can cause an immediate fatal exit. In this situation, any partially-built command lines will not be invoked (this prevents possible infinite loops).

At first sight, it looks like the list of filenames to be processed can only be at the end of the command line, and that this might be a problem for some commands (cp and rsync for example).

However, there is a slightly obscure but powerful workaround for this problem which takes advantage of the behaviour of sh -c:

find startpoint -tests … -exec sh -c 'scp "$@" remote:/dest' sh {} +

In the example above, the filenames we want to work on need to occur on the scp command line before the name of the destination. We use the shell to invoke the command scp "$@" remote:/dest and the shell expands "$@" to the list of filenames we want to process.

Another, but less secure, way to run a command on more than one file at once, is to use the xargs command, which is invoked like this:

xargs [option…] [command [initial-arguments]]

xargs normally reads arguments from the standard input. These arguments are delimited by blanks (which can be protected with double or single quotes or a backslash) or newlines. It executes the command (the default is echo) one or more times with any initial-arguments followed by arguments read from standard input. Blank lines on the standard input are ignored. If the ‘-L’ option is in use, trailing blanks indicate that xargs should consider the following line to be part of this one.

Instead of blank-delimited names, it is safer to use ‘find -print0’ or ‘find -fprint0’ and process the output by giving the ‘-0’ or ‘--null’ option to GNU xargs, GNU tar, GNU cpio, or perl. The locate command also has a ‘-0’ or ‘--null’ option which does the same thing.

You can use shell command substitution (backquotes) to process a list of arguments, like this:

grep -l sprintf `find $HOME -name '*.c' -print`

However, that method produces an error if the length of the ‘.c’ file names exceeds the operating system’s command line length limit. xargs avoids that problem by running the command as many times as necessary without exceeding the limit:

find $HOME -name '*.c' -print | xargs grep -l sprintf

However, if the command needs to have its standard input be a terminal (less, for example), you have to use the shell command substitution method or use either the ‘--arg-file’ option or the ‘--open-tty’ option of xargs.

The xargs command will usually process all of its input, building command lines and executing them. The processing stops earlier and immediately if the tool reads a line containing the end-of-file marker string specified with the ‘--eof’ option, or if one of the launched commands exits with a status of 255. The latter will cause xargs to issue an error message and exit with status 124.

• Unsafe File Name Handling
• Safe File Name Handling
• Unusual Characters in File Names
• Limiting Command Size
• Controlling Parallelism
• Interspersing File Names

find / -name '#*' -atime +7 -print | xargs rm

eg$ echo > '#
vmunix'

eg$ mkdir '#
'
eg$ cd '#
'
eg$ mkdir u u/joeuser u/joeuser/.plan'
'
eg$ echo > u/joeuser/.plan'
/#foo'
eg$ cd ..
eg$ find . -name '#*' -print | xargs echo
./# ./# /u/joeuser/.plan /#foo

find / -name xyzzy -print0 > list
xargs --null --arg-file=list munge

xargs --null --arg-file=<(find / -name xyzzy -print0) munge

find originals -name '*.jpg' | xargs -L 1 makeallsizes

find originals -name '*.jpg' | xargs -L 1 -P 2 makeallsizes

shell$ xargs <allimages -L 1 -P 4 makeallsizes &
[4] 27643
   ... at some later point ...
shell$ kill -USR2 27643
shell$ kill -USR2 %4

find bills -type f | xargs -I XX sort -o XX.sorted XX

find bills -type f -execdir sort -o '{}.sorted' '{}' ';'

somecommand | xargs -s 50000 echo | xargs -I '{}' -s 100000 rm '{}'

3.3.3 Querying

To ask the user whether to execute a command on a single file, you can use the find primary ‘-okdir’ instead of ‘-execdir’, and the find primary ‘-ok’ instead of ‘-exec’:

Action: -okdir command ; ¶

Like ‘-execdir’ (see Single File), but ask the user first. If the user does not agree to run the command, just return false. Otherwise, run it, with standard input redirected from /dev/null.

This action may not be specified together with the ‘-files0-from’ option.

The response to the prompt is matched against a pair of regular expressions to determine if it is a yes or no response. These regular expressions are obtained from the system (nl_langinfo items YESEXPR and NOEXPR are used) if the POSIXLY_CORRECT environment variable is set and the system has such patterns available. Otherwise, find’s message translations are used. In either case, the LC_MESSAGES environment variable will determine the regular expressions used to determine if the answer is affirmative or negative. The interpretation of the regular expressions themselves will be affected by the environment variables LC_CTYPE (character classes) and LC_COLLATE (character ranges and equivalence classes).

Action: -ok command ; ¶

This insecure variant of the ‘-okdir’ action is specified by POSIX. The main difference is that the command is executed in the directory from which find was invoked, meaning that ‘{}’ is expanded to a relative path starting with the name of one of the starting directories, rather than just the basename of the matched file. If the command is run, its standard input is redirected from /dev/null.

This action may not be specified together with the ‘-files0-from’ option.

When processing multiple files with a single command, to query the user you give xargs the following option. When using this option, you might find it useful to control the number of files processed per invocation of the command (see Limiting Command Size).

--interactive

-p

Prompt the user about whether to run each command line and read a line from the terminal. Only run the command line if the response starts with ‘y’ or ‘Y’. Implies ‘-t’.

4.2.1 LOCATE02 Database Format

updatedb runs a program called frcode to front-compress the list of file names, which reduces the database size by a factor of 4 to 5. Front-compression (also known as incremental encoding) works as follows.

The database entries are a sorted list (case-insensitively, for users’ convenience). Since the list is sorted, each entry is likely to share a prefix (initial string) with the previous entry. Each database entry begins with an offset-differential count byte, which is the additional number of characters of prefix of the preceding entry to use beyond the number that the preceding entry is using of its predecessor. (The counts can be negative.) Following the count is a null-terminated ASCII remainder – the part of the name that follows the shared prefix.

If the offset-differential count is larger than can be stored in a byte (+/-127), the byte has the value 0x80 and the count follows in a 2-byte word, with the high byte first (network byte order).

Every database begins with a dummy entry for a file called LOCATE02, which locate checks for to ensure that the database file has the correct format; it ignores the entry in doing the search.

Databases cannot be concatenated together, even if the first (dummy) entry is trimmed from all but the first database. This is because the offset-differential count in the first entry of the second and following databases will be wrong.

In the output of ‘locate --statistics’, the new database format is referred to as ‘LOCATE02’.

4.2.2 Sample LOCATE02 Database

Sample input to frcode:

/usr/src
/usr/src/cmd/aardvark.c
/usr/src/cmd/armadillo.c
/usr/tmp/zoo

Length of the longest prefix of the preceding entry to share:

0 /usr/src
8 /cmd/aardvark.c
14 rmadillo.c
5 tmp/zoo

Output from frcode, with trailing nulls changed to newlines and count bytes made printable:

0 LOCATE02
0 /usr/src
8 /cmd/aardvark.c
6 rmadillo.c
-9 tmp/zoo

(6 = 14 - 8, and -9 = 5 - 14)

4.2.3 slocate Database Format

The slocate program uses a database format similar to, but not quite the same as, GNU locate. The first byte of the database specifies its security level. If the security level is 0, slocate will read, match and print filenames on the basis of the information in the database only. However, if the security level byte is 1, slocate omits entries from its output if the invoking user is unable to access them. The second byte of the database is zero. The second byte is immediately followed by the first database entry. The first entry in the database is not preceded by any differential count or dummy entry. Instead the differential count for the first item is assumed to be zero.

Starting with the second entry (if any) in the database, data is interpreted as for the GNU LOCATE02 format.

4.2.4 Old Database Format

The old database format is used by Unix locate and find programs and pre-4.0 releases of GNU findutils. locate understands this format, though updatedb will no longer produce it.

The old format differs from ‘LOCATE02’ in the following ways. Instead of each entry starting with an offset-differential count byte and ending with a null, byte values from 0 through 28 indicate offset-differential counts from -14 through 14. The byte value indicating that a long offset-differential count follows is 0x1e (30), not 0x80. The long counts are stored in host byte order, which is not necessarily network byte order, and host integer word size, which is usually 4 bytes. They also represent a count 14 less than their value. The database lines have no termination byte; the start of the next line is indicated by its first byte having a value <= 30.

In addition, instead of starting with a dummy entry, the old database format starts with a 256 byte table containing the 128 most common bigrams in the file list. A bigram is a pair of adjacent bytes. Bytes in the database that have the high bit set are indexes (with the high bit cleared) into the bigram table. The bigram and offset-differential count coding makes these databases 20-25% smaller than the new format, but makes them not 8-bit clean. Any byte in a file name that is in the ranges used for the special codes is replaced in the database by a question mark, which not coincidentally is the shell wildcard to match a single character. The old format therefore cannot faithfully store entries with non-ASCII characters.

Because the long counts are stored as native-order machine words, the database format is not easily used in environments which differ in terms of byte order. If locate databases are to be shared between machines, the ‘LOCATE02’ database format should be used. This has other benefits as discussed above. However, the length of the filename currently being processed can normally be used to place reasonable limits on the long counts and so this information is used by locate to help it guess the byte ordering of the old format database. Unless it finds evidence to the contrary, locate will assume that the byte order of the database is the same as the native byte order of the machine running locate. The output of ‘locate --statistics’ also includes information about the byte order of old-format databases.

The output of ‘locate --statistics’ will give an incorrect count of the number of file names containing newlines or high-bit characters for old-format databases.

Old versions of GNU locate fail to correctly handle very long file names, possibly leading to security problems relating to a heap buffer overrun. See Security Considerations for locate, for a detailed explanation.

5.2.1 Setting Permissions

The basic symbolic operations on a file’s permissions are adding, removing, and setting the permission that certain users have to read, write, and execute the file. These operations have the following format:

users operation permissions

The spaces between the three parts above are shown for readability only; symbolic modes cannot contain spaces.

The users part tells which users’ access to the file is changed. It consists of one or more of the following letters (or it can be empty; see The Umask and Protection, for a description of what happens then). When more than one of these letters is given, the order that they are in does not matter.

u ¶

the user who owns the file;

g ¶

other users who are in the file’s group;

o ¶

all other users;

a

all users; the same as ‘ugo’.

The operation part tells how to change the affected users’ access to the file, and is one of the following symbols:

+ ¶

to add the permissions to whatever permissions the users already have for the file;

- ¶

to remove the permissions from whatever permissions the users already have for the file;

= ¶

to make the permissions the only permissions that the users have for the file.

The permissions part tells what kind of access to the file should be changed; it is normally zero or more of the following letters. As with the users part, the order does not matter when more than one letter is given. Omitting the permissions part is useful only with the ‘=’ operation, where it gives the specified users no access at all to the file.

r ¶

the permission the users have to read the file;

w ¶

the permission the users have to write to the file;

x ¶

the permission the users have to execute the file.

For example, to give everyone permission to read and write a file, but not to execute it, use:

a=rw

To remove write permission for all users other than the file’s owner, use:

go-w

The above command does not affect the access that the owner of the file has to it, nor does it affect whether other users can read or execute the file.

To give everyone except a file’s owner no permission to do anything with that file, use the mode below. Other users could still remove the file, if they have write permission on the directory it is in.

go=

Another way to specify the same thing is:

og-rwx

5.2.2 Copying Existing Permissions

You can base a file’s permissions on its existing permissions. To do this, instead of using a series of ‘r’, ‘w’, or ‘x’ letters after the operator, you use the letter ‘u’, ‘g’, or ‘o’. For example, the mode

o+g

adds the permissions for users who are in a file’s group to the permissions that other users have for the file. Thus, if the file started out as mode 664 (‘rw-rw-r--’), the above mode would change it to mode 666 (‘rw-rw-rw-’). If the file had started out as mode 741 (‘rwxr----x’), the above mode would change it to mode 745 (‘rwxr--r-x’). The ‘-’ and ‘=’ operations work analogously.

5.2.3 Changing Special Permissions

In addition to changing a file’s read, write, and execute permissions, you can change its special permissions. See Structure of File Permissions, for a summary of these permissions.

To change a file’s permission to set the user ID on execution, use ‘u’ in the users part of the symbolic mode and ‘s’ in the permissions part.

To change a file’s permission to set the group ID on execution, use ‘g’ in the users part of the symbolic mode and ‘s’ in the permissions part.

To change a file’s permission to set the restricted deletion flag or sticky bit, omit the users part of the symbolic mode (or use ‘a’) and put ‘t’ in the permissions part.

For example, to add set-user-ID permission to a program, you can use the mode:

u+s

To remove both set-user-ID and set-group-ID permission from it, you can use the mode:

ug-s

To set the restricted deletion flag or sticky bit, you can use the mode:

+t

The combination ‘o+s’ has no effect. On GNU systems the combinations ‘u+t’ and ‘g+t’ have no effect, and ‘o+t’ acts like plain ‘+t’.

The ‘=’ operator is not very useful with special permissions; for example, the mode:

o=t

does set the restricted deletion flag or sticky bit, but it also removes all read, write, and execute permissions that users not in the file’s group might have had for it.

5.2.4 Conditional Executability

There is one more special type of symbolic permission: if you use ‘X’ instead of ‘x’, execute permission is affected only if the file is a directory or already had execute permission.

For example, this mode:

a+X

gives all users permission to search directories, or to execute files if anyone could execute them before.

5.2.5 Making Multiple Changes

The format of symbolic modes is actually more complex than described above (see Setting Permissions). It provides two ways to make multiple changes to files’ permissions.

The first way is to specify multiple operation and permissions parts after a users part in the symbolic mode.

For example, the mode:

og+rX-w

gives users other than the owner of the file read permission and, if it is a directory or if someone already had execute permission to it, gives them execute permission; and it also denies them write permission to the file. It does not affect the permission that the owner of the file has for it. The above mode is equivalent to the two modes:

og+rX
og-w

The second way to make multiple changes is to specify more than one simple symbolic mode, separated by commas. For example, the mode:

a+r,go-w

gives everyone permission to read the file and removes write permission on it for all users except its owner. Another example:

u=rwx,g=rx,o=

sets all of the non-special permissions for the file explicitly. (It gives users who are not in the file’s group no permission at all for it.)

The two methods can be combined. The mode:

a+r,g+x-w

gives all users permission to read the file, and gives users who are in the file’s group permission to execute it, as well, but not permission to write to it. The above mode could be written in several different ways; another is:

u+r,g+rx,o+r,g-w

5.2.6 The Umask and Protection

If the users part of a symbolic mode is omitted, it defaults to ‘a’ (affect all users), except that any permissions that are set in the system variable umask are not affected. The value of umask can be set using the umask command. Its default value varies from system to system.

Omitting the users part of a symbolic mode is generally not useful with operations other than ‘+’. It is useful with ‘+’ because it allows you to use umask as an easily customizable protection against giving away more permission to files than you intended to.

As an example, if umask has the value 2, which removes write permission for users who are not in the file’s group, then the mode:

+w

adds permission to write to the file to its owner and to other users who are in the file’s group, but not to other users. In contrast, the mode:

a+w

ignores umask, and does give write permission for the file to all users.

8.1.1 Filesystem Traversal Options

The options ‘-H’, ‘-L’ or ‘-P’ may be specified at the start of the command line (if none of these is specified, ‘-P’ is assumed). If you specify more than one of these options, the last one specified takes effect (but note that the ‘-follow’ option is equivalent to ‘-L’).

-P

Never follow symbolic links (this is the default), except in the case of the ‘-xtype’ predicate.

-L

Always follow symbolic links, except in the case of the ‘-xtype’ predicate.

-H

Follow symbolic links specified in the list of files to search, or which are otherwise specified on the command line.

If find would follow a symbolic link, but cannot for any reason (for example, because it has insufficient permissions or the link is broken), it falls back on using the properties of the symbolic link itself. Symbolic Links for a more complete description of how symbolic links are handled.

8.1.2 Warning Messages

If there is an error on the find command line, an error message is normally issued. However, there are some usages that are inadvisable but which find should still accept. Under these circumstances, find may issue a warning message.

By default, warnings are enabled only if find is being run interactively (specifically, if the standard input is a terminal) and the POSIXLY_CORRECT environment variable is not set. Warning messages can be controlled explicitly by the use of options on the command line:

-warn

Issue warning messages where appropriate.

-nowarn

Do not issue warning messages.

These options take effect at the point on the command line where they are specified. Therefore it’s not useful to specify ‘-nowarn’ at the end of the command line. The warning messages affected by the above options are triggered by:

• Use of the ‘-d’ option which is deprecated; please use ‘-depth’ instead, since the latter is POSIX-compliant.
• Specifying an option (for example ‘-mindepth’) after a non-option (for example ‘-type’ or ‘-print’) on the command line.
• Use of the ‘-name’ or ‘-iname’ option with a slash character in the pattern. Since the name predicates only compare against the basename of the visited files, the only file that can match a slash is the root directory itself.

The default behaviour above is designed to work in that way so that existing shell scripts don’t generate spurious errors, but people will be made aware of the problem.

Some warning messages are issued for less common or more serious problems, and consequently cannot be turned off:

• Use of an unrecognised backslash escape sequence with ‘-fprintf’
• Use of an unrecognised formatting directive with ‘-fprintf’

8.1.3 Optimisation Options

The ‘-Olevel’ option sets find’s optimisation level to level. The default optimisation level is 1.

At certain optimisation levels (but not by default), find reorders tests to speed up execution while preserving the overall effect; that is, predicates with side effects are not reordered relative to each other. The optimisations performed at each optimisation level are as follows.

0

Currently equivalent to optimisation level 1.

1

This is the default optimisation level and corresponds to the traditional behaviour. Expressions are reordered so that tests based only on the names of files (for example‘ -name’ and ‘-regex’) are performed first.

2

Any ‘-type’ or ‘-xtype’ tests are performed after any tests based only on the names of files, but before any tests that require information from the inode. On many modern versions of Unix, file types are returned by readdir() and so these predicates are faster to evaluate than predicates which need to stat the file first.

If you use the ‘-fstype FOO’ predicate and specify a filsystem type ‘FOO’ which is not known (that is, present in /etc/mtab) at the time find starts, that predicate is equivalent to ‘-false’.

3

At this optimisation level, the full cost-based query optimizer is enabled. The order of tests is modified so that cheap (i.e., fast) tests are performed first and more expensive ones are performed later, if necessary. Within each cost band, predicates are evaluated earlier or later according to whether they are likely to succeed or not. For ‘-o’, predicates which are likely to succeed are evaluated earlier, and for ‘-a’, predicates which are likely to fail are evaluated earlier.

The re-ordering of operations performed by the cost-based optimizer can result in user-visible behaviour change. For example, the ‘-readable’ and ‘-empty’ predicates are sensitive to re-ordering. If they are run in the order ‘-empty -readable’, an error message will be issued for unreadable directories. If they are run in the order ‘-readable -empty’, no error message will be issued. This is the reason why such operation re-ordering is not performed at the default optimisation level.

8.1.4 Debug Options

The ‘-D’ option makes find produce diagnostic output. Much of the information is useful only for diagnosing problems, and so most people will not find this option helpful.

The list of debug options should be comma separated. Compatibility of the debug options is not guaranteed between releases of findutils. For a complete list of valid debug options, see the output of find -D help.

Valid debug options include:

‘tree’

Show the expression tree in its original and optimized form.

‘stat’

Print messages as files are examined with the stat and lstat system calls. The find program tries to minimise such calls.

‘opt’

Prints diagnostic information relating to the optimisation of the expression tree; see the ‘-O’ option.

‘rates’

Prints a summary indicating how often each predicate succeeded or failed.

‘all’

Enable all of the other debug options (but ‘help’).

‘help’

Explain the debugging options.

8.1.5 Find Expressions

The final part of the find command line is a list of expressions. See find Primary Index, for a summary of all of the tests, actions, and options that the expression can contain. If the expression is missing, ‘-print’ is assumed.

8.4.1 xargs options

-- ¶

Delimit the option list. Later arguments, if any, are treated as operands even if they begin with ‘-’. For example, ‘xargs -- --help’ runs the command ‘--help’ (found in PATH) instead of printing the usage text, and ‘xargs -- --mycommand’ runs the command ‘--mycommand’ instead of rejecting this as unrecognized option.

--arg-file=inputfile

-a inputfile

Read names from the file inputfile instead of standard input. If you use this option, the standard input stream remains unchanged when commands are run. Otherwise, stdin is redirected from /dev/null.

--null

-0

Input file names are terminated by a null character instead of by whitespace, and any quotes and backslash characters are not considered special (every character is taken literally). Disables the end of file string, which is treated like any other argument.

--delimiter delim

-d delim

Input file names are terminated by the specified character delim instead of by whitespace, and any quotes and backslash characters are not considered special (every character is taken literally). Disables the logical end-of-file marker string, which is treated like any other argument.

The specified delimiter may be a single character, a C-style character escape such as ‘\n’, or an octal or hexadecimal escape code. Octal and hexadecimal escape codes are understood as for the printf command. Multibyte characters are not supported.

-E eof-str

--eof[=eof-str]

-e[eof-str]

Set the logical end-of-file marker string to eof-str. If the logical end-of-file marker string occurs as a line of input, the rest of the input is ignored. If eof-str is omitted (‘-e’) or blank (either ‘-e’ or ‘-E’), there is no logical end-of-file marker string. The ‘-e’ form of this option is deprecated in favour of the POSIX-compliant ‘-E’ option, which you should use instead. As of GNU xargs version 4.2.9, the default behaviour of xargs is not to have a logical end-of-file marker string. The POSIX standard (IEEE Std 1003.1, 2004 Edition) allows this.

The logical end-of-file marker string is not treated specially if the ‘-d’ or the ‘-0’ options are in effect. That is, when either of these options are in effect, the whole input file will be read even if ‘-E’ was used.

--help

Print a summary of the options to xargs and exit.

-I replace-str

--replace[=replace-str]

-i[replace-str]

Replace occurrences of replace-str in the initial arguments with names read from standard input. Also, unquoted blanks do not terminate arguments; instead, the input is split at newlines only. If replace-str is omitted (omitting it is allowed only for ‘-i’ and ‘--replace’), it defaults to ‘{}’ (like for ‘find -exec’). Implies ‘-x’ and ‘-L 1’. The ‘-i’ option is deprecated in favour of the ‘-I’ option.

-L max-lines

--max-lines[=max-lines]

-l[max-lines]

Use at most max-lines non-blank input lines per command line. For ‘-l’, max-lines defaults to 1 if omitted. For ‘-L’, the argument is mandatory. Trailing blanks cause an input line to be logically continued on the next input line, for the purpose of counting the lines. Implies ‘-x’. The ‘-l’ form of this option is deprecated in favour of the POSIX-compliant ‘-L’ option.

--max-args=max-args

-n max-args

Use at most max-args arguments per command line. Fewer than max-args arguments will be used if the size (see the ‘-s’ option) is exceeded, unless the ‘-x’ option is given, in which case xargs will exit.

--open-tty

-o

Reopen stdin as /dev/tty in the child process before executing the command, thus allowing that command to be associated to the terminal while xargs reads from a different stream, e.g. from a pipe. This is useful if you want xargs to run an interactive application.

grep -lZ PATTERN * | xargs -0o -n1 vi

--interactive

-p

Prompt the user about whether to run each command line and read a line from the terminal. Only run the command line if the response starts with ‘y’ or ‘Y’. Implies ‘-t’.

--no-run-if-empty

-r

If the standard input is completely empty, do not run the command. By default, the command is run once even if there is no input.

--max-chars=max-chars

-s max-chars

Use at most max-chars characters per command line, including the command, initial arguments and any terminating nulls at the ends of the argument strings.

--show-limits

Display the limits on the command-line length which are imposed by the operating system, xargs’ choice of buffer size and the ‘-s’ option. Pipe the input from /dev/null (and perhaps specify ‘--no-run-if-empty’) if you don’t want xargs to do anything.

--verbose

-t

Print the command line on the standard error output before executing it.

--version

Print the version number of xargs and exit.

--exit

-x

Exit if the size (see the ‘-s’ option) is exceeded.

--max-procs=max-procs

-P max-procs

Run simultaneously up to max-procs processes at once; the default is 1. If max-procs is 0, xargs will run as many processes as possible simultaneously. See Controlling Parallelism, for information on dynamically controlling parallelism.

--process-slot-var=environment-variable-name

Set the environment variable environment-variable-name to a unique value in each running child process. Each value is a decimal integer. Values are reused once child processes exit. This can be used in a rudimentary load distribution scheme, for example.

8.4.2 Conflicting options

The options ‘--max-lines’ (‘-L’, ‘-l’), ‘--replace’ (‘-I’, ‘-i’) and ‘--max-args’ (‘-n’) are mutually exclusive.

If some of them are specified at the same time, then xargs will generally use the option specified last on the command line, i.e., it will reset the value of the offending option (given before) to its default value. Additionally, xargs will issue a warning diagnostic on stderr.

$ seq 4 | xargs -L2 -n3
xargs: warning: options --max-lines and --max-args/-n are \
  mutually exclusive, ignoring previous --max-lines value
1 2 3
4

The exception to this rule is that the special max-args value 1 is ignored after the ‘--replace’ option and its short-option aliases ‘-I’ and ‘-i’, because it would not actually conflict.

$ seq 2 | xargs --replace -n1 echo a-{}-b
a-1-b
a-2-b

8.4.3 Invoking the shell from xargs

Normally, xargs will exec the command you specified directly, without invoking a shell. This is normally the behaviour one would want. It’s somewhat more efficient and avoids problems with shell metacharacters, for example. However, sometimes it is necessary to manipulate the environment of a command before it is run, in a way that xargs does not directly support.

Invoking a shell from xargs is a good way of performing such manipulations. However, some care must be taken to prevent problems, for example unwanted interpretation of shell metacharacters.

This command moves a set of files into an archive directory:

find /foo -maxdepth 1 -atime +366 -exec mv {} /archive \;

However, this will only move one file at a time. We cannot in this case use -exec ... + because the matched file names are added at the end of the command line, while the destination directory would need to be specified last. We also can’t use xargs in the obvious way for the same reason. One way of working around this problem is to make use of the special properties of GNU mv; it has a -t option that allows specifying the target directory before the list of files to be moved. However, while this technique works for GNU mv, it doesn’t solve the more general problem.

Here is a more general technique for solving this problem:

find /foo -maxdepth 1 -atime +366 -print0 |
xargs -r0 sh -c 'mv "$@" /archive' move

Here, a shell is being invoked. There are two shell instances to think about. The first is the shell which launches the xargs command (this might be the shell into which you are typing, for example). The second is the shell launched by xargs (in fact it will probably launch several, one after the other, depending on how many files need to be archived). We’ll refer to this second shell as a subshell.

Our example uses the -c option of sh. Its argument is a shell command to be executed by the subshell. Along with the rest of that command, the $@ is enclosed by single quotes to make sure it is passed to the subshell without being expanded by the parent shell. It is also enclosed with double quotes so that the subshell will expand $@ correctly even if one of the file names contains a space or newline.

The subshell will use any non-option arguments as positional parameters (that is, in the expansion of $@). Because xargs launches the sh -c subshell with a list of files, those files will end up as the expansion of $@.

You may also notice the ‘move’ at the end of the command line. This is used as the value of $0 by the subshell. We include it because otherwise the name of the first file to be moved would be used instead. If that happened it would not be included in the subshell’s expansion of $@, and so it wouldn’t actually get moved.

Another reason to use the sh -c construct could be to perform redirection:

find /usr/include -name '*.h' | xargs grep -wl mode_t |
xargs -r sh -c 'exec emacs "$@" < /dev/tty' Emacs

Notice that we use the shell builtin exec here. That’s simply because the subshell needs to do nothing once Emacs has been invoked. Therefore instead of keeping a sh process around for no reason, we just arrange for the subshell to exec Emacs, saving an extra process creation.

Although GNU xargs and the implementations on some other platforms like BSD support the ‘-o’ option to achieve the same, the above is the portable way to redirect stdin to /dev/tty.

Sometimes, though, it can be helpful to keep the shell process around:

find /foo -maxdepth 1 -atime +366 -print0 |
xargs -r0 sh -c 'mv "$@" /archive || exit 255' move

Here, the shell will exit with status 255 if any mv failed. This causes xargs to stop immediately.

8.5.1 ‘findutils-default’ regular expression syntax

The character ‘.’ matches any single character.

‘+’

indicates that the regular expression should match one or more occurrences of the previous atom or regexp.

‘?’

indicates that the regular expression should match zero or one occurrence of the previous atom or regexp.

‘\+’

matches a ‘+’

‘\?’

matches a ‘?’.

Bracket expressions are used to match ranges of characters. Bracket expressions where the range is backward, for example ‘[z-a]’, are ignored. Within square brackets, ‘\’ is taken literally. Character classes are not supported, so for example you would need to use ‘[0-9]’ instead of ‘[[:digit:]]’.

GNU extensions are supported:

1. ‘\w’ matches a character within a word
2. ‘\W’ matches a character which is not within a word
3. ‘\<’ matches the beginning of a word
4. ‘\>’ matches the end of a word
5. ‘\b’ matches a word boundary
6. ‘\B’ matches characters which are not a word boundary
7. ‘\`’ matches the beginning of the whole input
8. ‘\'’ matches the end of the whole input

Grouping is performed with backslashes followed by parentheses ‘\(’, ‘\)’. A backslash followed by a digit acts as a back-reference and matches the same thing as the previous grouped expression indicated by that number. For example ‘\2’ matches the second group expression. The order of group expressions is determined by the position of their opening parenthesis ‘\(’.

The alternation operator is ‘\|’.

The character ‘^’ only represents the beginning of a string when it appears:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘\(’
3. After the alternation operator ‘\|’

The character ‘$’ only represents the end of a string when it appears:

1. At the end of a regular expression
2. Before a close-group, signified by ‘\)’
3. Before the alternation operator ‘\|’

‘*’, ‘+’ and ‘?’ are special at any point in a regular expression except:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘\(’
3. After the alternation operator ‘\|’

The longest possible match is returned; this applies to the regular expression as a whole and (subject to this constraint) to subexpressions within groups.

8.5.2 ‘emacs’ regular expression syntax

The character ‘.’ matches any single character except newline.

‘+’

indicates that the regular expression should match one or more occurrences of the previous atom or regexp.

‘?’

indicates that the regular expression should match zero or one occurrence of the previous atom or regexp.

‘\+’

matches a ‘+’

‘\?’

matches a ‘?’.

Bracket expressions are used to match ranges of characters. Bracket expressions where the range is backward, for example ‘[z-a]’, are ignored. Within square brackets, ‘\’ is taken literally. Character classes are not supported, so for example you would need to use ‘[0-9]’ instead of ‘[[:digit:]]’.

GNU extensions are supported:

1. ‘\w’ matches a character within a word
2. ‘\W’ matches a character which is not within a word
3. ‘\<’ matches the beginning of a word
4. ‘\>’ matches the end of a word
5. ‘\b’ matches a word boundary
6. ‘\B’ matches characters which are not a word boundary
7. ‘\`’ matches the beginning of the whole input
8. ‘\'’ matches the end of the whole input

Grouping is performed with backslashes followed by parentheses ‘\(’, ‘\)’. A backslash followed by a digit acts as a back-reference and matches the same thing as the previous grouped expression indicated by that number. For example ‘\2’ matches the second group expression. The order of group expressions is determined by the position of their opening parenthesis ‘\(’.

The alternation operator is ‘\|’.

The character ‘^’ only represents the beginning of a string when it appears:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘\(’
3. After the alternation operator ‘\|’

The character ‘$’ only represents the end of a string when it appears:

1. At the end of a regular expression
2. Before a close-group, signified by ‘\)’
3. Before the alternation operator ‘\|’

‘*’, ‘+’ and ‘?’ are special at any point in a regular expression except:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘\(’
3. After the alternation operator ‘\|’

The longest possible match is returned; this applies to the regular expression as a whole and (subject to this constraint) to subexpressions within groups.

8.5.3 ‘gnu-awk’ regular expression syntax

The character ‘.’ matches any single character.

‘+’

indicates that the regular expression should match one or more occurrences of the previous atom or regexp.

‘?’

indicates that the regular expression should match zero or one occurrence of the previous atom or regexp.

‘\+’

matches a ‘+’

‘\?’

matches a ‘?’.

Bracket expressions are used to match ranges of characters. Bracket expressions where the range is backward, for example ‘[z-a]’, are invalid. Within square brackets, ‘\’ can be used to quote the following character. Character classes are supported; for example ‘[[:digit:]]’ will match a single decimal digit.

GNU extensions are supported:

1. ‘\w’ matches a character within a word
2. ‘\W’ matches a character which is not within a word
3. ‘\<’ matches the beginning of a word
4. ‘\>’ matches the end of a word
5. ‘\b’ matches a word boundary
6. ‘\B’ matches characters which are not a word boundary
7. ‘\`’ matches the beginning of the whole input
8. ‘\'’ matches the end of the whole input

Grouping is performed with parentheses ‘()’. An unmatched ‘)’ matches just itself. A backslash followed by a digit acts as a back-reference and matches the same thing as the previous grouped expression indicated by that number. For example ‘\2’ matches the second group expression. The order of group expressions is determined by the position of their opening parenthesis ‘(’.

The alternation operator is ‘|’.

The characters ‘^’ and ‘$’ always represent the beginning and end of a string respectively, except within square brackets. Within brackets, ‘^’ can be used to invert the membership of the character class being specified.

‘*’, ‘+’ and ‘?’ are special at any point in a regular expression except:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘(’
3. After the alternation operator ‘|’

Intervals are specified by ‘{’ and ‘}’. Invalid intervals are treated as literals, for example ‘a{1’ is treated as ‘a\{1’

The longest possible match is returned; this applies to the regular expression as a whole and (subject to this constraint) to subexpressions within groups.

8.5.4 ‘grep’ regular expression syntax

The character ‘.’ matches any single character.

‘\+’

indicates that the regular expression should match one or more occurrences of the previous atom or regexp.

‘\?’

indicates that the regular expression should match zero or one occurrence of the previous atom or regexp.

‘+ and ?’

match themselves.

Bracket expressions are used to match ranges of characters. Bracket expressions where the range is backward, for example ‘[z-a]’, are invalid. Within square brackets, ‘\’ is taken literally. Character classes are supported; for example ‘[[:digit:]]’ will match a single decimal digit.

GNU extensions are supported:

1. ‘\w’ matches a character within a word
2. ‘\W’ matches a character which is not within a word
3. ‘\<’ matches the beginning of a word
4. ‘\>’ matches the end of a word
5. ‘\b’ matches a word boundary
6. ‘\B’ matches characters which are not a word boundary
7. ‘\`’ matches the beginning of the whole input
8. ‘\'’ matches the end of the whole input

Grouping is performed with backslashes followed by parentheses ‘\(’, ‘\)’. A backslash followed by a digit acts as a back-reference and matches the same thing as the previous grouped expression indicated by that number. For example ‘\2’ matches the second group expression. The order of group expressions is determined by the position of their opening parenthesis ‘\(’.

The alternation operator is ‘\|’.

The character ‘^’ only represents the beginning of a string when it appears:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘\(’
3. After a newline
4. After the alternation operator ‘\|’

The character ‘$’ only represents the end of a string when it appears:

1. At the end of a regular expression
2. Before a close-group, signified by ‘\)’
3. Before a newline
4. Before the alternation operator ‘\|’

‘\*’, ‘\+’ and ‘\?’ are special at any point in a regular expression except:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘\(’
3. After a newline
4. After the alternation operator ‘\|’

Intervals are specified by ‘\{’ and ‘\}’. Invalid intervals such as ‘a\{1z’ are not accepted.

The longest possible match is returned; this applies to the regular expression as a whole and (subject to this constraint) to subexpressions within groups.

8.5.5 ‘posix-awk’ regular expression syntax

The character ‘.’ matches any single character except the null character.

‘+’

indicates that the regular expression should match one or more occurrences of the previous atom or regexp.

‘?’

indicates that the regular expression should match zero or one occurrence of the previous atom or regexp.

‘\+’

matches a ‘+’

‘\?’

matches a ‘?’.

Bracket expressions are used to match ranges of characters. Bracket expressions where the range is backward, for example ‘[z-a]’, are invalid. Within square brackets, ‘\’ can be used to quote the following character. Character classes are supported; for example ‘[[:digit:]]’ will match a single decimal digit.

GNU extensions are not supported and so ‘\w’, ‘\W’, ‘\<’, ‘\>’, ‘\b’, ‘\B’, ‘\`’, and ‘\'’ match ‘w’, ‘W’, ‘<’, ‘>’, ‘b’, ‘B’, ‘`’, and ‘'’ respectively.

Grouping is performed with parentheses ‘()’. An unmatched ‘)’ matches just itself. A backslash followed by a digit acts as a back-reference and matches the same thing as the previous grouped expression indicated by that number. For example ‘\2’ matches the second group expression. The order of group expressions is determined by the position of their opening parenthesis ‘(’.

The alternation operator is ‘|’.

The characters ‘^’ and ‘$’ always represent the beginning and end of a string respectively, except within square brackets. Within brackets, ‘^’ can be used to invert the membership of the character class being specified.

‘*’, ‘+’ and ‘?’ are special at any point in a regular expression except the following places, where they are not allowed:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘(’
3. After the alternation operator ‘|’

Intervals are specified by ‘{’ and ‘}’. Invalid intervals are treated as literals, for example ‘a{1’ is treated as ‘a\{1’

The longest possible match is returned; this applies to the regular expression as a whole and (subject to this constraint) to subexpressions within groups.

8.5.6 ‘awk’ regular expression syntax

The character ‘.’ matches any single character except the null character.

‘+’

indicates that the regular expression should match one or more occurrences of the previous atom or regexp.

‘?’

indicates that the regular expression should match zero or one occurrence of the previous atom or regexp.

‘\+’

matches a ‘+’

‘\?’

matches a ‘?’.

Bracket expressions are used to match ranges of characters. Bracket expressions where the range is backward, for example ‘[z-a]’, are invalid. Within square brackets, ‘\’ can be used to quote the following character. Character classes are supported; for example ‘[[:digit:]]’ will match a single decimal digit.

GNU extensions are not supported and so ‘\w’, ‘\W’, ‘\<’, ‘\>’, ‘\b’, ‘\B’, ‘\`’, and ‘\'’ match ‘w’, ‘W’, ‘<’, ‘>’, ‘b’, ‘B’, ‘`’, and ‘'’ respectively.

Grouping is performed with parentheses ‘()’. An unmatched ‘)’ matches just itself. A backslash followed by a digit matches that digit.

The alternation operator is ‘|’.

The characters ‘^’ and ‘$’ always represent the beginning and end of a string respectively, except within square brackets. Within brackets, ‘^’ can be used to invert the membership of the character class being specified.

‘*’, ‘+’ and ‘?’ are special at any point in a regular expression except:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘(’
3. After the alternation operator ‘|’

The longest possible match is returned; this applies to the regular expression as a whole and (subject to this constraint) to subexpressions within groups.

8.5.7 ‘posix-basic’ regular expression syntax

The character ‘.’ matches any single character except the null character.

‘\+’

indicates that the regular expression should match one or more occurrences of the previous atom or regexp.

‘\?’

indicates that the regular expression should match zero or one occurrence of the previous atom or regexp.

‘+ and ?’

match themselves.

Bracket expressions are used to match ranges of characters. Bracket expressions where the range is backward, for example ‘[z-a]’, are invalid. Within square brackets, ‘\’ is taken literally. Character classes are supported; for example ‘[[:digit:]]’ will match a single decimal digit.

GNU extensions are supported:

1. ‘\w’ matches a character within a word
2. ‘\W’ matches a character which is not within a word
3. ‘\<’ matches the beginning of a word
4. ‘\>’ matches the end of a word
5. ‘\b’ matches a word boundary
6. ‘\B’ matches characters which are not a word boundary
7. ‘\`’ matches the beginning of the whole input
8. ‘\'’ matches the end of the whole input

Grouping is performed with backslashes followed by parentheses ‘\(’, ‘\)’. A backslash followed by a digit acts as a back-reference and matches the same thing as the previous grouped expression indicated by that number. For example ‘\2’ matches the second group expression. The order of group expressions is determined by the position of their opening parenthesis ‘\(’.

The alternation operator is ‘\|’.

The character ‘^’ only represents the beginning of a string when it appears:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘\(’
3. After the alternation operator ‘\|’

The character ‘$’ only represents the end of a string when it appears:

1. At the end of a regular expression
2. Before a close-group, signified by ‘\)’
3. Before the alternation operator ‘\|’

‘\*’, ‘\+’ and ‘\?’ are special at any point in a regular expression except:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘\(’
3. After the alternation operator ‘\|’

Intervals are specified by ‘\{’ and ‘\}’. Invalid intervals such as ‘a\{1z’ are not accepted.

The longest possible match is returned; this applies to the regular expression as a whole and (subject to this constraint) to subexpressions within groups.

8.5.8 ‘posix-egrep’ regular expression syntax

The character ‘.’ matches any single character.

‘+’

indicates that the regular expression should match one or more occurrences of the previous atom or regexp.

‘?’

indicates that the regular expression should match zero or one occurrence of the previous atom or regexp.

‘\+’

matches a ‘+’

‘\?’

matches a ‘?’.

Bracket expressions are used to match ranges of characters. Bracket expressions where the range is backward, for example ‘[z-a]’, are invalid. Within square brackets, ‘\’ is taken literally. Character classes are supported; for example ‘[[:digit:]]’ will match a single decimal digit.

GNU extensions are supported:

1. ‘\w’ matches a character within a word
2. ‘\W’ matches a character which is not within a word
3. ‘\<’ matches the beginning of a word
4. ‘\>’ matches the end of a word
5. ‘\b’ matches a word boundary
6. ‘\B’ matches characters which are not a word boundary
7. ‘\`’ matches the beginning of the whole input
8. ‘\'’ matches the end of the whole input

Grouping is performed with parentheses ‘()’. An unmatched ‘)’ matches just itself. A backslash followed by a digit acts as a back-reference and matches the same thing as the previous grouped expression indicated by that number. For example ‘\2’ matches the second group expression. The order of group expressions is determined by the position of their opening parenthesis ‘(’.

The alternation operator is ‘|’.

The characters ‘^’ and ‘$’ always represent the beginning and end of a string respectively, except within square brackets. Within brackets, ‘^’ can be used to invert the membership of the character class being specified.

The characters ‘*’, ‘+’ and ‘?’ are special anywhere in a regular expression.

Intervals are specified by ‘{’ and ‘}’. Invalid intervals are treated as literals, for example ‘a{1’ is treated as ‘a\{1’

The longest possible match is returned; this applies to the regular expression as a whole and (subject to this constraint) to subexpressions within groups.

8.5.9 ‘egrep’ regular expression syntax

This is a synonym for posix-egrep.

8.5.10 ‘posix-extended’ regular expression syntax

The character ‘.’ matches any single character except the null character.

‘+’

indicates that the regular expression should match one or more occurrences of the previous atom or regexp.

‘?’

indicates that the regular expression should match zero or one occurrence of the previous atom or regexp.

‘\+’

matches a ‘+’

‘\?’

matches a ‘?’.

Bracket expressions are used to match ranges of characters. Bracket expressions where the range is backward, for example ‘[z-a]’, are invalid. Within square brackets, ‘\’ is taken literally. Character classes are supported; for example ‘[[:digit:]]’ will match a single decimal digit.

GNU extensions are supported:

1. ‘\w’ matches a character within a word
2. ‘\W’ matches a character which is not within a word
3. ‘\<’ matches the beginning of a word
4. ‘\>’ matches the end of a word
5. ‘\b’ matches a word boundary
6. ‘\B’ matches characters which are not a word boundary
7. ‘\`’ matches the beginning of the whole input
8. ‘\'’ matches the end of the whole input

Grouping is performed with parentheses ‘()’. An unmatched ‘)’ matches just itself. A backslash followed by a digit acts as a back-reference and matches the same thing as the previous grouped expression indicated by that number. For example ‘\2’ matches the second group expression. The order of group expressions is determined by the position of their opening parenthesis ‘(’.

The alternation operator is ‘|’.

The characters ‘^’ and ‘$’ always represent the beginning and end of a string respectively, except within square brackets. Within brackets, ‘^’ can be used to invert the membership of the character class being specified.

‘*’, ‘+’ and ‘?’ are special at any point in a regular expression except the following places, where they are not allowed:

1. At the beginning of a regular expression
2. After an open-group, signified by ‘(’
3. After the alternation operator ‘|’

Intervals are specified by ‘{’ and ‘}’. Invalid intervals such as ‘a{1z’ are not accepted.

The longest possible match is returned; this applies to the regular expression as a whole and (subject to this constraint) to subexpressions within groups.

10.1.1 The Traditional Way

The traditional way to delete files in /var/tmp/stuff that have not been modified in over 90 days would have been:

find /var/tmp/stuff -mtime +90 -exec /bin/rm {} \;

The above command uses ‘-exec’ to run the /bin/rm command to remove each file. This approach works and in fact would have worked in Version 7 Unix in 1979. However, there are a number of problems with this approach.

The most obvious problem with the approach above is that it causes find to fork every time it finds a file that needs to delete, and the child process then has to use the exec system call to launch /bin/rm. All this is quite inefficient. If we are going to use /bin/rm to do this job, it is better to make it delete more than one file at a time.

The most obvious way of doing this is to use the shell’s command expansion feature:

/bin/rm `find /var/tmp/stuff -mtime +90 -print`

or you could use the more modern form

/bin/rm $(find /var/tmp/stuff -mtime +90 -print)

The commands above are much more efficient than the first attempt. However, there is a problem with them. The shell has a maximum command length which is imposed by the operating system (the actual limit varies between systems). This means that while the command expansion technique will usually work, it will suddenly fail when there are lots of files to delete. Since the task is to delete unwanted files, this is precisely the time we don’t want things to go wrong.

10.1.2 Making Use of xargs

So, is there a way to be more efficient in the use of fork() and exec() without running up against this limit? Yes, we can be almost optimally efficient by making use of the xargs command. The xargs command reads arguments from its standard input and builds them into command lines. We can use it like this:

find /var/tmp/stuff -mtime +90 -print | xargs /bin/rm

For example if the files found by find are /var/tmp/stuff/A, /var/tmp/stuff/B and /var/tmp/stuff/C then xargs might issue the commands

/bin/rm /var/tmp/stuff/A /var/tmp/stuff/B
/bin/rm /var/tmp/stuff/C

The above assumes that xargs has a very small maximum command line length. The real limit is much larger but the idea is that xargs will run /bin/rm as many times as necessary to get the job done, given the limits on command line length.

This usage of xargs is pretty efficient, and the xargs command is widely implemented (all modern versions of Unix offer it). So far then, the news is all good. However, there is bad news too.

10.1.3 Unusual characters in filenames

Unix-like systems allow any characters to appear in file names with the exception of the ASCII NUL character and the slash. Slashes can occur in path names (as the directory separator) but not in the names of actual directory entries. This means that the list of files that xargs reads could in fact contain white space characters – spaces, tabs and newline characters. Since by default, xargs assumes that the list of files it is reading uses white space as an argument separator, it cannot correctly handle the case where a filename actually includes white space. This makes the default behaviour of xargs almost useless for handling arbitrary data.

To solve this problem, GNU findutils introduced the ‘-print0’ action for find. This uses the ASCII NUL character to separate the entries in the file list that it produces. This is the ideal choice of separator since it is the only character that cannot appear within a path name. The ‘-0’ option to xargs makes it assume that arguments are separated with ASCII NUL instead of white space. It also turns off another misfeature in the default behaviour of xargs, which is that it pays attention to quote characters in its input. Some versions of xargs also terminate when they see a lone ‘_’ in the input, but GNU find no longer does that (since it has become an optional behaviour in the Unix standard).

So, putting find -print0 together with xargs -0 we get this command:

find /var/tmp/stuff -mtime +90 -print0 | xargs -0 /bin/rm

The result is an efficient way of proceeding that correctly handles all the possible characters that could appear in the list of files to delete. This is good news. However, there is, as I’m sure you’re expecting, also more bad news. The problem is that this is not a portable construct; although other versions of Unix (notably BSD-derived ones) support ‘-print0’, it’s not universal. So, is there a more universal mechanism?

10.1.4 Going back to -exec

There is indeed a more universal mechanism, which is a slight modification to the ‘-exec’ action. The normal ‘-exec’ action assumes that the command to run is terminated with a semicolon (the semicolon normally has to be quoted in order to protect it from interpretation as the shell command separator). The SVR4 edition of Unix introduced a slight variation, which involves terminating the command with ‘+’ instead:

find /var/tmp/stuff -mtime +90 -exec /bin/rm {} \+

The above use of ‘-exec’ causes find to build up a long command line and then issue it. This can be less efficient than some uses of xargs; for example xargs allows building up new command lines while the previous command is still executing, and allows specifying a number of commands to run in parallel. However, the find … -exec … + construct has the advantage of wide portability. GNU findutils did not support ‘-exec … +’ until version 4.2.12; one of the reasons for this is that it already had the ‘-print0’ action in any case.

10.1.5 A more secure version of -exec

The command above seems to be efficient and portable. However, within it lurks a security problem. The problem is shared with all the commands we’ve tried in this worked example so far, too. The security problem is a race condition; that is, if it is possible for somebody to manipulate the filesystem that you are searching while you are searching it, it is possible for them to persuade your find command to cause the deletion of a file that you can delete but they normally cannot.

The problem occurs because the ‘-exec’ action is defined by the POSIX standard to invoke its command with the same working directory as find had when it was started. This means that the arguments which replace the {} include a relative path from find’s starting point down the file that needs to be deleted. For example,

find /var/tmp/stuff -mtime +90 -exec /bin/rm {} \+

might actually issue the command:

/bin/rm /var/tmp/stuff/A /var/tmp/stuff/B /var/tmp/stuff/passwd

Notice the file /var/tmp/stuff/passwd. Likewise, the command:

cd /var/tmp && find stuff -mtime +90 -exec /bin/rm {} \+

might actually issue the command:

/bin/rm stuff/A stuff/B stuff/passwd

If an attacker can rename stuff to something else (making use of their write permissions in /var/tmp) they can replace it with a symbolic link to /etc. That means that the /bin/rm command will be invoked on /etc/passwd. If you are running your find command as root, the attacker has just managed to delete a vital file. All they needed to do to achieve this was replace a subdirectory with a symbolic link at the vital moment.

There is however, a simple solution to the problem. This is an action which works a lot like -exec but doesn’t need to traverse a chain of directories to reach the file that it needs to work on. This is the ‘-execdir’ action, which was introduced by the BSD family of operating systems. The command,

find /var/tmp/stuff -mtime +90 -execdir /bin/rm {} \+

might delete a set of files by performing these actions:

1. Change directory to /var/tmp/stuff/foo
2. Invoke /bin/rm ./file1 ./file2 ./file3
3. Change directory to /var/tmp/stuff/bar
4. Invoke /bin/rm ./file99 ./file100 ./file101

This is a much more secure method. We are no longer exposed to a race condition. For many typical uses of find, this is the best strategy. It’s reasonably efficient, but the length of the command line is limited not just by the operating system limits, but also by how many files we actually need to delete from each directory.

Is it possible to do any better? In the case of general file processing, no. However, in the specific case of deleting files it is indeed possible to do better.

10.1.6 Using the -delete action

The most efficient and secure method of solving this problem is to use the ‘-delete’ action:

find /var/tmp/stuff -mtime +90 -delete

This alternative is more efficient than any of the ‘-exec’ or ‘-execdir’ actions, since it entirely avoids the overhead of forking a new process and using exec to run /bin/rm. It is also normally more efficient than xargs for the same reason. The file deletion is performed from the directory containing the entry to be deleted, so the ‘-delete’ action has the same security advantages as the ‘-execdir’ action has.

The ‘-delete’ action was introduced by the BSD family of operating systems.

10.1.7 Improving things still further

Is it possible to improve things still further? Not without either modifying the system library to the operating system or having more specific knowledge of the layout of the filesystem and disk I/O subsystem, or both.

The find command traverses the filesystem, reading directories. It then issues a separate system call for each file to be deleted. If we could modify the operating system, there are potential gains that could be made:

• We could have a system call to which we pass more than one filename for deletion
• Alternatively, we could pass in a list of inode numbers (on GNU/Linux systems, readdir() also returns the inode number of each directory entry) to be deleted.

The above possibilities sound interesting, but from the kernel’s point of view it is difficult to enforce standard Unix access controls for such processing by inode number. Such a facility would probably need to be restricted to the superuser.

Another way of improving performance would be to increase the parallelism of the process. For example if the directory hierarchy we are searching is actually spread across a number of disks, we might somehow be able to arrange for find to process each disk in parallel. In practice GNU find doesn’t have such an intimate understanding of the system’s filesystem layout and disk I/O subsystem.

However, since the system administrator can have such an understanding they can take advantage of it like so:

find /var/tmp/stuff1 -mtime +90 -delete &
find /var/tmp/stuff2 -mtime +90 -delete &
find /var/tmp/stuff3 -mtime +90 -delete &
find /var/tmp/stuff4 -mtime +90 -delete &
wait

In the example above, four separate instances of find are used to search four subdirectories in parallel. The wait command simply waits for all of these to complete. Whether this approach is more or less efficient than a single instance of find depends on a number of things:

• Are the directories being searched in parallel actually on separate disks? If not, this parallel search might just result in a lot of disk head movement and so the speed might even be slower.
• Other activity - are other programs also doing things on those disks?

10.1.8 Conclusion

The fastest and most secure way to delete files with the help of find is to use ‘-delete’. Using xargs -0 -P N can also make effective use of the disk, but it is not as secure.

In the case where we’re doing things other than deleting files, the most secure alternative is ‘-execdir … +’, but this is not as portable as the insecure action ‘-exec … +’.

The ‘-delete’ action is not completely portable, but the only other possibility which is as secure (‘-execdir’) is no more portable. The most efficient portable alternative is ‘-exec …+’, but this is insecure and isn’t supported by versions of GNU findutils prior to 4.2.12.

10.3.1 Updating the Timestamp The Wrong Way

The obvious but wrong answer is just to use ‘-newer’:

find subdir -newer timestamp -exec touch -r {} timestamp \;

This does the right sort of thing but has a bug. Suppose that two files in the subdirectory have been updated, and that these are called file1 and file2. The command above will update timestamp with the modification time of file1 or that of file2, but we don’t know which one. Since the timestamps on file1 and file2 will in general be different, this could well be the wrong value.

One solution to this problem is to modify find to recheck the modification time of timestamp every time a file is to be compared against it, but that will reduce the performance of find.

10.3.2 Using the test utility to compare timestamps

The test command can be used to compare timestamps:

find subdir -exec test {} -nt timestamp \; -exec touch -r {} timestamp \;

This will ensure that any changes made to the modification time of timestamp that take place during the execution of find are taken into account. This resolves our earlier problem, but unfortunately this runs much more slowly.

10.3.3 A combined approach

We can of course still use ‘-newer’ to cut down on the number of calls to test:

find subdir -newer timestamp -and \
     -exec test {} -nt timestamp \; -and \
     -exec touch -r {} timestamp \;

Here, the ‘-newer’ test excludes all the files which are definitely older than the timestamp, but all the files which are newer than the old value of the timestamp are compared against the current updated timestamp.

This is indeed faster in general, but the speed difference will depend on how many updated files there are.

10.3.4 Using -printf and sort to compare timestamps

It is possible to use the ‘-printf’ action to abandon the use of test entirely:

newest=$(find subdir -newer timestamp -printf "%A@:%p\n" |
           sort -n |
           tail -n1 |
           cut -d: -f2- )
touch -r "${newest:-timestamp}" timestamp

The command above works by generating a list of the timestamps and names of all the files which are newer than the timestamp. The sort, tail and cut commands simply pull out the name of the file with the largest timestamp value (that is, the latest file). The touch command is then used to update the timestamp,

The "${newest:-timestamp}" expression simply expands to the value of $newest if that variable is set, but to timestamp otherwise. This ensures that an argument is always given to the ‘-r’ option of the touch command.

This approach seems quite efficient, but unfortunately it has a problem. Many operating systems now keep file modification time information at a granularity which is finer than one second. Findutils version 4.3.3 and later will print a fractional part with %A@, but older versions will not.

10.3.5 Solving the problem with make

Another tool which often works with timestamps is make. We can use find to generate a Makefile file on the fly and then use make to update the timestamps:

makefile=$(mktemp)
find subdir \
        \( \! -xtype l \) \
        -newer timestamp \
        -printf "timestamp:: %p\n\ttouch -r %p timestamp\n\n" > "$makefile"
make -f "$makefile"
rm   -f "$makefile"

Unfortunately although the solution above is quite elegant, it fails to cope with white space within file names, and adjusting it to do so would require a rather complex shell script.

10.3.6 Coping with odd filenames too

We can fix both of these problems (looping and problems with white space), and do things more efficiently too. The following command works with newlines and doesn’t need to sort the list of filenames.

find subdir -newer timestamp -printf "%A@:%p\0" |
   perl -0 newest.pl |
   xargs --no-run-if-empty --null --replace \
      find {} -maxdepth 0 -newer timestamp -exec touch -r {} timestamp \;

The first find command generates a list of files which are newer than the original timestamp file, and prints a list of them with their timestamps. The newest.pl script simply filters out all the filenames which have timestamps which are older than whatever the newest file is:

#! /usr/bin/perl -0
my @newest = ();
my $latest_stamp = undef;
while (<>) {
    my ($stamp, $name) = split(/:/);
    if (!defined($latest_stamp) || ($tstamp > $latest_stamp)) {
        $latest_stamp = $stamp;
        @newest = ();
    }
    if ($tstamp >= $latest_stamp) {
        push @newest, $name;
    }
}
print join("\0", @newest);

This prints a list of zero or more files, all of which are newer than the original timestamp file, and which have the same timestamp as each other, to the nearest second. The second find command takes each resulting file one at a time, and if that is newer than the timestamp file, the timestamp is updated.

11.2.1 Problems with -exec and filenames

It is safe in many cases to use the ‘-execdir’ action with any file name. Because ‘-execdir’ prefixes the arguments it passes to programs with ‘./’, you will not accidentally pass an argument which is interpreted as an option. For example the file -f would be passed to rm as ./-f, which is harmless.

However, your degree of safety does depend on the nature of the program you are running. For example constructs such as these two commands

# risky
find -exec sh -c "something {}" \;
find -execdir sh -c "something {}" \;

are very dangerous. The reason for this is that the ‘{}’ is expanded to a filename which might contain a semicolon or other characters special to the shell. If for example someone creates the file /tmp/foo; rm -rf $HOME then the two commands above could delete someone’s home directory.

So for this reason do not run any command which will pass untrusted data (such as the names of files) to commands which interpret arguments as commands to be further interpreted (for example ‘sh’).

In the case of the shell, there is a clever workaround for this problem:

# safer
find -exec sh -c 'something "$@"' sh {} \;
find -execdir sh -c 'something "$@"' sh {} \;

This approach is not guaranteed to avoid every problem, but it is much safer than substituting data of an attacker’s choice into the text of a shell command.

11.2.2 Changing the Current Working Directory

As find searches the filesystem, it finds subdirectories and then searches within them by changing its working directory. First, find reaches and recognises a subdirectory. It then decides if that subdirectory meets the criteria for being searched; that is, any ‘-xdev’ or ‘-prune’ expressions are taken into account. The find program will then change working directory and proceed to search the directory.

A race condition attack might take the form that once the checks relevant to ‘-xdev’ and ‘-prune’ have been done, an attacker might rename the directory that was being considered, and put in its place a symbolic link that actually points somewhere else.

The idea behind this attack is to fool find into going into the wrong directory. This would leave find with a working directory chosen by an attacker, bypassing any protection apparently provided by ‘-xdev’ and ‘-prune’, and any protection provided by being able to not list particular directories on the find command line. This form of attack is particularly problematic if the attacker can predict when the find command will be run, as is the case with cron tasks for example.

GNU find has specific safeguards to prevent this general class of problem. The exact form of these safeguards depends on the properties of your system.

• O_NOFOLLOW
• Systems without O_NOFOLLOW

find --version | grep Features

Features enabled: D_TYPE O_NOFOLLOW(enabled) LEAF_OPTIMISATION \
FTS(FTS_CWDFD) CBO(level=2)

11.2.3 Race Conditions with -exec

The ‘-exec’ action causes another program to be run. It passes to the program the name of the file which is being considered at the time. The invoked program will typically then perform some action on that file. Once again, there is a race condition which can be exploited here. We shall take as a specific example the command

find /tmp -path /tmp/umsp/passwd -exec /bin/rm

In this simple example, we are identifying just one file to be deleted and invoking /bin/rm to delete it. A problem exists because there is a time gap between the point where find decides that it needs to process the ‘-exec’ action and the point where the /bin/rm command actually issues the unlink() system call to delete the file from the filesystem. Within this time period, an attacker can rename the /tmp/umsp directory, replacing it with a symbolic link to /etc. There is no way for /bin/rm to determine that it is working on the same file that find had in mind. Once the symbolic link is in place, the attacker has persuaded find to cause the deletion of the /etc/passwd file, which is not the effect intended by the command which was actually invoked.

One possible defence against this type of attack is to modify the behaviour of ‘-exec’ so that the /bin/rm command is run with the argument ./passwd and a suitable choice of working directory. This would allow the normal sanity check that find performs to protect against this form of attack too. Unfortunately, this strategy cannot be used as the POSIX standard specifies that the current working directory for commands invoked with ‘-exec’ must be the same as the current working directory from which find was invoked. This means that the ‘-exec’ action is inherently insecure and can’t be fixed.

GNU find implements a more secure variant of the ‘-exec’ action, ‘-execdir’. The ‘-execdir’ action ensures that it is not necessary to dereference subdirectories to process target files. The current directory used to invoke programs is the same as the directory in which the file to be processed exists (/tmp/umsp in our example, and only the basename of the file to be processed is passed to the invoked command, with a ‘./’ prepended (giving ./passwd in our example).

The ‘-execdir’ action refuses to do anything if the current directory is included in the PATH environment variable. This is necessary because ‘-execdir’ runs programs in the same directory in which it finds files – in general, such a directory might be writable by untrusted users. For similar reasons, ‘-execdir’ does not allow ‘{}’ to appear in the name of the command to be run.

11.2.4 Race Conditions with -print and -print0

The ‘-print’ and ‘-print0’ actions can be used to produce a list of files matching some criteria, which can then be used with some other command, perhaps with xargs. Unfortunately, this means that there is an unavoidable time gap between find deciding that one or more files meet its criteria and the relevant command being executed. For this reason, the ‘-print’ and ‘-print0’ actions are just as insecure as ‘-exec’.

In fact, since the construction

find …  -print | xargs ...

does not cope correctly with newlines or other “white space” in file names, and copes poorly with file names containing quotes, the ‘-print’ action is less secure even than ‘-print0’.

11.4.1 Race Conditions

It is fairly unusual for the output of locate to be fed into another command. However, if this were to be done, this would raise the same set of security issues as the use of ‘find … -print’. Although the problems relating to whitespace in file names can be resolved by using locate’s ‘-0’ option, this still leaves the race condition problems associated with ‘find … -print0’. There is no way to avoid these problems in the case of locate.