Working with Directories with a Large Number of Files

Contents

1. 1. Show first files
2. 2. Count the number of files
3. 3. Remove files
   1. 3.1. Using for Loop
   2. 3.2. Using Find

Today I try to delete a directory with 485,000+ files on Linux server. It shows the error message below.

/bin/rm: Argument list too long.

If you’re trying to delete a very large number of files in a single directory at one time, you will probably run into this error. The problem is that when you type something like rm -f /data/*. So how do we working with directories with a large number of files? Let talk about it below.

Show first files

To show first n files in the directory instead of to show all files.

ls -U | head -10

Count the number of files

ls -f | wc -l

Remove files

Using for Loop

for f in *.pdf; do echo rm "$f"; done
for f in *.pdf; do rm "$f"; done

Using Find

Using find -delete. (2000 files/second, it is about three time faster rm)

find . -maxdepth 1 -type f -name "$.split-*" -print -delete

Using find and xargs. To find every file and pass them one-by-one to the “rm” command (not recommend, xargs is dangerous*)

find . -type f -exec rm -v {} \;

find . -maxdepth 1 -name "$.split-*" -print0 | xargs -0 rm