hadoop fs -test example

To test if file or directory exists in HDFS

If try to use normal shell syntax like

           if [ `hadoop fs -test -d /dev/pathToTest` -ne 0 ]; then
               hadoop fs -mkdir /dev/pathToTest
               echo “Creating  directory”
           fi

Then it does not works.

Reason being the result of the test is reported in the shell exit code, not as a textual output
from the command, just like the UNIX /usr/bin/test command.

The correct usage is given as below


if hadoop fs -test –d /dev/pathToTest ; then
                                echo "Directory  exists"
else
                                hadoop fs -mkdir /dev/pathToTest
                                 echo “Creating  directory”
fi



if hadoop fs -test –e /dev/pathToTest/file.html ; then
                                echo "File  exists"
else
                                echo “File does not exists ”
fi

For more details about HDFS commands please see Hadoop command guide at our favorite website Apache 

http://hadoop.apache.org/docs/r1.1.0/file_system_shell.html

Happy Hadooping :)

1 comment:

  1. Thank you for the examples. Really helped.

    ReplyDelete

Please share your views and comments below.

Thank You.