Shulou(Shulou.com)06/03 Report--

The following error is reported when using impala to query distinct with a large amount of data

WARNINGS: Create file / tmp/impala-scratch/24414dab2c19caca:e54b206c5ab149d_24414dab2c19caca:e54b206c5ab149f_91001337-9d70-4c93-84ce-e7916c1ae804 failed with errno=2 description=Error (2): No such file or directoryBackend 4:Create file / tmp/impala-scratch/24414dab2c19caca:e54b206c5ab149d_24414dab2c19caca:e54b206c5ab149f_91001337-9d70-4c93-84ce-e7916c1ae804 failed with errno=2 description=Error (2): No such file or directory

Consult the document and know that impala will use disk to save data when dealing with a large amount of data.

By default, intermediate files used during large sort, join, aggregation, or analytic function operations are stored in the directory / tmp/impala-scratch. These files are removed when the operation finishes. Multiple concurrent queries can perform operations that use the "spill to disk" technique, without any name conflicts for these temporary files. You can specify a different location by starting the impalad daemon with the-scratch_dirs= "path_to_directory" configuration option or the equivalent configuration option in the Cloudera Manager user interface. You can specify a single directory, or a comma-separated list of directories. The scratch directories must be on the local filesystem, not in HDFS. You might specify different directory paths for different hosts, depending on the capacity and speed of the available storage devices. Impala will not start if it cannot create or read and write files in the "scratch" directory. If there is less than 1 GB free on the filesystem where that directory resides, Impala still runs, but writes a warning message to its log.

Create / tmp/impala-scratch directory directly on each impalad node and grant read and write permissions

Mkdir / tmp/impala-scratch chmod 777 / tmp/impala-scratch

Welcome to subscribe "Shulou Technology Information " to get latest news, interesting things and hot topics in the IT industry, and controls the hottest and latest Internet news, technology news and IT industry trends.