Input path does not exist

Spark-sql 报错：org.apache.hadoop.mapred.InvalidInputException: Input path does not exist:
现象
spark-sql手动提交任务，报错提示Input path does not exist
PS:查资料过程中，有的用scala\java提交spark任务时，本应该读取HDFS路径，但是spark却读到了local path，有参数控制或者是dfs配置有操作，可以通过参数解决。

分析：
查tablename的hdfs路径，然后查看报错的分区\路径是否存在
desc formatted table_name;
dfs -ls ;

发现spark报错的路径不存在
原因&解决
平时新增数据都是msck repair table_name更新分区，对于历史数据的清理，都是直接操作hdfs路径\文件，没有进行hive元数据清理，导致hive中仍然存储此路径，但是实际文件已不在。
可以alter table test drop partition 手工管理hive元数据，保证spark读取到及时更新的分区。

Input path does not exist相关推荐

Spark读Hive分区表报错：Caused by: org.apache.hadoop.mapred.InvalidInputException: Input path does not exist
前言之前删除Hive分区表的时候,直接用hadoop fs -rm -r 命令删除了hdfs上的分区目录,然后用spark读取hive的这个表的时候,就报错了: Caused by: org.apa ...
Provided path doesn't exist. pylama Please fix pylama path or install latest version（UBuntu版本下的情况）
atom编辑器运行python出现如下报错 Provided path doesn't exist. pylama Please fix pylama path or install latest v ...
【k8s】path does not exist, ignoring“ path=“/etc/kubernetes/manifests“
参考查看kubelet日志 [root@k2c2g3m-worker0 ~]# journalctl -f -u kubelet – Logs begin at Thu 2021-06-24 21: ...
readxl包导入excel报错Error: `path` does not exist解决方法
readxl包导入excel报错Error: `path` does not exist解决方法方法1:File - import dataset - from excel 方法2:检查语法错误正 ...
docker bind source path does not exist: /etc/timezone“
原因 docker容器启动报错,文件挂载失败 invalid mount config for type "bind": bind source path does not exi ...
Fluent报错cl-set-default-save-dir: path does not exist.
Error: cl-set-default-save-dir: path does not exist. Error Object: "E:\???\ansyslainxi\WB_ZHANG ...
checkmarx : Input path not canocalized 问题解决方法
https://www.gyanblog.com/tutorials/resolving-checkmarx-issues-reported/ https://wiki.sei.cmu.edu/con ...
AssertionError: Path does not exist: py-faster-rcnn/data/VOCdevkit2007/VOC2007/ImageSets/Main
copied the VOCdevkit to data folder and renamed it to VOCdevkit2007 and have no problem now. I might ...
python os path isfile_Python path.isfile方法代码示例
本文整理汇总了Python中os.path.isfile方法的典型用法代码示例.如果您正苦于以下问题:Python path.isfile方法的具体用法?Python path.isfile怎么用?P ...

Input path does not exist

Input path does not exist相关推荐

最新文章

热门文章