hive select product_id, track_time from trackinfo limit 5; Total MapReduce jobs = 1Launching Job 1 out of 1Number of reduce tasks is set to 0 since there‘s no reduce operatoro...
hive> select product_id, track_time from trackinfo limit 5;
Total MapReduce jobs = 1
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there‘s no reduce operator
org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException: The number of tasks for this job 156028 exceeds the configured limit 5000
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
Caused by: java.io.IOException: The number of tasks for this job 156028 exceeds the configured limit 5000
at org.apache.hadoop.mapred.JobInProgress.checkTaskLimits(JobInProgress.java:509)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:485)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
... 10 more
at org.apache.hadoop.ipc.Client.call(Client.java:1066)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:225)
at org.apache.hadoop.mapred.$Proxy11.submitJob(Unknown Source)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:921)
at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:850)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:850)
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:824)
at org.apache.hadoop.hive.ql.exec.ExecDriver.execute(ExecDriver.java:447)
at org.apache.hadoop.hive.ql.exec.MapRedTask.execute(MapRedTask.java:136)
at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:138)
at org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:57)
at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1336)
at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1122)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:935)
at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:755)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
Job Submission failed with exception ‘org.apache.hadoop.ipc.RemoteException(java.io.IOException: java.io.IOException: The number of tasks for this job 156028 exceeds the configured limit 5000
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3943)
at sun.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:563)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1388)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1384)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1093)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1382)
Caused by: java.io.IOException: The number of tasks for this job 156028 exceeds the configured limit 5000
at org.apache.hadoop.mapred.JobInProgress.checkTaskLimits(JobInProgress.java:509)
at org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:485)
at org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:3941)
... 10 more
)‘
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
错误原因:
由于trackinfo表的数据太庞大,而且写的sql语句
select product_id, track_time from trackinfo limit 5
性能耗损太大了,将会导致调度的Map数太多,超出了job的限制。
来看看hive中trackinfo表的数据量有多大:
-bash-3.2$ hadoop fs -dus /data/share/trackinfo Warning: $HADOOP_HOME is deprecated. hdfs://yhd-hadoop06.int.yihaodian.com:9000/data/share/trackinfo 19387740988708
可以看到,trackinfo的数据量大概有19TB这么大
改写sql语句,指定where条件,性能得到了提升
select product_id, track_time from trackinfo where ds=‘2014-5-13‘ limit 5
问题同时也被解决了
原文:http://blog.csdn.net/yeweiouyang/article/details/25806815
织梦狗教程
本文标题为:hive执行query语句时提示错误:org.apache.hadoop.ipc.RemoteException: java.io.IOException: java.io.IOException:
基础教程推荐
猜你喜欢
- windows环境下apache-apollo服务器搭建 2023-09-10
- Apache CarbonData 1.0.0发布及其新特性介绍 2023-09-11
- 为Win2003服务器打造铜墙铁壁的方法步骤 2022-09-01
- Docker容器操作方法详解 2022-11-13
- RFO SIG之openEuler AWS AMI 制作详解 2022-12-28
- 通过StatefulSet部署有状态服务应用实现方式 2022-10-01
- Apache Kafka 2.5 稳定版发布,新特性抢先看 2023-09-11
- P3 利用Vulnhub复现漏洞 - Apache SSI 远程命令执行漏洞 2023-09-10
- Centos 安装Django2.1 2023-09-24
- Centos7 nginx的安装以及开机自启动的设置 2023-09-22
