hadoop mapreduce 串行执行,比较慢,请问可能什么原因



 hive>insert overwrite table archive_seller_by_geo_per_day partition(partDate, shard) select sellerTokenID, partDate archiveDate, countryCode, country, countryState, countryCity, count(*) pv, count(distinct ip) uv, partDate, shard from page_visit where partDate >= 20140105 and partDate <= 20140107 group by sellerTokenID, countryCode, country, countryState, countryCity, partDate, shard having sellerTokenID>0;


2014-01-06 10:48:31,590 null map = 0%,  reduce = 0%
2014-01-06 10:48:46,353 null map = 1%,  reduce = 0%
2014-01-06 10:48:48,775 null map = 2%,  reduce = 0%
2014-01-06 10:49:04,329 null map = 5%,  reduce = 0%
2014-01-06 10:49:22,239 null map = 7%,  reduce = 0%
2014-01-06 10:49:25,585 null map = 100%,  reduce = 0%
2014-01-06 10:49:27,814 null map = 12%,  reduce = 0%
2014-01-06 10:49:46,299 null map = 15%,  reduce = 0%
2014-01-06 10:50:06,873 null map = 19%,  reduce = 0%
2014-01-06 10:50:11,868 null map = 100%,  reduce = 0%
2014-01-06 10:50:14,524 null map = 22%,  reduce = 0%
2014-01-06 10:50:30,130 null map = 25%,  reduce = 0%
2014-01-06 10:50:33,458 null map = 26%,  reduce = 0%
2014-01-06 10:50:44,565 null map = 28%,  reduce = 0%
2014-01-06 10:50:51,427 null map = 30%,  reduce = 0%
2014-01-06 10:50:53,719 null map = 100%,  reduce = 0%
2014-01-06 10:50:56,403 null map = 34%,  reduce = 0%
2014-01-06 10:51:15,112 null map = 37%,  reduce = 0%
2014-01-06 10:51:29,410 null map = 41%,  reduce = 0%
2014-01-06 10:51:32,758 null map = 100%,  reduce = 0%
2014-01-06 10:51:35,993 null map = 45%,  reduce = 0%
2014-01-06 10:51:50,467 null map = 48%,  reduce = 0%
2014-01-06 10:52:06,274 null map = 52%,  reduce = 0%
2014-01-06 10:52:07,383 null map = 100%,  reduce = 0%
2014-01-06 10:52:10,282 null map = 56%,  reduce = 0%
2014-01-06 10:52:25,854 null map = 59%,  reduce = 0%
2014-01-06 10:52:40,382 null map = 63%,  reduce = 0%
2014-01-06 10:52:41,565 null map = 100%,  reduce = 0%
2014-01-06 10:52:44,892 null map = 67%,  reduce = 0%
2014-01-06 10:52:55,913 null map = 70%,  reduce = 0%
2014-01-06 10:53:11,398 null map = 74%,  reduce = 0%
2014-01-06 10:53:13,589 null map = 100%,  reduce = 0%
2014-01-06 10:53:15,911 null map = 78%,  reduce = 0%
2014-01-06 10:53:28,068 null map = 82%,  reduce = 0%
2014-01-06 10:53:36,848 null map = 84%,  reduce = 0%
2014-01-06 10:53:43,340 null map = 85%,  reduce = 0%
2014-01-06 10:53:44,579 null map = 100%,  reduce = 0%
2014-01-06 10:53:46,846 null map = 89%,  reduce = 0%
2014-01-06 10:53:56,057 null map = 96%,  reduce = 0%
2014-01-06 10:53:57,162 null map = 100%,  reduce = 0%
2014-01-06 10:54:03,018 null map = 100%,  reduce = 67%
2014-01-06 10:54:06,450 null map = 100%,  reduce = 68%
2014-01-06 10:54:08,777 null map = 100%,  reduce = 69%
2014-01-06 10:54:14,706 null map = 100%,  reduce = 70%
2014-01-06 10:54:17,933 null map = 100%,  reduce = 71%
2014-01-06 10:54:21,429 null map = 100%,  reduce = 72%
2014-01-06 10:54:24,682 null map = 100%,  reduce = 73%
2014-01-06 10:54:26,845 null map = 100%,  reduce = 74%
2014-01-06 10:54:30,375 null map = 100%,  reduce = 75%
2014-01-06 10:54:33,781 null map = 100%,  reduce = 76%
2014-01-06 10:54:36,054 null map = 100%,  reduce = 77%
2014-01-06 10:54:39,347 null map = 100%,  reduce = 78%
2014-01-06 10:54:42,764 null map = 100%,  reduce = 79%
2014-01-06 10:54:44,912 null map = 100%,  reduce = 80%
2014-01-06 10:54:48,182 null map = 100%,  reduce = 81%
2014-01-06 10:54:51,553 null map = 100%,  reduce = 82%
2014-01-06 10:54:53,814 null map = 100%,  reduce = 83%
2014-01-06 10:54:57,005 null map = 100%,  reduce = 84%
2014-01-06 10:55:00,303 null map = 100%,  reduce = 85%
2014-01-06 10:55:06,918 null map = 100%,  reduce = 86%
2014-01-06 10:55:09,084 null map = 100%,  reduce = 87%
2014-01-06 10:55:12,400 null map = 100%,  reduce = 88%
2014-01-06 10:55:15,788 null map = 100%,  reduce = 89%
2014-01-06 10:55:17,930 null map = 100%,  reduce = 90%
2014-01-06 10:55:21,452 null map = 100%,  reduce = 91%
2014-01-06 10:55:24,815 null map = 100%,  reduce = 92%
2014-01-06 10:55:27,028 null map = 100%,  reduce = 93%
2014-01-06 10:55:30,222 null map = 100%,  reduce = 94%
2014-01-06 10:55:33,535 null map = 100%,  reduce = 95%
2014-01-06 10:55:36,771 null map = 100%,  reduce = 96%
2014-01-06 10:55:39,249 null map = 100%,  reduce = 97%
2014-01-06 10:55:42,651 null map = 100%,  reduce = 98%
2014-01-06 10:55:45,888 null map = 100%,  reduce = 99%
2014-01-06 10:55:50,289 null map = 100%,  reduce = 100%

Time taken: 499.09 seconds

日志没有错误,结果也执行正常,但速度太慢,请问可能是什么原因导致的呢?谢谢~~

hadoop mapreduce hive

当麻当麻- 11 years, 11 months ago

Your Answer