To set the number of reducers for all Pig opeations, you can use the default_parallelproperty – but this means every single step will use a single reducer, decreasing throughput:set default_parallel 1; Prior to calling STORE, if one of the... Read More | Share it now!
Tag: hadoop
Apache Pig教程
https://www.w3cschool.cn/apache_pig/ ... Read More | Share it now!
【HDFS】hive任务报HDFS异常：last block does not have enough number of replicas
【HDFS】hive任务报HDFS异常：last block does not have enough number of replicas 【问题现象】 TSP客户端在2015-05-27 19:00:52.811时出现close ERROR，日志如下： 2015-05-27 19:00:52.811 ERROR:... Read More | Share it now!
Exception in thread “main” org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /testtxt. could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
Issue: When you upload a file to HDFS, you may encounter following exception. Exception: Exception in thread “main” org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /testtxt. could only be replicated to 0 nodes instead of... Read More | Share it now!
MapReduce:详解Shuffle过程
http://langyu.iteye.com/blog/992916 Shuffle过程是MapReduce的核心，也被称为奇迹发生的地方。要想理解MapReduce，... Read More | Share it now!
MapReduce: Job提交过程
初学Hadoop，准备用几篇日志来陈述MapReduce job的生命周期中job提交、task分发和task执行，以及JT... Read More | Share it now!
java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy
11:25:58.779 DEBUG o.a.h.h.p.d.s.DataTransferSaslUtil - DataTransferProtocol not using SaslPropertiesResolver, no QOP found in configuration for dfs.data.transfer.protection Exception in thread "main" java.lang.UnsatisfiedLinkError:... Read More | Share it now!
install snappy lib
2.2 安装依赖 首先在服务器上安装snappy，而安装snappy需要一系列的依赖，为了避免麻烦我们首先安装好这些依赖 $ sudo yum -y install gcc c++ autoconf automake libtool 1 2.3... Read More | Share it now!
mapreduce的二次排序 SecondarySort
关于二次排序主要涉及到这么几个东西： 在0.20.0 以前使用的是 setPartitionerClass setOutputkeyComparatorClass setOutputValueGroupingComparator 在0.20.0以后使用是 job.setPartitionerClass(Partitioner... Read More | Share it now!
“bad substitution” error running Spark on Yarn
you can update the hdp.version property through Ambari GUI. below are the steps: 1. Go to ‘Ambari -> YARN -> configs’ and go to ‘Advanced’ tab. 2. scroll down the page to till end, there will find an option to add custom... Read More | Share it now!