Integrate LZO with Hadoop2

Apache Hadoop 

---

Apache Hadoop is open source framework  written in java that is design to  store large shemaless/schema dataset in distrbute manner( HDFS)  and computing useful insights from stored data using programming model (Map Reduce).

In HDFS, the data can be compressed for storage to reduce the space it takes to store it and shorter disk read time and also speed the data transfer across network.  

Compression and De-Compression

---

There are many compression-decompression algorithms available such as GZIP, bzip2, LZO, LZ4, Snappy,Avro, Sequence file or Parpet file. Each algorithm has it own space time trade off- faster compression/decompression will space less space and vice-versa.

LZO is distributed under the term of GNU general purpose licence.Therefore, it is not included in apache hadoop and so need to install seperately.

In this tutorial, I will write down the steps to configure LZO with Hadoop.

LZO Format

---

LZO format compress and decompress file pretty efficiently. Although, it compression size is not efficient than GZIP or bzip2 but time taken to decompress is pretty fast. It can also be indexed to support split able.

Configure LZO with Hadoop

---

In the steps below we will demonstrate the steps to configure LZO.

Get LZO native library

---

pooja@pooja:~$ sudo apt-get install liblzo2-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  liblzo2-dev
0 upgraded, 1 newly installed, 0 to remove and 377 not upgraded.
Need to get 93.1 kB of archives.
After this operation, 690 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty-updates/main liblzo2-dev amd64 2.06-1.2ubuntu1.1 [93.1 kB]
Fetched 93.1 kB in 0s (244 kB/s)     
Selecting previously unselected package liblzo2-dev:amd64.
(Reading database ... 184739 files and directories currently installed.)
Preparing to unpack .../liblzo2-dev_2.06-1.2ubuntu1.1_amd64.deb ...
Unpacking liblzo2-dev:amd64 (2.06-1.2ubuntu1.1) ...
Setting up liblzo2-dev:amd64 (2.06-1.2ubuntu1.1) ...

Install LZO

---

Now, download the lzo from the url or use below command to download it.

pooja@pooja:~/dev$ wget http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz
--2017-08-28 16:03:11--  http://www.oberhumer.com/opensource/lzo/download/lzo-2.10.tar.gz
Resolving www.oberhumer.com (www.oberhumer.com)... 193.170.194.40
Connecting to www.oberhumer.com (www.oberhumer.com)|193.170.194.40|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 600622 (587K) [application/x-gzip]
Saving to: ‘lzo-2.10.tar.gz’

100%[=====================================================================================================>] 600,622      601KB/s   in 1.0s   

2017-08-28 16:03:13 (601 KB/s) - ‘lzo-2.10.tar.gz’ saved [600622/600622]

Then, untar the file using the below command.
pooja@pooja:~/dev$ tar xvzf lzo-2.10.tar.gz

Then, configure the lzo.
pooja@pooja:~/dev/lzo-2.10$ ./configure --enable-shared --prefix /usr/local/lzo-2.10
configure: Configuring LZO 2.10
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking target system type... x86_64-pc-linux-gnu
checking whether to enable maintainer-specific portions of Makefiles... no
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking whether gcc understands -c and -o together... yes
checking for ar... ar
checking the archiver (ar) interface... ar
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking whether the C preprocessor needs special flags... none needed
checking for an ANSI C-conforming const... yes
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking whether byte ordering is bigendian... no
checking for special C compiler options needed for large files... no
checking for _FILE_OFFSET_BITS value needed for large files... no
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... no
checking for mawk... mawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking dependency style of gcc... gcc3
checking whether make supports nested variables... (cached) yes
checking how to print strings... printf
checking for a sed that does not truncate output... /bin/sed
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking whether ln -s works... yes
checking the maximum length of command line arguments... 1572864
checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop
checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for dlltool... no
checking how to associate runtime and link libraries... printf %s\n
checking for archiver @FILE support... @
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for sysroot... no
checking for mt... mt
checking if mt is a manifest tool... no
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... yes
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
checking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking whether time.h and sys/time.h may both be included... yes
checking assert.h usability... yes
checking assert.h presence... yes
checking for assert.h... yes
checking ctype.h usability... yes
checking ctype.h presence... yes
checking for ctype.h... yes
checking dirent.h usability... yes
checking dirent.h presence... yes
checking for dirent.h... yes
checking errno.h usability... yes
checking errno.h presence... yes
checking for errno.h... yes
checking fcntl.h usability... yes
checking fcntl.h presence... yes
checking for fcntl.h... yes
checking float.h usability... yes
checking float.h presence... yes
checking for float.h... yes
checking limits.h usability... yes
checking limits.h presence... yes
checking for limits.h... yes
checking malloc.h usability... yes
checking malloc.h presence... yes
checking for malloc.h... yes
checking for memory.h... (cached) yes
checking setjmp.h usability... yes
checking setjmp.h presence... yes
checking for setjmp.h... yes
checking signal.h usability... yes
checking signal.h presence... yes
checking for signal.h... yes
checking stdarg.h usability... yes
checking stdarg.h presence... yes
checking for stdarg.h... yes
checking stddef.h usability... yes
checking stddef.h presence... yes
checking for stddef.h... yes
checking for stdint.h... (cached) yes
checking stdio.h usability... yes
checking stdio.h presence... yes
checking for stdio.h... yes
checking for stdlib.h... (cached) yes
checking for string.h... (cached) yes
checking for strings.h... (cached) yes
checking time.h usability... yes
checking time.h presence... yes
checking for time.h... yes
checking for unistd.h... (cached) yes
checking utime.h usability... yes
checking utime.h presence... yes
checking for utime.h... yes
checking sys/mman.h usability... yes
checking sys/mman.h presence... yes
checking for sys/mman.h... yes
checking sys/resource.h usability... yes
checking sys/resource.h presence... yes
checking for sys/resource.h... yes
checking for sys/stat.h... (cached) yes
checking sys/time.h usability... yes
checking sys/time.h presence... yes
checking for sys/time.h... yes
checking for sys/types.h... (cached) yes
checking sys/wait.h usability... yes
checking sys/wait.h presence... yes
checking for sys/wait.h... yes
checking whether limits.h is sane... yes
checking for off_t... yes
checking for ptrdiff_t... yes
checking for size_t... yes
checking return type of signal handlers... void
checking size of short... 2
checking size of int... 4
checking size of long... 8
checking size of long long... 8
checking size of __int16... 0
checking size of __int32... 0
checking size of __int64... 0
checking size of void *... 8
checking size of size_t... 8
checking size of ptrdiff_t... 8
checking size of __int32... (cached) 0
checking size of intmax_t... 8
checking size of uintmax_t... 8
checking size of intptr_t... 8
checking size of uintptr_t... 8
checking size of float... 4
checking size of double... 8
checking size of long double... 16
checking size of dev_t... 8
checking size of fpos_t... 16
checking size of mode_t... 4
checking size of off_t... 8
checking size of ssize_t... 8
checking size of time_t... 8
checking for access... yes
checking for alloca... no
checking for atexit... yes
checking for atoi... yes
checking for atol... yes
checking for chmod... yes
checking for chown... yes
checking for clock_getcpuclockid... yes
checking for clock_getres... yes
checking for clock_gettime... yes
checking for ctime... yes
checking for difftime... yes
checking for fstat... yes
checking for getenv... yes
checking for getpagesize... yes
checking for getrusage... yes
checking for gettimeofday... yes
checking for gmtime... yes
checking for isatty... yes
checking for localtime... yes
checking for longjmp... yes
checking for lstat... yes
checking for memcmp... yes
checking for memcpy... yes
checking for memmove... yes
checking for memset... yes
checking for mkdir... yes
checking for mktime... yes
checking for mmap... yes
checking for mprotect... yes
checking for munmap... yes
checking for qsort... yes
checking for raise... yes
checking for rmdir... yes
checking for setjmp... yes
checking for signal... yes
checking for snprintf... yes
checking for strcasecmp... yes
checking for strchr... yes
checking for strdup... yes
checking for strerror... yes
checking for strftime... yes
checking for stricmp... no
checking for strncasecmp... yes
checking for strnicmp... no
checking for strrchr... yes
checking for strstr... yes
checking for time... yes
checking for umask... yes
checking for utime... yes
checking for vsnprintf... yes
checking whether to build assembly versions... no
checking whether your compiler passes the LZO conformance test... yes
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating lzo2.pc
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands

   LZO configuration summary
   -------------------------
   LZO version                : 2.10
   configured for host        : x86_64-pc-linux-gnu
   source code location       : .
   compiler                   : gcc
   preprocessor definitions   : -DLZO_HAVE_CONFIG_H=1
   preprocessor flags         : 
   compiler flags             : -g -O2
   build static library       : yes
   build shared library       : yes
   enable i386 assembly code  : no


   LZO 2.10 configured.

   Copyright (C) 1996-2017 Markus Franz Xaver Johannes Oberhumer
   All Rights Reserved.

   The LZO library is free software; you can redistribute it and/or
   modify it under the terms of the GNU General Public License as
   published by the Free Software Foundation; either version 2 of
   the License, or (at your option) any later version.

   The LZO library is distributed in the hope that it will be useful,
   but WITHOUT ANY WARRANTY; without even the implied warranty of
   MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
   GNU General Public License for more details.

   Markus F.X.J. Oberhumer
   <markus@oberhumer.com>
   http://www.oberhumer.com/opensource/lzo/


Type 'make' to build LZO.
Type 'make check' and 'make test' to test LZO.
Type 'make install' to install LZO.
After installing LZO, please have a look at 'examples/simple.c'.

Finally, make and install LZO.

pooja@pooja:~/dev/lzo-2.10$ make && sudo make install

make  all-am

make[1]: Entering directory `/home/pooja/dev/lzo-2.10'

  CC       src/lzo1.lo

  CC       src/lzo1_99.lo

  CC       src/lzo1a.lo

  CC       src/lzo1a_99.lo

  CC       src/lzo1b_1.lo

  CC       src/lzo1b_2.lo

  CC       src/lzo1b_3.lo

  CC       src/lzo1b_4.lo

  CC       src/lzo1b_5.lo

  CC       src/lzo1b_6.lo

  CC       src/lzo1b_7.lo

  CC       src/lzo1b_8.lo

  CC       src/lzo1b_9.lo

  CC       src/lzo1b_99.lo

  CC       src/lzo1b_9x.lo

  CC       src/lzo1b_cc.lo

  CC       src/lzo1b_d1.lo

  CC       src/lzo1b_d2.lo

  CC       src/lzo1b_rr.lo

  CC       src/lzo1b_xx.lo

  CC       src/lzo1c_1.lo

  CC       src/lzo1c_2.lo

  CC       src/lzo1c_3.lo

  CC       src/lzo1c_4.lo

  CC       src/lzo1c_5.lo

  CC       src/lzo1c_6.lo

  CC       src/lzo1c_7.lo

  CC       src/lzo1c_8.lo

  CC       src/lzo1c_9.lo

  CC       src/lzo1c_99.lo

  CC       src/lzo1c_9x.lo

  CC       src/lzo1c_cc.lo

  CC       src/lzo1c_d1.lo

  CC       src/lzo1c_d2.lo

  CC       src/lzo1c_rr.lo

  CC       src/lzo1c_xx.lo

  CC       src/lzo1f_1.lo

  CC       src/lzo1f_9x.lo

  CC       src/lzo1f_d1.lo

  CC       src/lzo1f_d2.lo

  CC       src/lzo1x_1.lo

  CC       src/lzo1x_1k.lo

  CC       src/lzo1x_1l.lo

  CC       src/lzo1x_1o.lo

  CC       src/lzo1x_9x.lo

  CC       src/lzo1x_d1.lo

  CC       src/lzo1x_d2.lo

  CC       src/lzo1x_d3.lo

  CC       src/lzo1x_o.lo

  CC       src/lzo1y_1.lo

  CC       src/lzo1y_9x.lo

  CC       src/lzo1y_d1.lo

  CC       src/lzo1y_d2.lo

  CC       src/lzo1y_d3.lo

  CC       src/lzo1y_o.lo

  CC       src/lzo1z_9x.lo

  CC       src/lzo1z_d1.lo

  CC       src/lzo1z_d2.lo

  CC       src/lzo1z_d3.lo

  CC       src/lzo2a_9x.lo

  CC       src/lzo2a_d1.lo

  CC       src/lzo2a_d2.lo

  CC       src/lzo_crc.lo

  CC       src/lzo_init.lo

  CC       src/lzo_ptr.lo

  CC       src/lzo_str.lo

  CC       src/lzo_util.lo

  CCLD     src/liblzo2.la

  CC       examples/dict.o

  CCLD     examples/dict

  CC       examples/lzopack.o

  CCLD     examples/lzopack

  CC       examples/overlap.o

  CCLD     examples/overlap

  CC       examples/precomp.o

  CCLD     examples/precomp

  CC       examples/precomp2.o

  CCLD     examples/precomp2

  CC       examples/simple.o

  CCLD     examples/simple

  CC       lzotest/lzotest.o

  CCLD     lzotest/lzotest

  CC       tests/align.o

  CCLD     tests/align

  CC       tests/chksum.o

  CCLD     tests/chksum

  CC       tests/promote.o

  CCLD     tests/promote

  CC       tests/sizes.o

  CCLD     tests/sizes

  CC       minilzo/t-testmini.o

  CC       minilzo/t-minilzo.o

  CCLD     minilzo/testmini

make[1]: Leaving directory `/home/pooja/dev/lzo-2.10'

make[1]: Entering directory `/home/pooja/dev/lzo-2.10'

 /bin/mkdir -p '/usr/local/lzo-2.10/lib'

 /bin/bash ./libtool   --mode=install /usr/bin/install -c   src/liblzo2.la '/usr/local/lzo-2.10/lib'

libtool: install: /usr/bin/install -c src/.libs/liblzo2.so.2.0.0 /usr/local/lzo-2.10/lib/liblzo2.so.2.0.0

libtool: install: (cd /usr/local/lzo-2.10/lib && { ln -s -f liblzo2.so.2.0.0 liblzo2.so.2 || { rm -f liblzo2.so.2 && ln -s liblzo2.so.2.0.0 liblzo2.so.2; }; })

libtool: install: (cd /usr/local/lzo-2.10/lib && { ln -s -f liblzo2.so.2.0.0 liblzo2.so || { rm -f liblzo2.so && ln -s liblzo2.so.2.0.0 liblzo2.so; }; })

libtool: install: /usr/bin/install -c src/.libs/liblzo2.lai /usr/local/lzo-2.10/lib/liblzo2.la

libtool: install: /usr/bin/install -c src/.libs/liblzo2.a /usr/local/lzo-2.10/lib/liblzo2.a

libtool: install: chmod 644 /usr/local/lzo-2.10/lib/liblzo2.a

libtool: install: ranlib /usr/local/lzo-2.10/lib/liblzo2.a

libtool: finish: PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/sbin" ldconfig -n /usr/local/lzo-2.10/lib

----------------------------------------------------------------------

Libraries have been installed in:

   /usr/local/lzo-2.10/lib

If you ever happen to want to link against installed libraries

in a given directory, LIBDIR, you must either use libtool, and

specify the full pathname of the library, or use the '-LLIBDIR'

flag during linking and do at least one of the following:

   - add LIBDIR to the 'LD_LIBRARY_PATH' environment variable

     during execution

   - add LIBDIR to the 'LD_RUN_PATH' environment variable

     during linking

   - use the '-Wl,-rpath -Wl,LIBDIR' linker flag

   - have your system administrator add LIBDIR to '/etc/ld.so.conf'

See any operating system documentation about shared libraries for

more information, such as the ld(1) and ld.so(8) manual pages.

----------------------------------------------------------------------

 /bin/mkdir -p '/usr/local/lzo-2.10/share/doc/lzo'

 /usr/bin/install -c -m 644 AUTHORS COPYING NEWS THANKS doc/LZO.FAQ doc/LZO.TXT doc/LZOAPI.TXT '/usr/local/lzo-2.10/share/doc/lzo'

 /bin/mkdir -p '/usr/local/lzo-2.10/lib/pkgconfig'

 /usr/bin/install -c -m 644 lzo2.pc '/usr/local/lzo-2.10/lib/pkgconfig'

 /bin/mkdir -p '/usr/local/lzo-2.10/include/lzo'

 /usr/bin/install -c -m 644 include/lzo/lzo1.h include/lzo/lzo1a.h include/lzo/lzo1b.h include/lzo/lzo1c.h include/lzo/lzo1f.h include/lzo/lzo1x.h include/lzo/lzo1y.h include/lzo/lzo1z.h include/lzo/lzo2a.h include/lzo/lzo_asm.h include/lzo/lzoconf.h include/lzo/lzodefs.h include/lzo/lzoutil.h '/usr/local/lzo-2.10/include/lzo'

make[1]: Leaving directory `/home/pooja/dev/lzo-2.10'

Clone Hadoop LZO source code

---

Now, clone the hadoop-lzo project.

pooja@pooja:~/dev$ git clone https://github.com/twitter/hadoop-lzo.git
Cloning into 'hadoop-lzo'...
remote: Counting objects: 1889, done.
remote: Total 1889 (delta 0), reused 0 (delta 0), pack-reused 1889
Receiving objects: 100% (1889/1889), 16.45 MiB | 4.56 MiB/s, done.
Resolving deltas: 100% (745/745), done.
Checking connectivity... done.

pooja@pooja:~/dev/hadoop-lzo$ C_INCLUDE_PATH=/usr/local/lzo-2.10/include \
> LIBRARY_PATH=/usr/local/lzo-2.10/lib \
>   mvn clean package

[INFO] Scanning for projects...

[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building hadoop-lzo 0.4.21-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-lzo ---
[INFO] Deleting /home/pooja/dev/hadoop-lzo/target
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (check-platform) @ hadoop-lzo ---
[INFO] Executing tasks

check-platform:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (set-props-non-win) @ hadoop-lzo ---
[INFO] Executing tasks

set-props-non-win:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (set-props-win) @ hadoop-lzo ---
[INFO] Executing tasks

set-props-win:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ hadoop-lzo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/pooja/dev/hadoop-lzo/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:compile (default-compile) @ hadoop-lzo ---
[INFO] Compiling 25 source files to /home/pooja/dev/hadoop-lzo/target/classes
[WARNING] bootstrap class path not set in conjunction with -source 1.6
/home/pooja/dev/hadoop-lzo/src/main/java/com/hadoop/compression/lzo/DistributedLzoIndexer.java:[52,20] [deprecation] isDir() in FileStatus has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/main/java/com/hadoop/compression/lzo/DistributedLzoIndexer.java:[112,14] [deprecation] Job(Configuration) in Job has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/main/java/com/hadoop/compression/lzo/LzoIndexer.java:[82,18] [deprecation] isDir() in FileStatus has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/main/java/com/hadoop/mapreduce/LzoIndexOutputFormat.java:[31,28] [deprecation] cleanupJob(JobContext) in OutputCommitter has been deprecated
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (build-info-non-win) @ hadoop-lzo ---
[INFO] Executing tasks

build-info-non-win:
[propertyfile] Creating new property file: /home/pooja/dev/hadoop-lzo/target/classes/hadoop-lzo-build.properties
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (build-info-win) @ hadoop-lzo ---
[INFO] Executing tasks

build-info-win:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (check-native-uptodate-non-win) @ hadoop-lzo ---
[INFO] Executing tasks

check-native-uptodate-non-win:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (check-native-uptodate-win) @ hadoop-lzo ---
[INFO] Executing tasks

check-native-uptodate-win:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (build-native-non-win) @ hadoop-lzo ---
[INFO] Executing tasks

build-native-non-win:
    [mkdir] Created dir: /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/lib
    [mkdir] Created dir: /home/pooja/dev/hadoop-lzo/target/classes/native/Linux-amd64-64/lib
    [mkdir] Created dir: /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/src/com/hadoop/compression/lzo
    [javah] [Forcefully writing file RegularFileObject[/home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/src/com/hadoop/compression/lzo/com_hadoop_compression_lzo_LzoCompressor.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/src/com/hadoop/compression/lzo/com_hadoop_compression_lzo_LzoCompressor_CompressionStrategy.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/src/com/hadoop/compression/lzo/com_hadoop_compression_lzo_LzoDecompressor.h]]
    [javah] [Forcefully writing file RegularFileObject[/home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/src/com/hadoop/compression/lzo/com_hadoop_compression_lzo_LzoDecompressor_CompressionStrategy.h]]
     [exec] checking for a BSD-compatible install... /usr/bin/install -c
     [exec] checking whether build environment is sane... yes
     [exec] checking for a thread-safe mkdir -p... /bin/mkdir -p
     [exec] checking for gawk... no
     [exec] checking for mawk... mawk
     [exec] checking whether make sets $(MAKE)... yes
     [exec] checking whether to enable maintainer-specific portions of Makefiles... no
     [exec] checking for style of include used by make... GNU
     [exec] checking for gcc... gcc
     [exec] checking whether the C compiler works... yes
     [exec] checking for C compiler default output file name... a.out
     [exec] checking for suffix of executables... 
     [exec] checking whether we are cross compiling... no
     [exec] checking for suffix of object files... o
     [exec] checking whether we are using the GNU C compiler... yes
     [exec] checking whether gcc accepts -g... yes
     [exec] checking for gcc option to accept ISO C89... none needed
     [exec] checking dependency style of gcc... gcc3
     [exec] checking how to run the C preprocessor... gcc -E
     [exec] checking for grep that handles long lines and -e... /bin/grep
     [exec] checking for egrep... /bin/grep -E
     [exec] checking for ANSI C header files... yes
     [exec] checking for sys/types.h... yes
     [exec] checking for sys/stat.h... yes
     [exec] checking for stdlib.h... yes
     [exec] checking for string.h... yes
     [exec] checking for memory.h... yes
     [exec] checking for strings.h... yes
     [exec] checking for inttypes.h... yes
     [exec] checking for stdint.h... yes
     [exec] checking for unistd.h... yes
     [exec] checking minix/config.h usability... no
     [exec] checking minix/config.h presence... no
     [exec] checking for minix/config.h... no
     [exec] checking whether it is safe to define __EXTENSIONS__... yes
     [exec] checking for gcc... (cached) gcc
     [exec] checking whether we are using the GNU C compiler... (cached) yes
     [exec] checking whether gcc accepts -g... (cached) yes
     [exec] checking for gcc option to accept ISO C89... (cached) none needed
     [exec] checking dependency style of gcc... (cached) gcc3
     [exec] checking build system type... x86_64-unknown-linux-gnu
     [exec] checking host system type... x86_64-unknown-linux-gnu
     [exec] checking for a sed that does not truncate output... /bin/sed
     [exec] checking for fgrep... /bin/grep -F
     [exec] checking for ld used by gcc... /usr/bin/ld
     [exec] checking if the linker (/usr/bin/ld) is GNU ld... yes
     [exec] checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
     [exec] checking the name lister (/usr/bin/nm -B) interface... BSD nm
     [exec] checking whether ln -s works... yes
     [exec] checking the maximum length of command line arguments... 1572864
     [exec] checking whether the shell understands some XSI constructs... yes
     [exec] checking whether the shell understands "+="... yes
     [exec] checking for /usr/bin/ld option to reload object files... -r
     [exec] checking for objdump... objdump
     [exec] checking how to recognize dependent libraries... pass_all
     [exec] checking for ar... ar
     [exec] checking for strip... strip
     [exec] checking for ranlib... ranlib
     [exec] checking command to parse /usr/bin/nm -B output from gcc object... ok
     [exec] checking for dlfcn.h... yes
     [exec] checking for objdir... .libs
     [exec] checking if gcc supports -fno-rtti -fno-exceptions... no
     [exec] checking for gcc option to produce PIC... -fPIC -DPIC
     [exec] checking if gcc PIC flag -fPIC -DPIC works... yes
     [exec] checking if gcc static flag -static works... yes
     [exec] checking if gcc supports -c -o file.o... yes
     [exec] checking if gcc supports -c -o file.o... (cached) yes
     [exec] checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes
     [exec] checking whether -lc should be explicitly linked in... no
     [exec] checking dynamic linker characteristics... GNU/Linux ld.so
     [exec] checking how to hardcode library paths into programs... immediate
     [exec] checking whether stripping libraries is possible... yes
     [exec] checking if libtool supports shared libraries... yes
     [exec] checking whether to build shared libraries... yes
     [exec] checking whether to build static libraries... yes
     [exec] checking for dlopen in -ldl... yes
     [exec] checking for unistd.h... (cached) yes
     [exec] checking stdio.h usability... yes
     [exec] checking stdio.h presence... yes
     [exec] checking for stdio.h... yes
     [exec] checking stddef.h usability... yes
     [exec] checking stddef.h presence... yes
     [exec] checking for stddef.h... yes
     [exec] checking lzo/lzo2a.h usability... yes
     [exec] checking lzo/lzo2a.h presence... yes
     [exec] checking for lzo/lzo2a.h... yes
     [exec] checking Checking for the 'actual' dynamic-library for '-llzo2'... "liblzo2.so.2"
     [exec] checking for special C compiler options needed for large files... no
     [exec] checking for _FILE_OFFSET_BITS value needed for large files... no
     [exec] checking for stdbool.h that conforms to C99... yes
     [exec] checking for _Bool... yes
     [exec] checking for an ANSI C-conforming const... yes
     [exec] checking for off_t... yes
     [exec] checking for size_t... yes
     [exec] checking whether strerror_r is declared... yes
     [exec] checking for strerror_r... yes
     [exec] checking whether strerror_r returns char *... yes
     [exec] checking for mkdir... yes
     [exec] checking for uname... yes
     [exec] checking for memset... yes
     [exec] checking for JNI_GetCreatedJavaVMs in -ljvm... yes
     [exec] checking jni.h usability... yes
     [exec] checking jni.h presence... yes
     [exec] checking for jni.h... yes
     [exec] configure: creating ./config.status
     [exec] config.status: creating Makefile
     [exec] config.status: creating impl/config.h
     [exec] config.status: executing depfiles commands
     [exec] config.status: executing libtool commands
     [exec] depbase=`echo impl/lzo/LzoCompressor.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
     [exec] /bin/bash ./libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I/home/pooja/dev/hadoop-lzo/src/main/native -I./impl  -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux -I/home/pooja/dev/hadoop-lzo/src/main/native/impl -Isrc/com/hadoop/compression/lzo  -g -Wall -fPIC -O2 -m64 -g -O2 -MT impl/lzo/LzoCompressor.lo -MD -MP -MF $depbase.Tpo -c -o impl/lzo/LzoCompressor.lo /home/pooja/dev/hadoop-lzo/src/main/native/impl/lzo/LzoCompressor.c &&\
     [exec] mv -f $depbase.Tpo $depbase.Plo
     [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I/home/pooja/dev/hadoop-lzo/src/main/native -I./impl -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux -I/home/pooja/dev/hadoop-lzo/src/main/native/impl -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -g -O2 -MT impl/lzo/LzoCompressor.lo -MD -MP -MF impl/lzo/.deps/LzoCompressor.Tpo -c /home/pooja/dev/hadoop-lzo/src/main/native/impl/lzo/LzoCompressor.c  -fPIC -DPIC -o impl/lzo/.libs/LzoCompressor.o
     [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I/home/pooja/dev/hadoop-lzo/src/main/native -I./impl -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux -I/home/pooja/dev/hadoop-lzo/src/main/native/impl -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -g -O2 -MT impl/lzo/LzoCompressor.lo -MD -MP -MF impl/lzo/.deps/LzoCompressor.Tpo -c /home/pooja/dev/hadoop-lzo/src/main/native/impl/lzo/LzoCompressor.c -o impl/lzo/LzoCompressor.o >/dev/null 2>&1
     [exec] depbase=`echo impl/lzo/LzoDecompressor.lo | sed 's|[^/]*$|.deps/&|;s|\.lo$||'`;\
     [exec] /bin/bash ./libtool --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I/home/pooja/dev/hadoop-lzo/src/main/native -I./impl  -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux -I/home/pooja/dev/hadoop-lzo/src/main/native/impl -Isrc/com/hadoop/compression/lzo  -g -Wall -fPIC -O2 -m64 -g -O2 -MT impl/lzo/LzoDecompressor.lo -MD -MP -MF $depbase.Tpo -c -o impl/lzo/LzoDecompressor.lo /home/pooja/dev/hadoop-lzo/src/main/native/impl/lzo/LzoDecompressor.c &&\
     [exec] mv -f $depbase.Tpo $depbase.Plo
     [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I/home/pooja/dev/hadoop-lzo/src/main/native -I./impl -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux -I/home/pooja/dev/hadoop-lzo/src/main/native/impl -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -g -O2 -MT impl/lzo/LzoDecompressor.lo -MD -MP -MF impl/lzo/.deps/LzoDecompressor.Tpo -c /home/pooja/dev/hadoop-lzo/src/main/native/impl/lzo/LzoDecompressor.c  -fPIC -DPIC -o impl/lzo/.libs/LzoDecompressor.o
     [exec] libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I/home/pooja/dev/hadoop-lzo/src/main/native -I./impl -I/usr/lib/jvm/java-8-oracle/include -I/usr/lib/jvm/java-8-oracle/include/linux -I/home/pooja/dev/hadoop-lzo/src/main/native/impl -Isrc/com/hadoop/compression/lzo -g -Wall -fPIC -O2 -m64 -g -O2 -MT impl/lzo/LzoDecompressor.lo -MD -MP -MF impl/lzo/.deps/LzoDecompressor.Tpo -c /home/pooja/dev/hadoop-lzo/src/main/native/impl/lzo/LzoDecompressor.c -o impl/lzo/LzoDecompressor.o >/dev/null 2>&1
     [exec] /bin/bash ./libtool --tag=CC   --mode=link gcc -g -Wall -fPIC -O2 -m64 -g -O2 -L/usr/lib/jvm/java-8-oracle/jre/lib/amd64/server -Wl,--no-as-needed -o libgplcompression.la -rpath /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/../install/lib impl/lzo/LzoCompressor.lo impl/lzo/LzoDecompressor.lo  -ljvm -ldl 
     [exec] libtool: link: gcc -shared  impl/lzo/.libs/LzoCompressor.o impl/lzo/.libs/LzoDecompressor.o   -L/usr/lib/jvm/java-8-oracle/jre/lib/amd64/server -ljvm -ldl  -m64 -Wl,--no-as-needed   -Wl,-soname -Wl,libgplcompression.so.0 -o .libs/libgplcompression.so.0.0.0
     [exec] libtool: link: (cd ".libs" && rm -f "libgplcompression.so.0" && ln -s "libgplcompression.so.0.0.0" "libgplcompression.so.0")
     [exec] libtool: link: (cd ".libs" && rm -f "libgplcompression.so" && ln -s "libgplcompression.so.0.0.0" "libgplcompression.so")
     [exec] libtool: link: ar cru .libs/libgplcompression.a  impl/lzo/LzoCompressor.o impl/lzo/LzoDecompressor.o
     [exec] libtool: link: ranlib .libs/libgplcompression.a
     [exec] libtool: link: ( cd ".libs" && rm -f "libgplcompression.la" && ln -s "../libgplcompression.la" "libgplcompression.la" )
     [exec] libtool: install: cp /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/.libs/libgplcompression.so.0.0.0 /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/lib/libgplcompression.so.0.0.0
     [exec] libtool: install: warning: remember to run `libtool --finish /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/../install/lib'
     [exec] libtool: install: (cd /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/lib && { ln -s -f libgplcompression.so.0.0.0 libgplcompression.so.0 || { rm -f libgplcompression.so.0 && ln -s libgplcompression.so.0.0.0 libgplcompression.so.0; }; })
     [exec] libtool: install: (cd /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/lib && { ln -s -f libgplcompression.so.0.0.0 libgplcompression.so || { rm -f libgplcompression.so && ln -s libgplcompression.so.0.0.0 libgplcompression.so; }; })
     [exec] libtool: install: cp /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/.libs/libgplcompression.lai /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/lib/libgplcompression.la
     [exec] libtool: install: cp /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/.libs/libgplcompression.a /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/lib/libgplcompression.a
     [exec] libtool: install: chmod 644 /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/lib/libgplcompression.a
     [exec] libtool: install: ranlib /home/pooja/dev/hadoop-lzo/target/native/Linux-amd64-64/lib/libgplcompression.a
     [copy] Copying 5 files to /home/pooja/dev/hadoop-lzo/target/classes/native/Linux-amd64-64/lib
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (build-native-win) @ hadoop-lzo ---
[INFO] Executing tasks

build-native-win:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-resources-plugin:2.3:testResources (default-testResources) @ hadoop-lzo ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 12 resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.5.1:testCompile (default-testCompile) @ hadoop-lzo ---
[INFO] Compiling 6 source files to /home/pooja/dev/hadoop-lzo/target/test-classes
[WARNING] bootstrap class path not set in conjunction with -source 1.6
/home/pooja/dev/hadoop-lzo/src/test/java/com/hadoop/compression/lzo/TestDistLzoIndexerJobName.java:[14,14] [deprecation] Job(Configuration) in Job has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/test/java/com/hadoop/compression/lzo/TestDistLzoIndexerJobName.java:[30,14] [deprecation] Job(Configuration) in Job has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/test/java/com/hadoop/compression/lzo/TestDistLzoIndexerJobName.java:[44,14] [deprecation] Job(Configuration) in Job has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/test/java/com/hadoop/compression/lzo/TestDistLzoIndexerJobName.java:[61,14] [deprecation] Job(Configuration) in Job has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/test/java/com/hadoop/compression/lzo/TestDistLzoIndexerJobName.java:[82,14] [deprecation] Job(Configuration) in Job has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/test/java/com/hadoop/compression/lzo/TestDistLzoIndexerJobName.java:[96,16] [deprecation] Job(Configuration) in Job has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/test/java/com/hadoop/mapreduce/TestLzoTextInputFormat.java:[165,14] [deprecation] Job(Configuration) in Job has been deprecated
[WARNING] /home/pooja/dev/hadoop-lzo/src/test/java/com/hadoop/mapreduce/TestLzoTextInputFormat.java:[314,14] [deprecation] Job(Configuration) in Job has been deprecated
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (prep-test) @ hadoop-lzo ---
[INFO] Executing tasks

prep-test:
    [mkdir] Created dir: /home/pooja/dev/hadoop-lzo/target/test-classes/logs
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-surefire-plugin:2.14.1:test (default-test) @ hadoop-lzo ---
[INFO] Surefire report directory: /home/pooja/dev/hadoop-lzo/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running com.hadoop.mapreduce.TestLzoTextInputFormat
2017-08-28 15:14:19,876 INFO  lzo.GPLNativeCodeLoader (GPLNativeCodeLoader.java:<clinit>(52)) - Loaded native gpl library from the embedded binaries
2017-08-28 15:14:19,895 INFO  lzo.LzoCodec (LzoCodec.java:<clinit>(76)) - Successfully loaded & initialized native-lzo library [hadoop-lzo rev f12b7f24913ffbde938b8d140e8a7b22183221a0]
2017-08-28 15:14:20,226 WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-08-28 15:14:20,551 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1129)) - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2017-08-28 15:14:20,553 INFO  compress.CodecPool (CodecPool.java:getCompressor(151)) - Got brand-new compressor [.lzo]
2017-08-28 15:14:21,210 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:21,266 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 3
2017-08-28 15:14:21,273 INFO  compress.CodecPool (CodecPool.java:getDecompressor(179)) - Got brand-new decompressor [.lzo]
2017-08-28 15:14:21,503 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:21,541 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 3
2017-08-28 15:14:22,116 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:22,129 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 2
2017-08-28 15:14:22,274 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:22,289 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 2
2017-08-28 15:14:22,820 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:22,844 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 3
2017-08-28 15:14:23,023 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:23,040 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 3
2017-08-28 15:14:23,573 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:23,588 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 2
2017-08-28 15:14:23,709 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:23,726 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 2
2017-08-28 15:14:24,187 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:24,217 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 2
2017-08-28 15:14:24,358 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:24,375 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 2
2017-08-28 15:14:24,906 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:24,917 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 1
2017-08-28 15:14:25,032 INFO  output.FileOutputCommitter (FileOutputCommitter.java:commitTask(439)) - Saved output of task 'attempt_123_0001_r_000001_2' to file:/home/pooja/dev/hadoop-lzo/target/test-classes/data/outputDir/_temporary/0/task_123_0001_r_000001
2017-08-28 15:14:25,045 INFO  input.FileInputFormat (FileInputFormat.java:listStatus(281)) - Total input paths to process : 1
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.396 sec
Running com.hadoop.compression.lzo.TestLzopOutputStream
2017-08-28 15:14:25,706 WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-08-28 15:14:25,995 INFO  lzo.GPLNativeCodeLoader (GPLNativeCodeLoader.java:<clinit>(52)) - Loaded native gpl library from the embedded binaries
2017-08-28 15:14:26,002 INFO  lzo.LzoCodec (LzoCodec.java:<clinit>(76)) - Successfully loaded & initialized native-lzo library [hadoop-lzo rev f12b7f24913ffbde938b8d140e8a7b22183221a0]
2017-08-28 15:14:26,018 INFO  lzo.TestLzopOutputStream (TestLzopOutputStream.java:runTest(134)) - Comparing files /home/pooja/dev/hadoop-lzo/target/test-classes/data/100000.txt and /home/pooja/dev/hadoop-lzo/target/test-classes/data/100000.txt.lzo
2017-08-28 15:14:26,217 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1129)) - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2017-08-28 15:14:26,263 INFO  lzo.TestLzopOutputStream (TestLzopOutputStream.java:runTest(134)) - Comparing files /home/pooja/dev/hadoop-lzo/target/test-classes/data/1000.txt and /home/pooja/dev/hadoop-lzo/target/test-classes/data/1000.txt.lzo
2017-08-28 15:14:26,299 INFO  lzo.TestLzopOutputStream (TestLzopOutputStream.java:runTest(134)) - Comparing files /home/pooja/dev/hadoop-lzo/target/test-classes/data/100.txt and /home/pooja/dev/hadoop-lzo/target/test-classes/data/100.txt.lzo
2017-08-28 15:14:26,328 INFO  lzo.TestLzopOutputStream (TestLzopOutputStream.java:runTest(134)) - Comparing files /home/pooja/dev/hadoop-lzo/target/test-classes/data/issue20-lzop.txt and /home/pooja/dev/hadoop-lzo/target/test-classes/data/issue20-lzop.txt.lzo
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.06 sec
Running com.hadoop.compression.lzo.TestLzoCodec
2017-08-28 15:14:26,732 INFO  lzo.GPLNativeCodeLoader (GPLNativeCodeLoader.java:<clinit>(52)) - Loaded native gpl library from the embedded binaries
2017-08-28 15:14:26,740 INFO  lzo.LzoCodec (LzoCodec.java:<clinit>(76)) - Successfully loaded & initialized native-lzo library [hadoop-lzo rev f12b7f24913ffbde938b8d140e8a7b22183221a0]
2017-08-28 15:14:26,824 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1129)) - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2017-08-28 15:14:26,994 INFO  compress.CodecPool (CodecPool.java:getCompressor(151)) - Got brand-new compressor [.lzo_deflate]
2017-08-28 15:14:27,058 INFO  compress.CodecPool (CodecPool.java:getCompressor(151)) - Got brand-new compressor [.lzo_deflate]
2017-08-28 15:14:27,138 INFO  compress.CodecPool (CodecPool.java:getCompressor(151)) - Got brand-new compressor [.lzo_deflate]
Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.572 sec
Running com.hadoop.compression.lzo.TestLzopInputStream
2017-08-28 15:14:27,544 INFO  lzo.GPLNativeCodeLoader (GPLNativeCodeLoader.java:<clinit>(52)) - Loaded native gpl library from the embedded binaries
2017-08-28 15:14:27,573 INFO  lzo.LzoCodec (LzoCodec.java:<clinit>(76)) - Successfully loaded & initialized native-lzo library [hadoop-lzo rev f12b7f24913ffbde938b8d140e8a7b22183221a0]
2017-08-28 15:14:27,575 INFO  lzo.TestLzopInputStream (TestLzopInputStream.java:runTest(119)) - Comparing files /home/pooja/dev/hadoop-lzo/target/test-classes/data/100000.txt and /home/pooja/dev/hadoop-lzo/target/test-classes/data/100000.txt.lzo
2017-08-28 15:14:27,650 INFO  lzo.TestLzopInputStream (TestLzopInputStream.java:runTest(119)) - Comparing files /home/pooja/dev/hadoop-lzo/target/test-classes/data/1000.txt and /home/pooja/dev/hadoop-lzo/target/test-classes/data/1000.txt.lzo
2017-08-28 15:14:27,653 INFO  lzo.TestLzopInputStream (TestLzopInputStream.java:runTest(119)) - Comparing files /home/pooja/dev/hadoop-lzo/target/test-classes/data/100.txt and /home/pooja/dev/hadoop-lzo/target/test-classes/data/100.txt.lzo
2017-08-28 15:14:27,655 INFO  lzo.TestLzopInputStream (TestLzopInputStream.java:runTest(119)) - Comparing files /home/pooja/dev/hadoop-lzo/target/test-classes/data/0.txt and /home/pooja/dev/hadoop-lzo/target/test-classes/data/0.txt.lzo
2017-08-28 15:14:27,657 INFO  lzo.TestLzopInputStream (TestLzopInputStream.java:runTest(119)) - Comparing files /home/pooja/dev/hadoop-lzo/target/test-classes/data/100000-truncated.txt and /home/pooja/dev/hadoop-lzo/target/test-classes/data/100000-truncated.txt.lzo
2017-08-28 15:14:27,691 WARN  lzo.LzopInputStream (LzopInputStream.java:close(347)) - Incorrect LZO file format: file did not end with four trailing zeroes.
java.io.IOException: Corrupted uncompressed block
at com.hadoop.compression.lzo.LzopInputStream.verifyChecksums(LzopInputStream.java:220)
at com.hadoop.compression.lzo.LzopInputStream.close(LzopInputStream.java:343)
at sun.nio.cs.StreamDecoder.implClose(StreamDecoder.java:378)
at sun.nio.cs.StreamDecoder.close(StreamDecoder.java:193)
at java.io.InputStreamReader.close(InputStreamReader.java:199)
at java.io.BufferedReader.close(BufferedReader.java:525)
at com.hadoop.compression.lzo.TestLzopInputStream.runTest(TestLzopInputStream.java:147)
at com.hadoop.compression.lzo.TestLzopInputStream.testTruncatedFile(TestLzopInputStream.java:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at junit.framework.TestCase.runTest(TestCase.java:168)
at junit.framework.TestCase.runBare(TestCase.java:134)
at junit.framework.TestResult$1.protect(TestResult.java:110)
at junit.framework.TestResult.runProtected(TestResult.java:128)
at junit.framework.TestResult.run(TestResult.java:113)
at junit.framework.TestCase.run(TestCase.java:124)
at junit.framework.TestSuite.runTest(TestSuite.java:243)
at junit.framework.TestSuite.run(TestSuite.java:238)
at org.junit.internal.runners.JUnit38ClassRunner.run(JUnit38ClassRunner.java:83)
at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.maven.surefire.util.ReflectionUtils.invokeMethodWithArray2(ReflectionUtils.java:208)
at org.apache.maven.surefire.booter.ProviderFactory$ProviderProxy.invoke(ProviderFactory.java:159)
at org.apache.maven.surefire.booter.ProviderFactory.invokeProvider(ProviderFactory.java:87)
at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:95)
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.324 sec
Running com.hadoop.compression.lzo.TestDistLzoIndexerJobName
2017-08-28 15:14:28,435 WARN  util.NativeCodeLoader (NativeCodeLoader.java:<clinit>(62)) - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2017-08-28 15:14:28,628 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1129)) - mapred.job.queue.name is deprecated. Instead, use mapreduce.job.queuename
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.707 sec
Running com.hadoop.compression.lzo.TestLzoRandData
2017-08-28 15:14:29,130 INFO  lzo.GPLNativeCodeLoader (GPLNativeCodeLoader.java:<clinit>(52)) - Loaded native gpl library from the embedded binaries
2017-08-28 15:14:29,136 INFO  lzo.LzoCodec (LzoCodec.java:<clinit>(76)) - Successfully loaded & initialized native-lzo library [hadoop-lzo rev f12b7f24913ffbde938b8d140e8a7b22183221a0]
2017-08-28 15:14:29,200 INFO  Configuration.deprecation (Configuration.java:warnOnceIfDeprecated(1129)) - hadoop.native.lib is deprecated. Instead, use io.native.lib.available
2017-08-28 15:14:29,277 INFO  compress.CodecPool (CodecPool.java:getCompressor(151)) - Got brand-new compressor [.lzo]
Start to write to file...
Closed file.
2017-08-28 15:14:29,391 INFO  compress.CodecPool (CodecPool.java:getDecompressor(179)) - Got brand-new decompressor [.lzo]
Start to write to file...
Closed file.
Start to write to file...
Closed file.
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.325 sec

Results :

Tests run: 27, Failures: 0, Errors: 0, Skipped: 0

[INFO] 
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hadoop-lzo ---
[INFO] Building jar: /home/pooja/dev/hadoop-lzo/target/hadoop-lzo-0.4.21-SNAPSHOT.jar
[INFO] 
[INFO] >>> maven-source-plugin:2.2.1:jar (attach-sources) @ hadoop-lzo >>>
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (check-platform) @ hadoop-lzo ---
[INFO] Executing tasks

check-platform:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (set-props-non-win) @ hadoop-lzo ---
[INFO] Executing tasks

set-props-non-win:
[INFO] Executed tasks
[INFO] 
[INFO] --- maven-antrun-plugin:1.7:run (set-props-win) @ hadoop-lzo ---
[INFO] Executing tasks

set-props-win:
[INFO] Executed tasks
[INFO] 
[INFO] <<< maven-source-plugin:2.2.1:jar (attach-sources) @ hadoop-lzo <<<
[INFO] 
[INFO] --- maven-source-plugin:2.2.1:jar (attach-sources) @ hadoop-lzo ---
[INFO] Building jar: /home/pooja/dev/hadoop-lzo/target/hadoop-lzo-0.4.21-SNAPSHOT-sources.jar
[INFO] 
[INFO] --- maven-javadoc-plugin:2.9:jar (attach-javadocs) @ hadoop-lzo ---
[INFO] 
Loading source files for package org.apache.hadoop.io.compress...
Loading source files for package com.hadoop.mapred...
Loading source files for package com.hadoop.mapreduce...
Loading source files for package com.hadoop.compression.lzo...
Loading source files for package com.hadoop.compression.lzo.util...
Loading source files for package com.quicklz...
Constructing Javadoc information...
Standard Doclet version 1.8.0_131
Building tree for all the packages and classes...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/org/apache/hadoop/io/compress/LzoCodec.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapred/DeprecatedLzoLineRecordReader.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapred/DeprecatedLzoTextInputFormat.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/LzoIndexOutputFormat.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/LzoIndexRecordWriter.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/LzoLineRecordReader.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/LzoSplitInputFormat.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/LzoSplitRecordReader.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/LzoSplitRecordReader.Counters.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/LzoTextInputFormat.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/CChecksum.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/DChecksum.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/DistributedLzoIndexer.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/GPLNativeCodeLoader.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/LzoCodec.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/LzoIndex.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/LzoIndexer.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/LzoInputFormatCommon.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/LzopCodec.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/LzopDecompressor.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/LzopInputStream.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/LzopOutputStream.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/util/CompatibilityUtil.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/quicklz/QuickLZ.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/overview-frame.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/package-frame.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/package-summary.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/package-tree.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/util/package-frame.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/util/package-summary.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/util/package-tree.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapred/package-frame.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapred/package-summary.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapred/package-tree.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/package-frame.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/package-summary.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/package-tree.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/quicklz/package-frame.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/quicklz/package-summary.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/quicklz/package-tree.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/org/apache/hadoop/io/compress/package-frame.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/org/apache/hadoop/io/compress/package-summary.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/org/apache/hadoop/io/compress/package-tree.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/constant-values.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/org/apache/hadoop/io/compress/class-use/LzoCodec.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapred/class-use/DeprecatedLzoLineRecordReader.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapred/class-use/DeprecatedLzoTextInputFormat.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/class-use/LzoIndexRecordWriter.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/class-use/LzoIndexOutputFormat.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/class-use/LzoTextInputFormat.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/class-use/LzoSplitInputFormat.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/class-use/LzoSplitRecordReader.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/class-use/LzoSplitRecordReader.Counters.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/class-use/LzoLineRecordReader.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/LzoIndexer.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/LzopOutputStream.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/LzopInputStream.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/GPLNativeCodeLoader.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/LzoCodec.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/LzoInputFormatCommon.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/CChecksum.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/DChecksum.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/LzoIndex.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/DistributedLzoIndexer.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/LzopCodec.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/class-use/LzopDecompressor.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/util/class-use/CompatibilityUtil.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/quicklz/class-use/QuickLZ.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/package-use.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/compression/lzo/util/package-use.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapred/package-use.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/hadoop/mapreduce/package-use.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/com/quicklz/package-use.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/org/apache/hadoop/io/compress/package-use.html...
Building index for all the packages and classes...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/overview-tree.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/index-all.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/deprecated-list.html...
Building index for all classes...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/allclasses-frame.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/allclasses-noframe.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/index.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/overview-summary.html...
Generating /home/pooja/dev/hadoop-lzo/target/apidocs/help-doc.html...
[INFO] Building jar: /home/pooja/dev/hadoop-lzo/target/hadoop-lzo-0.4.21-SNAPSHOT-javadoc.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30.462s
[INFO] Finished at: Mon Aug 28 15:14:35 PDT 2017
[INFO] Final Memory: 33M/253M
[INFO] ------------------------------------------------------------------------

Copy the LZO-Hadoop files to Hadoop 

---

We need to copy the hadoop-lzo jar and native file to all hadoop cluster machine as shown below.

pooja@pooja:~/dev/hadoop-lzo$ sudo cp  target/hadoop-lzo-0.4.21-SNAPSHOT.jar /home/hduser/lzo/
pooja@pooja:~/dev/hadoop-lzo$ sudo cp -R target/native /home/hduser/lzo/native

Change configuration file

---

We need to change hadoop property files to include both hadoop-lzo-*.jar and native file to classpath and  library path.

Edit hadoop-env.sh 

Add hadoop-lzo-*.jar file to classpath and native files to library path as shown below:

#Set LZO compression path

export HADOOP_CLASSPATH=/home/hduser/lzo/hadoop-lzo-0.4.21-SNAPSHOT.jar:$HADOOP_CLASSPATH

export HADOOP_OPTS="$HADOOP_OPTS -Djava.library.path=$HADOOP_HOME/lib/native:/home/hduser/lzo/native/Linux-amd64-64/lib" 
                   

                                                                                                         

Edit core-site.xml

Add LZO compression codec as shown below

<property>

  <name>io.compression.codecs</name>

  <value>org.apache.hadoop.io.compress.GzipCodec, org.apache.hadoop.io.compress.DefaultCodec, org.apache.hadoop.io.compress.BZip2Codec, com.hadoop.compression.lzo.LzoCodec, com.hadoop.compression.lzo.LzopCodec

  </value>

</property>

<property>

  <name>io.compression.codec.lzo.class</name>

  <value>com.hadoop.compression.lzo.LzoCodec</value>

</property>

Edit yarn-site.xml

Here, set the below property:

<property>
   <name>mapreduce.map.output.compress</name>
   <value>true</value>
</property>
<property>
   <name>mapreduce.map.output.compress.codec</name>
   <value>com.hadoop.compression.lzo.LzoCodec</value>
</property>

<property>

     <name>yarn.application.classpath</name>
     <value>$HADOOP_CONF_DIR, $HADOOP_COMMON_HOME/share/hadoop/common/*, $HADOOP_COMMON_HOME/share/hadoop/common/lib/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/*, $HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*, $HADOOP_YARN_HOME/share/hadoop/yarn/*, $HADOOP_YARN_HOME/share/hadoop/yarn/lib/*,/home/hduser/lzo/hadoop-lzo-0.4.21-SNAPSHOT.jar</value>
</property>

Finally, restart the hadoop cluster and you can write Map Reduce program to include LZO compression.

Verify Hadoop Classpath

---

Verify if lzo is set on hadoop classpath using below command:

ps -eaf | grep lzo

Verify LZO setup

---

Write a Map Reduce program with LZO Codec as output

 FileOutputFormat.setCompressOutput(job, true);
 FileOutputFormat.setOutputCompressorClass(job,LzoCodec.class);

Now, run the job as shown below
hduser@pooja:~/hadoop-data-files$ yarn jar weatherHadoop.jar com.jbksoft.WeatherJob /usr/WeatherData output cache
17/08/29 07:38:54 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/08/29 07:38:55 INFO client.RMProxy: Connecting to ResourceManager at master/192.168.1.101:8032
17/08/29 07:38:56 INFO input.FileInputFormat: Total input files to process : 1
17/08/29 07:38:56 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries
17/08/29 07:38:56 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev f12b7f24913ffbde938b8d140e8a7b22183221a0]
17/08/29 07:38:57 INFO mapreduce.JobSubmitter: number of splits:1
17/08/29 07:38:57 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1504017422827_0001
17/08/29 07:38:58 INFO impl.YarnClientImpl: Submitted application application_1504017422827_0001
17/08/29 07:38:58 INFO mapreduce.Job: The url to track the job: http://master:8088/proxy/application_1504017422827_0001/
17/08/29 07:38:58 INFO mapreduce.Job: Running job: job_1504017422827_0001
17/08/29 07:39:07 INFO mapreduce.Job: Job job_1504017422827_0001 running in uber mode : false
17/08/29 07:39:07 INFO mapreduce.Job:  map 0% reduce 0%
17/08/29 07:39:12 INFO mapreduce.Job:  map 100% reduce 0%
17/08/29 07:39:18 INFO mapreduce.Job:  map 100% reduce 100%
17/08/29 07:39:19 INFO mapreduce.Job: Job job_1504017422827_0001 completed successfully
17/08/29 07:39:19 INFO mapreduce.Job: Counters: 49
File System Counters
FILE: Number of bytes read=2555
FILE: Number of bytes written=282357
FILE: Number of read operations=0
FILE: Number of large read operations=0
FILE: Number of write operations=0
HDFS: Number of bytes read=52455
HDFS: Number of bytes written=2430
HDFS: Number of read operations=6
HDFS: Number of large read operations=0
HDFS: Number of write operations=2
Job Counters 
Launched map tasks=1
Launched reduce tasks=1
Data-local map tasks=1
Total time spent by all maps in occupied slots (ms)=3010
Total time spent by all reduces in occupied slots (ms)=3286
Total time spent by all map tasks (ms)=3010
Total time spent by all reduce tasks (ms)=3286
Total vcore-milliseconds taken by all map tasks=3010
Total vcore-milliseconds taken by all reduce tasks=3286
Total megabyte-milliseconds taken by all map tasks=3082240
Total megabyte-milliseconds taken by all reduce tasks=3364864
Map-Reduce Framework
Map input records=366
Map output records=358
Map output bytes=4646
Map output materialized bytes=2551
Input split bytes=117
Combine input records=0
Combine output records=0
Reduce input groups=358
Reduce shuffle bytes=2551
Reduce input records=358
Reduce output records=358
Spilled Records=716
Shuffled Maps =1
Failed Shuffles=0
Merged Map outputs=1
GC time elapsed (ms)=131
CPU time spent (ms)=1810
Physical memory (bytes) snapshot=454684672
Virtual memory (bytes) snapshot=3855101952
Total committed heap usage (bytes)=309854208
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters 
Bytes Read=52338
File Output Format Counters 
Bytes Written=2430

Finally, verify the output as shown below

hduser@pooja:~/hadoop-data-files$ hdfs dfs -text /user/hduser/output/part-r-00000.lzo_deflate
17/08/29 08:23:14 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library from the embedded binaries
17/08/29 08:23:14 INFO lzo.LzoCodec: Successfully loaded & initialized native-lzo library [hadoop-lzo rev f12b7f24913ffbde938b8d140e8a7b22183221a0]
17/08/29 08:23:14 INFO Configuration.deprecation: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
0.0
20000109 42.5
20000110 52.8
20000111 46.5
20000112 44.7

I hope you are able to follow my tutorial. If you still face any issues, please mail me, I will love to help you.

Happy Coding !!!