Install Spark on Standalone Mode

Apache Spark

Apache Spark is cluster computing framework written in Scala language. It is gaining popularity as it provides real-time solutions to big data ecosystem.

Installation

Apache spark can be installed on stand alone mode by simply placing the compile version of spark on each node or build it yourself using the source code.

In this tutorial, I will provide details of installation using compile version of spark.

a. Install Java 7+ on machine (if not already installed)

b. Download the Spark tar ball

Download the Spark tar ball using http://spark.apache.org/downloads.html as shown below.



We need to select the below parameter for download.

1. Choose a Spark release. You can choose the latest version


2. Choose the package type. You can select with Hadoop version or with user provided hadoop.Note: Spark uses core Hadoop Library to communicate to HDFS and other Hadoop-supported storage system.Because the protocol changed for different version o HDFS therefore select that build against the same version as version hadoop cluster runs. I have selected the "Pre-build with Hadoop 2.7 and later".


3. Choose the download type. Select "Direct download".


4. Download Spark. Click on the link for download tar ball on local machine.

c. Unzip the downloaded tar file

$tar -xvf spark-2.0.2-bin-hadoop2.7.tgz

Below is the folder structure after you extract the tar file as shown below.



The description of the important folders:

Folder	Usage
sbin	Contain start, stop master and slave scripts
bin	Contain Scala and Python Spark shell
conf	Contain configuration files
data	Contain graph, machine leraning and streaming job data
jars	Contains jar included in Spark Classpath
examples	Contain example for Spark job
logs	Contain all log file

d. Start the spark stand alone cluster using below command
cd <Spark Root directory>
sbin/start-master.sh

e. Check if master node is working properly.

In the console, type in the URL http://localhost:8080, it should show up the screen as shown below.



f. Start worker node

Now, we will run script sbin/start-slave.sh as shown below.

cd <spark-root-directory>

sbin/start-slave.sh spark://localhost:7077

g. Verify if the worker node is running.

Make sure the http://localhost:8080, UI console, you can see a new Worker Id (worker-20161215153905-192.168.1.142-57869) as shown below.



h.Running a Spark example

We can run the Spark example job

$./bin/spark-submit --class org.apache.spark.examples.SparkPi --master spark://localhost:7077 examples/jars/spark-examples_2.11-2.0.2.jar 1000


Verify if the console shows below output:

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/12/15 21:36:41 INFO SparkContext: Running Spark version 2.0.2
16/12/15 21:36:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
16/12/15 21:36:42 WARN Utils: Your hostname, localhost.localdomain resolves to a loopback address: 127.0.0.1; using 192.168.1.142 instead (on interface wlp18s0b1)

...........

...........

16/12/15 21:36:51 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 4.098445 s
Pi is roughly 3.143019143019143
16/12/15 21:36:51 INFO SparkUI: Stopped Spark web UI at http://192.168.1.142:4040
16/12/15 21:36:51 INFO StandaloneSchedulerBackend: Shutting down all executors
16/12/15 21:36:51 INFO CoarseGrainedSchedulerBackend$DriverEndpoint: Asking each executor to shut down
16/12/15 21:36:51 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
16/12/15 21:36:51 INFO MemoryStore: MemoryStore cleared
16/12/15 21:36:51 INFO BlockManager: BlockManager stopped
16/12/15 21:36:51 INFO BlockManagerMaster: BlockManagerMaster stopped
16/12/15 21:36:51 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!

Running multiple instance of Spark Worker on Standalone Mode

In the conf/spark-env.sh, set SPARK_WORKER_INSTANCES to number of worker you want to start and start with start-slave.sh script as shown below

#Add the below line to <Spark_home/conf/spark-env.sh>
export SPARK_WORKER_INSTANCES=2
#Then start the worker threads
sbin/start-slave.sh spark://localhost:7077 --cores 2 --memory 2g

By now, I hope you are able to configure the Spark Stand Alone cluster successfully. If facing any issues, please reply on comments.

Keep Reading and Learning.

Happy Coding!!!

Installing Scala on CentOS

Scala

Scala is a programming language, which support object oriented and functional programming paradigm. This language is gaining popularity as it  is becoming good choice for scalable distributed systems.

There are many ways the language can be configured on CentOS machine.

1. Install SBT

Refer to the article Spark Development using SBT in IntelliJ

2. Download Scala

Download the tar file and set the environment variables as shown below

[code language="java"]

#Download the tar file
wget http://downloads.lightbend.com/scala/2.12.1/scala-2.12.1.tgz
....
HTTP request sent, awaiting response... 200 OK
Length: 19700349 (19M) [application/octet-stream]
Saving to: ‘scala-2.12.1.tgz’

100%[======================================&gt;] 19,700,349 214KB/s in 57s
2016-12-13 21:42:31 (336 KB/s) - ‘scala-2.12.1.tgz’ saved [19700349/19700349]

#Extract the file
tar xvf scala-2.12.1.tgz
.....
scala-2.12.1/doc/tools/css/style.css
scala-2.12.1/doc/tools/scalap.html
scala-2.12.1/doc/tools/scaladoc.html
scala-2.12.1/doc/License.rtf
scala-2.12.1/doc/licenses/
scala-2.12.1/doc/licenses/bsd_asm.txt
scala-2.12.1/doc/licenses/mit_jquery.txt
scala-2.12.1/doc/licenses/bsd_jline.txt
scala-2.12.1/doc/licenses/mit_sizzle.txt
scala-2.12.1/doc/licenses/mit_tools.tooltip.txt
scala-2.12.1/doc/licenses/apache_jansi.txt
scala-2.12.1/doc/LICENSE.md
scala-2.12.1/doc/README

#Move folder to /usr/local
sudo mv scala-2.12.1 /usr/local/scala

#Edit .bashrc file add the below lines
export SCALA_HOME=/usr/local/scala
export PATH=$PATH:$SCALA_HOME/bin

#Source .bashrc file
source ~/.bashrc

#Verify if installed properly
scala -version
Scala code runner version 2.12.1 -- Copyright 2002-2016, LAMP/EPFL and Lightbend, Inc.
[/code]

3. Install one of the Scala IDE plugin

Download one Scala IDE plugin, I have used   Eclipse or IntelliJ. Both works great.

Hope this tutorial help you in installation. Please let me know if you face ran into any problems.

Keep Reading and Learning...

Happy Coding!!!

Install Maven on CentOS

Maven is open source (written in java) powerful build tool for java development projects. We can automate task such as compile,clean, build, deploy and also dependency management.

In order to install Maven on CentOS, follow the below steps:

1.Download the Maven tar ball

Download the tar in the folder you want to extract to using below command

[code]

wget http://mirror.reverse.net/pub/apache/maven/maven-3/3.3.9/binaries/apache-maven-3.3.9-bin.tar.gz

[/code]

2. Extract the tar ball

[code]

tar xvf apache-maven-3.3.9-bin.tar.gz

[/code]

3. Move Maven to /usr/local directory. This is a optional step.

[code]
sudo mv apache-maven-3.3.9 /usr/local/apache-maven

[/code]

4. Edit ~/.bashrc to set env. properties

[code]

# Add below lines to .bashrc

export M2_HOME=/usr/local/apache-maven
export PATH=$M2_HOME/bin:$PATH

[/code]

4. Execute below command to set the env. properties

[code]

source ~/.bashrc

[/code]

5. Verify Maven is installed

[code]

$ mvn -version
Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; 2015-11-10T08:41:47-08:00)
Maven home: /usr/local/apache-maven
Java version: 1.8.0_72, vendor: Oracle Corporation
Java home: /usr/java/jdk1.8.0_72/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-327.el7.x86_64", arch: "amd64", family: "unix"

[/code]

Please let me know if you face any problem.

Happy Coding!!!!

Functional vs Object Oriented Programming

There are many programming model exists today such as Procedural Programming, Structural Programming,Event Driven Programming, Object Oriented Programming, Functional Programming and many more. In this tutorial, we will be focus on just two programming models.

Overview of Object Oriented Programming

In this model, there is a concept of Object which consists of set of data and operations on it.

The object is basically the blue print of any real thing such as aeroplane, car,chair, table, any living person and so on. Or object can be represents any conceptual term such as Company, Bank, Account,Customer and so on.

Each object will have attributes called data associated to it. And we can manipulate the data in the object using the method/operations defined in the it. This concept is called Data encapsulation.

Also, the object can inherit attributes and operations from other object. This terminology is called inheritance. For eg. Horse is a 'kind of' Animal. Therefore, it can inherit the data and operations from Animal.

Many programming language support object oriented model such as Java, C++, PHP,Python, Scala and many more.

Lets take a example of the Scala code wherein we will demonstrate a class called Account and inherited class Saving Account.

[code language="scala"]
package com.xxx

class Account(val AccountId:String, var AccountBalance:Double,val CustomerName: String,val AccountType:String ="ACC") {

def accId=AccountId
def accBal=AccountBalance
def custName=CustomerName
def acctType=AccountType
def setacctType(bal:Double):Unit= {AccountBalance=bal}
override def toString() =
"" + accId + (if (accBal < 0) "" else "+") + accBal + custName+ "i"
}
[/code]

[code language="scala"]

package com.xxx

class SavingsAccount(AccountId:String, AccountBalance:Double, CustomerName: String,var MinimumBalance:Double,AccountType:String ="SAV") extends Account(AccountId,AccountBalance,CustomerName,AccountType){

def minBal=MinimumBalance

override def toString() =
"" + accId + (if (accBal < 0) "" else "+") + accBal + custName+minBal+ "i"

}

[/code]

Overview of Functional Programming

In this model, the computation are evaluated as mathematical operations. Thus, output of the mathematical function f(x) depends only on the input data 'x'  and not any external variables.  For same input data 'x' there should be same output irrespective of the number of times the function is executes.

The mathematical function can be defined as high order function, lambda function, pure function, recursive function and many more.

Mostly, the programming language is hybrid of one or more models. The programming languages that support functional programming are Lisp, Scala, Clojure ,Standard ML.

Lets take the example of Scala code to define mathematical function

High order function: In this function can be passed as a parameter to a function as shown below

[code language="scala"]

def cube(a:Int):Int={
return a*a*a
}

def sum(f:Int=>Int,a:Int):Int={
if(a<=0) return 0
f(a)+sum(f,a-1)
}

//Now,execute the method as sum of cube

sum(cube,3)

[/code]

Recursive function

[code language="scala"]

//Recursive function
def fibonacci(a:Int):Int={
if(a <= 0) return 0
if ( a == 1) return 1
else return fibonacci(a-1)+fibonacci(a-2)

}
//Now, execute function and get fabonacci series

fibonacci(6)

[/code]

Difference between two with an example

We now understand some computation problem which can be solved using OOPs concept and some using functional based programming.

Suppose we have a list and we need to compute the total deposit of the bank.

In Object Oriented approach, we will loop through the list of account and find the results as shown below(using Scala Language)

[code language="scala"]

//Call the above method using for loop as shown below
var accList=List(new Account("Acc1",100,"XXX1"),new SavingsAccount("Sav",500,"XXX1",250))

var sum:Double = 0
for(i<-0 to accList.length-1){
sum+=accList(i).accBal
}

print(sum)

[/code]

In Functional programming approach, we can call list function "foldLeft" to perform operations on list.

[code language="scala"]

print(accList.foldLeft(0.0){(z,f)=> z+f.accBal})

[/code]

Spark Development using SBT in IntelliJ

Apache Spark

Apache Spark is open source big data computational system. It is developed using Scala programming language which run on JVM ( Java Virtual Machine) platform. Today, popularity of Spark is increasing due to it's in-memory data storage  and real time processing capabilities. This computational system provides high level API in Java, Scala and Python. Therefore, we can run data analytical queries using these high level API on Spark system and get desire insights. Spark can deployed to standalone cluster, Hadoop 2 (YARN) or Mesos.

SBT Overview

SBT is Simple Build Tool. A build tool help in automation of tasks like build,compile, test, package, run, deploy. Other build tools are like Maven, Ant, Gradle, Ivy. SBT is also one othe build tool that focus mainly on Scala projects.

Today, I am going to explore to write a basic query using Spark high level API in Scala 2.10. Also, I will be using IntelliJ as IDE for development.

Now, all set. Let get our hands dirty with some actual coding.

Prerequisite (Make sure your machine has below components already installed):

1. Install  Java JDK 7+.


2. Install SBT.


3. Unzip IntelliJ.

Working on Code

a. Creating project structure

There are different ways project structure can be created. We can even use the existing project templates to create it automatically.Today, we are going to create the project structure  manually. In below code, we have create the root directory/project name ( scalaProjectDemo) and folder src/main/scala inside it as shown below:

[code language="java"]

mkdir scalaProjectDemo

cd scalaProjectDemo
mkdir project
mkdir -p src/main/scala
mkdir -p src/main/resources
touch project/build.properties
touch project/plugins.sbt
touch project/assembly.sbt
[/code]

b. Creating a build file

We will be creating the build file "build.sbt" in the root directory as shown below:

[code language="java"]

import AssemblyKeys._

assemblySettings

name := "scalaProjectDemo"

version := "1.0"

scalaVersion := "2.10.4"

libraryDependencies += "org.apache.spark" %% "spark-core" % "1.1.0" % "provided"

libraryDependencies += "org.apache.spark" %% "spark-streaming" % "1.1.0"

libraryDependencies += "org.apache.hadoop" % "hadoop-client" % "2.0.0-mr1-cdh4.2.0" % "provided"

resolvers ++= Seq(
"Cloudera Repository" at "https://repository.cloudera.com/artifactory/cloudera-repos/",
"Akka Repository" at "http://repo.akka.io/releases/",
"Spray Repository" at "http://repo.spray.cc/")

[/code]



The project can be imported here into IntelliJ. Please refer the item " Importing the code into IntelliJ "

c. Creating a Scala file for testing

Next, we create a sample Scala file which just a single print statement as shown below:

[code language="text"]

package com.xxx

object HelloWorld {
def main(args: Array[String]){
println("Hello World")
}
}

[/code]

d. Run the code

Next, we will run the code and make sure the code compile successfully as shown below:

[code language="java"]
$ cd scalaProjectDemo
$ sbt run
Getting org.scala-sbt sbt 0.13.6 ...

downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/sbt/0.13.6/jars/sbt.jar ...
[SUCCESSFUL ] org.scala-sbt#sbt;0.13.6!sbt.jar (1481ms)
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/main/0.13.6/jars/main.jar ...
[SUCCESSFUL ] org.scala-sbt#main;0.13.6!main.jar (3868ms)
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/compiler-interface/0.13.6/jars/compiler-interface-bin.jar ...
[SUCCESSFUL ] org.scala-sbt#compiler-interface;0.13.6!compiler-interface-bin.jar (1653ms)
......
[SUCCESSFUL ] org.scala-sbt#test-agent;0.13.6!test-agent.jar (1595ms)
downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-sbt/apply-macro/0.13.6/jars/apply-macro.jar ...
[SUCCESSFUL ] org.scala-sbt#apply-macro;0.13.6!apply-macro.jar (1619ms)
:: retrieving :: org.scala-sbt#boot-app
confs: [default]
44 artifacts copied, 0 already retrieved (13750kB/320ms)
[info] Loading project definition from /home/xxx/dev/scalaProjectDemo/project
[info] Updating {file:/home/xxx/dev/scalaProjectDemo/project/}scalaprojectdemo-build...
[info] Resolving org.scala-sbt.ivy#ivy;2.3.0-sbt-14d4d23e25f354cd296c73bfff40554[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] downloading https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/com.eed3si9n/sbt-assembly/scala_2.10/sbt_0.13/0.11.2/jars/sbt-assembly.jar ...
[info] [SUCCESSFUL ] com.eed3si9n#sbt-assembly;0.11.2!sbt-assembly.jar (2136ms)
[info] downloading https://repo1.maven.org/maven2/org/apache/spark/spark-core_2.10/1.1.0/spark-core_2.10-1.1.0.jar ...
[info] [SUCCESSFUL ] org.slf4j#slf4j-log4j12;1.7.5!slf4j-log4j12.jar (78ms)
1.14.v20131031!jetty-webapp.jar (103ms)
............
[info] downloading https://repo1.maven.org/maven2/com/google/protobuf/protobuf-java/2.4.0a/protobuf-java-2.4.0a.jar ...
[info] [SUCCESSFUL ] com.google.protobuf#protobuf-java;2.4.0a!protobuf-java.jar (387ms)
[info] downloading https://repo1.maven.org/maven2/asm/asm/3.2/asm-3.2.jar ...
[info] [SUCCESSFUL ] asm#asm;3.2!asm.jar (195ms)
[info] Done updating.
[info] Compiling 1 Scala source to /home/pooja/dev/scalaProjectDemo/target/scala-2.10/classes...
[info] Running com.jbksoft.HelloWorld
Hello World
[success] Total time: 97 s, completed Dec 8, 2016 2:58:29 PM
[/code]

e. Importing the code into IntelliJ

Edit the file plugins.sbt

[code]

addSbtPlugin("com.github.mpeltonen" % "sbt-idea" % "1.5.2")

[/code]

Edit the assembly.sbt

[code]

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

[/code]

Edit the build.properties

[code]sbt.version=0.13.6[/code]

Open the IntelliJ



Then from Menu choose File> Open



The open dialog opens



choose the project path and click OK



Leave default and click OK



The project will take time as it downloads the dependencies. Click OK



Select the new window as it open the project in new window.



The project will be imported in IntelliJ



Expand the src > main > scala



You can now add more files, run and debug the code in IntelliJ



The code will get compiled by scala compiler and with then be executed.



The output is run the window.

Let me know i you face any issues.

Happy Coding

CentOS 6 Installation using pen drive

On my laptop Dell Inspiron 5010, operating system windows 7 got crashed and I was not able to restore it.  Another problem was that it was running too slow so decided to format it and freshly install CentOS 6.

Below are the steps for installation of CentOS:

1. Created a bootable pen drive using Mac machine (used dd command)

Prerequisite:
Pen drive should have ample space (350 MB for minimal boot media and 4.5 GB for full installation media).I have done minimal installation. It will be formatted before iso image is copied to it.

a. Download your favourite iso image using the url https://www.centos.org/download/ (Latest CentOS) or http://isoredirect.centos.org/centos/6/isos/x86_64/ (Centos 6).

b. Figure out the device of USB stick.
First, list all the disk attached to Mac machine

[code language="text"]
diskutil list
[/code]


The one at the bottom is my USB drive (look at its memory or name to identify the USB stick)

c. Detached USB stick from current accessible filesystem
The USB stick data will be earsed before iso image, therefore if it's file is accessed by user then for dd command resource will keep on waiting. Therefore, it is required to umount pen drive using command below.

[code language="text"]
diskutil unmountDisk /dev/disk2
Unmount of all volumes on disk2 was successful
[/code]

d. Copy the iso image over USB Stick
Lets type in dd command to copy the iso images to USB stick. It may take time but don't terminate the session.

[code language="text"]
sudo dd if=CentOS-6.7-x86_64-minimal.iso of=/dev/disk2
[/code]

For i copied Centos 6 minimal iso image and it took me around 1 min.

2. Make the laptop boot from pen drive.
Go into laptop set up configuration (Press F2) and then select boot tab and then change the 1st boot priority to USB Storage Device. Exit and save configuration.

3. Installing from Pen drive to hard disk

Plug-in the bootable usb in the laptop and start on the machine. The screen will show you the installation step for CentOS.
a. Welcome to CentOS screen choose "Install or upgrade an existing system"
Library installation with start and it will take few minutes.
b. Then choose the language (chosen english). Then type of keyboard (chosen us). Then choose installation method as "Hard Drive"
c. Select Partition screen will show partition on the disk drive holding the CentOS iso image.
d. What type of devices will your installation involve screen show up. I have choose "Basic Storage Device" and press Next button.
e. Now, "Please name this computer. The hostname identifies the computer on network." screen shows up. Didn't touch localhost name. On the screen there is "Configure Network" button also. Just configure network and press next.
f. Next, screen shows Select your nearest city in the timezone (Select your city). And press Next.
g. Next screen show up the root password. Specify it and press next.
h. Next screen show "What type of installation would you like". "Use All Space" chosen (Not selected any checkbox "Encrypt System" or "Review and modify partitioning layout" and press next.
i. Then warning popup message shows up "Writing storage configuration to disk". Press "Write changes to disk" button.Result in formatting the hard drives.
j. The next screen will show up the "Boot loader operating system list" (you can add/delete device). Just press next.
k. The next screen show up with message "The default installation of CentOS is a minimum install. You can optionally to select different set of Software now".Chosen "Minimal" only and press next

Now the "CentOS installation starting" popup message will show up.

Lastly, "Congratulations, your CentOS installation is complete" screen shows up.

Conclusion

In this tutorial, I  had written the step followed to install the centOS 7 minimal operating system on Dell machine.

AWS Lambda S3 integration

AWS recently added new compute service called Lambda. The lambda compute service can process the data from S3, Dynamodb, SQS etc without provisioning the required compute explicitly. The lambda service can listen to S3 and can process the file as it is put into the S3 bucket.

For this you need to create a s3 bucket.



Name the bucket as per your choice.



Log in the AWS console.



Choose Lambda. If you are accessing it the first time then you will see below page.



Click Get Started.

If you already have lambda functions in your account the screen will be as below, Click create a Lambda Function.





The lambda function is 2 step process. You can skip the blueprint for this tutorial.



Name your Lambda function. Provide short description. Choose the Java 8 as runtime.



Now we require java code in the form of zip file. Lets create the java file using maven project.

Create a maven project using below command

[code language="xml"]
mvn archetype:generate -DarchetypeArtifactId=maven-archetype-quickstart -DgroupId=com.jbksoft.lambda -DartifactId=lambda-s3-demo -DinteractiveMode=false -DarchetypeVersion=1.1
[/code]

Download the below versions of AWS-Java-SDK, AWS-Lambda-java-core and aws-lambda-java-events using the maven dependency. Update pom.xml with below dependency.

 <dependency>
   <groupId>com.amazonaws</groupId>
   <artifactId>aws-lambda-java-core</artifactId>
   <version>1.0.0</version>
 </dependency>
 <dependency>
   <groupId>com.amazonaws</groupId>
   <artifactId>aws-lambda-java-events</artifactId>
   <version>1.0.0</version>
 </dependency>
 <dependency>
   <groupId>com.amazonaws</groupId>
   <artifactId>aws-java-sdk-s3</artifactId>
   <version>1.10.36</version>
 </dependency>

 <build>
   <finalName>lambdas3demo</finalName>
   <plugins>
     <plugin>
       <groupId>org.apache.maven.plugins</groupId>
       <artifactId>maven-shade-plugin</artifactId>
       <version>2.3</version>
       <configuration>
         <createDependencyReducedPom>false</createDependencyReducedPom>
       </configuration>
       <executions>
         <execution>
           <phase>package</phase>
             <goals>
               <goal>shade</goal>
             </goals>
         </execution>
       </executions>
     </plugin>
   </plugins>
 </build>

Create a java file S3LambdaDemo.java. This handler returns the bucket concat with filename like sourcebucket\HappyFace.jpg.

[code language="java"]
package com.jbksoft.lambda;

import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.amazonaws.services.lambda.runtime.events.S3Event;
import com.amazonaws.services.s3.event.S3EventNotification;

public class S3LambdaDemo implements RequestHandler&amp;lt;S3Event, String&amp;gt; {
public String handleRequest(S3Event s3Event, Context context) {
S3EventNotification.S3EventNotificationRecord record = s3Event.getRecords().get(0);
String srcBucket = record.getS3().getBucket().getName();
String srcKey = record.getS3().getObject().getKey().replace('+', ' ');
return srcBucket+"/"+srcKey;
}
}
[/code]

Run on terminal

mvn clean package

Upload the jar file.



Provide Handler: com.jbksoft.lambda.S3LambdaDemo and Choose the role: lambda-execution-s3-role(Refer the execution role for role creation).



Click create.



Click test and choose s3 put from drop down.



Click 'save and test'. The lambda will provision the aws service and trigger the lambda with test file HappyFace.jpg(line 14) and source bucket 'sourcebucket' (line 19).

The output is name of bucket concat with filename.

Similarly the handler can process the content of the file on s3 as well.

Reference:

• AWS S3


• AWS Lambda