Send data between 2 JVMs using protocol buffer RPC

Protocol Buffer is flexible,efficient,automated,inter-operable solution for serialization.It makes writing remote procedure call(rpc) simpler and easy to implement by serializing/deserializing request/response objects. But, it doesn't implement any transport details by itself.
Now, protobuf-socket-rpc is a simple tcp/ip socket based rpc implementation in java, python for protobuf rpc services.

In this blog, we will discuss about implementing a rpc program
1. Creating the request/response object using .proto
2. Creating the Service class using .proto
3. Implementing the Service method
4. Write the Server application
5. Write Client application

Prerequisites:
1. Java installed. If not, install java.
2. Protocol Buffer 2.4.0 installed. If not, install
3. If not maven project download protobuf-java-2.4.1.jar and also download protobuf-socket-rpc-2.0.jar.
If maven project add dependency and download protobuf-socket-rpc-2.0.jar in local machine in directory ${project.basedir}/thirdparty/protobuf-socket-rpc-2.0.jar.
[code language="text"]
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.4.1</version>
</dependency>
<dependency>
<groupId>com.googlecode.protobuf.socketrpc</groupId>
<artifactId>protobuf-socket-rpc</artifactId>
<version>2.0</version>
<scope>system</scope>
<systemPath>${project.basedir}/thirdparty/protobuf-socket-rpc-2.0.jar</systemPath>
</dependency>
</dependencies>
[/code]
Note: protoc-java.jar version should be same as protoc –version. Otherwise error will popup java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

Now, we will be creating a simple greeting service.

Creating the request/response object using .proto.
We will be creating the objects for remote communication. We have named the file as 'testData.proto'.

[code language="text"]
package protobufDemo; //this is protobuf namespace, not java's
option java_package = "com.xxx.protobufferrpc.protobufferrpcData";
option java_outer_classname = "GreetingProtos";
message HelloRequest
{
required string name = 1;
}
message HelloReply
{
required string message = 1;
}
[/code]

After it generate the automatic classes using below command:

[code language="text"]
$ protoc --java_out=. name_of_protoc.proto
[/code]

Creating the Service class using .proto
We will be creating the service class.
[code language="text"]
package protobufDemo;
import "testData.proto";
option java_package = "com.xxx.protobufferrpc.protobufferrpcData";
option java_outer_classname = "MyGreetingService";
option java_generic_services = true; //if you don't do this, protoc wont generate the stubs you need for rpc

service Greeting //In generated class, this class is abstract class that extends service method need to extends this
{
rpc sayHello(HelloRequest) returns (HelloReply);
}
[/code]
After it generate the automatic classes using below command:
[code language="text"]
$ protoc --java_out=. name_of_protoc.proto
[/code]

Implementing the Service method
Now, we will be implementing rpc sayHello method defined above.
[code language="text"]
package com.xxx.protobufferrpc;

import com.google.protobuf.RpcCallback;
import com.google.protobuf.RpcController;
import com.xxx.protobufferrpc.protobufferrpcData.GreetingProtos;
import com.xxx.protobufferrpc.protobufferrpcData.MyGreetingService.Greeting;

public class MyGreetingServiceImpl extends Greeting {

@Override
public void sayHello(RpcController controller, GreetingProtos.HelloRequest request, RpcCallback<GreetingProtos.HelloReply> done) {
GreetingProtos.HelloReply.Builder build= GreetingProtos.HelloReply.newBuilder();
if(request.getName().equalsIgnoreCase("namenode")){
build.setMessage("This is message for namenode only");

}else{
build.setMessage("Please see person sending message");
}
done.run(build.build());
}
}
[/code]

Write the Server application
The server application is listening on port 4446.

[code language="text"]
package com.xxx.protobufferrpc;

import java.util.concurrent.Executors;
import com.googlecode.protobuf.socketrpc.RpcServer;
import com.googlecode.protobuf.socketrpc.ServerRpcConnectionFactory;
import com.googlecode.protobuf.socketrpc.SocketRpcConnectionFactories;

public class ServerCode {
/**
* @param args the command line arguments
*/
public static void main(String[] args)
{
ServerRpcConnectionFactory rpcConnectionFactory = SocketRpcConnectionFactories.createServerRpcConnectionFactory(4446);
RpcServer server = new RpcServer(rpcConnectionFactory, Executors.newFixedThreadPool(5), true);
server.registerService(new MyGreetingServiceImpl());
server.run();
}

}
[/code]

Write Client application
In the client code, we will be passing the IP Address of server machine and port (4446) for calling remote procedure "sayHello".

[code language="text"]
package com.xxx.protobufferrpc;

import com.google.protobuf.RpcCallback;
import com.google.protobuf.RpcChannel;
import com.google.protobuf.RpcController;
import com.googlecode.protobuf.socketrpc.RpcChannels;
import com.googlecode.protobuf.socketrpc.RpcConnectionFactory;
import com.googlecode.protobuf.socketrpc.SocketRpcConnectionFactories;
import com.googlecode.protobuf.socketrpc.SocketRpcController;
import com.xxx.protobufferrpc.protobufferrpcData.GreetingProtos;
import com.xxx.protobufferrpc.protobufferrpcData.MyGreetingService;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;

public class ProtoClient {
public static void main(String[] args)
{
// Create a thread pool
ExecutorService threadPool = Executors.newFixedThreadPool(1);
// Create channel
String host =args[0]; //IP Address of machine "192.16.42.22";
int port = 4446;
RpcConnectionFactory connectionFactory = SocketRpcConnectionFactories.createRpcConnectionFactory(host, port);
RpcChannel channel = RpcChannels.newRpcChannel(connectionFactory, threadPool);

// Call service
MyGreetingService.Greeting.Stub myService = MyGreetingService.Greeting.newStub(channel);
RpcController controller = new SocketRpcController();

GreetingProtos.HelloRequest.Builder cr = GreetingProtos.HelloRequest.newBuilder();
cr.setName("Hello");
myService.sayHello(controller, cr.build(),
new RpcCallback<GreetingProtos.HelloReply>()
{
public void run(GreetingProtos.HelloReply myResponse)
{
System.out.println("Received Response: " + myResponse);
}
});
// Check success
if (controller.failed())
{
System.err.println(String.format("Rpc failed %s ", controller.errorText()));
}
}

}
[/code]

Now, run the server and client application.

Hope you follow the discussion.

References:
Protocol Socket RPC
Google Protocol Buffer

Protocol Buffer Serializing in Java

Google Protocol Buffer is platform neutral, extensible tool for serializing structure data. It is inter-operable and not specific to any language.

In java, we have number of way we can serialized the object, some of the ways are listed below:
1. Java Build-in library : Marker class java.io.Serializable and just implement read,write method in class. But this way, serialized stream can be read/unmarshalled by Java program only.
2. Simple String : Need to implement code to covert to bytes and vice versa.
3. XML : XML DOM parsing can be memory intensive.
Protocol Buffer is flexible,efficient,automated solution to above problem.

In this post, we will be discussing about
1. Creating the .proto file (Structure Data)
2. Generating the automated classes
3. Writing Serialized Data
4. Reading Serialized Data

Prerequisites:
1. Java installed. If not, install.
2. Protocol Buffer installed. If not, install
3. If not maven project download protobuf-java-2.5.0.jar.
If maven project add dependency.

[code language="text"]
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>2.5.0</version>
</dependency>
[/code]
Note: protoc-java.jar version should be same as protoc --version. Otherwise error will popup java.lang.UnsupportedOperationException: This is supposed to be overridden by subclasses.

Creating the .proto file
Let create a simple structure object PersonAccountsBook embedding the Account object as as shown below:

[code language="text"]
package protobufDemo; //this is protobuf namespace, not java's
option java_package = "com.xxx.protobuf.protobufferData";
option java_outer_classname = "PersonAccountsBookProtos";

message Person {
required string name = 1;
required int32 id = 2;
optional string email = 3;

enum AccountType{
CHECKING = 0;
SAVING = 1;
VISA = 2;
}

message AccountNumber {
required string number = 1;
optional AccountType type = 2 [default = SAVING];
}

repeated AccountNumber accounts= 4;
}

message PersonAccountsBook {
repeated Person person = 1;
}
[/code]

Now, the Data Type can be any predefined like bool, int32, float, double, and string or user defined as AccountType.The, number specified with fields "=1" or "=2" is unique tags given. Number 1-15 required less code encoding than higher number than 15.
The fields can be
required- If field value required by the structured data like person_id,AccountNumber
optional-If field value has 0 or 1 occurrence
repeated-If field value has 0 to many occurrence.

Generating the automated classes from .proto file

The below code will generate the class PersonAccountsBookProtos in package "com.xxx.protobuf.protobufferData".Note .proto in path project_path/src/java/PersonAccountsBook.proto

[code language="text"]
protoc --java_out=. PersonAccountsBook.proto
[/code]

Writing Serialized Structure Data
In this class we need to provide the person account book file path argument, if file not exist then it will create one. Note: We have Builder class to pass value to the fields as shown below.

[code langugae="text"]
package com.xxx.protobuf;

import com.xxx.protobuf.protobufferData.PersonAccountsBookProtos.PersonAccountsBook;
import com.xxx.protobuf.protobufferData.PersonAccountsBookProtos.Person;

import java.io.*;

public class AddPerson {

// This function fills in a Person message based on user input.
static Person PromptForAddress(BufferedReader stdin,
PrintStream stdout) throws IOException {
Person.Builder person = Person.newBuilder();

stdout.print("Enter person ID: ");
person.setId(Integer.valueOf(stdin.readLine()));

stdout.print("Enter name: ");
person.setName(stdin.readLine());

stdout.print("Enter email address (blank for none): ");
String email = stdin.readLine();
if (email.length() > 0) {
person.setEmail(email);
}

while (true) {
stdout.print("Enter a Account Number (or leave blank to finish): ");
String number = stdin.readLine();
if (number.length() == 0) {
break;
}

Person.AccountNumber.Builder accountNumber =
Person.AccountNumber.newBuilder().setNumber(number);

stdout.print("Is this a saving,checking,visa account? ");
String type = stdin.readLine();
if (type.equals("saving")) {
accountNumber.setType(Person.AccountType.SAVING);
} else if (type.equals("visa")) {
accountNumber.setType(Person.AccountType.VISA);
} else if (type.equals("checking")) {
accountNumber.setType(Person.AccountType.CHECKING);
} else {
stdout.println("Unknown account type. Using default.");
}

person.addAccounts(accountNumber);
}

return person.build();
}

// Main function: Reads the entire account book from a file,
// adds one person based on user input, then writes it back out to the same
// file.
public static void main(String[] args) throws Exception {
if (args.length != 1) {
System.err.println("Usage: AddPerson ACCOUNT_BOOK_FILE");
System.exit(-1);
}

PersonAccountsBook.Builder accountBook = PersonAccountsBook.newBuilder();

// Read the existing address book.
try {
accountBook.mergeFrom(new FileInputStream(args[0]));
} catch (FileNotFoundException e) {
System.out.println(args[0] + ": File not found. Creating a new file.");
}

// Add an address.
accountBook.addPerson(
PromptForAddress(new BufferedReader(new InputStreamReader(System.in)),
System.out));

// Write the new address book back to disk.
FileOutputStream output = new FileOutputStream(args[0]);
accountBook.build().writeTo(output);
output.close();
}
}
[/code]

Reading Serialized Data
We need to provide person account book file path argument to the java application.

[code langugae="text"]
package com.xxx.protobuf;

import com.xxx.protobuf.protobufferData.PersonAccountsBookProtos.PersonAccountsBook;
import com.xxx.protobuf.protobufferData.PersonAccountsBookProtos.Person;

import java.io.FileInputStream;

public class ListPerson {
// Iterates though all people in the AccountBook and prints info about them.
static void Print(PersonAccountsBook accountBook) {
for (Person person: accountBook.getPersonList()) {
System.out.println("Person ID: " + person.getId());
System.out.println(" Name: " + person.getName());
if (person.hasEmail()) {
System.out.println(" E-mail address: " + person.getEmail());
}

for (Person.AccountNumber accountNumber : person.getAccountsList()) {
switch (accountNumber.getType()) {
case VISA:
System.out.print(" Visa Account #: ");
break;
case SAVING:
System.out.print(" Saving Account #: ");
break;
case CHECKING:
System.out.print(" Checking Account #: ");
break;
}
System.out.println(accountNumber.getNumber());
}
}
}

// Main function: Reads the entire address book from a file and prints all
// the information inside.
public static void main(String[] args) throws Exception {
if (args.length != 1) {
System.err.println("Usage: ListPeople Account_BOOK_FILE");
System.exit(-1);
}

// Read the existing address book.
PersonAccountsBook accountBook =
PersonAccountsBook.parseFrom(new FileInputStream(args[0]));

Print(accountBook);
}
}
[/code]

Hope you follow the discussion and it would be useful to you.

References:
Google protocol buffer tutorial
Java Serialization API

Install Google Protocol Buffer 2.5.0 on CentOS (configure: error: C++preprocessor "/lib/cpp" fails sanity check See `config.log' for moredetails.)

Google Protocol Buffer is platform neutral, extensible tool for serializing structure data. It is inter-operable and not specific to any language.
While installing Protocol Buffer 2.5.0 on CentOS machine got configuration error, so document it as it may be useful for others.

Step 1:Download protobuf-2.5.0.tar.bz2
[code language="text"]
$ sudo wget https://github.com/google/protobuf/releases/download/v2.5.0/protobuf-2.5.0.tar.gz

--2015-09-15 11:33:14-- http://protobuf.googlecode.com/files/protobuf-2.5.0.tar.bz2
Resolving protobuf.googlecode.com... 2607:f8b0:400e:c04::52, 74.125.129.82
Connecting to protobuf.googlecode.com|2607:f8b0:400e:c04::52|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1866763 (1.8M) [application/x-bzip2]
Saving to: “protobuf-2.5.0.tar.bz2”

100%[======================================>] 1,866,763 2.45M/s in 0.7s
2015-09-15 11:33:15 (2.45 MB/s) - “protobuf-2.5.0.tar.bz2” saved [1866763/1866763]
[/code]

Step 2: Expand the downloaded tar file and remove the tar file
[code language="text"]
$ tar -xvf protobuf-2.5.0.tar.bz2
$ rm -rf protobuf-2.5.0.tar.bz2
[/code]

Step 3: Configure the protobuf
[code langugae="text"]
$ cd protobuf-2.5.0/
$ ./configure
[/code]
There I got error
[code langugae="text"]
configure: error: C++ preprocessor "/lib/cpp" fails sanity check See `config.log' for more details.
[/code]
Now, check if you have it the right package, links, and versions, run these commands:
[code language="text"]
$ cpp --version
cpp (GCC) 4.4.7 20120313 (Red Hat 4.4.7-16)
Copyright (C) 2010 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
[/code]
If output is there then cpp lack corresponding library
[code language="text"]
$ sudo yum install glibc-headers
$ sudo yum install gcc-c++
[/code]
Now, again run configure,
[code language="text"]
$ ./configure
checking whether to enable maintainer-specific portions of Makefiles... yes
checking build system type... x86_64-unknown-linux-gnu
......................
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/gtest-config
config.status: creating build-aux/config.h
config.status: executing depfiles commands
config.status: executing libtool commands
[/code]

Step 4: Make
[code language="text"]
$ make
pthread -Wall -Wwrite-strings -Woverloaded-virtual -Wno-sign-compare -O2 -g -DNDEBUG -MT plugin.lo -MD -MP -MF .deps/plugin.Tpo -c -o plugin.lo `test -f 'google/protobuf/compiler/plugin.cc' || echo './'`google/protobuf/compiler/plugin.cc......
............
libtool: link: ranlib .libs/libprotobuf-lite.a
libtool: link: ( cd ".libs" && rm -f "libprotobuf-lite.la" && ln -s "../libprotobuf-lite.la" "libprotobuf-lite.la" )
make[3]: Leaving directory `/home/pooja/protobuf-2.5.0/src'
make[2]: Leaving directory `/home/pooja/protobuf-2.5.0/src'
make[1]: Leaving directory `/home/pooja/protobuf-2.5.0
[/code]

Step 5: Install
[code language="text"]
$ sudo make install
Making install in .
make[1]: Entering directory `/home/pooja/protobuf-2.5.0'
make[2]: Entering directory `/home/pooja/protobuf-2.5.0'
make[2]: Nothing to be done for `install-exec-am'.
test -z "/usr/local/lib/pkgconfig" || /bin/mkdir -p "/usr/local/lib/pkgconfig"
/usr/bin/install -c -m 644 protobuf.pc protobuf-lite.pc '/usr/local/lib/pkgconfig'
make[2]: Leaving directory `/home/pooja/protobuf-2.5.0'
atomicops_internals_x86_msvc.h google/protobuf/stubs/common.h google/protobuf/stubs/platform_macros.h google/protobuf/stubs/once.h google/protobuf/stubs/template_util.h google/protobuf/stubs/type_traits.h '/usr/local/include/google/protobuf/stubs'
.......................................
make[3]: Leaving directory `/home/pooja/protobuf-2.5.0/src'
make[2]: Leaving directory `/home/pooja/protobuf-2.5.0/src'
make[1]: Leaving directory `/home/pooja/protobuf-2.5.0/src
[/code]

Step 6: Verify protobuf installed properly
[code language="text"]
$ protoc
Missing input file.

$ protoc --version
libprotoc 2.5.0
[/code]

Hope you installed Protocol Buffer without hassel. Feel free to write to me. Thanks.

References:
Google protobuf github
Google protocol buffer tutorial