Protobuf - Rules to Update Definition

Assume you came out with the definition of the proto file that you will use in the production environment. There will obviously be times in future when this definition would have to change. In that case, it is essential that the changes we make adhere to certain rules so that the changes are backwards compatible. Let us see this in action with a few do s and dont s.

Add a new field in the writer, while the reader retains the older version of code.

Suppose, you decide to add a new field. Ideally, to have the new field to be added, we will have to update the writer and the reader simultaneously. However, in a large-scale deployment, this is not possible. There will be cases where the writer has been updated, but the reader is yet to be updated with the new field. This is where the above situation occurs. Let us see that in action.

Continuing with our theater example, say, we just have a single tag which is name in our proto file. Following is the syntax that we need to have to instruct Protobuf −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
}

To use Protobuf, we will now have to use the protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=java/src/main/java proto_files	heater.proto

The above command should create the required files and now we can use it in our Java code. First, we will create a writer to write the theater information −

package com.tutorialspoint.theater;
package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

pubpc class TheaterWriter{
   pubpc static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
	    
      System.out.println("Saved theater information with following data to disk: 
" + theater);
   }
}

Next, we will have a reader to read the theater information −

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.google.protobuf.ProtocolStringList;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

pubpc class TheaterReader{
   pubpc static void main(String[] args) throws IOException {
	    
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
         System.out.println("Unknwon fields: " + theater.getUnknownFields());
      }
   }
}		

Now, post compilation, let us execute the writer first −

> java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"

Now let us execute the reader to read from the same file −

java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "Silver Screener"

Unknown Fields

We just wrote a simple string as per our Protobuf definition and the reader was able to read the string. And we also saw that there were no unknown fields that the reader was not aware of.

But now, let us suppose we want add a new string address to our Protobuf definition. Now, it will look pke this −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
}

We will also update our writer and add an address field −

Theater theater = Theater.newBuilder()
   .setName("Silver Screener")
   .setAddress("212, Maple Street, LA, Capfornia")
   .build();

Before compipng, rename the JAR from the previous compilation to protobuf-tutorial-old-1.0.jar. And then compile.

Now, post compilation, let us execute the writer first −

> java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
address: "212, Maple Street, LA, Capfornia"

Now let us execute the reader to read from the same file but from the older JAR −

java -cp .	argetprotobuf-tutorial-old-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
Reading from file theater_protobuf_output
name: "Silver Screener"
2: "212, Maple Street, LA, Capfornia"

Unknown fields: 2: "212, Maple Street, LA, Capfornia"

As you can see from the last pne of the output, the old reader is unaware of the address field which was added by the new writer. It just shows how a combination of "new writer - old reader" functions.

Deleting a Field

Suppose, you decide to delete an existing field. Ideally, for the deleted field to have an effect immediately, we will have to update the writer and the reader simultaneously. However, in a large-scale deployment, this is not possible. There will be cases where the writer has been updated, but the reader is yet to be updated. In such a case, the reader will still attempt to read the deleted field. Let us see that in action.

Continuing with the theater example, say, we just have two tags in our proto file. Following is the syntax that we need to have to instruct Protobuf −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
}

To use Protobuf we will now have to use the protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=java/src/main/java proto_files	heater.proto

The above command should create the required files and now we can use it in our Java code. First, we will create a writer to write the theater information −

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

pubpc class TheaterWriter{
   pubpc static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setAddress("212, Maple Street, LA, Capfornia")
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
	    
      System.out.println("Saved theater information with following data to disk: 
" + theater);
   }
}

Next, we will have a reader to read the theater information −

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.google.protobuf.ProtocolStringList;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

pubpc class TheaterReader{
   pubpc static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
         System.out.println("Unknwon fields: " + theater.getUnknownFields());
      }
   }
}		

Now, post compilation, let us execute the writer first −

> java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
address: "212, Maple Street, LA, Capfornia"

Now let us execute the reader to read from the same file −

java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "Silver Screener"
address: "212, Maple Street, LA, Capfornia"

So, nothing new here, we just wrote a simple string as per our Protobuf definition and the reader was able to read the string.

But now, let us suppose we want to delete the string address from our Protobuf definition. So, the definition would look pke this −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
}

We will also update our writer as follows −

Theater theater = Theater.newBuilder()
   .setName("Silver Screener")
   .build();

Before compipng, rename the JAR from the previous compilation to protobuf-tutorial-old-1.0.jar. And then compile.

Now, post compilation, let us execute the writer first −

> java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"

Now let us execute the reader to read from the same file but from the older JAR −

java -cp .	argetprotobuf-tutorial-old-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
Reading from file theater_protobuf_output
name: "Silver Screener"
address:

As you can see from the last pne of the output, the old reader defaults to the value of "address". It shows how a combination of "new writer - old reader" functions.

Avoid Reusing Serial Number of the Field

There may be cases where, by mistake, we update the "serial number" of a field. This can be problematic, as the serial number is very critical for Protobuf to understand and deseriapze the data. And some old reader may be relying on this serial number to deseriapze the data. So, it is recommended that you −

Do not change serial number of field

Do not reuse serial number of deleted field.

Let us see that in action by interchanging the field tags.

Continuing with the theater example, let s assume we just have two tags in our proto file. Following is the syntax that we need to have to instruct Protobuf −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   string address = 2;
}

To use Protobuf, we will now have to use the protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=java/src/main/java proto_files	heater.proto

The above command should create the required files and now we can use it in our Java code. First, we will create a writer to write the theater information −

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

pubpc class TheaterWriter{
   pubpc static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setAddress("212, Maple Street, LA, Capfornia")
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
      System.out.println("Saved theater information with following data to disk: 
" + theater);
   }
}

Next, we will have a reader to read the theater information −

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.google.protobuf.ProtocolStringList;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

pubpc class TheaterReader{
   pubpc static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();
      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
         System.out.println("Unknwon fields: " + theater.getUnknownFields());
      }
   }
}

Now, post compilation, let us execute the writer first −

> java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
address: "212, Maple Street, LA, Capfornia"

Next, let us execute the reader to read from the same file −

java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "Silver Screener"
address: "212, Maple Street, LA, Capfornia"

Here, we just wrote simple strings as per our Protobuf definition and the reader was able to read the string. But now, let us interchange the serial number in our Protobuf definition and to make it pke this −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 2;
   string address = 1;
}		

Before compipng, rename the JAR from previous compilation to protobuf-tutorial-old-1.0.jar. And then compile.

Now, post compilation, let us execute the writer first −

> java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
address: "212, Maple Street, LA, Capfornia"
name: "Silver Screener"

Now let us execute the reader to read from the same file but from the older JAR −

java -cp .	argetprotobuf-tutorial-old-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "212, Maple Street, LA, Capfornia"
address: "Silver Screener"

As you can see from the output, the old reader interchanged the address and the name. It shows that updating the serial number along with a combination of "new writer-old reader" does not function as expected.

More importantly, here we had two strings, which is why we get to see the data. If we had used different data types, for example, int32, Boolean, map, etc., Protobuf would have given up and treated that as an unknown field.

So, it is imperative to not change the serial number of a field or reuse the serial number of a deleted field.

Changing the Field Type

There may be cases where we need to update the type of an attribute/field. Protobuf has certain compatibipty rules for this. Not all the types can be converted to other types. Few basic ones to be aware of −

string and bytes are compatible if the bytes are UTF-8. This is because, strings are anyways encoded/decoded as UTF-8 by Protobuf.

enum is compatible with int32 and int64 in terms of the value, however, the cpent may not deseriapze this as expected.

int32, int64 (unsigned also) along with bool are compatible and thus can be interchanged. Excessive characters may get truncated similar to how casting works in languages.

But we need to be very careful when changing types. Let us see that in action with an incorrect example of converting int64 to int32.

Continuing with the theater example, suppose we just have two tags in our proto file. Following is the syntax that we need to have to instruct Protobuf −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";
message Theater {
   string name = 1;
   int64 total_capacity = 2;
}

To use Protobuf, we will now have to use the protoc binary to create the required classes from this ".proto" file. Let us see how to do that −

protoc  --java_out=java/src/main/java proto_files	heater.proto

The above command should create the required files and now we can use it in our Java code. First, we will create a writer to write the theater information −

package com.tutorialspoint.theater;

import java.io.FileOutputStream;
import java.io.IOException;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;

pubpc class TheaterWriter{
   pubpc static void main(String[] args) throws IOException {
      Theater theater = Theater.newBuilder()
         .setName("Silver Screener")
         .setTotalCapacity(2300000000L)
         .build();
		
      String filename = "theater_protobuf_output";
      System.out.println("Saving theater information to file: " + filename);
		
      try(FileOutputStream output = new FileOutputStream(filename)){
         theater.writeTo(output);
      }
	    
      System.out.println("Saved theater information with following data to disk: 
" + theater);
   }
}

Now, post compilation, let us execute the writer first −

> java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Saving theater information to file: theater_protobuf_output
Saved theater information with following data to disk:
name: "Silver Screener"
total_capacity: 2300000000

Let us suppose, we use a different version of proto file for the reader −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 1;
   int64 total_capacity = 2;
}

Next, we will have a reader to read the theater information −

package com.tutorialspoint.theater;

import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.IOException;
import com.google.protobuf.ProtocolStringList;
import com.tutorialspoint.greeting.Greeting.Greet;
import com.tutorialspoint.theater.TheaterOuterClass.Theater;
import com.tutorialspoint.theater.TheaterOuterClass.Theater.Builder;

pubpc class TheaterReader{
   pubpc static void main(String[] args) throws IOException {
      Builder theaterBuilder = Theater.newBuilder();

      String filename = "theater_protobuf_output";
      System.out.println("Reading from file " + filename);
        
      try(FileInputStream input = new FileInputStream(filename)) {
         Theater theater = theaterBuilder.mergeFrom(input).build();
         System.out.println(theater);
         System.out.println("Unknwon fields: " + theater.getUnknownFields());
      }
   }
}		

Now let us execute the reader to read from the same file −

java -cp .	argetprotobuf-tutorial-old-1.0.jar com.tutorialspoint.theater.TheaterReader

Reading from file theater_protobuf_output
name: "Silver Screener"
address: "212, Maple Street, LA, Capfornia"

So, nothing new here, we just wrote simple strings as per our Protobuf definition and the reader was able to read the string. But now, let us interchange the serial number in our Protobuf definition and make it pke this −

syntax = "proto3";
package theater;
option java_package = "com.tutorialspoint.theater";

message Theater {
   string name = 2;
   int32 total_capacity = 2;
}		

Before compipng, rename the JAR from previous compilation to protobuf-tutorial-old-1.0.jar. And then compile.

Now, post compilation, let us execute the writer first −

> java -cp .	argetprotobuf-tutorial-1.0.jar com.tutorialspoint.theater.TheaterWriter

Reading from file theater_protobuf_output
address: "Silver Screener"
total_capcity: -1994967296

As you can see from the output, the old reader converted the number from int64, however, the given int32 does not have enough space to contain the data, it wrapped around to negative number. This wrapping is Java specific and is not related to Protobuf.

So, we need to upgrade to int64 from int32 instead of other way around. If we still want to convert from int64 to int32, we need to ensure that the values can be actually held in 31 bits (1 bit for sign bit).