Friday, 22 August 2014

Use Avro in Mobile Apps

It could be a good idea to replace JSON(XML)  to Avro format as the response of Mobile apps.
Because it will provide schema and reduce the transmission data size.

But it is not that simple to implement it due to some features of Avro.
By default, the Avro schema is hashed and the hash is stored in every event in the event header. 
The body of the event is the binary Avro record data, not including the schema

Recently, Flurry successfully made Avro work for Android and iOS apps.
The challenges are below.

1. Schema Matter
"
We maintain the invariant that the schema is always present when reading Avro data, however, unlike Avro’s RPC mechanism we never transport the schema with the data. While the Avro docs note this can be optimized to reduce the need to transport the schema in a handshake, such optimization could prove difficult and would likely not provide much added benefit for us. In mobile, we deal with many small requests from millions of devices that have transient connections. Fast, reliable transmissions with as little overhead as possible are our driving factors.

To meet our network goals we version our endpoints to correspond to the protocol. We feel it is more transparent and maintainable to have a one to one schema mapping instead of relying on Avro to resolve the differences between a varied reader and writer schema. The result is a single call without the additional communication for schema negotiation while transporting only datum that has no per-value overhead of field identifiers.
"

2. Mobile Matter
"
As a 3rd party library on mobile we had some additional areas we needed to address. First on Android, we needed to maintain support for Version 1.5+. Avro uses a couple of methods that were introduced in version 2.3, namely Arrays.copyOf() and  String.getBytes(Charset). Those were easily changed to use System.arrayCopy and String.getBytes(String), but it did require us to build from source and repackage.
On iOS, we need to avoid naming conflicts in case someone else integrates Avro in their library. We are able to do this by stripping the symbols. See this guide from Apple on that process.  On Android, we use ProGuard, which solves conflict issues through obfuscation. Alternatively, you can use the jarjar tool to change your package naming
Once these troubles are efficiently solved. Avro provides you consistency through structured schemas and maintain high performance.

References:
http://www.flurry.com/2012/07/12/apache-avro-at-flurry
https://github.com/flurry/avro-mobile

No comments:

Post a Comment