Participants – made of developers, designers and citizens, were asked to leverage the Voxe data API and build a smart app aiming at helping people choose their candidate the clever way for the French presidential election, 2012.

Brainstorm

On my own I had no predefined idea at pitch time. Keeping in mind the smartphone explosion I started brainstorming around 2 simple facts:

• • there is a huge amount of political propositions among the set of candidates: at the time of writing Voxe references 1500+ propositions. This makes it hard for many of us to take time to read, understand and get an opinion on them,
• • on a daily basis, until the election D-day, we will go through the candidate official posters as we will commute to work – and this could really act as a kind of daily reminder.

So I thought it would be great if I could be provided with a random proposition each time I encounter the poster of a candidate – think of it as a kind of one proposition a day. And it would be cool too if I could record at that time if I agree or disagree with this precise proposition: from day to day, cumulating these small votes would lead to a good evaluation of what is my real favorite candidate!

Mashup

Such a concept was an ideal use case to leverage both Voxe and Moodstocks plaftorms since:

• • Voxe provides all the required data via its API from candidates’ propositions (of course) to topics and avatars,
• • the Moodstocks SDK is able to identify any candidate poster via the smartphone camera and makes it possible to display related content (here a random proposition) and actions (here vote buttons) directly in overlay of the video capture.

I then found a name for the app – Voxometre, a few fords to depict it – my very own citizen remote control, and started the technical design & implementation focusing on the iOS platform.

Implementation

To me such an application had to follow the serverless programming principle (i.e. full client-side app) since:

• • the app has no sharing or social features: each small vote made on the random propositions is personal and has to be stored directly on the client. There is no need to centralize this data on the server-side,
• • it is a mashup app that consumes all its data from external APIs,
• • it is important that the app keeps its data available locally to maximize the user experience via real-time interactions at scanning time (i.e. zero network latency to fetch the data).

So the main topics I had to focus on were:

• • generating the pre-defined data and assets: I have written some scripts to pre-fetch the data that will not change from Voxe API (candidacy IDs, candidate names, avatars, main tags, etc) and embed these PNG-s and custom JSON directly within the app,
• • designing the data synchronization phase: the goal was to trigger at cold start the Moodstocks sync (to retrieve the image signatures) and write some logic to fetch and store the whole set of propositions by calling the search propositions by candidacyID HTTP API provided by Voxe,
• • choosing an embedded database and defining the data model: since we need to store and persist the set of propositions plus the votes made by the end user on the client, I had to use an embedded database. To make sure things were fun enough (!) I decided to use Redisk – the not-so-stable-nor-production-ready Redis-API-compatible embedded database written in C I had co-created with Pierre at the former Hack Day Paris hackathon last november,
• • implementing the overlay made of the proposition & related tag (as info) plus the votes up / down buttons (as actions).

Being able to use a Redis-like API on the client-side was pretty handy since it offers simple and convenient data structures to achieve our job. Below is an overview of the (very) simple data model:

Propositions values are stored in raw JSON even if we could have used Redis hashes to store them unserialized. The key format allows to easily find a random proposition by filtering the set of propositions for a candidate via key forward matching.

A set is used to record every proposition that has been voted.

At last the INCR command is used to bump the votes counters per candidate.

Design

Then it was time to style the app! I worked with Aymeric that did a great job at choosing a color theme, designing a logo, creating styled buttons and producing a keynote for the final presentation (feel free to browse these slides for some more details and screenshots).

Once again I would like to thank the Voxe team for the great work they have made organizing this event! Also congrats to Pierre and his team for their Democrazy project!

Stay tuned until the next April, 3rd since we will present the project for this Mash Up Birthday event!

[UPDATE] We have now a landing page!

 Before going on to the bug in itself, let us give a little context on how it happened. We designed the Moodstocks SDK as a C library and we ported it on iOS then Android thanks to the Android Native Development Kit (NDK). For performance reasons, we decided to support only ARMv7 devices (non-ARMv7 devices being mainly older devices), and cross-compiled our library so that the non-ARMv7 version would only return errors to inform the user that his/her device was not compatible.

Bug

Everything went perfectly well until this terrible day of March 2012, when one of our developers informed us that his brand new Samsung Galaxy Nexus running ICS told him that it was not compatible with our SDK. Even worse: after a few tests, he determined that it worked like a charm if he did not bundle too many resources with his app (animation xml’s, graphical resources, etc…), and suddenly decided to become incompatible if he bundled more than 8 resources. Yes, you heard me right: apparently, one of the latest and most powerful smartphones available, running the latest version of Android on its dual-core 1.5GHz ARMv7 processor, chose the most absurd and unrelated reason to start thinking of itself as a 3-year-old ARMv6 device.

Let’s first give a little example of what theoretically happens when you cross-compile a native library for both these architectures and run it on an Android device. Using the NDK, let’s build a simple C library containing this function, cross-compiled for ARMv7 and non-ARMv7 architectures:

int 
Java_com_my_namespace_MyClass_MyNativeFunction
(JNIEnv *env, jobject obj) {
#ifdef __ARM_V7__
  return 1;
#else
  return 0;
#endif
}

This will result in two files: /lib/armeabi-v7a/libfoo.so compiled for ARMv7 devices, and /lib/armeabi/libfoo.so for non-ARMv7 devices. Let’s write the corresponding Java class:

package com.my.namespace;
public class MyClass {
  static {
    System.loadLibrary("foo");
  }
  public native int MyNativeFunction();
}

When you call this class in Java, the System.loadLibrary function will check your device, and decide which of the two libraries it will load and use at runtime. And, as expected, this function will return 1 on any ARMv7 device, and 0 otherwise.

Workaround

That was for theory. Because in practice, and as explained in this thread, ICS developers accidentally let this functionality go rogue on Android 4.0.1 – 4.0.3: when crawling the application’s apk file looking for the right version of the library to use, ICS “forgets” that it found an ARMv7 version and choses the non-ARMv7 version instead! Luckily for us, they provide this quite ugly but useful tip:

“ensure that the armeabi-v7a [i.e. ARMv7] binaries are packaged _after_ the armeabi [i.e. non-ARMv7] ones in the final .apk. This is not trivial, but one way to do it is remove the armeabi-v7a files from the package, then add them back, manually.”

OK, it looks quite annoying, but at least it shows some coherency with the fact that adding resources could mess up with our SDK, as adding files in an archive does not necessarily preserve the files order. We thus started testing this workaround: after all, the Android SDK contains a small tool called aapt made especially to manipulate apk files. Let’s try what is suggested:

$ aapt list MyApp.apk                               //shows the content of the apk, we shorten it to our libs only:
> lib/armeabi/libfoo.so
> lib/armeabi-v7a/libfoo.so

$ jar xf MyApp.apk                                  //extract
$ aapt remove MyApp.apk lib/armeabi-v7a/libfoo.so   //remove the ARMv7 lib
$ aapt add MyApp.apk lib/armeabi-v7a/libfoo.so      //put it back

$ aapt list MyApp.apk                               //check result
> lib/armeabi/libfoo.so
> libfoo.so

See the problem here? The file was added back, but not within the right folder. Let’s have a look at the interesting lines in aapt’s documentation:

aapt a[dd] [-v] file.{zip,jar,apk} file1 [file2 ...]
  Add specified files to Zip-compatible archive.
  [...]
  -k  junk path of file(s) added

As you can see, we did not use the -k option. Believe me or not, but the tool provided by Android is bugged too! Never mind… after all, an apk file is nothing more than a zip file, so why not use the usual zip tool? I’ll spare you the details, but applying the same method using zip and checking the files order using the following small python script, we realized that the suggested workaround simply did not work.

from zipfile import ZipFile,ZIP_DEFLATED,ZIP_STORED
from sys import argv

with ZipFile(argv[1],"r") as x:
  for y in x.infolist():
    if y.compress_type == ZIP_DEFLATED:
      print y.filename + " deflated"
    elif y.compress_type == ZIP_STORED:
      print y.filename + " stored"
    else:
      print y.filename + " WTF"

Working around the workaround

At this point, we had spent hours trying to work around a bug in Android, using a bugged Android tool, with a method that did not work. But all hope was not lost. After hiding under my desk to cry for a good 10 minutes a good coffee, we decided to try a more brutal method – and i don’t mean smashing the phone with a sledgehammer, however tempting it seemed at the moment.

So, as the problem comes from the fact that this System.loadLibrary function cannot be trusted to choose between the ARMv7 and non-ARMv7 libraries, we’ll simply do it ourselves. The problem divided in two main parts:

- System.loadLibrary can’t be trusted to choose between two libraries with the same name, even if they are placed in explicitly named directories. By renaming them to /lib/armeabi/libfoo-core.so and /lib/armeabi-v7a/libfoo-core-v7a.so, we would simply call System.loadLibrary(“foo-core-v7a”) if we were on an ARMv7 architecture, and System.loadLibrary(“foo-core”) otherwise.

- There is no direct way in Java to know if the device you’re using is ARMv7 or not: we had to create another native library that would be in charge of this choice, cross compiled for ARMv7 and non-ARMv7 architectures, and simply named libfoo.so. A good piece of code being worth a thousand words, here is an example of source code for this libfoo.so:

#include <jni.h>
#include <cpu-features.h>

jboolean
Java_com_my_namespace_MyClass_isARMv7
(JNIEnv *env, jclass class) {
  uint64_t features = android_getCpuFeatures();
  if ((android_getCpuFamily() != ANDROID_CPU_FAMILY_ARM) ||
      ((features & ANDROID_CPU_ARM_FEATURE_ARMv7) == 0) ||
      ((features & ANDROID_CPU_ARM_FEATURE_NEON) == 0)) {
    return JNI_FALSE;
  }
  else {
    return JNI_TRUE;
  }
}

And the corresponding Java class simply becomes:

package com.my.namespace;
public class MyClass {

  static {
    System.loadLibrary("foo");
    if (isARMv7()) {
      System.loadLibrary("foo-core-v7a");
    }
    else {
      System.loadLibrary("foo-core");
    }
  }

  public native boolean isARMv7();
  public native int MyNativeFunction();
}

This way, the first loaded library makes a native isARMv7() function available, and this function is used to decide which core library must be loaded immediately after.

In the end, this workaround happens to be far more reliable and viable in the long term. Even though Google promises that the bug will be fixed in the next release of ICS, experience shows that many users don’t update their device, or update them months after the release, which will lead to thousands and thousands of “corrupted” devices that, despite this bug, we want to support. This trick has the advantage of being far more practical and sure than tinkling with our apk files hoping that it magically fixes a bug, and we’ll be able to keep it in place for a long time without concern.

Here is how it would work:

- Moodstocks SDK processes the camera flow in real-time, and at some point matches the current frame with an object that belongs to its database. It passes this information to one of Qualcomm’s SDKs. This step has all the advantages of Moodstocks SDK, as it uses our synchronization service: it makes applications lighter to download than any application using a bundled database, and allows changes in the database without requiring an application update each time a change, however minor it may be, occurs in the database.

- Given that the image has been properly recognized, Qualcomm’s SDK locates this object on the current frame and starts tracking it on the following frames, with the impressive fluidity that we could observe in the demo application, to add to the user experience.

We decided to begin with Vuforia SDK, as it seemed easier of use. Its strength for most developers is that it’s quite easy to use: you add Vuforia SDK to your application, upload all the images you want to recognize and track (via a convenient web interface), and in return you download one single qcar-resource.dat file containing the data necessary to perform these operations on your application. However, this full packaging was the source of several problems in our case:

- the first is that you cannot separate the image recognition among a database from the location and tracking part. A possible trick to separate these two functions would have been to create several of these .dat files, one for each image we wanted to track: as image recognition among a database of only one image is a trivial task, Vuforia SDK would matter of factly do no more that tracking. This could have worked, if Vuforia had not supported a single .dat file that must be named qcar-resource.dat… and juggling with some on the fly file renaming would not have been a really practical solution.

- the second one is that these .dat files are quite heavy, and we wanted to keep the advantages of our synchronization. Using one (or several) of these files would require to manage a second, heavier database than the one used for our image recognition.

In the end, Vuforia appeared to be too high-level and not flexible enough for our needs… all these tricks were far too dirty for us to implement them, so we decided to try our chance with the lower-level FastCV SDK.

We began by searching inspiration in the sample codes and documentation of the SDK. That was quickly done: the two sample codes illustrate hardly a tenth of the functions available in the library, and the documentation of each functions holds in two lines tops. But well, ok, this SDK is still quite young and needs to mature for some time. We went on anyway. After a bit of fumbling through the documentation, FastCV SDK seemed to contain everything we needed to implement the tracking part in itself without too much pain: a ready-to-use tracking function, and an efficient features detector to feed it. Of course, it would require a bit of wrapping to work, but it had the advantage of doing what we needed it to do, nothing more, nothing less. So we had what we needed to track features… Cool. But we couldn’t just track any features on the frame: we first had to locate the object recognized by Moodstocks SDK and extract features to track only from this object. And that’s when things got messy.

We tested several different tactics using the different tools included in FastCV. Our first idea implied using a quite classical method in image matching: pre-process the image to track to extract interesting features, describe them, and search for these features in the camera frames at runtime. As the task of finding a reference image in a camera frame when we do know it’s there (from our own SDK) is quite simple, we hoped to do this using only a few features, which would have had the advantage of allowing to package the lightweight necessary information (aka the descriptors) in a smart way by encoding this information directly within each image identifier. That’s for the theory… but in practice, it appeared that FastCV proposed only one, never-heard-of descriptor. From its concise documentation, it seemed to be some kind of custom, lightweight SURF-like descriptor. We tested it, and realized that it would never be robust and discriminant enough to allow image detection with the small number of reference features we could afford… and we found ourselves once more back to the beginning of our research. We tried a few other methods, before realizing that as we went on, our ideas were becoming more and more heavy and impractical… we just couldn’t find a simple and smart enough way to go through this tracking initialization step using FastCV, and creating a 1000-line-long, barely understandable piece of code would not have served the purpose of illustrating what could be done using both FastCV and our SDK.

So after a few days of work, we simply decided to give up -at least for now- on this project. Let it not be said that we criticize Qualcomm’s SDKs: I would personally encourage developers to use them. Only, none of them was fitted to the specific use we wanted. Vuforia is a perfect SDK for any developer who wants to implement painless augmented reality and doesn’t specifically care about the size or flexibility of their apps. It won’t fail to amaze their users and make their eyes shine. Considering FastCV, I would encourage anyone who work in mobile computer vision to use it as an optimized, efficient toolbox from which to pick elementary functions that will boost your applications. As for us, we’ll keep an eye on the evolution of these two young SDKs and won’t miss the opportunity to resume this project if the opportunity arises.

The answer holds in one word: simplicity. We just didn’t need  90% of the possibilities offered by most of these libraries. No decoding, no color, no such ‘expert’ options as progressive encoding or variable block sizes: we wanted to be able to compress grayscale images, no more, no less. Sure, we could have used one of them anyway, as they all perfectly do their job… but we just didn’t want to burden our SDK with these unused functions. The other option was to use platform-specific dynamic libraries such as Apple’s Image I/O framework, but they are, well… platform-specific. So, we decided to implement our own, customized, vertical encoder. And yes, it was worth the effort:

- the result holds in about 600 LOC,
- compiled for ARMv7 architecture using LLVM-GCC, it weights no more than 12kB,
- it compresses a 360×480 image on an iPhone 4 in 35ms, versus 22ms for Apple’s Image I/O framework… and can probably still be optimized using NEON instruction set.

But let’s go back in time a little, and give a little hindsight on the sneaky little traps that made all the fun of this task.

The JPEG compression could in fact be split into two main steps: a more ‘mathematical’ part that compresses an image into a series of numbers, and an ‘encoding’ part where we store these numbers as efficiently as possible in a file. The mathematical part was far from being the hardest to implement: basically, following Wikipedia was nearly enough to implement it, and even provided some quite efficient tricks such as the use of Chen’s algorithm to compute a Discrete Cosine Transform with a reduced number of operations. The difficulty laid in the encoding part.

Strangely, that’s when Wikipedia (as well as a good part of the other resources on JPEG available on the web) becomes far more blurry on how to proceed. What makes this step tricky is that, to minimize the size of JPEG files, data is stored using Huffman encoding bit after bit, regardless of the usual byte structure. It looks like a good idea (and it is, in the end, a very powerful idea), but it makes debugging your output JPEG files absolutely awful. Some part of the data may take 3 bits and the next one 17, etc… which makes absolutely impossible to track the origin of your bugs.

And then, after a few hours (not among the most pleasant of my life) of bit by bit debugging using an hexadecimal editor, I discovered JPEGsnoop. This small software, whose highest flaw is probably to be available only for Windows, simply did in a handful of seconds what had caused my headaches for hours. It analyzes the structure of your JPEG file, and gives you some great hints on where and why it ended up corrupted. It simply saved my day, and in a matter of hours we had a functional library.

From ‘functional’ to ‘releasable’, there remained a last difficulty: clearly separate the different steps to make all of this understandable… and maintainable. Once the maths, the encoding and the actual bit by bit file writing were not a great soup of mixed functions anymore, we had our minimalistic encoder. And now you can have it too, you lucky ones!

The cool thing is that you don’t even need to download an app to use it. The developer behind the project has made the most of our tutorial about how to develop a Moodstocks – Twitter mashup to be up and running in a breeze. So just tweet the picture of an artwork mentioning @qrtrs to join the fun. It’s that easy!

Some really good talks were given this year. To name a few, I really appreciated:

- the Red programming language talk by Nenad Rakocevic,
- the Ruby / C bindings via FFI presentation by Marc-André Lafortune – that’s a really good alternative to the standard way to extend Ruby with C,
- the very good introduction to the Haskell functional programming language Valentin Robert gave, featuring interesting concepts such as lazy evaluation or pure functions.

But for sure the Lua language definitely made the buzz with no more than 3 talks – a talk covering the basics, another one focused on the really impressive LuaJIT project (aka Just-In-Time compiler for Lua), both by François Perrad, and a last one covering its native integration within Redis (aka scripting-branch) by Pierre Chapuis.

Nothing to say that the ability to extend Redis with Lua scripting is a killer feature! For those who are familiar with Tokyo products it is in some ways pretty similar to what has been done within Tokyo Tyrant.

So let met say congrats to all speakers and participants, thanks to the organization team and see you next year!

The rules are simple: you score each time you scan an ad in magazines or on billboards from the dedicated iPhone and Android apps.

Along with points and badges, you can win a wide range of stuff, from iPads to a real Golf Cabrio. Pretty cool, right?

So far the apps are only available in Belgium. But feel free to ask Volkswagen to expand it to the rest of the world!

We’d like to thank our partners for their awesome integration work: In The Pocket (app design and development) and Tribal DDB Brussel (project leader). You guys rock!

As you can see image recognition has never been faster! A lot of people gave it a try last week during an apero we organized in our loft and they were thrilled. If you’re a developer and want a beta access to the brand new scanner, just let us know!

For example, overlaying information directly on the screen and refreshing it depending on the object you’re pointing to. That’s what we did to “augment” the Citroën Creative Tour, celebrating the 80th birthday of the Yellow Cruise.

And thousands of people loved the experience be it in Paris or Berlin (soon in Moscow and Beijing). For the record, it’s been developed within a couple of days leveraging our image recognition API. Can you believe it?

A lot of you guys wondered how invisible QR codes work. Well, they actually don’t exist! We rely on our state-of-the-art image recognition tech to recognize the book cover itself. The invisible QR Code stuck to it is nothing but a joke!

We’re very sorry if we fooled you. We wanted to emphasize that image recognition works right here right now. It’s now easier than ever to deploy and use it. And thanks to Moodstocks it’s in real-time even on 3G networks:

Do you also want to use image recognition to turn any object into its own QR code? We have everything you need to get yourself up and running in minutes. And if you have any question: feel free to connect. We’re here to help!