CLAM on a phone: Android and Ubuntu phone

After some days of trying stuff on Android exploring the feasibility of porting some CLAM based applications to the mobile phone, I must say I am pretty glad that yesterday Canonical has finally announced their Ubuntu Phone.

It may be just a proposal to the industry, no manufacturer has actually taken the challenge yet, but it is a cool proposal, pretty close of what Meego intended to be. And above all, it would ease many projects I have in mind.

Ubuntu Phone

Surprisingly Canonical has taken the smart move of embracing Qt/QML as the recommended (but not unique) choice for native application development, taking profit of Nokia efforts to make Qt shine on the phone. You can also program using Gtk, HTML5, C++, Python or anything that works on Linux. They also have embraced the Android version of the Linux kernel, a smart move too, because manufacturers can jump Ubuntu with the same drivers they already wrote for Android.

Anyway all could end being vapor-ware if manufacturers do not bet on it. Many people already embraced either Android or iOS and the other ‘new-comer’ Windows 8 got a very cold welcome to the market.

The best aces for Ubuntu Phone are freedom and easy development, but the real criteria for success is more related to what users want, and those aces should be paid back as more applications available, if not, Ubuntu Phone is not getting far beyond what an Android is, and without a good reason for the switch, it won’t happen. I really hope it will, because it would ease the accomplishment of what I want…

CLAM on a phone? Really?

Why not? Everyday, I see more and more appealing the idea of having, for instance, a chord analyzer on the phone that you can bring to a gig, or even better, a voice morphing program to troll your friends, or even being able to prototype applications in your mobile that do complex audio stuff.

ipyclam has shown that by having the building blocks pre-compiled, you can build rich graphical application doing complex audio processing, with no compilation to native code by using Python, just like a QML application does.

So, yes, CLAM in a phone seems a good idea… once you have it. It seems quite straight forward in one of those Ubuntu Phones, which are essentially Ubuntu Linux with an ARM processor. So using CLAM should be a matter of apt-get installing packages like that. But while Ubuntu Phones get the market, and get it, we have to stick to something that already exists like Android.

Exploring Android option

I was exploring several issues:

  • Native C++ support, so that we can compile CLAM and its dependencies.
  • Qt support, so that we can reuse apps and architecture.
  • An audio backend
  • Python support so that we can use the new Python prototyping features

In summary, I found all quite frustrating, but sure, this is because my extreme ignorance regarding Android platform. And really I must be missing one big point because I can not believe is that hard. So, I will be glad if someone can highlight me with some insight of what I did not see.

Necessitas/Ministro, Qt on Android

Necessitas is an environment based on QtCreator that builds Qt applications for Android and deploys them on any emulator or device you have connected. It includes:

  • the Android SDK (Java Libraries, emulator, device tools…),
  • the Android NDK (C++ cross-compilers and libraries),
  • a version of QtCreator with extensions to develop for Android (cross-compile, deploy…),
  • wrappers to bridge between Java and Qt/C++

Qt applications are wrapped with a Java wrapper that loads the required libraries and calls the C++ entry points when some events are triggered. So, for Android that’s a regular Java program calling some big C++ library containing your whole program.

Instead of bundling Qt libraries with your application, the stub uses a service called Ministro that must be installed in your device. The stub, ask Ministro for the required Qt libraries, and Ministro installs or updates the required libraries that are shared among any Qt application..

BogDan Vatra and the rest of the people working on Necessitas did a quite nice job hiding all the complexity of Android development under QtCreator interface. Still I found many edges.

First of all, it was hard for me to figure out how to setup the target image for the emulator. One of the reasons is the messing version scheme in Android. On one side, each Android release has a dotted version number (2.3.4, 3,2, 4,2…), some of those releases are grouped under a generic commercial code name, corresponding to alphabetically ordered sweet stuff (Ginger Bread, Honeycomb, Icecream Sandwich, Jelly Bean…). As programmers, we are interested in the API level. The API level is an increasing number which changes whenever the programming interface changes. The rule of thumb is that anything you develop for a given API level should work on devices supporting a higher API level. Then Android SDK and NDK have their own versions. And then you have many architectures ARMv5, ARMv7, MIPS, Intel… Well, crazy.

I had problems running the emulator with a relatively modern version of Android. Necessitas installer does not install images for API levels beyond 10. That’s because beyond that version, images are not included in the SDK package anymore, but in a different one which is not selectable from Necessitas package updater. So instead of using Necessitas package manager you have to use the one that comes with the SDK at android-sdk/tools/android and install the ARM image from the Android version you want to target.

Another problem I found is that whenever I start the emulator from QtCreator, QtCreator freezes. I recommend using the AVD Manager for starting the emulator and once it is up and running, just let QtCreator to deploy in the already started emulator. That is going to Tools/Options menu, Android section, and clicking on the Start Android AVD Manager button which launches the AVD Manager, and doing there all the creation and start operations instead of using the QtCreator dialog.

Ministro is available on the Android Market but the Market is not installed on the emulator, so you need to download the APK for Ministro and install it with:

$ /path/to/sdk/tools/adb install /path/to/Ministro\ II.apk

At the end of the day I was able to run pure Qt applications, but the ones using OpenGl. Some of them did not compile because they used full OpenGL and not OpenGL-ES. The ones that compiled did not work on the emulator, because problems with the OpenGL.

With widget based interfaces I found some style problems, such as light text color by default when the background is light as well. Someone (Necessitas, Qt or Android) broke the law of setting the foreground whenever your style changes de background, as you never know which are the inherited colors.

External CLAM dependencies

Most NDK documentation just discourages you from using NDK to anything else than calling a C/C++ function from Java. But INDEED is what I want to do. Not helping! Then they offer you several cut and paste snippets without explaining what they do. Not helping either. How do I cross-compile a library? Which is the convention for deploying them? Is there any repository of compiled libraries?

Well, I did not find anything like that. So, if you want to help, please point me to the right direction.

I do not know whether we can use Ministro to extend the library gathering to other libraries than Qt. But it would be a cool solution because we could choose

Audio on Android

If we want to port CLAM into Android you should expect some way of accessing audio streams in real-time. Real-time audio in Android had serious limitations until recently because the original API was thought with event triggered samples and multimedia streaming (low latency) in mind. It seems that the situation has been improved with the introduction of the OpenSL ES API. But music oriented developers still complain.

Here, we could:

  • Reuse any of our current back-ends when they get ported to Android
  • Build a native Android back-end.

The status of our back-ends in Android is quite deceiving:

So the better option is either wait for PortAudio or build a simple OpenSL ES back-end for CLAM.

Another issue I would like to explore is intercepting audio between the mic and speaker and voice call line. This would be cool to implement call recording, automated answering, impersonation, speech to text for deaf people… By the threads I have been reading I deduce that Android has API to access the voice call but most integrators do not provide drivers for doing such thing.

Python/PySide on Android

Having Python and PySide ported to Android would enable for the mobile, not only ipyclam but also several projects I have been working on.

Lucky for me, there some projects around porting Python (Py2A, SL4A) and PySide (PySide for Android) to Android and there are some prebuilt binaries! Unlucky for me, none of them work out the box. I could install Py2A but no idea on how to run Python once installed. PySide looks for Ministro and does not find it even if I Installed it as service. The possible solution is to recompile it all. Of course, it does not compile as is with the instructions provided, so it requires more work.

..so

…I had to left that here. I guess that at this point you understand why i was so glad Ubuntu Phone was announced. I mean I could recompile the whole thing, hoping no problem arises and all gets working. But that never happens when you are cross compiling things and I have a PhD Thesis to write. In summary, a quite frustrating spike. I would be glad if anyone wanting to hack on it and providing light is welcome.

I hope that the next time I move my head up from my thesis either someone has make any progress on any of those points or Ubuntu Phones are ready to be sold. If not, well I would try it harder.

Comments

0 comments.

Leave a comment

 (optional)
 (optional)