Karaf

3 posts

Talking to the cloud

While working on Eclipse Kapua, I wanted to do different tests, pushing telemetry data into the system. So I started to work on the Kura simulator, which can used to simulator an Eclipse Kura IoT gateway in a plain Java project, no special setup required. Now that helped a lot for unit testing and scale testing. Even generating a few simple telemetry data streams for simulating data works out of the box.

But then again I wanted to have something more lightweight and controllable. With the simulator you actually derive some a simple class and get fully controlled by the simulator framework. That may work well in some cases, but in others you may want to turn over the control to the actual application. Assume you already have a component which is “in charge” of your data, and now you want to push this into the cloud. Of course you can do this somehow, working around that. But creating a nice API for that, which is simple and easy to understand is way more fun 😉

So here is my take on a Gateway Client API, sending IoT data to the cloud, consuming command & control from it.

Intentions

I wanted to have a simple API, easy to understand, readable. Preventing you from making mistakes in the first place. And if something goes wrong, it should go wrong right away. Currently we go with MQTT, but there would be an option to go with HTTP as well, or AMQP in the future. And also for MQTT we have Eclipse Paho and FUSE MQTT. Both should be available, both may have special properties, but share some common ground. So implementing new providers should be possible, while sharing code should be easy as well.

Example

Now here is with what I came up with:

try (Client client = KuraMqttProfile.newProfile(FuseClient.Builder::new)
  .accountName("kapua-sys")
  .clientId("foo-bar-1")
  .brokerUrl("tcp://localhost:1883")
  .credentials(userAndPassword("kapua-broker", "kapua-password"))
  .build()) {

  try (Application application = client.buildApplication("app1").build()) {

    // subscribe to a topic

    application.data(Topic.of("my", "receiver")).subscribe(message -> {
      System.out.format("Received: %s%n", message);
    });

    // cache sender instance

    Sender sender = application
      .data(Topic.of("my", "sender"))
      .errors(ignore());

    int i = 0;
    while (true) {
      // send
      sender.send(Payload.of("counter", i++));
      Thread.sleep(1000);
    }
  }
}

Looks pretty simple right? On the background the MQTT connection is managed, payload gets encoded, birth certificates get exchanges and subscriptions get managed. But still the main application is in control of the data flow.

How to do this at home

If you want to have a look at the code, it is available on GitHub (ctron/kapua-gateway-client) and ready to consume on Maven Central (de.dentrassi.kapua). But please be aware of the fact that this is a proof-of-concept, and may never become more than that.

Simply adding the following dependency to your project should be enough:

<dependency>
  <groupId>de.dentrassi.kapua</groupId>
  <artifactId>kapua-gateway-client-provider-mqtt-fuse</artifactId>
  <version>0.2.0</version> <!-- check for a more recent version -->
</dependency>

With this dependency you can use the example above. If you want to got for Paho instead of FUSE use kapua-gateway-client-provider-mqtt-paho instead.

Taking for a test drive

Now taking this for a test drive as even more fun. Eclipse SmartHome has the concept of a persistence system, where telemetry data gets stored in a time series like database. There exists a default implementation for rrdb4j. So re-implementing this interface for Kapua was quite easy and resulted in an example module which can be installed into the Karaf based OpenHAB 2 distribution with just a few commands:

openhab> repo-add mvn:de.dentrassi.kapua/karaf/0.2.0/xml/features
openhab> feature:install eclipse-smarthome-kapua-persistence

Then you need to re-configure the component over the “Paper UI” and point it towards your Kapua setup. Maybe you will need to tweak the “kapua.persist” file in order to define what gets persisted and when. And if everything goes well, your temperate readings will get pushed from SmartHome to Kapua.

More information

Remote managing Eclipse Kura on Apache Karaf with ECF

To be honest, I had my troubles in the past with the Eclipse Communication Framework (ECF), not that it is a bad framework, but whatever I started it was complicated and never really worked for me. This story is different!

A few months back the Eclipse Kura project ran into an issue that the plugin which was being used for remote managing a Kura instance (mToolkit) from an IDE just kind of went away (issue #496). There is some workaround for that now, but still the problems around mToolkit still exists. Beside the fact that it is no longer maintained, it is also rather buggy. Deploying a single bundle takes about a minute for me. Of course using the Apache File Install package for Kura would also help here 😉

But having a decent IDE integration would also be awesome. So when Scott Lewis from the ECF project contacted me about that, I was ready to give it a try. Unfortunately the whole setup required more than Kura could handle at that time. But now we do have support for Java 8 in Kura and there also is some basic support for running Kura on Karaf, including a docker image with the Kura emulator running on Karaf.

So I asked Scott for some help in getting this up and running and the set of instructions was rather short. In the following examples I am assuming your are running RHEL 7, forgive me if you are not 😉

First we need to spin up a new Kura emulator instance:

sudo docker run -ti --net=host ctron/kura:karaf-stable

We are mapping all network to the host instance, since we are using another port, which is not configured in the upstream Dockerfile. There is probably another way, but this is just a quick example.

Then, inside the Karaf instance install ECF. We configure it first to use “ecftcp” instead of MQTT. But of course you can also got with MQTT or some other adapter ECF provides:

property -p service.exported.configs ecf.generic.server
property -p ecf.generic.server.id ecftcp://localhost:3289/server

feature:repo-add http://download.eclipse.org/rt/ecf/kura.20161206/karaf4-features.xml
feature:install -v ecf-kura-karaf-bundlemgr

Now Kura is read to go. Following up in the Eclipse IDE, you will need Neon running on Java 8:

Add the ECF 3.13.3 P2 repository using http://download.eclipse.org/rt/ecf/3.13.3/site.p2 and install the following components:

  • ECF Remote Services SDK
  • ECF SDK for Eclipse

Next install the preview components for managing Karaf with ECF. Please note, those components are previews and may or may not be release at some point in the future. Add the following P2 repository: http://download.eclipse.org/rt/ecf/kura.20161206/eclipseui and install the following components (disable Group Items by Category):

  • Remote Management API
  • Remote Management Eclipse Consumer

Now comes the fiddly part, this UI is a work in progress, and you have been warned, but it works:

  • Switch to the Remote Services perspective
  • Open a new view: Window -> Show View -> Other… – Select Remote OSGi Bundles
  • Click one of the green + symbols (choose either MQTT or ECFTCP) and enter the address of your Karaf instance (localhost and 3289 for me)

You should already see some information about that target device now. But when you open a new view (as before) named Karaf Features you will also have the ability to tinker around with the Karaf installation.

If you just want to have a quick look, here it is:

ECF connecting to Kura on Karaf
ECF connecting to Kura on Karaf

Of course you don’t need to use an IDE for managing Karaf. But having such an integration as an option, is a nice addition. And it shows how powerful a great OSGi setup can be 😉

Eclipse Kura on Apache Karaf

It took quite a while and triggered a bunch of pull request upstream, but finally I do have Eclipse Kura™ running with Apache Karaf™.

Now of course I immediately got the question: Why is that good? After all the recent setup still uses Equinox underneath in Karaf. Now what? … isn’t Equinox and Karaf something different? Read on …

What is Karaf?

Ok, back to the basics. Eclipse Equinox™ is an implementation of OSGi. As is Apache Felix™. OSGi is a highly modular Java framework for creating modular applications. And it eats its own dogfood. So there is a thin layer of OSGi (the framework) and a bunch of “add-on” functionality (services) which is part of OSGi, but at the same time running on top of that thin OSGi layer. Now Equinox and Felix both provide this thin OSGi layer as well a set of “standard” OSGi services. But it is possible to interchange service implementations from Equinox to Felix, since both use OSGi. For example in Package Drone I started to use Equinox as OSGi container, but in some release switched to the Apache Felix ConfigurationAdmin since it didn’t have the bugs of the Equinox implementation. So the final target is a plain Equinox installation with some parts from Apache Felix.

Apache Karaf can be seen as a distribution of OSGi framework and services. Like you have a Linux distribution starting with a kernel and a set of packages which work well together, Karaf is a combination of an OSGi framework (either Felix or Equinox or one of others) and a set of add-on functionality which is tested against each other.

And those add-on services are the salt in the OSGi soup. You can have a command line shell with completion support, JMX support, OSGi blueprint, webserver and websocket suport, JAAS, SSH, JPA and way more. Installing bundles, features or complete archives directly from local sources, HTTP or Maven central. It wraps some non-OSGi dependencies automatically into OSGi bundles. And many 3rd party components are already taking advantage of this. So for example installing Apache Camel™ into an Karaf contain is done with two lines on the Karaf shell.

What Kura currently does

Eclipse Kura currently assembles its own distribution based on pure and simple Equinox components using a big Ant file. While this has an advantage when it comes to distribution size and the fact that you can control each an every aspect of that distribution, it also comes at a prize that you need to do exactly that. That Ant file needs to create each startup configuration file, knowing about each and every bundle version.

Karaf on the other hand already has a ready to run distribution in various flavors. And has some tooling to create new ones.

What is different now

In order to bring Kura to Karaf there were a few changes necessary in Kura. In addition to that a new set of Maven projects assembly a target distribution. Right now it is not possible to simply drop Kura into Karaf as a Karaf feature. A special Karaf assembly has to be prepared which can then be uploaded and installed at a specific location. This is mostly necessary due to the fact that Kura expects files at a specific location in the file system. Karaf on the other hand can simple be downloaded, unzipped and started at any location.

In addition to that a few bugs in Kura had to be fixed. Kura did make a few assumption on how Equinox and its service implementations handle things which may no longer be true in other OSGi environments. these fixes will probably make it into Kura 2.1.

Can I test it today?

If you just want to start up a stable build (not a release) you can easily do this with Docker:


sudo docker run -ti ctron/kura:karaf-stable # stable milestone
sudo docker run -ti ctron/kura:karaf-develop # latest develop build

This will drop you in a Karaf shell of Kura running inside docker. You can play around with the console, install some additional functionality or try out an example like OPC UA with Apache Camel.

You can also check out the branch feature/karaf in my fork of Kura which contains all the upstream patches and the Karaf distribution projects in one branch. This branch is also used to generate the docker images.

What is next?

Clean up the target assembly: Right now assembling the target platform for Karaf is not as simple as I had hoped. This is mostly due to the complexity of such a setup (mixing different variants of Linux, processors and devices) and the way Maven works. With a bit of work this can made even simpler by adding a specific Maven plugin and make one or two fixes in the Karaf Maven plugin.

Enable Network management: Right now I disabled network management for the two bare-metal Linux targets. Simply because I did have a few issues getting this running on RHEL 7 and Fedora. Things like ethernet devices which are no longer named “eth0” and stuff like that.

Bring Kura to Maven Central/More Kura bundles: These two things go hand in hand. Of course I could simply create more Karaf features, packaging more Kura bundles. But it would be much better to have Kura artifacts available on Maven Central, being able to consume them directly with Karaf. That way it would be possible to either just drop in new functionality or to create a custom Kura+Karaf distribution based on existing modules.

Will it become the default for Kura?

Hard to say. Hopefully in the future. My plan is to bring it into Kura in the 2.2 release cycle. But this also means that quite a set of dependencies has to go through the Eclipse CQ process I we want to provide Karaf ready distributions for download. A first step could be to just provide the recipes for creating your own Karaf distribution of Kura.

So the next step is to bring support for Karaf into Kura upstream.