Developing for Eclipse Kura on Windows

Every now and then it is fun to leave the environment you are used to and do something completely different. So this journey take me to IntelliJ and Windows 10. And yes, I am glad to be back in Linux/Eclipse-land. But still, I think something rather interesting came out of this.

It all started when I helped my colleague Aurélien Pupier to get his environment ready for his talk at the Eclipse IoT day Grenoble. If you missed this talk, you can watch a recording of it. He wanted to present the Camel Developer Tools. The problem was the he was working on a Windows laptop. And he wanted to demonstrate Eclipse Kura in combination JBoss Tools IDE. However Kura can only run on Linux and he wanted to run the JBoss Tools native on his Windows machine.

Of course you could come up with some sort of Virtual Machine setup, but we wanted something which was easier to re-produce in the case there would be some issue with the laptop for the presentation.

Creating a docker image of Kura

The first step was to create a docker image of Kura. Currently Kura doesn’t offer any support for Docker. So that had to be created from scratch. As there is even no x86_64 distribution of Kura and no emulator distribution, it was necessary to do some rather unusual hacks. The background is, that Kura has a rather crude build system which assembles a few distributions in the end of the build. Kura also requires some hardware interfaces in order to work properly. For those hardware interfaces there exist emulator replacements for using in a local developer setup. However there is neither a distribution assembly for x86_64 nor one using the emulator replacements. The whole build system in the end is focused around creating Linux-only images. The solution was to simply rip out all functionality which was in the way and create a patch file.

This patch file and the docker build instructions are now located at a different repository where I can easily modify those and hook it up to the DockerHub build system: ctron/kura-emulator. Currently there are three tags for docker images in the Kura Emulator DockerHub repository: latest (which is the most recent, but stable release), 3.0.0-RC1 and develop (most recent, less stable). As there is currently no released version of Kura 3.0.0, the latest tag is also using the develop branch of Kura. The 3.0.0-RC1 tag is a stable version of the emulator which is known to work and won’t be updated in the future.

There is a more detailed README file in the GitHub repository which explains how to use and build the emulator yourself. In a nutshell you can start it with:

docker run -ti -p 8080:8080 ctron/kura-emulator

And afterwards you can navigate with your browser to http://localhost:8080 and use the Kura Web UI.

As Docker is also available for Windows, this will work the same way on either Linux or Windows, and although I didn’t test it, it should also work on Mac OS X.

JMX & Debugging

As the Camel tooling makes use of JMX, it was necessary to also enable JMX support for Kura, which normally is not available with Kura. By setting the JAVA_OPTS environment variable it is not only possible to enable JMX, but also to enable plain Java debugging for the docker image. Of course you will need to publish the selected ports with -p when running the docker image. And for Windows you cannot simply use localhost but you will need to use the IP addresses created by docker for windows: also see README.md.

Drop in & activate

After the conference was over, I started to think about what we actually had achieved by doing all this. We had a read-to-run Kura image, dockerized, capable of running of Windows (with docker), debuggable. The only part which was still missing was the ability to add a new, custom bundle to the emulator.

Apache Felix File Install to the rescue! In the past I created an Apache Felix File Install DP for Kura (DP = deployment package for Kura). File Install works in a way that it monitors a directory and automatically loads, unloads and updates an OSGi JAR file which you drop into this directory. The DP can simply be dropped into Kura, which extends Kura with this File Install functionality.

So I pre-seeded the Kura docker image with the File Install DP and declared a volume mount, so that you can simply mount a path of the docker image onto your host system. Dropping a file into the directory on the host system will make it available to the docker container and File Install will automatically pick it up and start it, but inside the docker container.

docker run -ti -p 8080:8080 -v c:/path/to/bundles:/opt/eclipse/kura/load ctron/kura-emulator

And this even works with Docker for Windows, if you share your drive first:

Share drive with Docker

Choose your tools

Currently Kura requires you to use a rather complicated setup for developing applications for Kura. You will need to learn about Eclipse PDE, target platforms, Tycho for Maven and bunch of other things to get your Kura application developed, built and packaged.

I already created a GitHub repository for showing a different way to develop Kura applications: ctron/kura-examples. Those project use plain maven, the maven-bundle-plugin and my osgi-dp plugin to create the final DP. Those examples also make use of the newer OSGi annotations instead of requiring your to craft all OSGi metadata by hand.

So if you wanted, you could already use your favorite IDE and start developing Kura application with style. But in order to run them, you still needed a Kura device. But with this docker image you can now simply let the emulator run and let File Install pick up the compiled results:

Summary

So yes, it is possible to use IntelliJ on Windows to develop and debug your Kura application, in a stylish fashion. Or you can simply do the same, just using an excellent IDE like Eclipse and an awesome operating system like Linux, with the same stylish approach 😉

IEC 60870-5-104 with Apache Camel

Yesterday the release 0.4.0 of Eclipse NeoSCADA™ was made available. This release features a cool new feature, an IEC 60870-5-104 stack, written in Java, licensed under the EPL and available on Maven Central. See also the Eclipse Wiki: https://wiki.eclipse.org/EclipseNeoSCADA/Components/IEC60870

So it was time to update my Apache Camel component for IEC 60870 and finally release it to Maven Central with proper dependencies on Eclipse NeoSCADA 0.4.0.

For more information about the see my page about the IEC 60870 Apache Camel component.

In a nutshell you can install it with the following commands into a running Karaf container and start using it with Apache Camel:

feature:repo-add mvn:org.apache.camel.karaf/apache-camel/2.18.0/xml/features
feature:repo-add mvn:de.dentrassi.camel.iec60870/feature/0.1.1/xml/features
feature:install camel-iec60870

But of course it can also be used outside of OSGi. In a standalone Java application or in the various other ways you can use Apache Camel.

Testing Kapua with simulated Kura gateways

Now you got your pretty new OpenShift setup of Eclipse Kapua and want to give your IoT cloud a test run?! Testing it out with 100 devices, just for fun? Or even more? But you are too lazy to flash 1000 SD cards for your Raspberry Pi cluster? Here comes the Kura simulator framework. 😉

In order to provide some automatic testing for Kapua I started working on a simulator framework which does simulate Kura instances completely in Java. No backend needed, no hardware needed, able to run multiple instances in a single JVM. And all hosted on GitHub at ctron/kura-simulator.

A screenshot of Kura simulator instances in Kapua
Kura simulator instances in Kapua

The basic idea was to create a set of classes which can be used in automated unit tests in order to simulate a Kura gateway, but allow for a finer grained control over it for testing the good, the bad and the ugly. A real Kura instance would of course be a more realistic test partner, but then again this would have quite a few drawbacks. First of all, Kura cannot be embedded into a unit or integration test. It has far too many dependencies to directory structures, command line utilities, native libraries and it would also require an OSGi container to be started. Second, Kura would always behave like Kura. Now for some tests this may be fine, but if you want to test corner cases where the gateway responds in a way which is not expected by Kapua, then this cannot be done with Kura.

So running a single Kura simulator can be as easy as:

ScheduledExecutorService downloadExecutor = 
   Executors.newSingleThreadScheduledExecutor(new NameThreadFactory("DownloadSimulator"));

GatewayConfiguration configuration =
   new GatewayConfiguration("tcp://kapua-broker:kapua-password@localhost:1883", "kapua-sys", "sim-1");

Set<Application> apps = new HashSet<>();
apps.add(new SimpleCommandApplication(s -> String.format("Command '%s' not found", s)));
apps.add(AnnotatedApplication.build(new SimpleDeployApplication(downloadExecutor)));

try (MqttSimulatorTransport transport = new MqttSimulatorTransport(configuration);
     Simulator simulator = new Simulator(configuration, transport, apps);) {
    Thread.sleep(Long.MAX_VALUE);
    logger.info("Bye bye...");
} finally {
  downloadExecutor.shutdown();
}

Of course, scaling this up and running a few more instances of this isn’t a big deal either. Running this in a docker container and scaling this up even more with OpenShift works fine as well. So testing any number of Gateways just became a lot easier.

Currently the simulator can emulate the command service (V1) and most of the deploy service (V2). The configuration service is still missing, but should get implemented in the next few days. Of course it is also possible to register a custom application and provide some metrics yourself.

Camel and IEC 60870-5-104

With the upcoming release 0.4.0 of Eclipse NeoSCADA™, the IEC 60870-5-105 implementation will finally make its way back into NeoSCADA. This will allow me to finally release the IEC 60870 component Apache Camel to Maven Central.

The Camel components for IEC 60870 are based on the NeoSCADA implementation and provide both client and server side of the protocol. Although the implementation of IEC 60870 does not implement all message types defined, all relevant types for data transmission and control are implementation and other modules can be added by an extensible mechanism, using the core layers of the protocol.

For Camel there are two endpoint types iec60870-server and iec60870-client. These allow either to offer data as IEC 60870 or to actively request data from another 60870 server.

The client component will open a connection to the remote station and initiate the data transmission. 60780 will then send updates for all addresses but the Camel component will only forward events to connected endpoints. When the connection breaks, it will be periodically tried to re-establish the connection. All event coming from the IEC connection can of course be processed with Camel.

For the server side the Camel component will hold an intern data model which can be filled using the Camel routes. In internal state will then be published to IEC clients connecting to the server instance. It also allows the use of background transmission or batching of events when required.

Now what can you actually do with IEC 60870 and Apache Camel? Well, to be honest, if you never have heard about IEC 60870 and don’t have a proper use case or specific requirement for it, then you should probably look for something different to play with 😉 IEC 60870 is used to remotely control and monitor electrical systems and power control systems (see Wikipedia page about IEC 60870-5). On the other hand, if you do want to use 60870, then the Apache Camel component can make it pretty easy to provide a data over the IEC protocol or get data out of an 60870 based system.

As routing data with Camel is easy, you can for example create a very simple Mock device in a Raspberry Pi for testing your system with an IEC component. And you can do all of this with pure open source (EPL licensed) software. You can also extract data out of your application and offer it towards another system, which explicitly requires a transmission based on IEC 60870.

When the component will be released on the next few weeks I will hopefully find the time to provide some example, showing what you can do with IEC 60870 and Apache Camel.

Released version 0.1.0 of OPC UA component for Camel

After Eclipse Milo™ 0.1.0 was released a few days back and is available on Maven Central since this week it was time to update my OPC UA component for Apache Camel to use the release version of Milo:

This means that there is now a released version of, available on Maven Central as well, of the Apache Camel Milo component which can either be used standalone or dropped in directly to some OSGi container like Apache Karaf.

The basics

The component is available from Maven Central under the group ID de.dentrassi.camel.milo and the source code is available on GitHub: ctron/de.dentrassi.camel.milo

For more details also see: Apache Camel component for OPC UA

If you want to use is as a dependency use:


  de.dentrassi.camel.milo
  camel-milo
  0.1.0

Or for the Apache Karaf feature:

mvn:de.dentrassi.camel.milo/feature/0.1.0/xml/features

Plain Java

If you want to have a quick example you can clone the GitHub repository and simply compile and run an example using the following commands:

git clone https://github.com/ctron/de.dentrassi.camel.milo
cd de.dentrassi.camel.milo/examples/milo-example1
mvn camel:run

This will compile and run a simple example which transfers all temperate measurements from the iot.eclipse.org MQTT server from the topic javaonedemo/eclipse-greenhouse-9home/sensors/temperature to the OPC UA tag item-GreenHouse.Temperature, namespace urn:org:apache:camel on the connection opc.tcp://localhost:12685.

The project is a simple OSGi Blueprint bundle which can be also be run by Apache Camel directly. The only configuration is the blueprint file:

<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0">

    <bean id="milo-server" class="org.apache.camel.component.milo.server.MiloServerComponent">
        <property name="enableAnonymousAuthentication" value="true"/>
    </bean>

    <camelContext xmlns="http://camel.apache.org/schema/blueprint">
      <route id="milo1">
        <from uri="paho:javaonedemo/eclipse-greenhouse-9home/sensors/temperature?brokerUrl=tcp://iot.eclipse.org:1883"/>
        <convertBodyTo type="java.lang.String"/>
        <log message="iot.eclipse.org - temperature: ${body}"/>
        <to uri="milo-server:GreenHouse.Temperature"/>
      </route>
    </camelContext>

</blueprint>

This configures a Camel Milo server component and routes the data from MQTT to OPC UA.

Apache Karaf

If you compile the previous example using:

mvn package

You can download and start an Apache Karaf instance, add the Camel Milo component as a feature and deploy the bundle:

feature:repo-add mvn:de.dentrassi.camel.milo/feature/0.1.0/xml/features
feature:repo-add mvn:org.apache.camel.karaf/apache-camel/2.18.0/xml/features
feature:install aries-blueprint shell-compat camel camel-blueprint camel-paho camel-milo

The next step will download and install the example bundle. If you did compile this yourself, then use the
path of your locally compiled JAR. Otherwise you can also use a pre-compiled example bundle:

bundle:install -s https://dentrassi.de/download/camel-milo/milo-example1-0.1.0-SNAPSHOT.jar

To check if it works you can cannot using an OPC UA client or peek into the log file of Karaf:

karaf> log:tail
2017-01-11 15:11:45,348 | INFO  | -930541343163004 | milo1  | 146 - org.apache.camel.camel-core - 2.18.0 | iot.eclipse.org - temperature: 21.19
2017-01-11 15:11:45,958 | INFO  | -930541343163004 | milo1  | 146 - org.apache.camel.camel-core - 2.18.0 | iot.eclipse.org - temperature: 21.09
2017-01-11 15:11:49,648 | INFO  | -930541343163004 | milo1  | 146 - org.apache.camel.camel-core - 2.18.0 | iot.eclipse.org - temperature: 21.19

FUSE tooling

If you want some more IDE integration you can quickly install the JBoss FUSE tooling and connect via JMX to either the Maven controlled instance (mvn camel:run) or the Karaf instance and monitor, debug and trace the active Camel routes:

FUSE tooling with Milo
FUSE tooling with Milo

What is next?

For one this component will hopefully become part of Apache Camel itself. And of course there is always something to improve 😉

I also did update the Kura Addon for Milo, which provides the Milo Camel component for Eclipse Kura 2.1.0 which was recently released. This component is now also available on Maven Central and can easily be deployed into Kura. See the Kura Addons page for more information.

Then there are a few location where I used SNAPSHOT versions of Milo and for some I did promise an update. So I will try to update as many locations as I can with links to the released version of those components.

Remote managing Eclipse Kura on Apache Karaf with ECF

To be honest, I had my troubles in the past with the Eclipse Communication Framework (ECF), not that it is a bad framework, but whatever I started it was complicated and never really worked for me. This story is different!

A few months back the Eclipse Kura project ran into an issue that the plugin which was being used for remote managing a Kura instance (mToolkit) from an IDE just kind of went away (issue #496). There is some workaround for that now, but still the problems around mToolkit still exists. Beside the fact that it is no longer maintained, it is also rather buggy. Deploying a single bundle takes about a minute for me. Of course using the Apache File Install package for Kura would also help here 😉

But having a decent IDE integration would also be awesome. So when Scott Lewis from the ECF project contacted me about that, I was ready to give it a try. Unfortunately the whole setup required more than Kura could handle at that time. But now we do have support for Java 8 in Kura and there also is some basic support for running Kura on Karaf, including a docker image with the Kura emulator running on Karaf.

So I asked Scott for some help in getting this up and running and the set of instructions was rather short. In the following examples I am assuming your are running RHEL 7, forgive me if you are not 😉

First we need to spin up a new Kura emulator instance:

sudo docker run -ti --net=host ctron/kura:karaf-stable

We are mapping all network to the host instance, since we are using another port, which is not configured in the upstream Dockerfile. There is probably another way, but this is just a quick example.

Then, inside the Karaf instance install ECF. We configure it first to use “ecftcp” instead of MQTT. But of course you can also got with MQTT or some other adapter ECF provides:

property -p service.exported.configs ecf.generic.server
property -p ecf.generic.server.id ecftcp://localhost:3289/server

feature:repo-add http://download.eclipse.org/rt/ecf/kura.20161206/karaf4-features.xml
feature:install -v ecf-kura-karaf-bundlemgr

Now Kura is read to go. Following up in the Eclipse IDE, you will need Neon running on Java 8:

Add the ECF 3.13.3 P2 repository using http://download.eclipse.org/rt/ecf/3.13.3/site.p2 and install the following components:

  • ECF Remote Services SDK
  • ECF SDK for Eclipse

Next install the preview components for managing Karaf with ECF. Please note, those components are previews and may or may not be release at some point in the future. Add the following P2 repository: http://download.eclipse.org/rt/ecf/kura.20161206/eclipseui and install the following components (disable Group Items by Category):

  • Remote Management API
  • Remote Management Eclipse Consumer

Now comes the fiddly part, this UI is a work in progress, and you have been warned, but it works:

  • Switch to the Remote Services perspective
  • Open a new view: Window -> Show View -> Other… – Select Remote OSGi Bundles
  • Click one of the green + symbols (choose either MQTT or ECFTCP) and enter the address of your Karaf instance (localhost and 3289 for me)

You should already see some information about that target device now. But when you open a new view (as before) named Karaf Features you will also have the ability to tinker around with the Karaf installation.

If you just want to have a quick look, here it is:

ECF connecting to Kura on Karaf
ECF connecting to Kura on Karaf

Of course you don’t need to use an IDE for managing Karaf. But having such an integration as an option, is a nice addition. And it shows how powerful a great OSGi setup can be 😉

Apache Camel Logo

Providing telemetry data with OPC UA on Eclipse Kura

The upcoming version 2.1.0 of Eclipse Kura™ will feature an enhanced version of the Apache Camel™ integration which was introduced in Kura 2.0.0. There are various new ways on how to run Camel routes, configured either by XML routes or using the Java DSL. Apache Camel can act as a Kura application but, new in this release, there is also a way to simply configure Camel as a “cloud service”. In past releases of Kura, applications could only push data to one cloud target. The new 2.1.0 release will add the functionality of adding multiple cloud targets and one of those targets can be Apache Camel router instances.

With Camel you can have different ways of achieving this goal, but in this post I would like to focus on the “out of the box” way, by simply configuring (not developing) a set of Camel routes, which act as cloud service. Traditional instances of cloud services in Kura are only capable of delivering data to one cloud target or subscribing to one cloud infrastructure. But using Apache Camel as a technology it is possible to connect to a bunch of technologies at the same time.

The setup

The setup will be a Kura instance, running a pre-release version of Kura 2.1.0. The final version should be out in a few weeks and won’t differ much from the current version. We will be configuring a new cloud service instance which takes Kura application payload data and provide it as OPC UA, using the Camel OPC UA adapter. As payload provider (aka Kura application) we will be using the “Example publisher” from my Kura addons project.

Open up the Kura Web UI, navigate to “Packages” and select “Install/Update”. Switch to “URL” and provide the following URL:

https://dentrassi.de/download/kura/de.dentrassi.kura.addons.example.publisher_0.1.0-SNAPSHOT.dp

Note: As an alternative you can also download the “dp” package with your desktop browser and deploy the file using the “file” upload instead of “URL”.

Adding packages to Kura
Adding packages to Kura

The installation may take a bit and it may be necessary to press the “Refresh” button in order to see the installed package. After the packages was installed you should be able to see the service “Camel example publisher” on the left side.

Now we need to install the “Milo component for Camel”. Press “Install/Update” again and enter the following URL:

http://central.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.milo/0.2.2/de.dentrassi.kura.addons.milo-0.2.2.dp

This installation will take a lot longer and you will need to check again by pressing the “Refresh” button in the Web UI.

We will also need to allow TCP access to port 12685. If you have the network managed version of Kura installed switch to the UI section “Firewall” and open a new port “12685” allowing access from “0.0.0.0/0” (Permitted Network) and press “Apply”.

A new cloud service

By default the “example publisher” will publish to the default Kura cloud service instance. We will now create a new Cloud service instance and then redirect the data to OPC UA. The data will be available as an OPC UA server. OPC UA differs between client and server. And while the Camel component does provide both ways, in this case we want others to consume our data, so offering data as an OPC UA server is the way to go.

Navigate to “Cloud Services” and press the “New” button. From the list of possible providers select org.eclipse.kura.camel.cloud.factory.CamelFactory, enter a cloud service PID (e.g. camel-opcua) and press “Create”.

After the instance has been created select it and configure it with the following options:

Router XML:

<routes xmlns="http://camel.apache.org/schema/spring">
    <route id="opc-ua-example">
       <from uri="vm:camel:example"/>
       <split>
           <simple>${body.metrics().entrySet()}</simple>
           <setHeader headerName="item">
               <simple>${body.key()}</simple>
           </setHeader>
           <setBody>
               <simple>${body.value()}</simple>
           </setBody>
           <toD uri="my-milo:${header.item}"/>
       </split>
    </route>
</routes>

Initialzation Code:

var milo = new org.apache.camel.component.milo.server.MiloServerComponent();
milo.setEnableAnonymousAuthentication(true);
camelContext.addComponent("my-milo", milo);
Screenshot of cloud service configuration
OPC UA configuration

Assigning the cloud service

Now we need to configure the example publisher to actually use our new cloud service instance. Select “Camel example publisher” from the left navigation bar and enter “opcua” (or whatever PID you used before) as “Cloud Service PID”. Apply the changes.

Testing the result

First of all, if you log in into your device using SSH, you should be able to see that port 12685 is opened:

root@raspberrypi:/home/pi# ss -nlt | grep 12685
LISTEN     0      128                      :::12685                   :::*     

Now you can connect to your device using any OPC UA explorer to the URI: opc.tcp://<my-ip>:12685

I am using Android and the “ProSYS OPC UA Client”

Summing it up

This tutorial uses a SNAPSHOT version of Eclipse Milo. Simply due to the fact that no version of Milo is released just yet. This should change in the following weeks and my play is to update the blog post once it is available. However the functionality of Milo will not change and using the Camel component, most internals of Milo are hidden anyway.

Update: As Milo and the Camel Milo component are released now I did update the links.

Apache Camel on Eclipse Kura can provide a complete new way of communication. This example was a rather simple one, Camel can do a lot more when it comes to processing data. And not all real-life applications may be as easy as that. But of course the intention of this blog post was to give a quick introduction into Camel and Kura in combination. Using the Camel Java DSL or the Kura Camel programmatic API can give greater flexibility. And yet, the example shows that even with a few lines of Camel XML, amazing things can be achieved.

Dropping Apache File Install into Eclipse Kura

Sometimes the simple things may be the most valuable. Testing with Eclipse Kura™ on a Raspberry Pi (or any other Eclipse Kura device) may be a bit tricky. Of course can use the Eclipse UI in combination with mToolkit. But if you want to edit, compile, deploy from a local desktop machine, to a Kura device, then you either need to click through the Web UI for uploading your application. But for this to work you also need to assembly a DP (distribution package).

But what if you could simply drop an OSGi bundle into a directory and let it get picked up by Kura automatically. Thanks to Apache File Install, there already is such a solution. File Install scans a folder and loads every OSGi bundle located in this folder. If a bundle is started and it gets overwritten in the file system, then File Install will reload and restart the bundle.

So deploying and re-deploying to a Kura device is as easy a copying a file to your target with SCP (or the remote copy tool of your choice).

And installing Apache File Install into Eclipe Kura now just got a bit simpler.

Using Maven Central

Simply navigate to the Packages section of the Eclipse Kura Web UI. Press the “Install” button and choose “URL” and enter the following URL:

https://repo1.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.utils.fileinstall/0.1.0/de.dentrassi.kura.addons.utils.fileinstall-0.1.0.dp

After confirming using the Submit button it will take a bit and then File Install will be installed into your Kura installation. Sometimes it takes as bit longer than Kura expects and you need to reload the Web UI (Ctrl-R) until Kura has performed the installation.

Using the Eclipse Marketplace

Currently the Eclipse Marketplace is focused on hosting plugins for the Eclipse IDE, but this should change rather soon. But still it is possible right now to drag and drop Apache File Install into Eclipse Kura with the use of the following install button:

Drag to your running Eclipse workspace to install Apache File Install for Eclipse Kura

Dragging this button into the Kura Web UI will bring up a confirmation dialog if you want to install the addon. After confirming it will go and fetch the DP and install Apache File Install into the running Kura instance.

Deploying bundles

Now it is time to deploy some bundles. By default the directory where Apache File Install looks for bundles is /opt/eclipse/kura/load. At first this directory will not exist, so it has to be created. Next we simply fetch and example bundle using wget:

$ cd /opt/eclipse/kura
$ mkdir load
$ wget "https://dentrassi.de/download/kura/org.eclipse.kura.example.camel.publisher-1.0.0.jar"

This is an example publisher bundle from the upcoming Kura 2.1.0 release. So if you are still using Kura 2.0, then you can either try a different OSGi bundle, or maybe give Kura 2.1 a try 😉

Due to a regression in Kura (eclipse/kura#123) there is currently no way to manually start OSGi bundles. So you need to stop Kura and start the local command console (/opt/eclipse/kura/bin/start_kura.sh). On the OSGi shell you can then:

osgi> ss example
"Framework is launched."


id	State       Bundle
116	RESOLVED    org.eclipse.kura.example.camel.publisher_1.0.0
osgi> start 116
osgi> ss example
"Framework is launched."


id	State       Bundle
116	ACTIVE      org.eclipse.kura.example.camel.publisher_1.0.0
osgi>

Now once this initial activate has been performed, File Install and OSGi will keep the bundle active. You can re-deploy this OSGi bundle by simply copying a new version over the old one. File Install will detect the change and refresh the bundle.

There is more…

Apache File Install can also update OSGi configurations and it can be configured using a set of system properties. For the full set of options check out the Apache File Install documentation.

Eclipse Kura on Apache Karaf

It took quite a while and triggered a bunch of pull request upstream, but finally I do have Eclipse Kura™ running with Apache Karaf™.

Now of course I immediately got the question: Why is that good? After all the recent setup still uses Equinox underneath in Karaf. Now what? … isn’t Equinox and Karaf something different? Read on …

What is Karaf?

Ok, back to the basics. Eclipse Equinox™ is an implementation of OSGi. As is Apache Felix™. OSGi is a highly modular Java framework for creating modular applications. And it eats its own dogfood. So there is a thin layer of OSGi (the framework) and a bunch of “add-on” functionality (services) which is part of OSGi, but at the same time running on top of that thin OSGi layer. Now Equinox and Felix both provide this thin OSGi layer as well a set of “standard” OSGi services. But it is possible to interchange service implementations from Equinox to Felix, since both use OSGi. For example in Package Drone I started to use Equinox as OSGi container, but in some release switched to the Apache Felix ConfigurationAdmin since it didn’t have the bugs of the Equinox implementation. So the final target is a plain Equinox installation with some parts from Apache Felix.

Apache Karaf can be seen as a distribution of OSGi framework and services. Like you have a Linux distribution starting with a kernel and a set of packages which work well together, Karaf is a combination of an OSGi framework (either Felix or Equinox or one of others) and a set of add-on functionality which is tested against each other.

And those add-on services are the salt in the OSGi soup. You can have a command line shell with completion support, JMX support, OSGi blueprint, webserver and websocket suport, JAAS, SSH, JPA and way more. Installing bundles, features or complete archives directly from local sources, HTTP or Maven central. It wraps some non-OSGi dependencies automatically into OSGi bundles. And many 3rd party components are already taking advantage of this. So for example installing Apache Camel™ into an Karaf contain is done with two lines on the Karaf shell.

What Kura currently does

Eclipse Kura currently assembles its own distribution based on pure and simple Equinox components using a big Ant file. While this has an advantage when it comes to distribution size and the fact that you can control each an every aspect of that distribution, it also comes at a prize that you need to do exactly that. That Ant file needs to create each startup configuration file, knowing about each and every bundle version.

Karaf on the other hand already has a ready to run distribution in various flavors. And has some tooling to create new ones.

What is different now

In order to bring Kura to Karaf there were a few changes necessary in Kura. In addition to that a new set of Maven projects assembly a target distribution. Right now it is not possible to simply drop Kura into Karaf as a Karaf feature. A special Karaf assembly has to be prepared which can then be uploaded and installed at a specific location. This is mostly necessary due to the fact that Kura expects files at a specific location in the file system. Karaf on the other hand can simple be downloaded, unzipped and started at any location.

In addition to that a few bugs in Kura had to be fixed. Kura did make a few assumption on how Equinox and its service implementations handle things which may no longer be true in other OSGi environments. these fixes will probably make it into Kura 2.1.

Can I test it today?

If you just want to start up a stable build (not a release) you can easily do this with Docker:


sudo docker run -ti ctron/kura:karaf-stable # stable milestone
sudo docker run -ti ctron/kura:karaf-develop # latest develop build

This will drop you in a Karaf shell of Kura running inside docker. You can play around with the console, install some additional functionality or try out an example like OPC UA with Apache Camel.

You can also check out the branch feature/karaf in my fork of Kura which contains all the upstream patches and the Karaf distribution projects in one branch. This branch is also used to generate the docker images.

What is next?

Clean up the target assembly: Right now assembling the target platform for Karaf is not as simple as I had hoped. This is mostly due to the complexity of such a setup (mixing different variants of Linux, processors and devices) and the way Maven works. With a bit of work this can made even simpler by adding a specific Maven plugin and make one or two fixes in the Karaf Maven plugin.

Enable Network management: Right now I disabled network management for the two bare-metal Linux targets. Simply because I did have a few issues getting this running on RHEL 7 and Fedora. Things like ethernet devices which are no longer named “eth0” and stuff like that.

Bring Kura to Maven Central/More Kura bundles: These two things go hand in hand. Of course I could simply create more Karaf features, packaging more Kura bundles. But it would be much better to have Kura artifacts available on Maven Central, being able to consume them directly with Karaf. That way it would be possible to either just drop in new functionality or to create a custom Kura+Karaf distribution based on existing modules.

Will it become the default for Kura?

Hard to say. Hopefully in the future. My plan is to bring it into Kura in the 2.2 release cycle. But this also means that quite a set of dependencies has to go through the Eclipse CQ process I we want to provide Karaf ready distributions for download. A first step could be to just provide the recipes for creating your own Karaf distribution of Kura.

So the next step is to bring support for Karaf into Kura upstream.

Collecting data to OpenTSDB with Apache Camel

OpenTSDB is an open source, scalable, <buzzword>big data</buzzword> solution for storing time series data. Well, that what the name of the project actually says 😉 In combination with Grafana you can pretty easy build a few nice dashboards for visualizing that data. The only question is of course, how do you get your data into that system.

My intention was to provide a simple way to stream metrics into OpenTSDB using Apache Camel. A quick search did not bring up any existing solutions for pushing data from Apache Camel to OpenTSDB, so I decided to write a small Camel component which can pick up data and send this to OpenTSDB using the HTTP API. Of course having a plain OpenTSDB HTTP Collector API for that would be fine as well. So I did split up the component into three different modules. A generic OpenTSDB collector API, an HTTP implementation of that and finally the Apache Camel component.

All components are already available in Maven Central, and although they have the version number 0.0.3, they are working quite well 😉


<dependency>
<groupId>de.dentrassi.iot</groupId>
<artifactId>de.dentrassi.iot.opentsdb.collector</artifactId>
<version>0.0.3</version>
</dependency>
<dependency>
<groupId>de.dentrassi.iot</groupId>
<artifactId>de.dentrassi.iot.opentsdb.collector.http</artifactId>
<version>0.0.3</version>
</dependency>
<dependency>
<groupId>de.dentrassi.iot</groupId>
<artifactId>de.dentrassi.iot.opentsdb.collector.camel</artifactId>
<version>0.0.3</version>
</dependency>

Dropping those dependencies into your classpath, or into your OSGi container ;-), you can use the following code to easily push data coming from MQTT directly into OpenTSDB:


CamelContext context = new DefaultCamelContext();

// add routes

context.addRoutes(new RouteBuilder() {

@Override
public void configure() throws Exception {
from("paho:sensors/test2/temperature?brokerUrl=tcp://iot.eclipse.org")
.log("${body}")
.convertBodyTo(String.class).convertBodyTo(Float.class)
.to("open-tsdb:http://localhost:4242#test2/value=temp");

from("paho:tele/devices/TEMP?brokerUrl=tcp://iot.eclipse.org")
.log("${body}")
.convertBodyTo(String.class).convertBodyTo(Float.class)
.to("open-tsdb:http://localhost:4242#test3/value=temp");
}
});

// start the context
context.start();

You can directly push Floats or Integers into OpenTSDB. The example above shows a “to” component which does directly address a metric. Using #test3/value=temp If you need more tags, then those can be added using #test3/value=temp/foo=bar.

But it is also possible to have a generic endpoint and provide the metric information in the actual payload. In this case you have to use the type de.dentrassi.iot.opentsdb.collector.Data and fill in the necessary information. It is also possible to publish an array of Data[].