Kura

12 posts

Apache Camel Java DSL in combination Eclipse Kura Wires

In part #1 and part #2, we saw how easy it is to interface Apache Camel with Kura Wires. Simply by re-using some existing functionality. A few lines of XML, Groovy and you can already build an IoT solution based on the Camel ecosystem and the Eclipse Kura runtime. This part will focus on the Java DSL of Apache Camel.

It will also take into account, that when you develop and deploy an application, you need some kind of development, test and integration environment. When you build something, no matter how big, based on Camel or Kura Wires, you do want to test it. You want to have unit tests, and the capability to automatically test if your solution works, or still works after you made changes.

Using Kura Wires alone, this can be a problem. But Camel offers you a way to easily run your solution in a local IDE, debugging the whole process. You can have extra support for debugging Camel specific constructs like routes and endpoints. Camel has support for JUnit and e.g. using the “seda” endpoints, you can in create an abstraction layer between Camel and Wires.

The goal

I’ll make this one up (and yes, let’s try to keep it realistic). We have a device, and his device allows to set two parameters for its operation (P1 and P2, both floating points). Now we already have the device connection set up in Kura. Maybe using Modbus, or something else. Kura can talk to it using Kura Wires and that is all that counts.

Now we do get two additional requirements. There is some kind of operating panel next to the device, which should allow viewing and setting those parameters locally. Also, those parameters should be accessible, using IEC 60870-5-104, for an additional device, right next to the Kura gateway.

All of those operations have to be local only, and still work when no connection to the cloud is possible. But of course, we don’t want to lose the ability to monitor the parameters from our cloud system.

The operator panel will, of course, be programmed in the newest and hippest Web UI technology possible. It is super easy to fire a few HTTP API calls and encode everything in JSON. While, at the same time, the IEC 60870 layer has no idea about complex data structures. The panel application will send a full update of both parameters, while the 60870 client, due to the nature of this protocol, will only send one parameter at a time.

Doesn’t sound too unrealistic, does it?

The project structure

The full project source code is available at ctron/kura-examples, on GitHub. So this blog post will only focus on some important spots of the whole project.

The project is a standard Maven project, and thus has the typical project structure:

Maven Camel project structure

There are only three important differences to a standard Java Maven project:

The packaging type is bundle, which requires the maven-bundle-plugin. It will create an OSGi bundle JAR, instead of a plain JAR file. This is required as the Kura IoT gateway is based on OSGi.

We will create a “DP” package at the end of the build, using the OSGi DP Maven Plugin. This package can directly be uploaded to the Kura instance. As this plugin does include direct dependencies, but doesn’t include transient dependencies (on purpose), the project declares a few dependencies as “provided” to prevent them from being re-packaged in the final DP package.

The project uses the maven-antrun-plugin to download and unpack the static Swagger UI resources. Swagger UI is just a convenience for playing around with the REST API later on. Camel will take care of creating the OpenAPI (Swagger) JSON description, even if the SwaggerUI part is removed. So in a production setup, you most likely would not add Swagger UI to the deployment.

Starting it up

The project has three entry points:

  • CamelApplicationComponent is the OSGi service, which will be managed by the OSGi service component runtime (SCR) when the component is uploaded to Kura.
  • TestApplication is a local test application, which is intended to be started from the local IDE for manual testing.
  • CamelApplicationComponentTest is the JUnit 4 based test for testing the Camel routes.

All three entry points will have a slightly different creation process for the Camel Context. This is simply due to the fact that different environments (like plain Java, OSGI and JUnit) have different requirements.

The routes configuration, which is the same for all entry points, is located in Routes.

Let’s have a quick look at the OSGi startup:

@Activate
public void start(final BundleContext context) throws Exception {
  this.context = new OsgiDefaultCamelContext(context, SwaggerUi.createRegistry());
  this.context.addRoutes(new Routes(this.state));
  this.context.start();

  final Dictionary<String, Object> properties = new Hashtable<>();
  properties.put("camel.context.id", "camel.example.4");
  this.registration = context.registerService(CamelContext.class, this.context, properties);
}

Once the component is placed inside an OSGi container, the start method will be called and set up the Camel context. This is all pretty straightforward Camel code. As the last step, the Camel context will be registered with the OSGi service layer. Setting the service property camel.context.id in the process. This property is important, as we will, later on, use it to locate the Camel context from the Kura Wires graph by it.

The Java DSL routes

The routes configuration is pretty simple Camel stuff. First, the REST DSL will be used to configure the REST API. For example, the “GET” operation to receive the currently active parameters:

…
  .get()
  .description("Get the current parameters")
  .outType(Parameters.class)
  .to("direct:getParameters")
…

This creates a get operation, which is being redirected to the internal “direct:getParameters” endpoint. Which is a way of forwarding that call to another Camel Route. This way Camel routes can be re-used from different callers.

Like for example the `direct:updateParameters` route, which will be called by all routes which want to update the parameters, no matter if that call originated in the IEC 60870, the REST or the Kura Wires component:

from("direct:updateParameters")
  .routeId("updateParameters")
  .bean(this.state, "updateCurrentParameters")
  .multicast()
  .to("direct:update.wires", "direct:update.iec.p1", "direct:update.iec.p2").end();

The route will forward the new parameters to the method updateCurrentParameters of the State class. This class is a plain Java class, holding the state and filling in null parameters with the current state. The result of this method will be forwarded to the other routes, for updating Kura Wires and the two parameters in the IEC 60870 data layer.

Trying it out

If you have Java and Maven installed, then you can simply compile the package by running:

cd camel/camel-example4
mvn clean package

This will compile, run the unit tests and create the .dp package in the folder target.

You can upload the package directly to your Kura instance. Please note that you do need the dependencies installed in part #1 of the tutorial. In additional you will need to install the following dependencies:

  • https://repo1.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.camel.iec60870/0.6.1/de.dentrassi.kura.addons.camel.iec60870-0.6.1.dp
  • https://repo1.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.camel.jetty/0.6.1/de.dentrassi.kura.addons.camel.jetty-0.6.1.dp
  • https://repo1.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.camel.swagger/0.6.1/de.dentrassi.kura.addons.camel.swagger-0.6.1.dp
  • https://repo1.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.camel.http/0.6.1/de.dentrassi.kura.addons.camel.http-0.6.1.dp

This will install the support for REST APIs, backed by Jetty. As Kura already contains Jetty, it only makes sense to re-use those existing components.

Once the component is deployed and started, you can navigate your web browser to http://:8090/api. This should bring up the Swagger UI, showing the API of the routes:

SwaggerUI of Camel example for Kura

Next, you can create the following components in the Kura wires graph:

  • Create a new “Camel consumer”, named consumer2
    • Set the ID to camel.example.4
    • Set the endpoint URI to seda:wiresOutput1
  • Create a new “Logger”, named logger2
    • Set it to “verbose”
  • Connect consumer2 with logger2
  • Click on “Apply” to activate the changes

Open the console of Kura and then open the Swagger UI page with the Web browser. Click on ““Try Out” of the “PUT” operation, enter some new values for setpoint1 and/or setpoint2 and click on the blue “Execute” button.

In the console of Kura you should see the following output:

2018-09-17T13:35:49,589 [Camel (camel-10) thread #27 - seda://wiresOutput1] INFO  o.e.k.i.w.l.Logger - Received WireEnvelope from org.eclipse.kura.wire.camel.CamelConsume-1537188764126-1
2018-09-17T13:35:49,589 […] INFO  o.e.k.i.w.l.Logger - Record List content:
2018-09-17T13:35:49,589 […] INFO  o.e.k.i.w.l.Logger -   Record content:
2018-09-17T13:35:49,589 […] INFO  o.e.k.i.w.l.Logger -     P1 : 3.0
2018-09-17T13:35:49,589 […] INFO  o.e.k.i.w.l.Logger -     P2 : 2.0
2018-09-17T13:35:49,589 […] INFO  o.e.k.i.w.l.Logger -

This is the result of the “Logger” component from Kura Wires. Which did receive the new parameter updates from the Camel Context, as they got triggered through the Web UI. At the same time, the IEC 60870 server would update all clients being subscribed to those data items.

Wrapping it up

The last part of this tutorial showed that, if the prep-prepared XML router component of Eclipse Kura, is not enough, then you can drop in your own and powerful replacements. Developing with all the bells and whistles of Apache Camel, and still integrate with Kura Wires if necessary.

Sunny weather with Apache Camel and Kura Wires

Part #1 of the Apache Camel to Kura Wires integration tutorial did focus on pushing data from Kura Wires to Camel and processing it there. But part #1 already mentioned that it is also possible to pull in data from Camel into Kura Wires.

Apache Camel consumer node in Kura Wires

Preparations

For the next step, you again need to install a Camel package, for interfacing with Open Weather Map: https://repo1.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.camel.weather/0.6.0/de.dentrassi.kura.addons.camel.weather-0.6.0.dp The installation follows the same way as already described in part #1.

In addition to the installation of the package, you will also need to create an account at https://openweathermap.org/ and create an API key. You can select the free tier plan, it is more than enough for our example.

Back to Wires

Next, create a new Camel context, like before, and give it the ID “camel2”. Add the required component weather, the required language groovy and set the following XML router content (be sure to replace <appid> with your API token):

<routes xmlns="http://camel.apache.org/schema/spring">

  <route>

    <from uri="weather:dummy?appid=<YOUR API TOKEN>&amp;lat=48.1351&amp;lon=11.5820"/>
    <to uri="stream:out"/>

    <unmarshal><json library="Gson"></json></unmarshal>
    <transform><simple>${body["main"]["temp"]}</simple></transform>
    <convertBodyTo type="java.lang.Double"/>
    <to uri="stream:out"/>

    <transform><groovy>["TEMP": request.body-273.15]</groovy></transform>
    <to uri="stream:out"/>
    <to uri="seda:output1"/>

  </route>

</routes>

After applying the changes, you can create two new components in the Wire graph:

  • Create a new “Camel Consumer”, name it consumer1
    • Set the Camel context ID camel2
    • Set the endpoint URI seda:output1
  • Create a new “Logger”, name it logger1
    • Set it to “verbose”
  • Connect consumer1 with logger1
  • Click on “Apply” to activate the changes

What does it do?

What this Camel context does, is to first start polling information from the Open Weather Map API. It requests with a manually provided GPS location, Munich.

It then parses the JSON, so that we can work with the data. Then it extracts the current temperature from the rather complex Open Weather Map structure. Of course, we could also use a different approach and extract additional or other information.

The extracted value could still be a number, represented internally by a string. So we ask Camel to ensure that the body of the message gets converted to a Double. If the body already is a double, then nothing will be done. But, if necessary, Camel will pull in its type converter system and optionally convert e.g. a string to a double by parsing it.

Now the body contains the raw value, as a Java double. But we still have two issues with that. The first one is, that the value is in degree Kelvin. Living in Germany, I would expect degree Celsius ;-) The second issue is, that Kura Wires requires some kind of key to that value, like a Map structure.

Fortunately, we easily can solve both issues with a short snippet of Groovy: ["TEMP": request.body-273.15]. This will take the message (request) body, convert it to degree Celsius, and using this as a value for the key TEMP in the newly created map.

Checking the result

As soon as you apply the changes, you should see some output on the console, which shows the incoming weather data:

{"coord":{"lon":11.58,"lat":48.14},"weather":[{"id":801,"main":"Clouds","description":"few clouds","icon":"02d"}],"base":"stations","main":{"temp":297.72,"pressure":1021,"humidity":53,"temp_min":295.15,"temp_max":299.15},"visibility":10000,"wind":{"speed":1.5},"clouds":{"all":20},"dt":1537190400,"sys":{"type":1,"id":4914,"message":0.0022,"country":"DE","sunrise":1537160035,"sunset":1537204873},"id":2867714,"name":"Muenchen","cod":200}
297.72
{TEMP=24.57000000000005}

Every change, which should happen every second, shows three lines. First the raw JSON data, directly from the Open Weather Map API. Then the raw temperature in degree Kelvin, parsed by Camel and converted into a Java type already. Followed by the custom Map structure, created by the Groovy script. The beauty here is again, that you don’t need to fiddle around with custom data structures of the Kura Wires system, but can rely on standard data structures likes plain Java maps.

Looking at the Kura log file, which is by default /var/log/kura.log, you should see some output like this:

2018-09-17T13:57:10,117 [Camel (camel-15) thread #31 - seda://output1] INFO  o.e.k.i.w.l.Logger - Received WireEnvelope from org.eclipse.kura.wire.camel.CamelConsume-1537188764126-1
2018-09-17T13:57:10,117 [Camel (camel-15) thread #31 - seda://output1] INFO  o.e.k.i.w.l.Logger - Record List content:
2018-09-17T13:57:10,118 [Camel (camel-15) thread #31 - seda://output1] INFO  o.e.k.i.w.l.Logger -   Record content:
2018-09-17T13:57:10,118 [Camel (camel-15) thread #31 - seda://output1] INFO  o.e.k.i.w.l.Logger -     TEMP : 24.57000000000005
2018-09-17T13:57:10,118 [Camel (camel-15) thread #31 - seda://output1] INFO  o.e.k.i.w.l.Logger -

This shows the same value, as processed by the Camel context but received by Kura Wires.

Wrapping it up

Now, of course, a simple logger component isn’t really useful. But as you might now, Kura has the ability to connect to a GPS receiver. So you could also take the current position as an input to the Open Weather Map request. And instead of using my static GPS coordinates of Munich, you could query for the nearby weather information. So this might allow you to create some amazing IoT applications.

Stay tuned for Part #3, where we will look at a Camel based solution, which can run inside of Kura, as well as outside. Including actual unit tests, ready for continuous delivery.

Leveraging the power of Apache Camel in Eclipse Kura

With the upcoming version of Eclipse Kura 4, we will see some nice new features for the embedded Apache Camel runtime. This tutorial walks you through the Camel integration of Kura wires, which allows you to bridge both technologies, and leverage the power of Apache Camel for your solutions on the IoT gateway.

Kura Wires is a graph-oriented programming model of Eclipse Kura. It allows wiring up different components, like a Modbus client to the internal Kura Cloud Service. It is similar to Node-RED.

Apache Camel is a message-oriented integration platform with a rule-based routing approach. It has a huge eco-system of components, allowing to integrate numerous messaging endpoints, data formats, and scripting languages.

A graphical approach, like Kura Wires may be interesting for a single instance, which is manually administered. But assume that you want to re-deploy the same solution multiple times. In this case you would want to locally develop and test it. Have proper tooling like validation and debugging. And then you want to automatically package it and run a set of unit and integration tests. And only after that you would want to deploy this. This model is supported when you are using Apache Camel. There is a lot of tooling available, tutorials, training, books on how to work with Apache Camel. And you can make use of the over 100 components which Camel itself provides. In addition to that, you have a whole ecosystem around Apache Camel, which can extend this even more. So it is definitely worth a look.

Prerequisites

As a prerequisite, you will need an instance of Kura 4. As this is currently not yet released, you can also use a snapshot build of Kura 3.3, which will later become Kura 4.

If you don’t want to set up a dedicated device just for playing around, you can always use the Kura container image and it e.g. with Docker. There is a short introduction on how to get started with this at the DockerHub repository: https://hub.docker.com/r/ctron/kura/

Starting a new Kura instance is as easy as:

docker run -ti ctron/kura:develop -p 8080:8080

The following tutorial assumes that you have already set up Kura, and started with a fresh instance.

Baby Steps

The first step we take is to create a very simple, empty, Camel Context and hook and directly hook up a Camel endpoint without further configuration.

New Camel Context

As a first step, we create a new XML Router Camel context:

  • Open the Kura Web UI
  • Click on the “+” button next to the services search box
  • Select the org.eclipse.kura.camel.xml.XmlRouterComponent factory
  • Enter the name camel1
  • Press “Submit”

New Camel Context Component

A new service should appear in the left side navigation area. Sometimes it happens that the service does not show up, but reloading the Web UI will reveal the newly created service.

Now select the service and edit the newly created context. Clear out the “Router XML” and only leave the root element:

<routes xmlns="http://camel.apache.org/schema/spring">
</routes>

In the field “Required Camel Components” add the stream component. Click on “Apply” to activate the changes. This will configure the Camel context to have no routes, but wait for the stream component to be present in the OSGi runtime. The stream component is a default component, provided by the Eclipse Kura Camel runtime. The Camel context should be ready immediately and will be registered as an OSGi service for others to consume.

The Wires Graph

The next step is to configure the Kura Wires graph:

  • Switch to “Wire Graph” in the navigation pane
  • Add a new “Timer” component named timer1
    • Configure the component to fire every second
  • Add a new “Camel Producer” named producer1
    • Set the Context ID field of the component to camel1
    • Set the endpoint URI to stream:out
  • Connect the nodes timer1 and producer1
  • Click on Apply to activate the changes

If you look at the console of the Kura instance, then you should see something like this:

org.eclipse.kura.wire.WireEnvelope@bdc823c
org.eclipse.kura.wire.WireEnvelope@5b1f50f4
org.eclipse.kura.wire.WireEnvelope@50851555
org.eclipse.kura.wire.WireEnvelope@34cce95d

Note: If you are running Kura on an actual device, then the output might be in the file /var/log/kura-console.log.

What is happening is, that the Kura wires timer component will trigger a Wires event every second. That event is passed along to the Camel endpoint stream:out in the Camel context camel1. This isn’t using any Camel routes yet. But this is a basic integration, which allows you to access all available Camel endpoints directly from Kura Wires.

Producer, Consumer, Processor

In addition to the “Producer” component, it is also possible to use the “Consumer”, or the “Processor”. The Consumer takes events from the Camel context and forwards them to the Kura Wires graph. While the “Processor” takes an event from the Wire Graph, processes it using Camel, and passes along the result to Wires again:


For Producer and Consumer, this would be a unidirectional message exchange from a Camel point of view. The Processor component would use an “in”/”out” message exchange, which is more like “request/response”. Of course that only makes sense when you have an endpoint which actually hands back a response, like the HTTP client endpoint.

In the following sections, we will see that in most cases there will be a more complex route set up that the Camel Wire component would interact with, proxied by a seda Camel component. Still, the “in”, “out” flow of the Camel message exchange would be end-to-end between whatever endpoint you have and the Wires graph.

Getting professional

Apache Camel mostly uses the concept of routes. And while accessing an endpoint directly from the Kura Camel component technically works, I wouldn’t recommend it. Mainly due to the fact that you would be missing an abstraction layer, there is no way to inject anything between the Kura Wires component and the final destination at the Camel endpoint. You directly hook up Kura Wires with the endpoint and thus lose all ways that Camel allows you to work with your data.

So as a first step, let’s decouple the Camel endpoint from Kura Wires and provide an API for our Camel Context.

In the camel1 configurations screen, change the “Router XML” to:

<routes xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="seda:input1"/>
        <to uri="stream:out"/>
    </route>
</routes>

Then configure the producer1 component in the Wire Graph to use the “Endpoint URI” seda:input1 instead of directly using stream:out.

If everything is right, then you should still see the same output on the Kura console, but now Wires and Camel are decoupled and properly interfaced using an internal event queue, which allows us to use Camel routes for the following steps.

One benefit of this approach also is that you can now take the XML route definitions outside of Kura and test them in your local IDE. There are various IDE extensions for Eclipse, IntelliJ and Visual Studio, which can help to work with Camel XML route definitions. And of course, there are the JBoss Tools as well ;-). So you can easily test the routes outside of a running Kura instance and feed in emulated Kura Wires events using the seda endpoints.

To JSON

This first example already shows a common problem, when working with data, and even so for IoT use cases. The output of org.eclipse.kura.wire.WireEnvelope@3e0cef10 is definitely not what is of much use. But Camel is great a converting data, so let’s make use of that.

As a first step we need to enable the JSON support for Camel:

  • Navigate to “Packages”
  • Click on “Install/Upgrade”
  • Enter the URL: https://repo1.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.camel.gson/0.6.0/de.dentrassi.kura.addons.camel.gson-0.6.0.dp
  • Click on “Submit”

After a while, the package de.dentrassi.kura.addons.gson should appear in the list of installed packages. It may happen that the list doesn’t properly refresh. Clicking on “refresh” or reloading the Web page will help.

Instead of downloading the package directly to the Kura installation you can also download the file to your local machine and then upload it by providing the file in the “Install/Upgrade” dialog box.

As a next step, you need to change the “Router XML” of the Camel context camel1 to the following configuration:

<routes xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="seda:input1"/>
        <marshal><json library="Gson"/></marshal>
        <transform><simple>${body}\n</simple></transform>
        <to uri="stream:out"/>
    </route>
</routes>

In the Kura console you will now see that we successfully transformed the internal Kura Wires data format to simple JSON:

{"value":[{"properties":{"TIMER":{}}}],"identification":"org.eclipse.kura.wire.Timer-1536913933101-5","scope":"WIRES"}

This change did intercept the internal Kura wires objects and serialized them into proper JSON structures. The following step simply appends the content with a “newline” character in order to have a more readable output on the command line.

Transforming data

Depending on your IoT use case, transforming data can become rather complex. Camel is good at handling this. Transforming, filtering, splitting, aggregating, … for this tutorial I want to stick to a rather simple example, in order to focus in the integration between Kura and Camel, and less on the powers of Camel itself.

As the next step will use the “Groovy” script language to transform data, we will need to install an additional package using the same way as before: https://repo1.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.camel.groovy/0.6.0/de.dentrassi.kura.addons.camel.groovy-0.6.0.dp

Then go ahead and modify the “Router XML” to include a transformation step, add the following content before the JSON conversion:

<transform><groovy>
return  ["value": new Random().nextInt(10), "timer": request.body.identification ];
</groovy></transform>

The full XML context should now be:

<routes xmlns="http://camel.apache.org/schema/spring">
    <route>
        <from uri="seda:input1"/>
        <transform><groovy>
        return  ["value": new Random().nextInt(10), "timer": request.body.identification ];
        </groovy></transform>
        <marshal><json library="Gson"/></marshal>
        <transform><simple>${body}\n</simple></transform>
        <to uri="stream:out"/>
    </route>
</routes>

After applying the changes, the output on the console should change to something like:

{"value":2,"timer":"org.eclipse.kura.wire.Timer-1536913933101-5"}

As you can see, we now created a new data structure, based on generated content and based on the original Kura Wires event information.

Off to the Eclipse Hono HTTP Adapter

Printing out JSON to the console is nice, but let’s get a bit more professional. Yes, Kura allows you to use its Kura specific MQTT data format. But what we want to send this piece of JSON to some HTTP endpoint, like the Eclipse Hono HTTP protocol adapter?

Camel has a huge variety of endpoints for connecting to various APIs, transport mechanisms and protocols. I doubt you directly would like your IoT gateway to contact Salesforce or Twitter, but using OPC UA, MQTT, HTTP, IEC 60870, might be a reasonable use case for IoT.

As a first step, we need to install Camel HTTP endpoint support: https://repo1.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.camel.http/0.6.0/de.dentrassi.kura.addons.camel.http-0.6.0.dp

The next step requires an instance of Eclipse Hono, thankfully there is a Hono sandbox server running at hono.eclipse.org.

In the XML Router we need two steps for this. You can add them after the to element, so that we still see the JSON on the command line:

<setHeader headerName=”Content-Type”><constant>application/json</constant></setHeader>
<to uri="https4://hono.eclipse.org:28080/telemetry?authenticationPreemptive=true&amp;authUsername=sensor1@DEFAULT_TENANT&amp;authPassword=hono-secret"/>

The first step sets the content type to application/json, which is passed along by Hono to the AMQP network.

Yes, it really is http4://, this is not a typo but the Camel endpoint using Apache HttpClient 4.

You may need to register the device with Hono before actually publishing data to the instance. Also, it is necessary that a consumer is attached, which receives the data. Hono rejects devices publish data if no consumer is attached. Also see: https://www.eclipse.org/hono/getting-started/#publishing-data

If you are using a custom deployment of Hono using the OpenShift S2I approach, then the to URL would look more like:

<to uri="https4://hono-adapter-http-vertx-sec-hono.my.openshift.cluster/telemetry?authenticationPreemptive=true&amp;authUsername=sensor1@DEFAULT_TENANT&amp;authPassword=hono-secret"/>

Wrapping it up

What we have seen so far is that, with a few lines of XML, it is possible to interface with Kura Wires, and start processing data that was originally not supported by Kura, sending to a target that also isn’t supported by Kura. On for that we only used a few lines of XML.

In addition to that, you can test and develop everything in a local, confined space. Without having to worry too much about actually running a Kura instance.

In Part #2, we will have a look at ways to get data from Camel back into Kura Wires. And in Part #3 of this tutorial, we will continue with this approach and develop a Camel based solution, which can run inside of Kura, as well as outside, including actual unit tests.

Eclipse Kura on the Intel UP² with CentOS

Intel UP² In the past I was testing modifications to Kura with a Raspberry Pi 3 and Fedora for ARM. But I got a nice little Intel UP² just recently, and so I decided to perform my next Kura tests, with the modifications to the Apache Camel runtime in Kura, on this nice board. Creating a new device profile for Kura using CentOS 7 and the Intel UP² looked like a good idea anyway.

At the time of writing, the PR for merging the device profile into Kura is still pending (PR #2093). But my hope is that this will be merged before Kura 4 comes out.

Build your own Kura image

But it is possible to try this out right now by using the preview branch (preview/intel_up2_1) on my forked repository: ctron/kura.

The following commands use the kura-build container. For more information about building Kura with this container see: https://github.com/ctron/kura-build and https://hub.docker.com/r/ctron/kura-build/.

So for the moment you will need to build this image yourself. But if you have Docker installed, then it only needs a few minutes to create your own build of Kura:

docker run -v /path/to/output:/output -ti ctron/kura-build -r ctron/kura -b preview/intel_up2_1 -- -Pintel-up2-centos-7

Where /path/to/output must be replaced with a local directory where the resulting output should be placed. If you are running Docker with SElinux enabled, then you might need to append :z to the volume:

docker run -v /path/to/output:/output:z -ti ctron/kura-build -r ctron/kura -b preview/intel_up2_1 -- -Pintel-up2-centos-7

As you might guess, it is also possible to build other branches and repositories of Kura in the same way. That docker image only ensures that all the necessary build dependencies are present when executing the build.

If you are running on Linux and do have all the dependencies installed locally. Then of course there is no need to run through Docker, you can simply call the build-kura script directly:

./build-kura preview/intel_up2_1 -r ctron/kura -b preview/intel_up2_1 -- -Pintel-up2-centos-7

Setting up CentOS 7

This is rather simple step, you simply need to download CentOS from https://www.centos.org/download/ (the Minimal ISO is just fine). Copy the ISO to a USB stick (https://wiki.centos.org/HowTos/InstallFromUSBkey). On a Linux-ish system this should work like (where /dev/sdX is the USB stick, all data on this stick will be lost!):

sudo dd if=CentOS-7-x86_64-Minimal-1804.iso of=/dev/sdX bs=8M status=progress oflag=direct

Rebooting your UP with the USB stick attached, this should reboot into the CentOS installer from where you can perform a standard installation.

After the installation is finished and you booted into CentOS, you will need to enable EPEL, as Kura requires some extra components (like wireless-tools and hostapd). You can do this by executing:

sudo yum install epel-release

You might also want to install a more recent kernel into CentOS. All the core things works with the default CentOS kernel. However some things like support for the GPIO support is still missing in the default CentOS kernel. But the mainline kernel from ELRepo can easily be installed:

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-3.el7.elrepo.noarch.rpm
yum --enablerepo=elrepo-kernel install kernel-ml

For more information check e.g.: https://www.howtoforge.com/tutorial/how-to-upgrade-kernel-in-centos-7-server/

Installing Kura on the Intel UP²

Copy the RPM you just created from the build process over to the UP, e.g. by:

scp kura-build-output/2018XXXX-YYYY/kura-intel-up2-centos-7*.rpm user@my-up:

And then on the device run:

yum install kura-*.rpm

This will install the Kura package as well as any required dependencies. After the installation has completed, reboot the machine and navigate your web browser to “http://my-up”, using the credentials “admin” / “admin”.

More information

CEP & Machine learning for IoT – Drools on Kura

Machine learning and predicate maintenance we are the role models of IoT use cases. Having an IoT gateway allows you to pre-process data before you send it upstream to your cloud. It also allows you to do local decisions, without the actual need for a cloud upload. But as you know from sitting at your favorite restaurant, staring at the menu, making decisions can be quite hard ;-) Complex event process and machine learning models can help you with IoT use cases though.

Eclipse Kura is an open source IoT gateway with a focus on industrial use cases. Drools is an open source rule engine and, with Drools Fusion, provides complex event processing. It also supports making decisions based on Predictive Model Markup Language (PMML) based models. So why not bring both components together?! PMML’s can be used for all kind of scenarios. But the classic IoT use case would probably be to do predicate maintenance, based on a PMML generated by your machine learning solution on the cloud. Sending back the “learned” knowledge to the edge gateway for local processing.

Installation

The Drools addon for Kura provide two DPs (the package type used by Kura) which can be deployed in order to extend Kura with Drools.

Note: If you don’t have a Raspberry Pi at hand, or don’t want to install Kura on some “real” device. You can always use the Kura Emulator docker image (see also Developing for Eclipse Kura in Windows).

Setup

Once the components are installed, it is possible to create a new Drools instance by clicking in the blue “+” on the left side of the Kura services area. Create a new component of the type de.dentrassi.kura.addons.drools.component.DroolsInstance. Choose any unused ID and click “Apply”. It might be necessary to reload the Web UI of Kura at this point as the refresh doesn’t properly work. When the service is listed in the left hand side “services” list, select it in order to configure:

The actual rules document comes from the file SimpleScorecard.pmml of the PMML drools example mpbravo/brms-pmml-example. Also be sure to set the file type to “Predictive Model Markup Language”. Save the changes by clicking on “Apply”.

Next we will use Kura Wires in order to mesh up the model with some “data”. A need to use a timer as input source, as Kura currently doesn’t offer any kind of value creating like a function or sine curve. So create a new “timer”, you can leave the default of 10 seconds. Add a new logger, which we simply use for testing, you should set the “verbosity” to “VERBOSE”. And then create a new “DroolsProcess” component with the following configuration:

ID
pmml1 – The ID of the drools session
Fire all rules
true – After the fact has been injected, rules have to fired
Delete after fire
true – After the rules have been fired, we can remove the fact from the session
Fact Package
org.drools.scorecards.example – The package name, from the rules model
Fact Type
SampleScore – The type name, from the rules model
Inputs
age=TIMER – comma separated list for Wire record names to fact object properties
Outputs
result=scorecard_calculatedScore – comma separated list for Wire record names to fact object properties

Finally wire that all up:

Results

Looking at the Kura log file /var/log/kura.log should show you something like (where result is coming from the PMML model):

2018-03-15 16:07:12,165 [DefaultQuartzScheduler_Worker-6] INFO  d.d.k.a.d.c.w.DroolsProcess - Result - type: class java.lang.Double, value: 24.0
2018-03-15 16:07:12,165 [DefaultQuartzScheduler_Worker-6] INFO  o.e.k.i.w.l.Logger - Received WireEnvelope from de.dentrassi.kura.addons.drools.component.wires.DroolsProcess-1521129031885-7
2018-03-15 16:07:12,165 [DefaultQuartzScheduler_Worker-6] INFO  o.e.k.i.w.l.Logger - Record List content: 
2018-03-15 16:07:12,165 [DefaultQuartzScheduler_Worker-6] INFO  o.e.k.i.w.l.Logger -   Record content: 
2018-03-15 16:07:12,165 [DefaultQuartzScheduler_Worker-6] INFO  o.e.k.i.w.l.Logger -     result : 24.0
2018-03-15 16:07:12,165 [DefaultQuartzScheduler_Worker-6] INFO  o.e.k.i.w.l.Logger - 

Conclusion

Of course the example is rather trivial. And the actual model and the way we wired it up is just an example. But of course you will bring your own model, based on your machine learning solutions, and have your own data to wire up. So go ahead and explore.

Just a bit of Apache Camel

Sometimes you write something and then you nearly forget that you did it … although it is quite handy sometimes, here are a few lines of Apache Camel XML running in Eclipse Kura:

I just wanted to publish some random data from Kura to Kapua, without the need to code, deploy or build anything. Camel came to the rescue:

<routes xmlns="http://camel.apache.org/schema/spring">
  <route id="route1">

    <from uri="timer:1"/>
    <setBody><simple>${bean:payloadFactory.create("value", ${random(100)})}</simple></setBody>
    <to uri="kura-cloud:myapp/topic"/>

  </route>
</routes>

Dropping this snippet into the default XML Camel router:

  • Registers a Kura application named myapp
  • Creates a random number between 0 and 100 every second
  • Converts this to the Kura Payload structure
  • And publishes it on the topic topic

Developing for Eclipse Kura on Windows

Every now and then it is fun to leave the environment you are used to and do something completely different. So this journey take me to IntelliJ and Windows 10. And yes, I am glad to be back in Linux/Eclipse-land. But still, I think something rather interesting came out of this.

It all started when I helped my colleague Aurélien Pupier to get his environment ready for his talk at the Eclipse IoT day Grenoble. If you missed this talk, you can watch a recording of it. He wanted to present the Camel Developer Tools. The problem was the he was working on a Windows laptop. And he wanted to demonstrate Eclipse Kura in combination JBoss Tools IDE. However Kura can only run on Linux and he wanted to run the JBoss Tools native on his Windows machine.

Of course you could come up with some sort of Virtual Machine setup, but we wanted something which was easier to re-produce in the case there would be some issue with the laptop for the presentation.

Creating a docker image of Kura

The first step was to create a docker image of Kura. Currently Kura doesn’t offer any support for Docker. So that had to be created from scratch. As there is even no x86_64 distribution of Kura and no emulator distribution, it was necessary to do some rather unusual hacks. The background is, that Kura has a rather crude build system which assembles a few distributions in the end of the build. Kura also requires some hardware interfaces in order to work properly. For those hardware interfaces there exist emulator replacements for using in a local developer setup. However there is neither a distribution assembly for x86_64 nor one using the emulator replacements. The whole build system in the end is focused around creating Linux-only images. The solution was to simply rip out all functionality which was in the way and create a patch file.

This patch file and the docker build instructions are now located at a different repository where I can easily modify those and hook it up to the DockerHub build system: ctron/kura-emulator. Currently there are three tags for docker images in the Kura Emulator DockerHub repository: latest (which is the most recent, but stable release), 3.0.0-RC1 and develop (most recent, less stable). As there is currently no released version of Kura 3.0.0, the latest tag is also using the develop branch of Kura. The 3.0.0-RC1 tag is a stable version of the emulator which is known to work and won’t be updated in the future.

There is a more detailed README file in the GitHub repository which explains how to use and build the emulator yourself. In a nutshell you can start it with:

docker run -ti -p 8080:8080 ctron/kura-emulator

And afterwards you can navigate with your browser to http://localhost:8080 and use the Kura Web UI.

As Docker is also available for Windows, this will work the same way on either Linux or Windows, and although I didn’t test it, it should also work on Mac OS X.

JMX & Debugging

As the Camel tooling makes use of JMX, it was necessary to also enable JMX support for Kura, which normally is not available with Kura. By setting the JAVA_OPTS environment variable it is not only possible to enable JMX, but also to enable plain Java debugging for the docker image. Of course you will need to publish the selected ports with -p when running the docker image. And for Windows you cannot simply use localhost but you will need to use the IP addresses created by docker for windows: also see README.md.

Drop in & activate

After the conference was over, I started to think about what we actually had achieved by doing all this. We had a read-to-run Kura image, dockerized, capable of running of Windows (with docker), debuggable. The only part which was still missing was the ability to add a new, custom bundle to the emulator.

Apache Felix File Install to the rescue! In the past I created an Apache Felix File Install DP for Kura (DP = deployment package for Kura). File Install works in a way that it monitors a directory and automatically loads, unloads and updates an OSGi JAR file which you drop into this directory. The DP can simply be dropped into Kura, which extends Kura with this File Install functionality.

So I pre-seeded the Kura docker image with the File Install DP and declared a volume mount, so that you can simply mount a path of the docker image onto your host system. Dropping a file into the directory on the host system will make it available to the docker container and File Install will automatically pick it up and start it, but inside the docker container.

docker run -ti -p 8080:8080 -v c:/path/to/bundles:/opt/eclipse/kura/load ctron/kura-emulator

And this even works with Docker for Windows, if you share your drive first:

Share drive with Docker

Choose your tools

Currently Kura requires you to use a rather complicated setup for developing applications for Kura. You will need to learn about Eclipse PDE, target platforms, Tycho for Maven and bunch of other things to get your Kura application developed, built and packaged.

I already created a GitHub repository for showing a different way to develop Kura applications: ctron/kura-examples. Those project use plain maven, the maven-bundle-plugin and my osgi-dp plugin to create the final DP. Those examples also make use of the newer OSGi annotations instead of requiring your to craft all OSGi metadata by hand.

So if you wanted, you could already use your favorite IDE and start developing Kura application with style. But in order to run them, you still needed a Kura device. But with this docker image you can now simply let the emulator run and let File Install pick up the compiled results:

Summary

So yes, it is possible to use IntelliJ on Windows to develop and debug your Kura application, in a stylish fashion. Or you can simply do the same, just using an excellent IDE like Eclipse and an awesome operating system like Linux, with the same stylish approach ;-)

Testing Kapua with simulated Kura gateways

Now you got your pretty new OpenShift setup of Eclipse Kapua and want to give your IoT cloud a test run?! Testing it out with 100 devices, just for fun? Or even more? But you are too lazy to flash 1000 SD cards for your Raspberry Pi cluster? Here comes the Kura simulator framework. ;-)

In order to provide some automatic testing for Kapua I started working on a simulator framework which does simulate Kura instances completely in Java. No backend needed, no hardware needed, able to run multiple instances in a single JVM. And all hosted on GitHub at ctron/kura-simulator.

A screenshot of Kura simulator instances in Kapua
Kura simulator instances in Kapua

The basic idea was to create a set of classes which can be used in automated unit tests in order to simulate a Kura gateway, but allow for a finer grained control over it for testing the good, the bad and the ugly. A real Kura instance would of course be a more realistic test partner, but then again this would have quite a few drawbacks. First of all, Kura cannot be embedded into a unit or integration test. It has far too many dependencies to directory structures, command line utilities, native libraries and it would also require an OSGi container to be started. Second, Kura would always behave like Kura. Now for some tests this may be fine, but if you want to test corner cases where the gateway responds in a way which is not expected by Kapua, then this cannot be done with Kura.

So running a single Kura simulator can be as easy as:

ScheduledExecutorService downloadExecutor = 
   Executors.newSingleThreadScheduledExecutor(new NameThreadFactory("DownloadSimulator"));

GatewayConfiguration configuration =
   new GatewayConfiguration("tcp://kapua-broker:kapua-password@localhost:1883", "kapua-sys", "sim-1");

Set<Application> apps = new HashSet<>();
apps.add(new SimpleCommandApplication(s -> String.format("Command '%s' not found", s)));
apps.add(AnnotatedApplication.build(new SimpleDeployApplication(downloadExecutor)));

try (MqttSimulatorTransport transport = new MqttSimulatorTransport(configuration);
     Simulator simulator = new Simulator(configuration, transport, apps);) {
    Thread.sleep(Long.MAX_VALUE);
    logger.info("Bye bye...");
} finally {
  downloadExecutor.shutdown();
}

Of course, scaling this up and running a few more instances of this isn’t a big deal either. Running this in a docker container and scaling this up even more with OpenShift works fine as well. So testing any number of Gateways just became a lot easier.

Currently the simulator can emulate the command service (V1) and most of the deploy service (V2). The configuration service is still missing, but should get implemented in the next few days. Of course it is also possible to register a custom application and provide some metrics yourself.

Remote managing Eclipse Kura on Apache Karaf with ECF

To be honest, I had my troubles in the past with the Eclipse Communication Framework (ECF), not that it is a bad framework, but whatever I started it was complicated and never really worked for me. This story is different!

A few months back the Eclipse Kura project ran into an issue that the plugin which was being used for remote managing a Kura instance (mToolkit) from an IDE just kind of went away (issue #496). There is some workaround for that now, but still the problems around mToolkit still exists. Beside the fact that it is no longer maintained, it is also rather buggy. Deploying a single bundle takes about a minute for me. Of course using the Apache File Install package for Kura would also help here ;-)

But having a decent IDE integration would also be awesome. So when Scott Lewis from the ECF project contacted me about that, I was ready to give it a try. Unfortunately the whole setup required more than Kura could handle at that time. But now we do have support for Java 8 in Kura and there also is some basic support for running Kura on Karaf, including a docker image with the Kura emulator running on Karaf.

So I asked Scott for some help in getting this up and running and the set of instructions was rather short. In the following examples I am assuming your are running RHEL 7, forgive me if you are not ;-)

First we need to spin up a new Kura emulator instance:

sudo docker run -ti --net=host ctron/kura:karaf-stable

We are mapping all network to the host instance, since we are using another port, which is not configured in the upstream Dockerfile. There is probably another way, but this is just a quick example.

Then, inside the Karaf instance install ECF. We configure it first to use “ecftcp” instead of MQTT. But of course you can also got with MQTT or some other adapter ECF provides:

property -p service.exported.configs ecf.generic.server
property -p ecf.generic.server.id ecftcp://localhost:3289/server

feature:repo-add http://download.eclipse.org/rt/ecf/kura.20161206/karaf4-features.xml
feature:install -v ecf-kura-karaf-bundlemgr

Now Kura is read to go. Following up in the Eclipse IDE, you will need Neon running on Java 8:

Add the ECF 3.13.3 P2 repository using http://download.eclipse.org/rt/ecf/3.13.3/site.p2 and install the following components:

  • ECF Remote Services SDK
  • ECF SDK for Eclipse

Next install the preview components for managing Karaf with ECF. Please note, those components are previews and may or may not be release at some point in the future. Add the following P2 repository: http://download.eclipse.org/rt/ecf/kura.20161206/eclipseui and install the following components (disable Group Items by Category):

  • Remote Management API
  • Remote Management Eclipse Consumer

Now comes the fiddly part, this UI is a work in progress, and you have been warned, but it works:

  • Switch to the Remote Services perspective
  • Open a new view: Window -> Show View -> Other… – Select Remote OSGi Bundles
  • Click one of the green + symbols (choose either MQTT or ECFTCP) and enter the address of your Karaf instance (localhost and 3289 for me)

You should already see some information about that target device now. But when you open a new view (as before) named Karaf Features you will also have the ability to tinker around with the Karaf installation.

If you just want to have a quick look, here it is:

ECF connecting to Kura on Karaf
ECF connecting to Kura on Karaf

Of course you don’t need to use an IDE for managing Karaf. But having such an integration as an option, is a nice addition. And it shows how powerful a great OSGi setup can be ;-)

Providing telemetry data with OPC UA on Eclipse Kura

The upcoming version 2.1.0 of Eclipse Kura™ will feature an enhanced version of the Apache Camel™ integration which was introduced in Kura 2.0.0. There are various new ways on how to run Camel routes, configured either by XML routes or using the Java DSL. Apache Camel can act as a Kura application but, new in this release, there is also a way to simply configure Camel as a “cloud service”. In past releases of Kura, applications could only push data to one cloud target. The new 2.1.0 release will add the functionality of adding multiple cloud targets and one of those targets can be Apache Camel router instances.

With Camel you can have different ways of achieving this goal, but in this post I would like to focus on the “out of the box” way, by simply configuring (not developing) a set of Camel routes, which act as cloud service. Traditional instances of cloud services in Kura are only capable of delivering data to one cloud target or subscribing to one cloud infrastructure. But using Apache Camel as a technology it is possible to connect to a bunch of technologies at the same time.

The setup

The setup will be a Kura instance, running a pre-release version of Kura 2.1.0. The final version should be out in a few weeks and won’t differ much from the current version. We will be configuring a new cloud service instance which takes Kura application payload data and provide it as OPC UA, using the Camel OPC UA adapter. As payload provider (aka Kura application) we will be using the “Example publisher” from my Kura addons project.

Open up the Kura Web UI, navigate to “Packages” and select “Install/Update”. Switch to “URL” and provide the following URL:

https://dentrassi.de/download/kura/de.dentrassi.kura.addons.example.publisher_0.1.0-SNAPSHOT.dp

Note: As an alternative you can also download the “dp” package with your desktop browser and deploy the file using the “file” upload instead of “URL”.

Adding packages to Kura
Adding packages to Kura

The installation may take a bit and it may be necessary to press the “Refresh” button in order to see the installed package. After the packages was installed you should be able to see the service “Camel example publisher” on the left side.

Now we need to install the “Milo component for Camel”. Press “Install/Update” again and enter the following URL:

http://central.maven.org/maven2/de/dentrassi/kura/addons/de.dentrassi.kura.addons.milo/0.2.2/de.dentrassi.kura.addons.milo-0.2.2.dp

This installation will take a lot longer and you will need to check again by pressing the “Refresh” button in the Web UI.

We will also need to allow TCP access to port 12685. If you have the network managed version of Kura installed switch to the UI section “Firewall” and open a new port “12685” allowing access from “0.0.0.0/0” (Permitted Network) and press “Apply”.

A new cloud service

By default the “example publisher” will publish to the default Kura cloud service instance. We will now create a new Cloud service instance and then redirect the data to OPC UA. The data will be available as an OPC UA server. OPC UA differs between client and server. And while the Camel component does provide both ways, in this case we want others to consume our data, so offering data as an OPC UA server is the way to go.

Navigate to “Cloud Services” and press the “New” button. From the list of possible providers select org.eclipse.kura.camel.cloud.factory.CamelFactory, enter a cloud service PID (e.g. camel-opcua) and press “Create”.

After the instance has been created select it and configure it with the following options:

Router XML:

<routes xmlns="http://camel.apache.org/schema/spring">
    <route id="opc-ua-example">
       <from uri="vm:camel:example"/>
       <split>
           <simple>${body.metrics().entrySet()}</simple>
           <setHeader headerName="item">
               <simple>${body.key()}</simple>
           </setHeader>
           <setBody>
               <simple>${body.value()}</simple>
           </setBody>
           <toD uri="my-milo:${header.item}"/>
       </split>
    </route>
</routes>

Initialzation Code:

var milo = new org.apache.camel.component.milo.server.MiloServerComponent();
milo.setEnableAnonymousAuthentication(true);
camelContext.addComponent("my-milo", milo);
Screenshot of cloud service configuration
OPC UA configuration

Assigning the cloud service

Now we need to configure the example publisher to actually use our new cloud service instance. Select “Camel example publisher” from the left navigation bar and enter “opcua” (or whatever PID you used before) as “Cloud Service PID”. Apply the changes.

Testing the result

First of all, if you log in into your device using SSH, you should be able to see that port 12685 is opened:

root@raspberrypi:/home/pi# ss -nlt | grep 12685
LISTEN     0      128                      :::12685                   :::*     

Now you can connect to your device using any OPC UA explorer to the URI: opc.tcp://<my-ip>:12685

I am using Android and the “ProSYS OPC UA Client”

Summing it up

This tutorial uses a SNAPSHOT version of Eclipse Milo. Simply due to the fact that no version of Milo is released just yet. This should change in the following weeks and my play is to update the blog post once it is available. However the functionality of Milo will not change and using the Camel component, most internals of Milo are hidden anyway.

Update: As Milo and the Camel Milo component are released now I did update the links.

Apache Camel on Eclipse Kura can provide a complete new way of communication. This example was a rather simple one, Camel can do a lot more when it comes to processing data. And not all real-life applications may be as easy as that. But of course the intention of this blog post was to give a quick introduction into Camel and Kura in combination. Using the Camel Java DSL or the Kura Camel programmatic API can give greater flexibility. And yet, the example shows that even with a few lines of Camel XML, amazing things can be achieved.