[WIP] Working with External Sensors


#1

This is somewhat of a work in progress. I’m sharing it here as I’m at a stopping point and I’m not yet sure where I want to go from here.

Goal

Many of us, myself included, have a number of sensors scattered throughout the house yet there is only one thermostat that controls the heat (forced air HVAC in my case). How to best use these various sensors?

I still don’t have the answer but I have the first part solved at least, how to get the readings into HestiaPi.

Prerequisites

I’m running the latest ONE branch of the code from GitHub. The code is significantly different from version 1.1. But over all I’m trying to provide all of the changes external to the stock openHAB configs so they are easier to retain over an upgrade/.

Requirements

  • Support more than one sensor
  • Require minimal configuration from users
  • Require minimal changes to existing openHAB configs
  • Have timeouts so if a sensor doesn’t report after too long of a time that sensor is ignored; if no external sensors are reporting default to the built in sensors

MQTT

To minimize the configuration necessary on the HestiaPi side of things I’ve defined a standardized MQTT topic structure:

hestia/external/<room>/temperature
hestia/external/<room>/humidity

The messages are expected to be just the values of the sensor readings without units. The code assumes that the units of the values posted match TempUnit (i.e. degrees F if TempUnit is set to F). For humidity, values between 0 and 100 are assumed.

It should be pretty self explanatory. <room> can be any arbitrary name supported by MQTT. Configure your sensors to publish to these topics on whatever MQTT broker HestiaPi subscribes to. If your sensors do not support publishing to these topics, you may need to bridge between where they do publish and these topics.

Personally, I have sensors of various technologies, not all MQTT, reporting to my main openHAB. Some of these sensors use Units of Measurement so I can’t forward the readings using the follow Profile. So I wrote a Rule to forward my various sensors to the above topics.

from core.rules import rule
from core.triggers import when

@rule("Forward temps to HestiaPi",
      description="Takes current readings and forwards them to the proper MQTT topic")
@when("Member of gIndoorTemps received update")
@when("Member of gIndoorHumidity received update")
def forward(event):
    broker = "mqtt:broker:spare"
    room = event.itemName.split("_")[0].replace("v", "")
    reading = event.itemState.toString() # may need to deal with UoM here
    units = "temperature" if "Temp" in event.itemName else "humidity"
    topic = "hestia/external/{}/{}".format(room, units)

    forward.log.debug("Forwarding {} to {}".format(reading, topic))

    action = actions.get("mqtt", broker)
    if action:
        action.publishMQTT(topic, reading)
    else:
        forward.log.error("There is no broker thing {}!".format(broker))

NOTE: The above is using the NGRE and is written in Python using the Helper Libraries. My temp Items have “Temp” in the name which is how I separate and publish to the right topics based on the sensor type.

If you don’t have the senors feeding to another OH you have other options. You can change the following code and configs to use whatever topic structure they already follow. You can add Rules to HestiaPi to do this. You might be able to set up the broker as a bridge to itself and forward the original topics to the above topics. I’ve never tried this last one though, may not be possible.

HestiaPi Thing Configuration

Create a new .things file, I’ll call it /etc/openhab2/things/personal.things. We will create a new Broker Thing to subscribe to the HestiaPi broker and create a couple of publishTrigger Channels which use a wild card subscription to the above MQTT structure.

Bridge mqtt:broker:mosquitto "External Sensors MQTT Broker" [ host="localhost", secure=false, port=1883, clientID="external"]{
  Channels:
    Type publishTrigger : externaltemp "External Temperature" [stateTopic="hestia/external/+/temperature", separator="#"]
    Type publishTrigger : externalhumi "External Humidity" [stateTopic="hestia/external/+/humidity", separator="#"]
}

A publishTrigger will trigger a Rule every time a message is received. The separator will be placed between the topic and the message in the receivedEvent implicit variable in the Rule which we can parse to figure out where the message came from.

You can add the Channels to the existing Bridge in the HestiaPi configuration, but make sure to put the Channels after the definition of all the Things.

HestiaPi Items Configuration

This one is pretty easy. Since we have multiple sensors, what we really care about is the minimum sensor reading and the maximum sensor readings. So we need four new Items, which I place in /etc/openhab2/items/personal.items.

Number ExternalTempMax "External Max Temperature [%.0f]"
Number ExternalTempMin "External Min Temperature [%.0f]"
Number ExternalHumiMax "External Max Humidity [%.0f %%]"
Number ExternalHumiMin "External Min Humidity [%.0f %%]"

HestiaPi Rules

The Rule gets triggered by the internal sensors as well as the MQTT Trigger Channels defined above. We parse out the source of the sensor reading and the sensor reading and keep them in a Map. For each new reading from MQTT we set a Timer that will eliminate the reading if there isn’t a new one after the right amount of time (10 minutes in the code below).

The tricky part is the end of the Rule. We go through the Map of sensor readings and find the minimum and maximum reading. If we have no external readings at all, we use the built in MyTempProxy/MyHumiProxy Item instead.

Finally, we publish the min and max readings to the new Items.

import java.util.Map

val Map<String, Number> readings = newHashMap
val Map<String, Timer> timers = newHashMap

// Keep track of external sensors
rule "External Temp"
when
  Channel "mqtt:broker:mosquitto:externaltemp" triggered or
  Channel "mqtt:broker:mosquitto:externalhumi" triggered or
  Item MyTempProxy changed or
  Item MyHumiProxy changed
then
  val logName = "external"

  var Number reading = null
  var String sensor = null

  if(receivedEvent !== null) {
    // receivedEvent is of the format: <channel ID> triggered <topic>#<message>
    // <topic> is of the format: hestai/external/<room>/<temperature|humidity>
    val payload = receivedEvent.toString.split(" ").get(2).split("#")
    val topic = payload.get(0)
    reading = Float::parseFloat(payload.get(1))
    sensor = topic.split("/").get(3)

    logDebug(logName, "Reading = {} Topic = {} Sensor = {}", reading, topic, sensor)

    readings.put(topic, reading)

    // Erase the value if it get's more than 10 minutes old
    if(timers.get(topic) === null) {
      timers.put(topic, createTimer(now.plusMinutes(10), [ |
        logWarn(logName, "{} has not reported in 10 minutes! Deleting reading.", topic)
        timers.put(topic, null)
        readings.put(topic, null)
      ]))
    }
    else {
      timers.get(topic).reschedule(now.plusMinutes(10))
    }
  }
  else {
      reading = triggeringItem.state as Number
      sensor = if(triggeringItem.name.contains("Temp")) "temperature" else "humidity"
  }

  // Get the min and max for that type of reading, if there are no external readings
  // default to the built in sensor
  var local = if(sensor == "temperature") MyTempProxy else MyHumiProxy
  var min = readings.entrySet.filter[ entry | entry.key.contains(sensor) ].map[ getValue ].reduce[ min, v | if(min > v) v else min ]
  if(min === null) min = local.state as Number
  var max = readings.entrySet.filter[ entry | entry.key.contains(sensor) ].map[ getValue ].reduce[ max, v | if(max < v) v else max ]
  if(max == null) max = local.state as Number
  logDebug("external", "{}: Min = {} Max = {}", sensor, min, max)

  // Publish the min and max values to the appropriate Items
  switch(sensor){
    case "temperature":{
      ExternalTempMin.postUpdate(min)
      ExternalTempMax.postUpdate(max)
    }
    case "humidity":{
      ExternalHumiMin.postUpdate(min)
      ExternalHumiMax.postUpdate(max)
    }
  }

end

Next Steps

The next steps are where I’ve stopped. Now that I have the min and max readings from around the house, what do I do with them? For heating, do I try to make the max temp equal to the SetPoint or the min temp equal to the setpoint? Or do I aim for the middle and try to get the mid point between the min temp and max temp be the setpoint?

There are lots of ways to handle this. Anyway, once you figure out what you want to do with these max and min values, some changes need to be made to the existing HestiaPi Rules.

  1. Update/change the PreviousTempReading/PreviousHumiReading Item to kick off the Rule that tries to figure out what to do with the heating/cooling/humidifier/dehumidifier in the Rule above.

  2. Remove the updates to the Previous Reading Items from “Process Sensor Changes” Rule so the internal sensor doesn’t drive the thermostat.

  3. Modify the heating/cooling/humidity lambdas to decide whether or not to turn on/off the device based on the External Min and Max Items based on the behavior you desire.

Future Ideas

I can make this much more dynamic and easier to customize using the NGRE. For example, I can dynamically create Items for the external sensors and move the min/max logic to a Group definition. I can use the Expire binding on these dynamically created Items to timeout the value. All of this will make the sitemap able to keep up with the external sensor readings, showing all the information available without needing to statically define it.

Also, with NGRE, Rules can call Rules, enable/disable Rules, and support library functions. Thus one could just load these new Rules, run a configuration Rule and it will cut out the old default Rules and insert this new Rule, freeing the user from needing to modify any of the code.


#2

My vote goes to solution 3 as it will be easier for issue 14 to be handled in the future too.
I still have something unclear and I believe this will be true for other people too. I have some remote sensors that are not in areas affected much from heating/cooling like cupboards, garages, attic.
Should I simply add them to a group different to gIndoorTemps and gIndoorHumidity ?

Additionally I have heard a lot of people that want parts of the house to take control of the setpoint throughout the day, like kitchen/dinning room during lunch time, living room in the afternoon and bedrooms in the night (we are still in a single-zone setup).
Can you think of a way to generalize support for multi remote sensor, single remote sensor and single internal sensor?
Could this be “enabled/disabled” without editing files?


#3

With the code as written above, you would need them to publish to a different topic. Alternatively, instead of the one Channel with a wild card subscription, you could create a separate Channel for each room and omit those Channels as triggers to the “External Temp” Rule that you do not want to use to drive the devices.

NOTE: The new Broker Thing and these Channels can be defined by the users using PaperUI instead of through the text based configs.

Implement something like the Time of Day Design Pattern to define the various time periods. Then one needs to make a mapping between a time of day and which room takes control of the heater. From there its a simple matter of populating the PrevousXReading with the reading from the right room. This could be expanded to support occupancy sensors (motion sensors with wasp in the box algorithm, FIND2, reelyActive BT sensing, etc.).

But the problem is the user needs to create the mapping between the room and the time of day. And they need to be able to define the times of day. And using Rules DSL, that is going to require editing .rules files. If we were using the NGRE, there are ways we can let the user define these using just Items which are created using PaperUI, no need to edit files.

To enable/disable we can use a Switch, but the actual configuration, using OH 2.4 and Rules DSL and with everything in text based configs rather than JSONDB means the user will have to edit files. The ability to access Item metadata which is a prerequisite for this.

NOTE: I’ve already an implementation of the Time of Day DP which doesn’t necessarily require users to edit any text configs in order to define the times of day and it would be trivial to add the mappings to the room that should control the heater. It even supports a different schedule for weekends and holidays. But it requires the NGRE and OH 2.5.

This is somewhat easier if we assume we only care about the max and min of all the external sensors like presented above. In that case the only difference between one external sensor and multiple external sensors is which topics are subscribed to that trigger the Rule. Just subscribe to one (or have one sensor publish to hestia/external/blah). The code would be the same.

To switch between the remote sensor(s) and internal sensor the easiest is probably to just use a Switch on BasicUI.

I’m really keen on moving to JSONDB for storing the configs over the text based configs. They will give us way more flexibility to support stuff like this and provide a UI for the users to customize everything (the UI doesn’t have to be PaperUI btw, it’s all REST API based). That sort of customization really isn’t available with the text based configs.

It also will let you develop an ecosystem of Rules that users can use and contribute to that they can just import as a template and use. Rules can enable/disable each other, you can turn Rules on and off through PaperUI and all that sort of thing which should make it much easier for users to customize their config, all without touching the text configs.

But I’m not sure how you build your images and don’t know the best way to source control them. I would imagine adding a var/lib/openhab2 folder along side the etc/openhab2 and home/pi/scripts folders and checking in at a minimum the etc, jsondb, and config folders.

That’s my inclination too, but I was trying to see if I could implement this without touching the main Rules. That could be easier for users to adopt and show how they can customize their config without requiring changes to the main .rules files. It doesn’t seem to be possible but I’m still thinking about ways to change default.rules in a way to make this possible.


#4

As long as JSONDB can be ‘popupaled’ with the help of a script then it shouldn’t be a problem. Usually the image process starts with calling packitupandgo.sh on a prep’ed Pi Zero W. It would be beneficial if we could see the rules etc of an image file without having to boot it. It also has to reside on GitHub for development history. Are these possible?


#5

It’s just a text file that follows a specific JSON format. The big difference is that file can be edited through the OH REST API and it is a lot less work for OH to load and parse it. And there is no need to hide and then restore the rules files during a boot.

It’s just a JSON formatted text file, but what is it you want to see in it. It’s not very fun to edit the JSON by hand, though it can be done. If you give me a little more of the why I can tell you the how.

Absolutely, it would reside in github and be source controlled. I do this with my personal OH instance where I have all of my Things managed by JSONDB and have started migrating my Rules to JSONDB. It’s just a text file so it works well with github. The only gotcha is, though I’ve never seen this, others have reported that sometimes the order of the entries in the JSONDB can move around which makes diffs a challenge.

I’ve never seen this myself so it’s hard to say how often it occurs, but it is a possibility.

Personally, the decrease in boot time, the ability to give users a UI to change configs and Rules, and other flexibility it will allow more than make up for the limitations.

Anyway, we would have a director structure in github like the following:

\
    etc
        openhab
            persistence
                mapdb.persist
            sitemaps
                default.sitemap
            transform? I didn't see these map files in use anywhere

    home
        pi
            scripts
                All the scripts, note that the rules won't live here any more

    var
        lib
            openhab2
                config
                    These files are generated by openHAB but this is where add-ons installed and configurations 
                    made through PaperUI get saved
                etc
                    This is where Karaf configs are stored. At a minimum this is where the changes to the logging 
                    config will reside. This is also where we would remove the LSP which is CPU and memory 
                    overhead we don't need running
                jsondb
                    This is where Things, Items, and Rules will live in JSON formatted text files. We need to exclude
                    the backups folder as that is where automatic backups of the JSON files are made by OH when 
                    there are changes.
                persistence
                    mapdb # optional, we can prepopulate initial states for some Items in the database instead of 
                          # the relatively complicated initialization Rule


Here is what a Thing looks like in the JSONDB (It's an MQTT Thing from my main OH instance):

```json
  "mqtt:topic:thermostat": {
    "class": "org.eclipse.smarthome.core.thing.internal.ThingImpl",
    "value": {
      "label": "HestiaPi Thermostat",
      "bridgeUID": {
        "segments": [
          "mqtt",
          "broker",
          "hestiapi"
        ]
      },
      "channels": [
        {
          "acceptedItemType": "Number",
          "kind": "STATE",
          "uid": {
            "segments": [
              "mqtt",
              "topic",
              "thermostat",
              "temperature"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "mqtt",
              "number"
            ]
          },
          "label": "Current Temperature",
          "configuration": {
            "properties": {
              "stateTopic": "hestia/local/temperature"
            }
          },
          "properties": {},
          "defaultTags": []
        },
        {
          "acceptedItemType": "Number",
          "kind": "STATE",
          "uid": {
            "segments": [
              "mqtt",
              "topic",
              "thermostat",
              "humidity"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "mqtt",
              "number"
            ]
          },
          "label": "Current Humidity",
          "configuration": {
            "properties": {
              "stateTopic": "hestia/local/humidity",
              "max": 100.0,
              "min": 0.0
            }
          },
          "properties": {},
          "defaultTags": []
        },
        {
          "acceptedItemType": "Number",
          "kind": "STATE",
          "uid": {
            "segments": [
              "mqtt",
              "topic",
              "thermostat",
              "tgttemp"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "mqtt",
              "number"
            ]
          },
          "label": "Taget Temp",
          "configuration": {
            "properties": {
              "commandTopic": "hestia/local/settempsetpoint",
              "stateTopic": "hestia/local/tempsetpoint",
              "retained": true
            }
          },
          "properties": {},
          "defaultTags": []
        },
        {
          "acceptedItemType": "Switch",
          "kind": "STATE",
          "uid": {
            "segments": [
              "mqtt",
              "topic",
              "thermostat",
              "heating"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "mqtt",
              "switch"
            ]
          },
          "label": "Heating",
          "configuration": {
            "properties": {
              "commandTopic": "",
              "stateTopic": "hestia/local/cmnd/heatingstate/POWER"
            }
          },
          "properties": {},
          "defaultTags": []
        },
        {
          "acceptedItemType": "String",
          "kind": "STATE",
          "uid": {
            "segments": [
              "mqtt",
              "topic",
              "thermostat",
              "heatingmode"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "mqtt",
              "string"
            ]
          },
          "label": "Heating Mode",
          "configuration": {
            "properties": {
              "commandTopic": "hestia/local/cmnd/heatingmode",
              "stateTopic": "hestia/local/cmnd/heatingmode",
              "retained": true
            }
          },
          "properties": {},
          "defaultTags": []
        },
        {
          "acceptedItemType": "Switch",
          "kind": "STATE",
          "uid": {
            "segments": [
              "mqtt",
              "topic",
              "thermostat",
              "fan"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "mqtt",
              "switch"
            ]
          },
          "label": "House Fan",
          "configuration": {
            "properties": {
              "commandTopic": "hestia/local/cmnd/fanstate/POWER",
              "stateTopic": "hestia/local/cmnd/fanstate/POWER"
            }
          },
          "properties": {},
          "defaultTags": []
        },
        {
          "acceptedItemType": "String",
          "kind": "STATE",
          "uid": {
            "segments": [
              "mqtt",
              "topic",
              "thermostat",
              "fanmode"
            ]
          },
          "channelTypeUID": {
            "segments": [
              "mqtt",
              "string"
            ]
          },
          "label": "House Fan Mode",
          "configuration": {
            "properties": {
              "commandTopic": "hestia/local/cmnd/fanmode",
              "stateTopic": "hestia/local/cmnd/fanmode",
              "retained": true
            }
          },
          "properties": {},
          "defaultTags": []
        }
      ],
      "configuration": {
        "properties": {}
      },
      "properties": {},
      "uid": {
        "segments": [
          "mqtt",
          "topic",
          "thermostat"
        ]
      },
      "thingTypeUID": {
        "segments": [
          "mqtt",
          "topic"
        ]
      },
      "location": "Entryway"
    }
  }
}

Here is an Item:

{
  "Test": {
    "class": "org.eclipse.smarthome.core.items.ManagedItemProvider$PersistedItem",
    "value": {
      "groupNames": [],
      "itemType": "Switch",
      "tags": [],
      "label": "Test"
    }
  }
}

Links to Things are managed separately.

Here is a rule:

{
  "d62407e8-2732-4415-9ec8-454e8a8d3c18": {
    "class": "org.openhab.core.automation.dto.RuleDTO",
    "value": {
      "triggers": [
        {
          "id": "1",
          "label": "an item receives a command",
          "description": "This triggers the rule if an item receives a command.",
          "configuration": {
            "itemName": "Test",
            "command": "ON"
          },
          "type": "core.ItemCommandTrigger"
        }
      ],
      "conditions": [],
      "actions": [
        {
          "inputs": {},
          "id": "2",
          "label": "execute a given script",
          "description": "Allows the execution of a user-defined script.",
          "configuration": {
            "type": "application/javascript",
            "script": "// Code goes here\nvar myLog \u003d Java.type(\"org.slf4j.LoggerFactory\").getLogger(\"org.eclipse.smarthome.model.script.Rules\");\nmyLog.info(\"Hello world!\")"
          },
          "type": "script.ScriptAction"
        }
      ],
      "configuration": {},
      "configDescriptions": [],
      "uid": "d62407e8-2732-4415-9ec8-454e8a8d3c18",
      "name": "Test Rule",
      "tags": [],
      "visibility": "VISIBLE",
      "description": "Simple test Rule"
    }
  }
}

The code part of the Rule can be found at script. As you can see, it’s not pretty to edit by hand but it can be done. Ultimately though, the NGRE will let us build lots of smaller Rules that can call each other and it has a “but only if…” section that acts as a guard on the rule so the actual code that goes in the “script” sections will be significantly shorter than we have now in Rules DSL.

In many cases there won’t be any code at all as there are a number of Actions that do not require special code. For example, we can have a Rule that does a bunch of checks on state (but only if section…) kind of like the lambdas and if they pass call another Rule to run directly, or send a command to an Item, or the like. No “code” required.


#6

Wow… I didn’t realise the change involved. Definitely agree for JSONDB and 2.5 but I already feel bad for you and the huge tasks you have already accomplished and this is not a little one for sure…


#7

It’s not that much really. And worth it in the long run. There is already code to import the Items from .items files in bulk. A similar script for Things wouldn’t be hard either, though there are not that many Things so doing it by hand wont take much.

It’s the Rules that will be work, but it’s work I already know how to do. :slight_smile: And the overall improvements will be very worth it.

So the question is should we upgrade to 2.5 first, or start moving stuff the JSONDB first? I’d prefer to hold off on moving Rules until after the upgrade, but Things and Items can be moved before that.


#8

Regarding the issue you already spotted with Exec binding in 2.5… Does it matter if we simply apply the workaround and leave the bug in so that we can move with the rest that need 2.5?


#9

I guess it depends on when you want to snap a baseline. The options are:

  1. Use OH 2.5.1 instead of 2.5.2 which doesn’t include the Exec binding whitelist change. There are some bug fixes to the MQTT binding between 2.5.1 and 2.5.2 I think but nothing that we need.
  2. Download the 2.5.1 Exec binding jar file and drop it into the addons folder instead of installing it through the REST API.
  3. Download the 2.5.3-SNAPSHOT version of the binding jar and drop it into the addons folder. the 2.5.3-SNAPSHOT version has (or will shortly have) the fix for this.
  4. Wait for OH 2.5.3, which should be released in a month before snapping the baseline.
  5. Add a line to the initializing rule to touch the whitelist file, though until the Rule runs the log will be full of the whitelist warning.

Personally I’d choose 4 and in the mean time use 5 for development.

So far I’m not seeing any errors beside that one. But I’m not where I can see the LCD to verify everything is working there. But watching MQTT Explorer and the events.log everything seems to be working as expected. I think everything is good with OH 2.5.

Under the assumption that this will work, I can start to migrate stuff to JSONDB. Should this be done on a separate branch or the ONE branch? I don’t know how long this will take so I could see doing this in parallel with what’s in ONE now so we don’t have something in a half working or transitioned state cluttering up the ONE branch.

In addition to the Fan returning to it’s previous mode after AUTO I noticed a little bug in the boost logic as I was testing the upgrade.