Are there any gotchas? I'm going to make some changes

[rlkoshak]

I’m very proficient with openHAB and there are a few things I’m seeing that I’d like to change. But sometimes things are done a way for a good reason and I’d hate to run into problems that are foreseeable.

My initial integration with OH went very smoothly using MQTT. I decided not to move it to use my main system broker because I think I read on a thread here that the UI uses MQTT to communicate with OH. If that’s the case I don’t want to introduce an external dependency if I don’t have to. But it was simple enough to create a couple Things on my main system OH and access the needed topics. So my initial pressure to read and understand how it all works is reduced.

Long term I’m looking to do the following:

• run reasonable frequent apt updates

• I’d like to try to get it to run on OH 2.5 release if that’s possible. I don’t see why it would be having scanned through the Things and Items and Rules.

• install and configure tripwire (I’m not paranoid, it’s a great way to monitor file system changes to discover potential problems, I discovered a failing SD card once through oddness in the tripwire report)

• if successful getting it to run on OH 2.5, I’d like to rewrite the Rules using Scripted Automation and Python or JavaScript. The Rules as they exist right now are not all that complicated really so it should be pretty easy for someone with my experience with both

• migrate much if not all of the config stuff to JSONDB (including Rules), I believe it will greatly improve the boot times on the RPi 0 as OH has to do a lot less work to parse and load them all and in my experience on OH it requires less ram and runs faster too.

• replace the system started Rule that resets everything with a Rule that only resets a couple of things and depend on restoreOnStartup to restore to the most recent settings. The Rule already checks for not NULL before updating the Items so I might not need to change much but there are a few hard coded settings (e.g. if my Heater was on Auto before HestiaPi closed down, it had better come back up on Auto. Starting up completely off is not an option and there’s no practical way I can put this machine on an UPS.

• add in a configurable hysteresis and/or a timeout to keep from cycling the unit too fast. As written it appears to turn on the furnace immediately when it goes below the setpoint and turns off when the setpoint is reached, rounding the decimal places giving a hysteresis of one degree F (I haven’t looked at how it would work with C). That’s too tight. The house should be allowed to get colder before starting the heater so it runs less often but for longer periods, at least when dealing with forced air furnaces like I have.

• Update the logger to suppress the warning from org.eclipse.jetty.server.HttpChannel about Asynchronous processing not supported on Servlet 2.x container.. It’s a warning and it doesn’t look like there is anything I can do about it. May as well suppress it in the pax logging config. Unless this is something I need to be worried about. I’m assuming it’s because the RPi0 is a single CPU machine.

These are just a few ideas I have after about a day with it installed and working. Obviously, if any of the above are good ideas I’m more than willing to contribute them to the project.

[HestiaPi]

Let me first reply to this thread and I’ll cover any remaining questions from the other thread afterwards.

@rlkoshak

I think I read on a thread here that the UI uses MQTT to communicate with OH.

That is correct. To decouple the UI from the backend we use MQTT and websockets. You can launch the UI subscribing to a different MQTT server instead of localhost and even different topics.

@rlkoshak

run reasonable frequent apt updates

apt updates should be fine but we haven’t tried it yet.

@rlkoshak

I’d like to try to get it to run on OH 2.5 release if that’s possible.

Coming from an update to 2.4 with all that was happening on MQTT we would never suggest it on a live instance. Obviously you know exactly what 2.5 affects in terms of functionality and syntax so go ahead but we strongly discourage beginners to be the first to try it.

@rlkoshak

install and configure tripwire

Haven’t used it before. How can you react to specific file changes? Is it system-specific or OH-specific catches you are after?

@rlkoshak

• I’d like to rewrite the Rules using Scripted Automation and Python or JavaScript.
  …

• migrate much if not all of the config stuff to JSONDB (including Rules)

I feel a total newb to openHAB compared to your experience and exposure! I need to do some homework tonight on these subjects but having coded scenarios with the existing rules engine I totally felt the lack of a higher level language to code. Rules have been the hardest bit to process during boot for the Pi Zero W but it was mostly CPU intensive rather than memory. We actually have a little trick to load a very basic rules set initially and when system is fully up, then load the actual full set of rules. Still boot time is toooo long. If you feel this will improve this you are the most appropriate person to take this decision and dig in.

@rlkoshak

replace the system started Rule that resets everything with a Rule that only resets a couple of things and depend on restoreOnStartup to restore to the most recent settings.

Didn’t know of restoreOnStartup till now. We have already a few issues open and one is addressing the lack of non volatile settings. If this is something you want to address too please feel free, but I feel like I gave you my month’s worth of tasks from day 1.

@rlkoshak

add in a configurable hysteresis and/or a timeout to keep from cycling the unit too fast.

At the moment only the second stage heating has these configurable (Settings>2nd Stage Heating>) but another user has mentioned it and it makes total sense. We feel some functionality slipped while we were debugging, manufacturing and chasing slipped delivery dates. . Rule checkcurrtemp is the place for anyone else who want to hardcode it, but you know that already
We thought of having a switch between an ECO (higher) and a Comfort (lower) hysteresis mode under settings. Having their actual values configurable from the UI or not is simply a matter of balance between simple UI and customisation.

@rlkoshak

Update the logger to suppress the warning from org.eclipse.jetty.server.HttpChannel

More homework here We saw this error but we couldn’t something about it. Suppressing it didn’t appear as an option then.

From your previous thread I kept in the ToDo list the obvious need for documentation on the UI which I will work on this weekend.

@rlkoshak

What exactly is backed up and restored when I click those buttons in BasicUI?

It is calling the openhab-cli to backup all openHAB files. Not system.

Please respond here with your view on all the above. It is a lot!
I cannot thank you enough for your valuable insight on our project.

[rlkoshak]

@HestiaPi

Haven’t used it before. How can you react to specific file changes? Is it system-specific or OH-specific catches you are after?

Tripwire is usually used as a part of a host based protection system. Essentially what it does is create a database of hashes of all the files on your system (you usually customize the list to ignore stuff you don’t care about changing). Then you configure a cron job to periodically check the current files with that database and any new or changed files get sent as a report (I use email).

With this you can get a nice little email listing all the files that have changed on the OS within a day of the change happening (depending on how often you run the cron job).

The tool was designed to try and catch root kits and other attempts to hack a machine. I mostly use it to keep track of when automatic updates happen or stuff starts changing that shouldn’t which indicates other problems. I’m not all that worried about hackers on my LAN.

I’m mostly concerned with system file changes, but I could include openHAB file changes too. I’m usually messing around there though so the fact that files are changing isn’t very meaningful to me.

@HestiaPi

If you feel this will improve this you are the most appropriate person to take this decision and dig in.

My main OH instance is on a VM and I have a remote OH on an RPi 3, and this remote instance is doing less than even HestiaPi’s rules do. So I can’t say for sure if this will improve performance on the RPi 0 (which really isn’t a supported platform for OH). But on these machines, I’ve seen a real improvement in boot time by moving all my Things to JSONDB. I’ve done some experiments moving my Items to JSONDB as well, but I’m not ready to make that move yet as managing Item metadata isn’t as nice.

So far Python verses Rules DSL is kind of a wash. They both take a bit of time to parse. However, if you use primitives in Rules DSL it will greatly extend your boot time. I don’t think I saw many of those in HestiaPi’s Rules but didn’t look for them so may have missed some. However, I expect to see a speedup once I move the Python Rules from .py files to JSONDB. The UI for creating JSONDB Rules isn’t all that functional (a replacement to PaperUI is in the works for OH 3 which will address this and many of the other limitations of PaperUI) but the REST API is there and the execution engine itself is rock solid. I’ve been learning to write the JSON for these Rules by hand. It’s not too hard actually once you create a few.

But, one really important thing that comes with using Scripted Automation (e.g. Python Rules) is there is no longer a boot order problem. I’m not sure if this is an issue on the single core RPi 0 but sometimes Rules defined in .rules files can start executing before all the Items have been loaded and made available resulting in lots of errors and a need to reboot and cross your fingers that it doesn’t happen again.

@HestiaPi

Didn’t know of restoreOnStartup till now. We have already a few issues open and one is addressing the lack of non volatile settings. If this is something you want to address too please feel free, but I feel like I gave you my month’s worth of tasks from day 1.

This is super simple. Install the MapDB Persistence add-on. MapDB is a good choice as it only saves the most recent value. It never grows in size. It’s also embedded so there is no external software required.

The config file would look something like: mapdb.persist

Strategies {
  default = everyChange
}
Items {
        TempStepoint,HeatingMode  : strategy = everyChange, restoreOnStartup
}

Obviously you would want to include all the settings Items on the list. Or you can cheat and add a Group (e.g. Group Restore, and add all the Items you want to be restored on startup to that Group. Then use Restore* in place of the list of Items in the mapdb.persist file.

Since you are using MQTT, you could also make more liberal use of retained messages and pull down the last commanded state for the settings from the topics as well. But I think restoreOnStartup would be less work and work better over all.

Hopefully you can get this solved from the above. If not, it may be a few weeks before I can get a PR prepared. It shouldn’t take more than an hour to set up for someone how is already used to submitting PRs on this repo though.

@HestiaPi

We thought of having a switch between an ECO (higher) and a Comfort (lower) hysteresis mode under settings. Having their actual values configurable from the UI or not is simply a matter of balance between simple UI and customisation.

So another nice thing with the Scripted Automation, especially for a project like this, is that you can put global configuration parameters that you don’t necessarily want to expose on the UI and it doesn’t make sense to put into an Item in a configuration.py file. I could see the hysteresis range being defined in there. Most users won’t know and won’t care and for the few who do care, well they can get their hands dirty and edit a config file.

@HestiaPi

More homework here We saw this error but we couldn’t something about it. Suppressing it didn’t appear as an option then.

This is one of my first tasks. I’ll post the config lines here as soon as I get it sorted out. It should just be a few lines in /var/lib/openhab2/etc/org.ops4j.pax.logging.cfg. It uses logj4’s properties file format. At one point I was going to do the work to convert it to XML but never got around to it.

@HestiaPi

It is calling the openhab-cli to backup all openHAB files. Not system.

Yep, I managed to find the script and saw that. Given you are calling a script to do the backup and restore, it probably wouldn’t be a bad idea to include some other stuff like the UI configuration and Mosquitto config. There might be other stuff as well.

Essentially, you would want to backup and restore anything an advanced user might want to muck with. We would definitely appreciate that.

I need to look back and seen how much that script actually backs up in OH 2.4. There might be some stuff missing from /var/lib/openhab2 and there might be some extra stuff in /var/lib/openhab2 as well. I can’t remember when that was firmed up (OH 2.4 was released over a year ago). For example, you definitely don’t want the cache or tmp folders backed up but you definitely want the mapdb folder which is where all those restoreOnStartup values will reside.

I’ll definitely be coming back and start to make a few small PRs over time. I think I can help firm up the OH configuration part.

Speaking of which, you might look into the ZRAM setup openHABian uses to move the logging folders at least into a compressed RAM folder. I don’t know if there is enough spare RAM on the RPi 0 but it will do wonders to save the SD card from wearing out. There are other read only file system stuff you can do to make the RPi more resistant to file system corruption on a power loss given how difficult it would be to give the RPi a battery backup. Or turning off the OH logging entirely and let users re-enable it when needed. I’m not sure what else might be doing a lot of logging.

Anyway, more to follow.

[rlkoshak]

Christmas is early. I had a few minutes.

Add the following to the end of /var/lib/openhab2/etc/org.ops4j.pax.logging.cfg

# HestiaPi

# Filter out Asynchronous processing not supported warnings
log4j2.logger.org_eclipse_jetty_server_httpchannel.level = ERROR
log4j2.logger.org_eclipse_jetty_server_httpchannel.name = org.eclipse.jetty.server.HttpChannel

Alternative, log into the Karaf console:

ssh -p 8101 openhab@localhost

And issue the following command:

log:set ERROR org.eclipse.jetty.server.HttpChannel

This will add the above lines (minus the comments) to the logging config.

If you issue the command through the Karaf console it immediately takes effect. If you just edit the file, you may need to reboot.

I submitted this information as an issue. https://github.com/HestiaPi/hestia-touch-openhab/issues/17

I would have submitted a PR as well but there is nothing from /var/lib/openhab2 in the repo so adding this would be more than just adding a couple of lines to a config file in the browser.

[HestiaPi]

Do you see any reason why not adding this file to the repo?
/var/lib/openhab2/etc/org.ops4j.pax.logging.cfg

EDIT:
Ignore me… These lines need to be “added” to the existing file. I’ll get the installation script do that.