![[personal profile]](https://www.dreamwidth.org/img/silk/identity/user.png){kind=small}
silveradeptThe last time we checked in on the automation efforts, I'd done some cron job work to keep my Pi-Hole updated when there was a new version, having it check and run once a week for new Pi-hole system software. This time around, we're back in Home Assistant, trying to get some useful information ingested for new functionality, and to refine one of the sensors to be a little more accurate.
So! The climate control systems here in the Dragon Conspiracy Territory work fine, but it also occasionally gets chilly enough at night that we can just pop open the windows and let the heat flow out, rather than trying to condition all of it all of the time. If there's enough of a temperature difference, the heat will flow out and the breeze will be appreciated.
To set up this particular automation, I started from a copy of the automation that I'd made that would suggest when it was an appropriate time to turn on the air conditioner on the inside and modified the values to be "the outside temperature is below 70 degrees F, the inside temperature is above 75 degrees F." Which is a good start, but we don't want to have the windows opening up just when the temperature dips below the threshold if it's going to shoot up later in the day, or the dip is temporary and it's going to go back up shortly. We want to be sure that the temperature is cooling off sufficiently that opening the windows will result in the desired temperature exchange. Thankfully, Home Assistant has a method to collect samples of a sensor and plot whether or not it's trending in a particular direction, and how sharply, over time. Which means we can be relatively certain that the temperature is on the downswing sharply enough, and for long enough, before triggering the automation to suggest opening the windows. (The household has at least a few 1776 fans, so I also had to snip a clip from the beginning of the movie, so the Home Assistant will chorus at us to open up a window when the automation is triggered.)
That said, the Dragon Conspiracy Territory is not immune to wildfire season, and we've had nasty ones in recent memory, so there needed to be an additional condition imposed, to make sure that we weren't being serenaded to open the window when the air quality outside is not what we want to let inside. The good thing is, one of the sensors we already had an API key for also publishes Air Quality Index (AQI) data, along with more specific concentrations of the tracked pollutants that's ingestible from a REST API. Working with a REST API at this point is becoming somewhat common for me, so after making sure I could hit the correct endpoint with the correct authorization and get back useful JSON, I started building my sensor in Home Assistant.
There was one complication, however - the data that came back from the endpoint had a few different arrays associated with it, and while the AQI data and pollutant data were in the same array, the sensor syntax for Home Assistant expects the information to be at most one more level deep from the point where it's told to look in the attributes definition of the sensor. So, if the attribute was set for "main", you could configure the sensor to look for the key "aqi" inside of "main" and return the value associated with it. But if you were looking at "main" and there was a sub-array called "second" and the "aqi" key was in "second," it's not possible (or at least, not easy) to reach into "second" from "main" and get the "aqi" key. Luckily, Home Assistant has anticipated the possible need to traverse the tree of a JSON payload delivered from a RESTful sensor to find the desired key-value pairs. By including an optional path statement, I could tell Home Assistant where to look in the payload for the attributes I was going to put to use. Which, of course, means learning the syntax of JSONPath, or, more practically, using an online path evaluator and pasting in the JSON I'd gotten from the successful endpoint hit at the beginning of the operation. Using the interactive evaluator, I was able to point the sensor at the right part of the data I wanted, and then build out sensors from the attributes of the main sensor I'd asked for. I'd used jq for this kind of local manipulation with the Pi-Hole API, so it wasn't completely out of the ordinary for me, but it still took me a couple tries before I had working data being fed in to the Home Assistant.
Once the sensors were up and running, I was able to add an additional condition to the Window-opening automation check to see if the AQI was sufficiently low that opening the windows wouldn't result in letting in wildfire smoke or other polluted air, even if it was cool air. It's a relatively complex set of conditions, relying on a device's temperature, data from outside temperatures, ingested through an API and plotted on a trend line, as well as the numeric value of the data itself, and AQI data also ingested through an API. It's four or five points that have to be tested when the automation is triggered. Which is easy comparison operations for Home Assistant, but it seems pretty magical to be able to gather in all this data and use it for the comparisons. And pretty magical that I was able to get it to work with only a few false starts and a little help from the outside tools for constructing the right JSONPath. (These are accomplishments, and it's why I'm documenting them, because my traitorous brain will tell me that it wasn't actually all that impressive if I didn't have to expend significant amount of learning or effort on getting it to work.)
In addition to that, we've been using the state of one of the API-called sensors to track how much our ISP goes on the blip, or slows down sufficiently as to be on the blip, even if the cable modem never blinks or otherwise goes through a reconnection phase where it's obvious that things have gone wrong. We've watched the device not move or register any kind of outage, even though we haven't been able to get any data to come down the pipe. Today, it seemed like there was a lot of blipping going on using the indicator lights, but the rest of the services and computers didn't seem to be as affected as to what was going on, so it's possible that the server where we were fetching API data from was having a glitch or some downtime and that was affecting the indicators. So I was asked to add a confirmation step to the process. The standard way of doing these kinds of confirmations in Home Assistant seems to be using a "ping" sensor and setting it up so the result of the ping controls something to indicate a lack of connectivity. (Many people use this as a way of setting up an automated reboot of their routers or cable modems when they lose the ping for a sufficiently long time.) To me, it feels somewhat wasteful to be constantly pinging a server at regular intervals, and the scope that we were working at would have the ping sensor pinging about once every minute to try and keep pace with and confirm the outages, since the rate limits for the API we're using as a proxy for network connectivity is at the moment quite generous.
So, instead, I decided to go about it in a valid, but probably backwards, way. Since Home Assistant does allow for shell scripts and shell commands to be executed, so long as they last less than 60 seconds, it is possible for me to set a simple script to fire a ping, pipe the result into grep to find a key phrase, cut out the extraneous letters so that the result I get is either a 0 or a 1 from the ping attempt, and then write that 0 or 1 to a file. So, I set a second automation that fires, as before, when the RESTful sensor goes to a state from a lack of data, and it calls the ping to confirm whether or not the Internet is out. The primary automation's trigger is changed to fire when the ping fails (0). It does a check to make sure the RESTful sensor is still out, and then it fires the indicator light to tell us there's an outage, however small it may be. When data comes back to the RESTful sensor, the light changes to indicate the resumption of connectivity, and things go on merrily. I think this sequence should work, but we'll see what happens, and whether the light changes as expected when the Internet blips. With the refined sensor, we should be able to exclude false positives from our data, unless both the place providing the API and the secondary ping's website are both down at the same time in an outage that doesn't affect other parts of the Web. This should be a rare occurrence. And then we can potentially yell at the ISP about their uptime and what they're doing to make it better.
[VICTORY FANFARE.]
I'd say that these things are becoming more common for me, but that's not quite true. It's because each of these new projects is building on something I've already done before that they seem to be easier to achieve. When I'm building the new sensors and the URIs to poke the endpoints with from the documentation provided and the code I've already put into place, it's copy-paste-tweak, and the new knowledge is in the tweaking. Or practicing various skills and thinking through the processes that I want to happen means the tweaking generally works once I've picked up the component that's new for this enterprise. It might be because I'm working with a project that has both documentation and a robust forum culture, so I can look at how someone else has achieved their results and then tweak accordingly for my own purposes.
If I really wanted to test myself and what I know and possibly some other things like whether I can learn or remember how to program in a language and then generate an application as well as play around with APIs, Space Traders is right there to use as a deep dive to build a client for the game. I have other things to do than build a client for a game just because I'm feeling hubristic about how I can work with APIs and parsing JSON.
So! The climate control systems here in the Dragon Conspiracy Territory work fine, but it also occasionally gets chilly enough at night that we can just pop open the windows and let the heat flow out, rather than trying to condition all of it all of the time. If there's enough of a temperature difference, the heat will flow out and the breeze will be appreciated.
To set up this particular automation, I started from a copy of the automation that I'd made that would suggest when it was an appropriate time to turn on the air conditioner on the inside and modified the values to be "the outside temperature is below 70 degrees F, the inside temperature is above 75 degrees F." Which is a good start, but we don't want to have the windows opening up just when the temperature dips below the threshold if it's going to shoot up later in the day, or the dip is temporary and it's going to go back up shortly. We want to be sure that the temperature is cooling off sufficiently that opening the windows will result in the desired temperature exchange. Thankfully, Home Assistant has a method to collect samples of a sensor and plot whether or not it's trending in a particular direction, and how sharply, over time. Which means we can be relatively certain that the temperature is on the downswing sharply enough, and for long enough, before triggering the automation to suggest opening the windows. (The household has at least a few 1776 fans, so I also had to snip a clip from the beginning of the movie, so the Home Assistant will chorus at us to open up a window when the automation is triggered.)
That said, the Dragon Conspiracy Territory is not immune to wildfire season, and we've had nasty ones in recent memory, so there needed to be an additional condition imposed, to make sure that we weren't being serenaded to open the window when the air quality outside is not what we want to let inside. The good thing is, one of the sensors we already had an API key for also publishes Air Quality Index (AQI) data, along with more specific concentrations of the tracked pollutants that's ingestible from a REST API. Working with a REST API at this point is becoming somewhat common for me, so after making sure I could hit the correct endpoint with the correct authorization and get back useful JSON, I started building my sensor in Home Assistant.
There was one complication, however - the data that came back from the endpoint had a few different arrays associated with it, and while the AQI data and pollutant data were in the same array, the sensor syntax for Home Assistant expects the information to be at most one more level deep from the point where it's told to look in the attributes definition of the sensor. So, if the attribute was set for "main", you could configure the sensor to look for the key "aqi" inside of "main" and return the value associated with it. But if you were looking at "main" and there was a sub-array called "second" and the "aqi" key was in "second," it's not possible (or at least, not easy) to reach into "second" from "main" and get the "aqi" key. Luckily, Home Assistant has anticipated the possible need to traverse the tree of a JSON payload delivered from a RESTful sensor to find the desired key-value pairs. By including an optional path statement, I could tell Home Assistant where to look in the payload for the attributes I was going to put to use. Which, of course, means learning the syntax of JSONPath, or, more practically, using an online path evaluator and pasting in the JSON I'd gotten from the successful endpoint hit at the beginning of the operation. Using the interactive evaluator, I was able to point the sensor at the right part of the data I wanted, and then build out sensors from the attributes of the main sensor I'd asked for. I'd used jq for this kind of local manipulation with the Pi-Hole API, so it wasn't completely out of the ordinary for me, but it still took me a couple tries before I had working data being fed in to the Home Assistant.
Once the sensors were up and running, I was able to add an additional condition to the Window-opening automation check to see if the AQI was sufficiently low that opening the windows wouldn't result in letting in wildfire smoke or other polluted air, even if it was cool air. It's a relatively complex set of conditions, relying on a device's temperature, data from outside temperatures, ingested through an API and plotted on a trend line, as well as the numeric value of the data itself, and AQI data also ingested through an API. It's four or five points that have to be tested when the automation is triggered. Which is easy comparison operations for Home Assistant, but it seems pretty magical to be able to gather in all this data and use it for the comparisons. And pretty magical that I was able to get it to work with only a few false starts and a little help from the outside tools for constructing the right JSONPath. (These are accomplishments, and it's why I'm documenting them, because my traitorous brain will tell me that it wasn't actually all that impressive if I didn't have to expend significant amount of learning or effort on getting it to work.)
In addition to that, we've been using the state of one of the API-called sensors to track how much our ISP goes on the blip, or slows down sufficiently as to be on the blip, even if the cable modem never blinks or otherwise goes through a reconnection phase where it's obvious that things have gone wrong. We've watched the device not move or register any kind of outage, even though we haven't been able to get any data to come down the pipe. Today, it seemed like there was a lot of blipping going on using the indicator lights, but the rest of the services and computers didn't seem to be as affected as to what was going on, so it's possible that the server where we were fetching API data from was having a glitch or some downtime and that was affecting the indicators. So I was asked to add a confirmation step to the process. The standard way of doing these kinds of confirmations in Home Assistant seems to be using a "ping" sensor and setting it up so the result of the ping controls something to indicate a lack of connectivity. (Many people use this as a way of setting up an automated reboot of their routers or cable modems when they lose the ping for a sufficiently long time.) To me, it feels somewhat wasteful to be constantly pinging a server at regular intervals, and the scope that we were working at would have the ping sensor pinging about once every minute to try and keep pace with and confirm the outages, since the rate limits for the API we're using as a proxy for network connectivity is at the moment quite generous.
So, instead, I decided to go about it in a valid, but probably backwards, way. Since Home Assistant does allow for shell scripts and shell commands to be executed, so long as they last less than 60 seconds, it is possible for me to set a simple script to fire a ping, pipe the result into grep to find a key phrase, cut out the extraneous letters so that the result I get is either a 0 or a 1 from the ping attempt, and then write that 0 or 1 to a file. So, I set a second automation that fires, as before, when the RESTful sensor goes to a state from a lack of data, and it calls the ping to confirm whether or not the Internet is out. The primary automation's trigger is changed to fire when the ping fails (0). It does a check to make sure the RESTful sensor is still out, and then it fires the indicator light to tell us there's an outage, however small it may be. When data comes back to the RESTful sensor, the light changes to indicate the resumption of connectivity, and things go on merrily. I think this sequence should work, but we'll see what happens, and whether the light changes as expected when the Internet blips. With the refined sensor, we should be able to exclude false positives from our data, unless both the place providing the API and the secondary ping's website are both down at the same time in an outage that doesn't affect other parts of the Web. This should be a rare occurrence. And then we can potentially yell at the ISP about their uptime and what they're doing to make it better.
[VICTORY FANFARE.]
I'd say that these things are becoming more common for me, but that's not quite true. It's because each of these new projects is building on something I've already done before that they seem to be easier to achieve. When I'm building the new sensors and the URIs to poke the endpoints with from the documentation provided and the code I've already put into place, it's copy-paste-tweak, and the new knowledge is in the tweaking. Or practicing various skills and thinking through the processes that I want to happen means the tweaking generally works once I've picked up the component that's new for this enterprise. It might be because I'm working with a project that has both documentation and a robust forum culture, so I can look at how someone else has achieved their results and then tweak accordingly for my own purposes.
If I really wanted to test myself and what I know and possibly some other things like whether I can learn or remember how to program in a language and then generate an application as well as play around with APIs, Space Traders is right there to use as a deep dive to build a client for the game. I have other things to do than build a client for a game just because I'm feeling hubristic about how I can work with APIs and parsing JSON.
