After adopting a "wait and see" approach toward whether Elon's Folly would go through with the threat to cut off all free API access (and smiling a knowledgeable smile when the threat was walked back to some degree,) and determining, as best I can tell, the status lights are still working properly, I had another idea of something I could get my external brain to do for me instead of requiring me to both remember and be motivated to do it. I have a book of excerpts and quotes that I have been meaning to use more as meditative prompts, and I added looking in it at a random page as a tickybox in my daily spread. I wasn't getting consistent results, based on a host of factors, most importantly whether I had time to do the thing as part of my morning routine or before going to bed. So, rather than relying on my internal brain to do the habit thing, I thought it might be possible to make the external brain do it. If the external brain could pull a random quote from a quotes file, and then sling the selected piece of wisdom to my tablet at some point during the day, I'd be more likely to examine it and meditate on it.
As a proof of the concept, and because I also needed something like this in my life anyway, I constructed a message patterned on the advice that the protagonist's abuela gives in Kat Fajardo's Miss Quinces, set it to send to my tablet through a notification service, and then scrounged up some Python code that would work to randomize when the message would be sent - the automation fires at 6am, but there's a random delay of up to 12 hours after the firing before the command to send the message is executed. (This project is so very much me being the information professional and reusing other people posting code snippets in forums. So, so much.)
Having successfully proof-of-concepted the idea and watched it work (sometimes my affirmation fires before I get to work, and that's okay), I started trying to figure out the main event - a rotating piece of wisdom that would appear at a random time of the day and be something I could meditate on it keep in mind while going about my day, or using it for something to help with my day. As with so many of my brilliant ideas involving this smart home brain, what looks to be simple turns out to be anything but. Knowing how sensors work in Home Assistant, I had already decided that I was going to use JavaScript Object Notation (JSON) to establish key-value pairs and let them be interpreted as the attributes to a sensor, because Home Assistant likes JSON and knows how to parse it well. So I wrote up a few pieces to use as test data and then went to work on the part of figuring out how to randomize them.
A whole lot of people have written randomizers in every conceivable language, as implementations of generalized randomization algorithms, and some of the optimized randomization algorithms that require fewer passes through a file and less time spent parsing. I initially gravitated toward a Python interpretation, on the assumption that because Home Assistant is written in Python, I would be able to call the function from an integration that allows you to execute a command line action. Turns out
My initial idea was to have the shell script output the chosen random line to a file on the Home Assistant's space for serving static files, and then pointing a set of File sensors (meant for ingesting a sensor's attributes from a static file) to collect the attributes needed and display them. Because the File sensor only reads the last line of the file, the output of the script has to be a single line, so the input JSON has to be a single line (which uglifies it significantly.) The file output of the shell script worked perfectly, as all I needed to do to modify from the original implementation was to take the echo output and write it to the file on disk instead of stdout. The File sensor, as implemented, worked as described. However, the File sensor ingests the data, by default, and dumps it into the state attribute of the sensor. The state attribute of a sensor has a maximum limit of 255 characters. Which would truncate most of the wisdom text of it only displayed the first 255, but also actually forces an "unknown" state if the sensor tries to ingest anything over 255 characters. Given that the sensors are more likely to be ingesting small-character things like integers, floats, or single words as states, the character limit makes sense. Unfortunately, it also means I can't use the File sensor, because the File sensor has no way of accepting the long strings.
The accepted workaround to the character limit on sensor states is to format the sensor data in such a way that instead of trying to update the state attribute directly, the data is carried in an array / dictionary of attributes, which do not have a character limit, accompanying the new sensor state. The REST integration has an option called "json_attributes" that allow Home Assistant to know where the attributes coming along with a state change are located in the data file. (As well as being able to define where in the data file the new state information is.) The thing about the REST integration is that it assumes, correctly, that the data is coming from an Application Programming Interface or other service that implements REpresentational State Transfer (a RESTful API or other service.) The shell script and the file it was generating? Not RESTful. And I definitely didn't want to spin up a server just to serve this file when it was requested. That's a waste of resources.
Additionally, as I would eventually figure out, trying to proclaim something a as RESTful when it wasn't caused Home Assistant to get suspicious of itself for trying to access a resource that it wasn't authorized to, although I didn't fully understand the error message at the time. Home Assistant also knows how to receive and interpret JSON payloads directly from entities that have been authorized to communicate with its endpoints. The documentation also says that on top of the integration, Home Assistant exposes several APIs for communication using various protocols, libraries, or programs, including REST. I misinterpreted the documentation to mean that I should be able to define a RESTful sensor, use one of the API endpoints, and Home Assistant would create the sensor dynamically. What was happening was that Home Assistant was trying to access a protected resource without the proper authorization to do so. Now that I understand the error, if I provide the correct authorization in the headers, I can probably avoid the error, and possibly get the sensor to persist across reboots.
This still leaves me with the problem of how to get the data fed into the sensor ending, so I went back to the drawing board about what I was doing. Knowing that it's possible to dynamically create sensors through the use of a Long-Lived Access Token, I can still move forward. Those dynamically created sensors won't survive a restart of the system, but once created, they behave like any other sensor and their JSON is parsed with some sensible defaults, like, as I would find out, an array/dictionary titled "attributes" with key-value pairs contained within would be treated as sensor attributes and propagate as such. (I'm hoping that if I can define this sensor as RESTful, Home Assistant will both save the information better and then repopulate it after a reboot. Unfortunately, it didn't, so I still need to figure out how to serialize the data and read it back in after a reboot. But I did successfully managed to avoid the authorization error, so my understanding was at least correct about what happened there.)
In the script I was running, the chosen random line was stored in a bash variable. Writing the contents of the variable to a local file was easy enough. Many of the examples present for working with the http sensor used the example of curl, (sometimes styled cURL) a terminal program that can send http requests with headers and payloads to specified URLs. (I'm sure that someone is going to start eye-twitching at what happens next, but it's a communication from the device to another endpoint on the same device, so I feel reasonably confident that even if I'm engaging in bad practice, it's very self-contained bad practice.) Since the examples used curl, I made the assumption that it was installed on the Home Assistant system, set up the method, the headers with the token, and tried to get the variable's contents to get sent as the payload of the message. There's just one tiny problem with that which I took forever to figure out. (And by "figure out," I mean "use my information professional skills and somehow find a Stack Overflow n question with an actually helpful answer.") curl requires the payload/data (-d) to be surrounded by single quotes to transmit correctly. So if I pasted in a line directly, the message would send and the sensor values would propagate! Success! However, trying to surround the variable name with the single quotes would result in a "Invalid JSON specified" error from the endpoint. I assumed it was because the script was sending a string consisting of the variable's name, not the contents of the variable. (Correct, self!) When I tried to omit the single quotes, or include them in the variable file itself, and then invoke the contents of the variable by surrounding it with double quotes, as I had for the echo command that wrote the variable's contents to stdout or to a file name, curl protested that I had an unmatched brace in the URL position, or a nested brace in a different position, which told me it was trying to interpret the JSON as a URL instead of as a data payload. (Correct again, self. 2 for 2!) Since the static payload transmitted just fine, I came to the conclusion that my bash-fu was wrong and I needed to figure out what the correct syntax was to get the variable's contents to come out with the appropriate single quotes around it. Now that I understood what I wanted to figure out, I found the helpful Stack Overflow question that gave an example of the differences between single quotes and double quotes in bash syntax. (Link is to the Linux Documentation Project.) Which means a way to achieve the desired result for me is to do -d ''"$variable"'' (which may be bad or insecure bash) The double single quotes produces a single single quote, as best as I can tell, and the double single quote also allows bash to expand the variable in the double quotes, rather than treating anything between the double quotes only as a string, including its special characters. Once I've arranged the quoting this way, the curl command works the way I want it to, and the data from the randomly selected line feeds into the sensor, and I can see the gloriousness of it. Success!
I think I've got it taken care of at this point, but there's still one more curveball in store for me. After setting up the notification to fire randomly, I get one of the text-heavier blocks of text sent and…the story is truncated. Turns out that even with the expanded notification structure, there's a character limit built into Android notifications and several of the pieces of wisdom exceed that, so my initial idea of having all the wisdom contained in a pop-up that could be expanded and then dismissed turns out not to work at all. That said, there's always a backup idea somewhere, and in this case, the rummaging around in the forums tells me I can put widgets on my home screen from the app, including a template widget where I can define what I want to see on the screen, and, most critically, the widget's ability to display text is only limited by the size of the widget, rather than a fixed character limit. So I can set up the widget on the home screen, run a script to choose and then push the changed wisdom to the widget, and then use the same notification system for the affirmation to randomly pop a reminder to myself to look at and contemplate the wisdom on offer. Success!
…Almost. The widget editing screen, where I can put in the template I want to use to display things helpfully tells me that I can use HTML to help format the information that's coming in, so I can wrap the data in header tags and paragraph tags and use some escape codes to prettify things. After a few times of looking at test wisdom cases, I realize the line breaks haven't translated between Home Assistant and the widget. If I look at the attributes in the developer tools, my newline characters have been accepted and rendered. Those newline characters, however, mean nothing to an HTML rendering engine. Because there are no explicit HTML line breaks, the entire attribute is treated as a single unbroken chunk of text. Which leaves me with a decision. I can either decide that I'm going to format all of these wisdom pieces as explicit HTML and do whatever merry escaping needs to be done to make those characters render properly, or I can figure out some way of dynamically substituting or adding explicit line breaks in the rendering template for the widget. Option two is more appealing, of course, because it leaves the data as untouched as possible before the rendering step. And, a short amount of poking around later, I have another Python/Jinja2 code snippet that loads the text into a variable, then runs the "split" function against it, such that the single line of text gets broken into multiple lines of text, with each new line beginning at the point where the function runs into the newline special character. Then the template runs a for loop for each of the lines in the split text and inserts the line break tag <br /> before rendering the line. Now I have wisdom with proper line breaks displaying on my widget, and the entire text displayed without truncation and character limits.
[Victory fanfare goes here!]
About the only thing that I haven't fully figured out is how to consistently escape the single quote / apostrophe in such a way that won't make anything believe it's the premature end of the file. If I can figure that part out, then I can do a quick pass back over the wisdom file and put all the contractions back in and not have to resort to an HTML escape to get all the possessives to render correctly in the widget.
(I probably could have figured out at the start whether the Home Assistant system had fortune installed on it and gone in that direction, but I doubt it does. Plus, there are some advantages to doing it this way as a sensor with JSON, like being able to define key-value pairs rather than have to deal with the text as a single chunk.)
This is the end of this particular adventure in automation, getting an external brain to remind me of things to do, say, or contemplate throughout the day. It's probably not something in the specification or original conception of Home Assistant as a thing that can monitor IoT devices and then issue commands to them based on the data inputs and rules that I set up. Because Home Assistant is flexible, modular, and extensible, though, when I get wild ideas like this in my head, it turns out that there are pathways to implementation that I can find, code snippets to examine and reuse and modify, and as I try to make it work the way I want to, I learn a little bit more about the systems, their limitations, how to format things, how to construct queries, troubleshooting errors and issues, and more. It probably also helps that I'm going at it with the understanding that I can conceive of what I want in my head, that I have been successfully tinkering with machines and their configurations since the MS-DOS era, and the real difficulty of the task is finding enough material so I can create or modify code in such a way to make it work.
It works. There are things I want to do to improve it, but it works, and that's what's important.
[Edited to Add: I did eventually figure out the correct escape sequences for the single apostrophe and several other special characters - it turns out that newlines and double quotes require a double backslash so as to generate the right escapes (\n for newline, \" for a double quotes), and everything else manages with a single backslash to escape correctly.
Beyond that, when doing formating for the widget, I started trying to figure out how to substitute characters, since I was using Markdownish characters to indicate emphasis and the like, which led me to discover that there's a Jinja filter that lets you perform replacements of characters using regular expressions. After thinking through this to achieve these particular ends, the template that reminders on the weather has now replaced the formating characters and several of the punctuation ones with their proper HTML entities, as well as being able to substitute HTML line breaks for the newline characters. So I substituted in several regular expression substitutions for the split and for loop operation, which puts the tag in the proper place in the document as well. I get why regexes are extremely powerful, and also very easy to mess up. And this, I gathered a new piece of information, and same new tools, because I have a sneaking suspicious that entities like awk and sed and the like run really beautifully for someone who can command regular expressions.
[Remixed Victory Fanfare here.]]
As a proof of the concept, and because I also needed something like this in my life anyway, I constructed a message patterned on the advice that the protagonist's abuela gives in Kat Fajardo's Miss Quinces, set it to send to my tablet through a notification service, and then scrounged up some Python code that would work to randomize when the message would be sent - the automation fires at 6am, but there's a random delay of up to 12 hours after the firing before the command to send the message is executed. (This project is so very much me being the information professional and reusing other people posting code snippets in forums. So, so much.)
Having successfully proof-of-concepted the idea and watched it work (sometimes my affirmation fires before I get to work, and that's okay), I started trying to figure out the main event - a rotating piece of wisdom that would appear at a random time of the day and be something I could meditate on it keep in mind while going about my day, or using it for something to help with my day. As with so many of my brilliant ideas involving this smart home brain, what looks to be simple turns out to be anything but. Knowing how sensors work in Home Assistant, I had already decided that I was going to use JavaScript Object Notation (JSON) to establish key-value pairs and let them be interpreted as the attributes to a sensor, because Home Assistant likes JSON and knows how to parse it well. So I wrote up a few pieces to use as test data and then went to work on the part of figuring out how to randomize them.
A whole lot of people have written randomizers in every conceivable language, as implementations of generalized randomization algorithms, and some of the optimized randomization algorithms that require fewer passes through a file and less time spent parsing. I initially gravitated toward a Python interpretation, on the assumption that because Home Assistant is written in Python, I would be able to call the function from an integration that allows you to execute a command line action. Turns out
- the implementation I was looking at used a fixed randomization seed, and therefore would return the same result every time
- I couldn't invoke a Python interpreter with the command line integration. There is a custom component from the community store that allows for this, but I'm trying to keep my dependence on custom components small, just so I don't end up with code that isn't maintained performing vital functions. (And because I don't know enough about Python and coding to do the really silly thing and take over a project if it goes unmaintained.)
My initial idea was to have the shell script output the chosen random line to a file on the Home Assistant's space for serving static files, and then pointing a set of File sensors (meant for ingesting a sensor's attributes from a static file) to collect the attributes needed and display them. Because the File sensor only reads the last line of the file, the output of the script has to be a single line, so the input JSON has to be a single line (which uglifies it significantly.) The file output of the shell script worked perfectly, as all I needed to do to modify from the original implementation was to take the echo output and write it to the file on disk instead of stdout. The File sensor, as implemented, worked as described. However, the File sensor ingests the data, by default, and dumps it into the state attribute of the sensor. The state attribute of a sensor has a maximum limit of 255 characters. Which would truncate most of the wisdom text of it only displayed the first 255, but also actually forces an "unknown" state if the sensor tries to ingest anything over 255 characters. Given that the sensors are more likely to be ingesting small-character things like integers, floats, or single words as states, the character limit makes sense. Unfortunately, it also means I can't use the File sensor, because the File sensor has no way of accepting the long strings.
The accepted workaround to the character limit on sensor states is to format the sensor data in such a way that instead of trying to update the state attribute directly, the data is carried in an array / dictionary of attributes, which do not have a character limit, accompanying the new sensor state. The REST integration has an option called "json_attributes" that allow Home Assistant to know where the attributes coming along with a state change are located in the data file. (As well as being able to define where in the data file the new state information is.) The thing about the REST integration is that it assumes, correctly, that the data is coming from an Application Programming Interface or other service that implements REpresentational State Transfer (a RESTful API or other service.) The shell script and the file it was generating? Not RESTful. And I definitely didn't want to spin up a server just to serve this file when it was requested. That's a waste of resources.
Additionally, as I would eventually figure out, trying to proclaim something a as RESTful when it wasn't caused Home Assistant to get suspicious of itself for trying to access a resource that it wasn't authorized to, although I didn't fully understand the error message at the time. Home Assistant also knows how to receive and interpret JSON payloads directly from entities that have been authorized to communicate with its endpoints. The documentation also says that on top of the integration, Home Assistant exposes several APIs for communication using various protocols, libraries, or programs, including REST. I misinterpreted the documentation to mean that I should be able to define a RESTful sensor, use one of the API endpoints, and Home Assistant would create the sensor dynamically. What was happening was that Home Assistant was trying to access a protected resource without the proper authorization to do so. Now that I understand the error, if I provide the correct authorization in the headers, I can probably avoid the error, and possibly get the sensor to persist across reboots.
This still leaves me with the problem of how to get the data fed into the sensor ending, so I went back to the drawing board about what I was doing. Knowing that it's possible to dynamically create sensors through the use of a Long-Lived Access Token, I can still move forward. Those dynamically created sensors won't survive a restart of the system, but once created, they behave like any other sensor and their JSON is parsed with some sensible defaults, like, as I would find out, an array/dictionary titled "attributes" with key-value pairs contained within would be treated as sensor attributes and propagate as such. (I'm hoping that if I can define this sensor as RESTful, Home Assistant will both save the information better and then repopulate it after a reboot. Unfortunately, it didn't, so I still need to figure out how to serialize the data and read it back in after a reboot. But I did successfully managed to avoid the authorization error, so my understanding was at least correct about what happened there.)
In the script I was running, the chosen random line was stored in a bash variable. Writing the contents of the variable to a local file was easy enough. Many of the examples present for working with the http sensor used the example of curl, (sometimes styled cURL) a terminal program that can send http requests with headers and payloads to specified URLs. (I'm sure that someone is going to start eye-twitching at what happens next, but it's a communication from the device to another endpoint on the same device, so I feel reasonably confident that even if I'm engaging in bad practice, it's very self-contained bad practice.) Since the examples used curl, I made the assumption that it was installed on the Home Assistant system, set up the method, the headers with the token, and tried to get the variable's contents to get sent as the payload of the message. There's just one tiny problem with that which I took forever to figure out. (And by "figure out," I mean "use my information professional skills and somehow find a Stack Overflow n question with an actually helpful answer.") curl requires the payload/data (-d) to be surrounded by single quotes to transmit correctly. So if I pasted in a line directly, the message would send and the sensor values would propagate! Success! However, trying to surround the variable name with the single quotes would result in a "Invalid JSON specified" error from the endpoint. I assumed it was because the script was sending a string consisting of the variable's name, not the contents of the variable. (Correct, self!) When I tried to omit the single quotes, or include them in the variable file itself, and then invoke the contents of the variable by surrounding it with double quotes, as I had for the echo command that wrote the variable's contents to stdout or to a file name, curl protested that I had an unmatched brace in the URL position, or a nested brace in a different position, which told me it was trying to interpret the JSON as a URL instead of as a data payload. (Correct again, self. 2 for 2!) Since the static payload transmitted just fine, I came to the conclusion that my bash-fu was wrong and I needed to figure out what the correct syntax was to get the variable's contents to come out with the appropriate single quotes around it. Now that I understood what I wanted to figure out, I found the helpful Stack Overflow question that gave an example of the differences between single quotes and double quotes in bash syntax. (Link is to the Linux Documentation Project.) Which means a way to achieve the desired result for me is to do -d ''"$variable"'' (which may be bad or insecure bash) The double single quotes produces a single single quote, as best as I can tell, and the double single quote also allows bash to expand the variable in the double quotes, rather than treating anything between the double quotes only as a string, including its special characters. Once I've arranged the quoting this way, the curl command works the way I want it to, and the data from the randomly selected line feeds into the sensor, and I can see the gloriousness of it. Success!
I think I've got it taken care of at this point, but there's still one more curveball in store for me. After setting up the notification to fire randomly, I get one of the text-heavier blocks of text sent and…the story is truncated. Turns out that even with the expanded notification structure, there's a character limit built into Android notifications and several of the pieces of wisdom exceed that, so my initial idea of having all the wisdom contained in a pop-up that could be expanded and then dismissed turns out not to work at all. That said, there's always a backup idea somewhere, and in this case, the rummaging around in the forums tells me I can put widgets on my home screen from the app, including a template widget where I can define what I want to see on the screen, and, most critically, the widget's ability to display text is only limited by the size of the widget, rather than a fixed character limit. So I can set up the widget on the home screen, run a script to choose and then push the changed wisdom to the widget, and then use the same notification system for the affirmation to randomly pop a reminder to myself to look at and contemplate the wisdom on offer. Success!
…Almost. The widget editing screen, where I can put in the template I want to use to display things helpfully tells me that I can use HTML to help format the information that's coming in, so I can wrap the data in header tags and paragraph tags and use some escape codes to prettify things. After a few times of looking at test wisdom cases, I realize the line breaks haven't translated between Home Assistant and the widget. If I look at the attributes in the developer tools, my newline characters have been accepted and rendered. Those newline characters, however, mean nothing to an HTML rendering engine. Because there are no explicit HTML line breaks, the entire attribute is treated as a single unbroken chunk of text. Which leaves me with a decision. I can either decide that I'm going to format all of these wisdom pieces as explicit HTML and do whatever merry escaping needs to be done to make those characters render properly, or I can figure out some way of dynamically substituting or adding explicit line breaks in the rendering template for the widget. Option two is more appealing, of course, because it leaves the data as untouched as possible before the rendering step. And, a short amount of poking around later, I have another Python/Jinja2 code snippet that loads the text into a variable, then runs the "split" function against it, such that the single line of text gets broken into multiple lines of text, with each new line beginning at the point where the function runs into the newline special character. Then the template runs a for loop for each of the lines in the split text and inserts the line break tag <br /> before rendering the line. Now I have wisdom with proper line breaks displaying on my widget, and the entire text displayed without truncation and character limits.
[Victory fanfare goes here!]
About the only thing that I haven't fully figured out is how to consistently escape the single quote / apostrophe in such a way that won't make anything believe it's the premature end of the file. If I can figure that part out, then I can do a quick pass back over the wisdom file and put all the contractions back in and not have to resort to an HTML escape to get all the possessives to render correctly in the widget.
(I probably could have figured out at the start whether the Home Assistant system had fortune installed on it and gone in that direction, but I doubt it does. Plus, there are some advantages to doing it this way as a sensor with JSON, like being able to define key-value pairs rather than have to deal with the text as a single chunk.)
This is the end of this particular adventure in automation, getting an external brain to remind me of things to do, say, or contemplate throughout the day. It's probably not something in the specification or original conception of Home Assistant as a thing that can monitor IoT devices and then issue commands to them based on the data inputs and rules that I set up. Because Home Assistant is flexible, modular, and extensible, though, when I get wild ideas like this in my head, it turns out that there are pathways to implementation that I can find, code snippets to examine and reuse and modify, and as I try to make it work the way I want to, I learn a little bit more about the systems, their limitations, how to format things, how to construct queries, troubleshooting errors and issues, and more. It probably also helps that I'm going at it with the understanding that I can conceive of what I want in my head, that I have been successfully tinkering with machines and their configurations since the MS-DOS era, and the real difficulty of the task is finding enough material so I can create or modify code in such a way to make it work.
It works. There are things I want to do to improve it, but it works, and that's what's important.
[Edited to Add: I did eventually figure out the correct escape sequences for the single apostrophe and several other special characters - it turns out that newlines and double quotes require a double backslash so as to generate the right escapes (\n for newline, \" for a double quotes), and everything else manages with a single backslash to escape correctly.
Beyond that, when doing formating for the widget, I started trying to figure out how to substitute characters, since I was using Markdownish characters to indicate emphasis and the like, which led me to discover that there's a Jinja filter that lets you perform replacements of characters using regular expressions. After thinking through this to achieve these particular ends, the template that reminders on the weather has now replaced the formating characters and several of the punctuation ones with their proper HTML entities, as well as being able to substitute HTML line breaks for the newline characters. So I substituted in several regular expression substitutions for the split and for loop operation, which puts the tag in the proper place in the document as well. I get why regexes are extremely powerful, and also very easy to mess up. And this, I gathered a new piece of information, and same new tools, because I have a sneaking suspicious that entities like awk and sed and the like run really beautifully for someone who can command regular expressions.
[Remixed Victory Fanfare here.]]
no subject
Date: 2023-02-18 04:39 pm (UTC)no subject
Date: 2023-02-18 05:03 pm (UTC)And using an external brain for reminders is something I'm used to doing, usually in a calendar app or similar, so getting it to do this specific thing is an extension of what I'm already used to.
no subject
Date: 2023-02-19 02:43 am (UTC)no subject
Date: 2023-02-19 05:55 am (UTC)no subject
Date: 2023-02-19 09:52 am (UTC)no subject
Date: 2023-02-19 04:43 pm (UTC)no subject
Date: 2023-02-19 04:54 pm (UTC)no subject
Date: 2023-02-19 05:14 pm (UTC)no subject
Date: 2023-02-19 06:14 pm (UTC)