Decoding and replicating IR signal (part 1)

Some home automation, although this is my home away from home we're talking about here. At our cottage we have a Panasonic heat pump, now about 10 years old so at point it is up for replacement (I asked for service for it and technician pretty much told us that there's no point, when it stops working you just replace it). 

For now I have used some remote controlled sockets (with good results) to control heating there. Direct heat is somewhat expensive though, and using said heat pump to do it would be more cost-effective. The problem is that it is controlled by proprietary IR remote, so I can't just tell it to switch to "+20C heat" .. 

Or can I?

I wanted to replicate IR with hardware I have already - effectively same I used for RF stuff. My idea had basically two steps;

1) Receive and decode code send by IR remote
2) Replicate said code.

Seems so simple. It turned out to be much less so.

My original idea was to make a simple IR receiver module (easy, there's plenty of schematics on net) and record sequence. IR signal has usually carrier (IR being turned on and off, generally at 40-80 kHz range) and data is modulated on top. In this case it (I found out after building signal receiver) was 75kHz-ish modulation, with PWM on top of it, starting with longer preamble.

My original idea was to modify same code I had used as RF receiver to serve as IR receiver. Well, long story short, that did not work. RF receiver module did AM decoding, so I was receiving (relatively) clean 1 or 0 data, and only had to decode from there. IR didn't. I was receiving that carrier signal directly. And as it turns out, software was not fast enough for clean reception. It lost pulses too often to be reliable, so that idea was out.

That was last autumn. I ran out of time, winter was coming so I had to install system as-is to have even current control over it (and oh boy has it been great even at its current state; it dropped my power usage by full two thirds since system can keep direct electric heating off when it's warm enough outside for heat pump to manage alone).

I couldn't drop the idea of heat pump control however... (this is to say; to be continued)

 

Boredom is essential

Prove me wrong: Boredom, or idleness, is essetial for creativity.

Over the years I've noticed that it is the periods of boredom, or not having things to do, or in other words, being idle, are the ones that bring out the creativity in me. 

Lately I've been extremely busy just keeping up with work (quite successfully.)  Several indenpendent things happened at same time (virus that messed with world being just one of them) and caused massive work load for us. During this time I've been able to do just the essentials. Yes, I kept things rolling and lights on, but it's just doing what is required. There is no creativity, no new things involved.

On the other hand, when this virus first hit, our business was second in line to get hit (first being our customers.)  While this was not a financial problem for us, it meant that we got almost no phone calls, no messages, nothing. Suddenly lots of time previously taken by other things -- was free. I suddenly had (almost) nothing urgent to do!

It took me a week or so before I installed full development environment at my home computer and in few weeks more that things rolling on things I almost never do. I had fun literally playing around with game ideas, and while nothing came out of them, they still were relaxing distraction.

Whenever I have had extended down time something similar has happened every time. Sometimes work related (some low-priority thing I might not normally touch), but I also get started with projects I haven't had energy to start before, like game development. I never get very far there (down time is almost always limited) but at least I get something done.

Years ago I read some blog post from someone, I don't remember the details, but gist was that as a business owner the goal is to make yourself unnecessary. Make your employers do the work - you teach them how, they keep the business rolling, and you make sure they have the tools to do that. Maybe then I could start thinking about other things to get done.

At this moment, I feel that I am on route to that goal. There's just this small issue of needing good software people. These days they're not exactly easily available...

 

Server overload

I run a server for IoT-like devices. They send data to server, which verifies it (transmission errors with checksums and so on), checks that it isn't duplicate entry and then stores and acknowledges it to device, after which device marks it handled on its end. 

This system has been going on for almost ten years now, gradually growing by number of users and, by extension, data amount.

Some time ago this failed hard. Server overloaded and requests started timing out. Restarts helped for a while but since devices were still trying to send data, it very quickly fell again. After some while, even more devices had data to send but couldn't, and they really didn't have any throttling implemented so server could them to cool down.

Things were looking quite bad.

I had no real great tools to profile system, so I effectively had to resort to using top to see what's going on. And what was going on was that SQL server (not the MS product with similar name)  was using vast majority of CPU power.

So, I started think back to what code is doing.

1) Parse incoming message, doing validations and checksums
2) Find client device from database, discard data if it doesn't exist
3) Look through device's data, looking for duplicate for this data. Ignore data if duplicate found.
4) Insert new data to database
5) Send device acknowledge of data (in case of duplicate, data is acknowledged without action)

The process is like this for a reason. Basically I want that all data produced by devices is stored exactly once, and process assumes that there will be hiccups. Data getting corrupt during transmission, connections dropping for some reason in mid-transmit, acknowledgements not received and so on.

At this point I had strong suspicions on step 3. After few experiments (trying and timing those duplicate search queries) I indeed confirmed that this step was taking way too long, even with fully indexed device measurement data tables and query optimizations. This was Not Good.

I already knew what I would have to do. I just didn't want to do it, since it would make a small but not insignificant dent on reliability. I had to separate data reception from data storage. But there was no other option.

So I created a new table that contains just raw measurement data. The process above was split in two separate parts, running independently.

A1) Parse incoming data, validating etc.
A2) Store data to intermediate table
A3) Send device acknowledge of data

Then B, which is another script on cronjob. Architecturally this could be a continuosly running service which is notified by (A) via signal or such when there is data available. Cronjob was however quick and good enough solution here, causing latency of few minutes before received data appears to main database. Basically non-issue here.

B1) Load bunch of data from intermediate table
B2) Find client device from database, discard data if not found
B3) Look for duplicates, discard if found
B4) Insert data

And just like that, entire server issue was solved, with a cost of very slight reduction of data reliability (adding very unlikely but non-zero possibility that corrupt-in-transport data that somehow passes A1 gets through but fails either at B2/B3, without possibility of re-transmit).

Since then site has been chugging along nicely. I have also added some throttling to devices (by using remote update capability) that should in future ease the worst-case scenario. Switching to beefier server is another future-proofing option, which I need to keep an eye on in case it seems that things start get iffy again. And of course switching to using separate database and front-end servers.