Dirty C

Sometimes C can make me feel ... dirty ... for writing code that should be relatively straightforward.

Case in point:

if ( ((unsigned int (*)(void))val)() )
   return;

I actually had to look up the syntax for that since this was the first time I actually have done this kind of operation, ever. Guess there is first time for everything...

(what? What this does, you want to know? I'll explain that in the next post ...)

Interesting bug tracking session.

Recently I was testing a new a board I was working on. I already knew that I need to make some minor changes here and there before committing to next iteration I wanted to check the remaining parts of the board. So far there was nothing major, few component changes, few tweaks here and there.

So I was going through thing quickly. Write small software stub to test a piece of hardware, upload, run, seems to be working, next... Until I got to a certain radio module. And it just crashed the software. Based on software traces (see this post about that) it seemed to crash in middle of short delay loop. Same loop that is used everywhere in same code.

...what?

This just didn't make any sense. I have said it before, I am not fan of embedded debuggers, but once again I had to dive in. So I traced the offending code and whenever software hit this one specific instruction it crashed.

The instruction: (ARM assembly, exact format IIRC)

str r3,[r0, #20]

Store contents of register r3 to memory, address (r0+20).

...what?

Everything about this instruction is correct. Register being stored, address where it is to be put to, no errors there. And still it just crashes. Every single time. Mind you, at this point I had fiddled with the code somewhat, so in the meantime the timings had changed slightly - yet it was crashing on this same instruction every time.

At this point I went back to my earlier hunch I had dismissed once already (after quick measurement that turned up nothing then.)  As this is a radio module we're talking about, it's a bit power-hungry. Especially at startup. And there are some large-ish (relatively speaking) capacitors there too. And part of the code that crashed was controlling the FET that fed power to the module...

Aaand yes, that was the issue, after a bit more careful measurement attempt. When power was enabled to the FET, there is a very short - few hundred microseconds long - drop in 3.3v rail. Not even a large drop - 300mV or so - but just enough to trigger external voltage monitor/reset circuit of the main MCU. Causing "crash". Not the kind of crash I was expecting but anyway.

So, is this board pulling that much current during startup that it actually makes power rail sag that badly? Doesn't sound likely but it seemed so. So I added some caps to supply side of mentioned FET. Then some more caps. And more. No change.

Feed the chip power directly. It takes 200mA, 500mA, 1A? No, this can't be right anymore. So on another hunch, take out multimeter and do some resistance measurements... Aaaand right. Supply of this module is shorted to ground.

This is professionally assembled board, by the way, with machine pick-and-place and reflow soldering. Should be all good, but guess not.

This module comes in QFN-type package, meaning that contacts are underneath it, completely inaccessible. Pitch isn't exactly tiny - 1.25mm, but the contacts have just 0.25mm of space between them. Very very close to each other. As it (obviously) had to be, this board I was testing happened to have mis-placed module, approximately 0.5mm away from where it should be. Just great.

Annoying this with these QFNs is that they're extremely hard to desolder, at least without destroying them in the process with excessive heat. Essentially impossible with my tools, unless I wanted to destroy most of the board too - and I do have relatively good hot air soldering station.

So I am kinda out of good options here. If this were production run, I'd be having a hard discussion with manufacturer (about 40% of the boards made had this issue), but as this is just prototype, it might be easier to just junk bad boards and try to figure out how to improve footprint of this part to be easier to solder. Not an easy job, there.

Power supply #2

I've had this lab power supply for close to ten years now. It was pretty cheap, with pretty typical specs (3 outputs, 1 with fixed 5v output, two adjustable 0-30v and 0-3A), and it wasn't used much. Lately it started to annoyingly click its relays when nothing was going on (supply was good all the time), so I moved it to aside and got another cheap one, with similar specs. After all, this was not used much so why spend too much...

Then that one broke too (not long after warranty period. Figures...)  Now it shows maximum current all the time and limits voltage accordingly. So it's useless. And I found out that no, that power supply was actually used a lot. All the time, actually, but since things I used it for were simple and brief - testing small things and what not - those didn't feel much. Until I didn't have that supply any more.

Suddenly things started to get difficult to get done, with that one testing station effectively out of order.

Guess it's time to find yet another supply. Maybe one that is bit more reliable (i.e., expensive) this time...

BT121 evaluating kit update

Bluegiga - now owned by Silicon Labs - makes pretty nice RF modules. I think I've mentioned their WLAN modules previously; not as cheap is those ESP8266-based ones, but at least these come with pre-tested and with actual paperwork needed for properly done CE marking & DoC (although I've heard that some ESPs these days do have necessary CE paperwork too)

Now, I just bought evaluation kit for BT121 (Bluetooth module), hoping to be able to use it to develop my new embedded BT library on PC without having to build actual hardware for it yet. When I can connect development kit with USB to PC and write and run the software there, it makes things so much easier.

I plugged USB in the development kit and ran their own interactive BGTool. Nifty one, that; it allows you to interactively run commands to the module, showing content of command, response and event packets, making starting up with module much easier as you don't necessarily need to read through full loooong API manual to get things started. 

Except no. Didn't work.

The development kit is by default in demo mode where you can connect to it various ways - remotely, serially and whatnot - but not control it directly via API like I wanted to. And the manual says absolutely nothing on how you can actually write - or even upload - your own software package to it!

I almost - almost - wrote entire thing off as wasted money. Just 100-ish € or so, but nevertheless.

Now, the very basic setup the chips (as in, on reel intended for PCB assembly) would be more than sufficient for my needs, though. 115kbps serial link and raw API commands I want to use to control it. With that knowledge (okay, assumption) and few educated guesses I was able to upload the software on the kit anyway. Despite completely useless instructions. Nice.

So if you want to update it too, here's quick how-to:

Set up the board as quick start guide instructs (especially "USB to UART" converter, that's needed).
Set board to be powered by USB.
Plug out current sense jumper labeled BAT, below the BNC connector, and plug it in so that it shorts BOOT0 pin (slightly below and to the left of header you took it from) to 3V3.
Press "Reset module".
Open BGTool, then go to "View -> Upload tool".
Select "project.xml" as project, and "bt121_bgapi_uart115k" as binary.
Change port to correct one, and hit Upload.
And if everything goes well, you should see green bar flashing and shortly it's done.
Take out jumper from BOOT0 and replace it to its original position.
Reset module again.

And now you should be able to connect to it via Interactive view.

Where did my code crash?

If you don't have suitable JTAG debugger handy, program crash in microcontroller can be annoyingly difficult to trace, especially if it happens only occasionally. Here is one trick for tracking those crashes with relatively low overhead;

char dtraceBuff[32];

void dtrace(char *data)
{
  memcpy(dtraceBuff, data, 32);
}


void dtraceOut()
{
  dtraceBuff[sizeof(dtraceBuff)-1] = 0; 
  printf("DTrace: %s\n", dtraceBuff);
}

And now you can sprinkle dtraces liberally around the code;

dtrace("intHandler");
dtrace("intHandlerEnd");
dtrace("mainloop 1");
dtrace("mainloop 2");
...

And in the beginning of main() (after your output system is initialized, of course) put call  to dtraceOut(). Replace print here with your favorite method of getting debug data out of your MCU.

So, when the code crashes, you can see immediately that DTrace says "mainloop 1". So what happens between "mainloop 1" and "mainloop 2"? If the issue isn't immediately clear, put more outputs there and try again. Eventually you will find the crash point.

But wait, if you just copy paste above to your program, you will quickly find out that this doesn't actually work. No matter what happens, DTrace will only print out empty string. Why is this?

The C standard requires that all uninitialized data is initialized to zeros. So when program crashes, MCU is reset and the C startup code will wipe all data - including dtraceBuff here.

There are however ways to prevent this, and exact method depends on compiler used. For Microchip's XC16, there is persistent-attribute:

char dtraceBuff[32] __attribute__((persistent));

For GCC (ARM), I couldn't quickly find similar simple attribute to make this easy. So next best option is to play around with linker script and related __attribute__s, allocating part of RAM for persistent (non-cleared) data and putting your data there. This is however a bit more complex task than simple attribute, so I will not go into details in this post. Personally I have modified the startup code I use to not clear any variables - but if you do that, you absolutely must remember to initialize everything yourself. And this can be cause for more subtle bugs by itself.

There is also option of using nonvolatile memory for trace data, but I'd recommend not going that route unless you absolutely, positively have to. Nonvolatile memories typically can only handle so many writes before they are worn out (Flash some 10k erase cycles; EEPROM some 10M; MRAM and FRAM on the other hand have "infinite" life), and they (unless embedded to MCU core) are slower to access. Sometimes, however, that may be only option.

Super AAA

It seems I found an supercharged AA battery today.

While for moment this seemed very interesting, after few seconds of consideration I came into a conclusion that my multimeter is more or less at fault here.

This specific meter one is one I have at home. It retails (in brick-and-mortar store) for something like 20€ (or at least it was, some 10 years ago), and today you could get similar one for some 5-10€ or so on ebay. While it claims to be CAT III rated, I wouldn't let let it anywhere close of high voltages. In other words, it's a PoS that is barely sufficient for checking if a battery is full or empty. And apparently it can't do even that that well.

But I don't blame the meter for all that. I knew what I was buying when I got that one.

Now, if you take a closed look at the picture above, you may see a small "battery" symbol on the screen of the meter, meaning that its battery is almost empty. Unlike newer models, this one has toggle-button for power. You always must remember to turn it off after you're done with it.

What isn't immediately obvious is that the same battery symbol also means that the readings aren't to be trusted anymore, as we can see here. I haven't really bothered to check how these things work (in circuit level), but I guess that there is a reference voltage somewhere - generated from battery voltage by some kind of accurate reference, be it of shunt or whatever type - which isn't correct anymore due to low battery voltage and this is causing readings to be higher than they should be.

As proven by quick battery change. With fresh 9v battery the reading is much closer to what I'd expect it to be. Old battery read about 4,9v when I took it out so it clearly was at end of its life.

So, in short, if your multimeter reading seems suspicious, take another meter and cross-check. And never, ever, take these cheap ones anywhere close to mains voltages, for any reason. The protections they contain are simply insufficient and they will kill you for that.

This is supposed to be a server?

Annoyed ranting ahead. I've been incredibly busy lately and will be for some time still, so I guess there'll be no exciting posts coming anytime soon.

I just bought a new server to our office. This is essentially backup machine, in case main system lets the operating smoke out, with planned use for other, less important purposes like staging/testing and whatnot in the mean time.

Since this is supposed to be server, it will eventually be headless, accessible via SSH key login only. Before that, however, I do need monitor of OS installation and so on.

But here I hit a problem: Some un-friggin-believeable idiot has decided to equip this server with no other display connections than two Displayport 1.2  connectors. Which cannot even be converted to more useful signals - like, oh, HDMI - without nothing less than active converters (so no, DP-HDMI cable won't just work. I tried.)

And really, two displayports? What is this machine supposed to be, some movie prop with separate huge monitors scrolling text logs for anyone to see?

Guess what: I don't have even one display that has displayport here anywhere. And by quick review, displays that have DP aren't exactly the cheapest ones there are. I don't like the idea of having to get a new display just for this, but damn it if I just have to... Or maybe I'll have to get a converter. We'll see...