Debugging Applications in Production

This evening, I noticed that an application running in a cluster of machines I look after was behaving oddly. It was using huge amounts of RAM and then crashing. What made this even more unusual was the fact that the code was written in Elixir and had been in production use for several years without any issue.

I looked in the server logs but could not see anything out of the ordinary, so I connected to the remote Elixir console and poked around a bit. In the end I came up with a nice little code fragment to find processes that are using lots of memory and produce a stack trace to help with tracking down the issue.

Process.list() |> Enum.map(fn p -> {p, :erlang.process_info(p, :memory)} end) |> Enum.filter(fn {pid, {memory, v}} -> v > 10000000 end) |> Enum.map(fn {pid, {_, m}} -> {pid, m, pid |> Process.info(), pid |> Process.info(:current_stacktrace)} end)

With the output from the above line in hand fixing the issue was fairly simple 🙂

ESP32 Nabaztag

My poor little Nabaztag rabbit is feeling the heavy weight of years. Modern WiFi means it is having trouble connecting to newer access points so I need another solution. With that in mind I decided to have a go at replacing the guts with something a little more modern and the ESP32 sounded perfect. I’ve gotten the firmware to compile and run on the the ESP32 but it needs more RAM for it to be fully functional. The ESP32 WROOM modules don’t have enough but the WROVER modules should be pleanty big enough. Hopefully once my WROVER module arrives I should be able to continue the development.

Roomba running on a Makita 18v cordless drill battery

I just got myself a nice new Makita 18v drill. It’s really nice 🙂

On thing about my battery operated tools I’ve noticed is that I tend to have one battery in the device and another charging. Since I don’t used them very often they batteries tend to degrade after a while, so I was thinking I could use the battery in something that I would run often to keep the battery in a nicer condition. It occurred to me that I have a pair of old Roomba’s in the shed that are just sitting there because the batteries are dead. What if I could use a cordless drill battery instead? I got a Makita battery adapter on Ebay and sacrificed one of the dead Roomba batteries to allow me to connect the two up. Now the interesting thing is the Roomba batteries are 14.4v and the Makita battery is 18v so I was expecting a bang and some smoke, but the Roomba seems to like the new batteries! I’ve since done some reading and apparently there is a version of the Roomba called the Create that can be run on 12 AA batteries. That would be the equivilent of 18v so it looks like the Roomba is built to take these kinds of voltages. I’ve yet to let the Roomba run through a full cycle as it is kind of noisy, but maybe later…

Windows

I’ve been running Macintosh machines for quite some time as my primary development environment. Recently I’ve been having to spend a lot of my time running Windows for my work and I’m finding it a lot less painful than I thought I would. Windows 10 has vastly improved over Windows 8 (although the bar was set very low on that on).

The Windows Subsystem for Linux makes doing a lot of things that were tricky under Windows in the past much easier. There are a few applications like Omnigraffle that I miss and I’ve yet to find an email client that feels right, but the thing I’m missing the most is my own QueryThing application that I’ll release for MacOS one of these days. There are many database query tools available for Windows but none of them works quite the way I want to work.

Now all I have to do is rewrite QueryThing to be a cross platform tool and I’ll be happy wherever I have to code 😉

Common Lisp, ASDF, Quicklisp and Windows

Today I spent quite a bit of time trying to get ASDF to find my local packages on my Windows development machine. It turns out that since ASDF 3.1.5 the place to put the config files has moved. Now the config file needs to be placed in $HOME/AppData/Local/config/common-lisp/source-registry.conf.d/
There’s a couple of hours of my life I’m never getting back 😉

Common Lisp, OpenGL and CEPL

Yesterday I spent a couple of hours playing with Common Lisp and OpenGL. It’s been a while since I’ve had a chance to play like this and I had a good time. I really need to do it more often 😉

I’ve been watching the CEPL videos on youtube for a while and wanted to give it a go and since I’ve gotten this lovely new HP 840 to work with it seemed like now was a good time to make it do something other than run Oracle database VMs.

The results of my efforts are here.

Data rescue

As with most “computer” guys (and gals), I often get asked to look at computer problems. This latest one has been interesting. A friend overseas had their USB hard disk (WD My Passport Ultra) fail so she shipped it to me to take a look at. When I plugged it into a Mac I got nothing, however the device did show up in the system report. Next I plugged it into a Linux box and got some strange errors in the kernel log about not being able to set the device configuration. I looked on the WD web site and found there was a firmware update for the drive but it could only be applied from a Windows machine, so I booted up my Windows machine and ran the firmware update. Just before the update finished the drive suddenly popped into the explorer window. I followed the instructions in the firmware updater and shut down the machine, unplugged the device and booted up again. Unfortunately the device didn’t come up after doing this. I decided to have another go with the firmware updater and this time I did not shut down and so while the device is available I am madly copying as much data off as I can.

Initially I tried using the Windows explorer to copy the files, but since there were so many FS errors and timeouts this didn’t work so well. What I eventually ended up doing was writing a little C# app to copy the files and ignore as many errors as I can. So far it seems to be working. The drive appears to be running VERY slowly, but still actually reading the data. Strangely some files are copying at the expected speed. It’s very weird. My guess is this drive is very sick.

I’ve been a bit quiet of late

It’s been quite a while since my last post. I’ve been really busy with my work and haven’t had much time for my many hobbies of late.

Over the last weekend I managed to dig out my 3D printer that I’d been working on for a while and finally managed to get it going. There were a couple of problems with it. The first was caused by corrupted settings in the EEPROM which once fixed started giving good prints almost straight away. The second problem was that unless I flexed the controller board in just the right way no data was transmitted over the USB. This made reflashing the device really tricky as I had to hold the board in just the right position during the flashing process and it was very easy to get it wrong resulting in a corrupted firmware.

Today on my lunch break I got out the oscilloscope and was able to see the serial data on the processor pin (Atmel Mega2560). I then looked up the datasheet for the USB chip (another Atmel processor) and found that there was serial data appearing there as well. A quick little go with the soldering iron on that pin (fortunately an easy one to get to as it was on a corner of the chip) and the data started appearing on the USB as expected 🙂

 

IMG_1078

Apps

I’ve been making some apps recently.  One for the new Windows 8 modern UI and one for iOS using swift.  One was a joy to code, the other was not 😉

Swift is an interesting language and I look forward to writing more code in it.  C# is also a nice language, however writing code for “modern” Windows apps is much harder than it should be.  This is mainly because many of the more useful parts of the .Net API just are available any more 🙁  No System.Data was the real problem for me and meant that I had to roll my own database access layer on top of a SQLite database.  An app that should have taken a couple of days to whip up took a couple of weeks.  Every time I thought I was getting somewhere I would hit yet another assembly or class that wasn’t available, or worked slightly differently to the “normal” way things should work.

Both apps are working now, so I’m happy 🙂