I've spent the past week troubleshooting a device which I shouldn't need to troubleshoot. It's an commercial device which we bought (several of) at work. They are not cheap. They have a Beaglebone Black inside, and one of the processes they have running on the Beaglebone consumes more and more memory over time, eventually running out after a few days, when it crashes and stops responding to my system, at which point, to the unitiated (which is everyone at work) it looks like my stuff is failing. I don't like that. And I'm not getting much help from the MFG of the devices, so I've been investing my own time trying to fix their code.
My boss called to check in on my progress and I spilled the beans about the whole thing and he started the dumb questions I was afraid of, about swapping out all the devices we have already installed (it would be a nightmare) and I was on the verge of offering to make rounds to all the local facilities every 24 hrs to cycle power to the devices, until the MFG can eventually figure what's wrong and issue a software update. I realized right before the words came out, that I can simply power the devices from a relay, controlled by the existing PLCs at every installation. I can automate the power cycle. It's obviously a kludge but not my fault.
I am glad he called. If he hadn't, I may have spent another week banging my head against the wall. Sometimes we get our heads buried so deep in a problem that we lose peripheral vision, and we need someone to come along and ask dumb questions to slap us out of it. Or at least I do. I suspect I'm not alone. That was my lesson learned from last week.
Maybe my next project should be finding some way to automate the asking of dumb questions. Maybe thermal camera could be linked to an Alexa device and pointed at my head to detect cranial overheating and start blurting out lines from the IT help desk script "is it plugged in?" "have you tried cycling power?" "What were you doing when it happened?" ...
My boss called to check in on my progress and I spilled the beans about the whole thing and he started the dumb questions I was afraid of, about swapping out all the devices we have already installed (it would be a nightmare) and I was on the verge of offering to make rounds to all the local facilities every 24 hrs to cycle power to the devices, until the MFG can eventually figure what's wrong and issue a software update. I realized right before the words came out, that I can simply power the devices from a relay, controlled by the existing PLCs at every installation. I can automate the power cycle. It's obviously a kludge but not my fault.
I am glad he called. If he hadn't, I may have spent another week banging my head against the wall. Sometimes we get our heads buried so deep in a problem that we lose peripheral vision, and we need someone to come along and ask dumb questions to slap us out of it. Or at least I do. I suspect I'm not alone. That was my lesson learned from last week.
Maybe my next project should be finding some way to automate the asking of dumb questions. Maybe thermal camera could be linked to an Alexa device and pointed at my head to detect cranial overheating and start blurting out lines from the IT help desk script "is it plugged in?" "have you tried cycling power?" "What were you doing when it happened?" ...