I have been spending a lot more time than I’d like debugging a problem with Heltec ESP8266 boards and the Arduino IDE.

I may be zeroing in on the problem, but mucking about in the deeper configuration of the Arduino IDE has given me new respect for the people who collectively maintain it. For most users, it will just work, and it make embedded program seem, and actually be, easy.

I remember earlier days when embedded was a black art practiced by a few twisted people in back rooms with secret documentation earned the really hard way. Unless you were a big customer, you couldn’t expect too much help with this or that SBC. The tricks to make things work were hungrily accumulated after trial and error. Complicated header files and arcane compiler invocations were painstakingly crafted and carefully preserved. It wasn’t something you could do casually.

But, with the Arduino bootloader and IDE, it was suddenly possible to do some pretty amazing stuff while concentrating mostly on the code. All of that other stuff… is still there, but the user doesn’t even know it exists. They don’t have to. It all happens like magic.

Until it doesn’t, and then, untangling can be a long, dark slog into the internal configurations that weren’t meant for ordinary humans to see.

So the ugly truth about the Arduino IDE is that behind the scenes, it can be very ugly—but the other truth is, you don’t even have to know about that if you just want to make your things work. Just avoid badly supported hardware, stick to the mainstream stuff, keep your IDE and OS updated properly, and make backups of working configurations before changing them.

There is ugliness, but the pretty curtains hide it and you don’t have to open them.

 

I knew a programmer who maintained that the purpose of writing an application is to hide complexity from the user. Sounds like the Arduino IDE is doing a good job hiding its uglier aspects of operation.

 

It does hide a lot of the dirty work for the ESP8266 in the various configuration menus. I downloaded the compiler and such from Espressif and want to try that route for my next project. I tried a couple examples from the download and found I have some configuring to figure out to make it all work

Although the Arduino IDE makes it easier there are some features it doesn't have that I have grown to like with Kate which I usually use for programming. It's just a matter of personal preference for me.

What sort of problems are you having (out of curiosity)?

 

It does hide a lot of the dirty work for the ESP8266 in the various configuration menus. I downloaded the compiler and such from Espressif and want to try that route for my next project. I tried a couple examples from the download and found I have some configuring to figure out to make it all work

Although the Arduino IDE makes it easier there are some features it doesn't have that I have grown to like with Kate which I usually use for programming. It's just a matter of personal preference for me.

What sort of problems are you having (out of curiosity)?

I just figured out the workaround, though not the cause. There is something not-quite-right about the Heltec distribution for the Arduino IDE. It doesn't properly work with upload.py which is a wrapper around esptool.py.

When the IDE tries to invoke upload.py there is an error emitted in the editor's output window:

pyserial or esptool directories not found next to this upload.py tool.
​

After some spellunking, I found the directory containing upload.py, and, guess what, pyserial/ and esptool/ are right there, "next to" upload.py. So, what is going on?

Frankly, I don't know, but looking into the upload.py program itself I found:

import sys
import os
import tempfile

sys.argv.pop(0) # Remove executable name
toolspath = os.path.dirname(os.path.realpath(__file__)).replace('\\', '/') # CWD in UNIX format
try:
    sys.path.insert(0, toolspath + "/pyserial") # Add pyserial dir to search path
    sys.path.insert(0, toolspath + "/esptool") # Add esptool dir to search path
    import esptool # If this fails, we can't continue and will bomb below
except:
    sys.stderr.write("pyserial or esptool directories not found next to this upload.py tool.\n")
    sys.exit(1)

cmdline = []
write_option = ''
write_addr = '0x0'
erase_addr = ''
erase_len = ''

As you can see, it is setting toolspath then attempting to add the pyserial/ and esptool/ directories to sys.path. Then it tries import esptool. The error message could be from any of the three statements failing, so I wrote a little while loop to emit the values in sys.path and both directories had been properly added, so it was failing to import.

I figured there must be something broken about Heltec's versions of those tools, so, in a fit of intuition, I commented lines 8 and 9. Tried to upload again, and huzzah, it worked perfectly. So, I have a workaround but I am not sure just what is wrong with the files provided by Heltec. The versions I installed myself in the system directories work just fine.

I might bother to figure it out, but probably not. It's working, I have notes, and I have other more interesting things to do.

 

it can be very ugly—but the other truth is, you don’t even have to know about that if you just want to make your things work

I've had several cases where adding trivial statements (like serial prints) causes a program to stop working. I've tried to reduce the code to something that I could submit as a bug report, but that always made the code start working. Because of that, I've adopted a methodology of versioning (using different sketch names) once I have something working and want to tweak it.

Even with all of its warts, I'm still using Arduino Uno's more because it can replace a lot of discrete logic chips. I bought some Raspberry Picos, verified I could program with the Arduino IDE, then stopped using them because they're 3.3V and I use 5V for most of my projects, so I'm back to Uno's so I don't have to deal with level shifting.

 

I also stopped going to the Arduino forum for information and assistance. I found many of the high post count members to be very condescending. I couldn't even make it to 100 posts.

 

I also stopped going to the Arduino forum for information and assistance. I found many of the high post count members to be very condescending. I couldn't even make it to 100 posts.

Don’t throw out the baby with the bath water. I treat the condescending posts with a little less respect. Ie, I look first to the other posts for a solution.

I had a strange bug in my last code. A forum member immediately caught the issue and I had a correct sketch with just one post in the Arduino forum.

 

Don’t throw out the baby with the bath water.

That ship has sailed. There is one thread with tips, unrelated to Arduino, that I check from time to time. It got broken during a forum update, but there's still an occasional gem.

 

I've read and used several Arduino books. The one that I advise for folks who have the basics down is a SAM's book, Sams Teach Yourself Arduino™ Programming in 24 Hours (pearsoncmg.com). 24 basically 1-hour lessons. It is not what I would call an Expert's book, but it does pull the curtain back a bit for those who know the basics. It is not what I would reccomend for someone to start learning Arduino with unless they are already into microprocessors. FWIW

 

I've read and used several Arduino books. The one that I advise for folks who have the basics down is a SAM's book, Sams Teach Yourself Arduino™ Programming in 24 Hours (pearsoncmg.com). 24 basically 1-hour lessons. It is not what I would call an Expert's book, but it does pull the curtain back a bit for those who know the basics. It is not what I would reccomend for someone to start learning Arduino with unless they are already into microprocessors. FWIW

The Arduino documentation here is the kind of stuff I was digging through. It's the kind of stuff that you hope is handled by the vendor who sold you the board, but sometimes with a change in OS, and version changes in the IDE, things break and no one in China is paying attention.

 

You buy clones with "a wing and a prayer" and hope they work most of the time. Who knows exactly how "compatible" they really are. But, hey, they are really cheap!

 

You buy clones with "a wing and a prayer" and hope they work most of the time. Who knows exactly how "compatible" they really are. But, hey, they are really cheap!

Most of the time I can make them work. Often, the key is jiggering those files hiding in the IDE mud.

 

I have never tried Python, but I do wonder what sys.path looks like. It seems like a string array of some sorts judging from the script. Maybe toolspath doesn't do what it is supposed to and making sys.path invalid. Maybe figure out how to print it out sometime and see what the result is.

At least you have it working for now and know where the problem is.

 

I have never tried Python, but I do wonder what sys.path looks like. It seems like a string array of some sorts judging from the script. Maybe toolspath doesn't do what it is supposed to and making sys.path invalid. Maybe figure out how to print it out sometime and see what the result is.

At least you have it working for now and know where the problem is.

It is an array, and it looked fine when I printed it. And, it did use the toolspath string properly to add entries pointing to the right places. It was the import esptool that failed when those directories were in the path.

I can only guess that versioning or something similar caused Python to barf when it tried to eat it. It was happy with the system versions, though.

This is the diagnostic loop I added to the error message:

while len(sys.path):
    thisarg = sys.path.pop(0)
    sys.stderr.write(thisarg + "\n")

And this is the output in the IDE:

/Users/yaakov/Library/Arduino15/packages/heltec-esp8266/hardware/esp8266/0.0.4/tools
/Library/Frameworks/Python.framework/Versions/3.9/lib/python39.zip
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/lib-dynload
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/esptool-2.8-py3.9.egg
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/ecdsa-0.18.0b2-py3.9.egg
/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/pyaes-1.6.1-py3.9.egg

 

It's not Python, and it's not C like a lot of people say. It is Arduino, and while it is similar (or a variant) to C it has some commands all its own. For me, it was a lot easier to learn than C. Maybe because I actually had something to DO with the code instead of just writing and compiling it to see if it ran. I always told people "I am not a programmer" but I really am. I don't build or write massive applications, but I've always written code to integrate and build subroutines for process control systems during my career.

 

It's not Python, and it's not C like a lot of people say. It is Arduino, and while it is similar (or a variant) to C it has some commands all its own. For me, it was a lot easier to learn than C. Maybe because I actually had something to DO with the code instead of just writing and compiling it to see if it ran. I always told people "I am not a programmer" but I really am. I don't build or write massive applications, but I've always written code to integrate and build subroutines for process control systems during my career.

He was referring to the esptool.py used in the ESP8266 toolchain. It's provided by Espressif, and for the Arduino IDE there i a wrapper called upload.py used to invoke it.

The Arduino code is a subset of C++ and the gnu g++ compiler is used to compile it. The libraries often have more C++ code variations in them, and you will see all of the libraries are <library>.cpp

 

If one wants all of the capabilities of the ESP8266 and ESP32, they would use the IDE provided by Espressif. The Arduino IDE was cobbled together to make the basic io and basic wifi work. Bits have been slowly fixed and enabled since. Last time I checked, there were interrupts and PWM issues still not addressed on Arduino IDE. Some of the lesser-needed bluetooth and wifi features ESP32 were also missing as of about April 2021 when I looked. I installed the Espressif ide and never looked back.

 

I never noticed the upload.py before in the Arduino IDE. After a little poking around I found it... it is one of the little hidden things. While attempting to upload into thin air (just the FTDI connected and no board) it shows no mention of upload.py... only esptool which appears to be the output of esptool itself if I remember right.

Interesting stuff... I learned a few things today. Thanks

 

I never noticed the upload.py before in the Arduino IDE. After a little poking around I found it... it is one of the little hidden things. While attempting to upload into thin air (just the FTDI connected and no board) it shows no mention of upload.py... only esptool which appears to be the output of esptool itself if I remember right.

Interesting stuff... I learned a few things today. Thanks

Yes, upload.py is just a wrapper for esptool.py to handle the requests from the idea by formatting the command line arguments with variables provided by it.

Unfortunately, the way I find out about these things is when they break. I am very familiar with that syndrome in my own work. When systems work well they are invisible, and you get little credit for making them work so transparently, but when they break they turn opaque and you get the blame for a system that doesn't work—no matter what tiny fraction of the time things are down, it is visible. Uptime is "normal".

This is why Scotty always tells Kirk the repair can't be done, then performs a "miracle" by doing it.

 

I’d like to make an observation.

Threads such as these feed into the myth that Arduinos are garbage. While I totally understand your post, it’s about a small subset of the Arduino IDE’s capability.

Noobies probably miss this point. I get it.