I am also trying this one today to see if anything else shows up, seems to be more advanced.
https://www.hhdsoftware.com/device-monitoring-studio
https://www.hhdsoftware.com/device-monitoring-studio
I don't know either what your log files mean, but I think this is showing what is happening in the driver on the PC side. I don't know that it actually represents the serial data sent/received. Notice I said "I don't know that it...." because I actually don't know what I'm looking at. But it smells funny. It smells like these comms might be communications between your PC and the FTDI chip over USB, maybe driver instructions for the FTDI chip and not necessarily representative of the serial comms issuing out from the FTDI chip.Attached is the log print outs from 3 different data views on the last program. The top of the file is connection and the bottom is just the program sending and updating the values. I am still unaware how to take what I see being sent and mimic it via python.
The scripts I've used in the past where dealing with much simpler context but seem so different from what I am seeing here.
import serial
ser = serial.Serial('/dev/ttyUSB1', baudrate=2400, timeout=5)
ser.write(bytes.fromhex('5c 00 3f 00 3f'))
response = ser.readline()
print(response.hex())
The cable that connects to the controller has 4 wires attached to the FTDI chip and usb connection. I could setup an extra arduino or raspberry pi to try and piggyback that serial connection if you think that would help.I don't know either what your log files mean, but I think this is showing what is happening in the driver on the PC side. I don't know that it actually represents the serial data sent/received. Notice I said "I don't know that it...." because I actually don't know what I'm looking at. But it smells funny. It smells like these comms might be communications between your PC and the FTDI chip over USB, maybe driver instructions for the FTDI chip and not necessarily representative of the serial comms issuing out from the FTDI chip.
You are using COM6 which is a FTDI virtual serial port on USB. Where is this FTDI chip? Inside the inverter unit? IOW are plugged into the inverter with a USB cable or are you using a an external FTDI USB serial converter to communicate with a hardware serial port on the inverter?
If there is an exposed hardware serial port I would recommend using a hardware sniffer to verify you are seeing the actual serial comms.
Also, serial.readline() assumes the termination character is '\n' (newline) which is pretty common but by no means absolute. Your device's protocol might specify some other character as a termination, in which case ser.readline() would never output anything until/unless by coincidence '\n' happened to be among the received bytes. If your device's protocol specifies another EOL character you'll need to use (ex) ser.read_until(b'\r') or (ex) ser.read_until(b'ff'). Or the protocol might not specify an EOL character at all. Some specify in the first byte(s) the length of the message to follow, and the message simply ends, so you have to parse the first byte and read until that number of bytes is received. Or there are even more schemes. It can be hell without the protocol specification.
But don't lose hope. maybe it's easy, you just don't have right data yet. Do you have the ability to sniff the actual serial lines with a hardware sniffer?
Perfect, yes that's what I would try. (Again, because I don't know what I'm looking at in your logs - could be that you already have the data but I don't recognize it)The cable that connects to the controller has 4 wires attached to the FTDI chip and usb connection. I could setup an extra arduino or raspberry pi to try and piggyback that serial connection if you think that would help.
The bus pirate looks interesting. I’ll try that and let you know what data I get. Thanks again for all your assistance with this.Perfect, yes that's what I would try. (Again, because I don't know what I'm looking at in your logs - could be that you already have the data but I don't recognize it)
For this I have used this cheap logic analyzer with sigrok PulseView. Also often recommended is bus pirate.
I've never used RPi or arduino as a sniffer. It's surely possible and I would expect you can find documentation on how to do it, but I don't think setting up either device as a normal serial device will work, as it will be expecting to participate in the conversation and cause problems.
@strantor, Just to confirm, I will use the bus pirate channel 0 piggy backed to the RX of the existing serial, and channel 1 of pirate piggy backed to TX. Run the sniffer program on the pirate and record the hex that is being sent by both devices via serial. With that information I should be able to mimic that by sending the exact same command that was sent from the PC to the controller from the pirate and receive the same response from the controller to know it is working correctly. Then I should hopefully get something from the controller that would match up to what I would see on the PC GUI program of the controller?The bus pirate looks interesting. I’ll try that and let you know what data I get. Thanks again for all your assistance with this.
Yes@strantor, Just to confirm, I will use the bus pirate channel 0 piggy backed to the RX of the existing serial, and channel 1 of pirate piggy backed to TX. Run the sniffer program on the pirate and record the hex that is being sent by both devices via serial. With that information I should be able to mimic that by sending the exact same command that was sent from the PC to the controller from the pirate and receive the same response from the controller to know it is working correctly.
Hopefully, yes. It may still require scaling and/or other conversionsThen I should hopefully get something from the controller that would match up to what I would see on the PC GUI program of the controller?
This is the data I gathered from the logic sniffer. Channel 1 is connected to tx which would be the controller and channel 2 is the rx which would be the computer program or software. The first set of files named connection, are when I connect to the com port the controller is connected to using the turbine software to auto connect. The 2nd set of files is the information page that shows all the controller data I would like to extract. The only values shown on the information page at the time of sniffing is battery V 53.2, Bat total KWH 0.4 Kwh, bat cap 100%. The rest of the values are 0. If you get a change to look at the readout and tell me what you think that would be much appreciated.Yes
Hopefully, yes. It may still require scaling and/or other conversions
Ok, here's what I did with your files (I only looked at the cyclic transmission files "information page from controller-hex.txt" and "information page from software-hex.txt").This is the data I gathered from the logic sniffer. Channel 1 is connected to tx which would be the controller and channel 2 is the rx which would be the computer program or software. The first set of files named connection, are when I connect to the com port the controller is connected to using the turbine software to auto connect. The 2nd set of files is the information page that shows all the controller data I would like to extract. The only values shown on the information page at the time of sniffing is battery V 53.2, Bat total KWH 0.4 Kwh, bat cap 100%. The rest of the values are 0. If you get a change to look at the readout and tell me what you think that would be much appreciated.
import pprint
def processLogFile(filename,source):
byteList = []
with open(filename) as txtFile:
for line in txtFile.readlines():
# The line will look like: 0.016639750000000,'255' (0xFF),,
# Split at the commas, this will look like: ['15.553970750000000', "'1' (0x01)", '', '\n']
cols = line.split(",")
if len(cols) != 4: # If the line contains an extra comma there will be issues
print("bad data:",cols)
# turn column 0 from '15.553970750000000' (string) into 15.55397075 (floating point number)
cols[0] = float(cols[0])
# turn column 1 from "'255' (0xFF)" (unusable string) to bytes object (
temp = cols[1]
cols[1] = cols[1].split("(")[1].rstrip(")") # this is now string "0xFF"
cols[1] = cols[1].split("x")[1] #this is now string "FF"
# (did not split on "x" originally because "x" might appear more than once)
cols[1] = int(cols[1],16) # convert "FF" (string) to 255 with 16 bit conversion
byteList.append([cols[0],cols[1]]) # we only care about the timestamp and the byte (int)
#cols[1] = bytes(cols[1])
#packet.append(cols[1])
#print(cols,temp)
packets = {}
thisPacket = bytearray(b'')
for i in range(0, len(byteList)):
if thisPacket == b'':
startTime = byteList[i][0]
# go thru the byte list one by one, concatenating the bytes onto the blank bytearray
thisPacket.append(byteList[i][1])
if i < len(byteList)-1:
# If this isn't the last byte, calculate the elapsed time
td = byteList[i+1][0] - byteList[i][0] # time delta between this byte and the next byte
else:
# but if it is, then set the time >.005 seconds to make sure the last packed is added
td=1
if td>0.005: # detect the end of a packet
# add the packet to the packets dictionary, using the timestamp of the final byte as the dict key
packets[byteList[i][0]] = {}
packets[byteList[i][0]]['packet'] = thisPacket
packets[byteList[i][0]]['startTime'] = startTime
packets[byteList[i][0]]['endTime'] = byteList[i][0]
packets[byteList[i][0]]['source'] = source
thisPacket = bytearray(b'')
# for packet in packets:
# print(packets[packet])
return packets
conversation = {}
controllerPackets = processLogFile("information page from controller-hex.txt","controller")
conversation.update(controllerPackets)
controllerPackets = processLogFile("information page from software-hex.txt","software ")
conversation.update(controllerPackets)
#pprint.pprint(conversation)
for k,v in sorted(conversation.items()):
print("at",str(v['startTime'])[0:7],",",v['source'],"said:",v['packet'])
Z:\G\solarData\venv\Scripts\python.exe Z:/G/solarData/solarData.py
at -0.0580 , software said: bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
at 0.0 , controller said: bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
at 2.16067 , software said: bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
at 2.21845 , controller said: bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
at 4.38062 , software said: bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
at 4.43841 , controller said: bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
at 6.59640 , software said: bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
at 6.65436 , controller said: bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
at 8.81719 , software said: bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
at 8.87531 , controller said: bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
at 11.0308 , software said: bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
at 11.0887 , controller said: bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
at 13.2529 , software said: bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
at 13.3107 , controller said: bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
at 15.4749 , software said: bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
at 15.5331 , controller said: bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
at 17.6919 , software said: bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
at 17.7501 , controller said: bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00')
Process finished with exit code 0
import struct
"""The only values shown on the information page at the time of sniffing is
battery V 53.2,
Bat total KWH 0.4 Kwh,
bat cap 100%.
The rest of the values are 0."""
controllerData =bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
softwareData = bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.')
# for bt in softwareData:
# print(bt)
print("len(controllerData) = ",len(controllerData), "and screenshot shows 24 values on the screen")
a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData)
print("could it be unsigned INTs? :",a)
a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData)
print("could it be signed INTs? : ",a)
print("moving first byte to end")
firstByte = controllerData[0]
controllerData = controllerData[1:]
controllerData.append(firstByte)
a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData)
print("could it be unsigned INTs? :",a)
a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData)
print("could it be signed INTs? : ",a)
Z:\G\solarData\venv\Scripts\python.exe Z:/G/solarData/decodeMessages.py
len(controllerData) = 50 and screenshot shows 24 values on the screen
could it be unsigned INTs? : (43775, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 51200, 20, 0, 1024, 0, 0, 0, 0, 25737, 13571)
could it be signed INTs? : (-21761, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, -14336, 20, 0, 1024, 0, 0, 0, 0, 25737, 13571)
moving first byte to end
could it be unsigned INTs? : (21930, 65326, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 5320, 0, 0, 4, 0, 0, 0, 35072, 868, 65333)
could it be signed INTs? : (21930, -210, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 5320, 0, 0, 4, 0, 0, 0, -30464, 868, -203)
Process finished with exit code 0
batV = controllerData[30:32]
a = struct.unpack("h" * ((len(batV)) // 2), batV)[0]/100
print("is this battery voltage? : ",a)
is this battery voltage? : 53.2
batKwH = controllerData[36:38]
a = struct.unpack("h" * ((len(batKwH)) // 2), batKwH)[0]/10
print("is this Bat total KWH? : ",a)
is this Bat total KWH? : 0.4
Wow! I was not expecting you to help me with this so much. I am grateful of your help and knowledge with this. I will have to read into your response to help learn more of it.I'm an insomniac and a liar; I wasn't really done for the night.
I gave decoding the controller's response a paltry effort and the results seem somewhat promising.
Here's another script I wrote to play with the data:
and here's what it yielded:Python:import struct """The only values shown on the information page at the time of sniffing is battery V 53.2, Bat total KWH 0.4 Kwh, bat cap 100%. The rest of the values are 0.""" controllerData =bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035') softwareData = bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.') # for bt in softwareData: # print(bt) print("len(controllerData) = ",len(controllerData), "and screenshot shows 24 values on the screen") a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData) print("could it be unsigned INTs? :",a) a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData) print("could it be signed INTs? : ",a) print("moving first byte to end") firstByte = controllerData[0] controllerData = controllerData[1:] controllerData.append(firstByte) a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData) print("could it be unsigned INTs? :",a) a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData) print("could it be signed INTs? : ",a)
note these two numbers that jumped out when I shifted the first byte to the end ( I did that just to change the starting point of the conversion from bytes to 16-bit INTs):Code:Z:\G\solarData\venv\Scripts\python.exe Z:/G/solarData/decodeMessages.py len(controllerData) = 50 and screenshot shows 24 values on the screen could it be unsigned INTs? : (43775, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 51200, 20, 0, 1024, 0, 0, 0, 0, 25737, 13571) could it be signed INTs? : (-21761, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, -14336, 20, 0, 1024, 0, 0, 0, 0, 25737, 13571) moving first byte to end could it be unsigned INTs? : (21930, 65326, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 5320, 0, 0, 4, 0, 0, 0, 35072, 868, 65333) could it be signed INTs? : (21930, -210, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 5320, 0, 0, 4, 0, 0, 0, -30464, 868, -203) Process finished with exit code 0
View attachment 287236
So it seems pretty likely to me that we learned at least one thing:
Because:Python:batV = controllerData[30:32] a = struct.unpack("h" * ((len(batV)) // 2), batV)[0]/100 print("is this battery voltage? : ",a)
and the odds of that being a coincidence are pretty low.Code:is this battery voltage? : 53.2
And since "Bat Total KwH" is 3 spots down in the UI from "Bat V" and there is a 4 in the list 3 spots down from 5320, it is plausible that:
Python:batKwH = controllerData[36:38] a = struct.unpack("h" * ((len(batKwH)) // 2), batKwH)[0]/10 print("is this Bat total KWH? : ",a)
Likely I had to shift bytes to reveal these values because not every piece of data in the controller's response is a 16 bit integer.Code:is this Bat total KWH? : 0.4
The first byte might be an acknowledgement of the software's command.
Things like "bat work status" might be a single byte that only has (ex) 3 possible values: 1(low), 2(med), 3(high)
Things like "first out status" might be a single bit of a byte representing up to 255 individual binary values
Things like "wind average DC I" might be a 32 bit float (context cue: in the UI it's the only value with two decimal points shown: 0.00A - or that could just be random)
In order to figure out what bytes the rest of the UI values correspond to, you'll need to play around with it as I demonstrated, and get familiar with struct. It will be really helpful if you can force some of these numbers to change and observe the difference in the message received from the controller. In my previous post with the packets, every reply from the controller was exactly the same. We have identified ONE variable out of (est) no less than 24. If just one bit associate with a known value could be changed in the controller for even just a second, it would be immediately obvious which part of the reply corresponds to that value.
For example, I think you said you don't have a wind turbine connected, so naturally you probably wouldn't care to identify which bytes correspond to RPM, but if you could simulate a few pulses to the turbine sensor input (just jab that input with a signal wire a few times) it should cause the RPM reading to jump by at least a fraction of an RPM, which would result in one or two bytes you can cross off the list when trying to solve for other variables.
You could force some other numbers too:
and so on.
- Extern input volts (hook up 120V/240V mains to it?)
- Solar input I, V, W, kWK (hook up a solar panel to it, even just a tiny cheap one)
- Wind average DC I, V, W, kWH (connect a bench power supply to the wind turbine power input)
Were you able to find any clues in the decompiled software? That may yet be the fastest option.
Neither was I. You can thank whatever undiagnosed disorder I have. Sometimes my mind and an idea get stuck together like two dogs humping, and the only way to get unstuck is to let it run its course. I would have done all this even if I knew you had died last night and would never see it.I was not expecting you to help me with this so much.
I am glad I did not die last night and I super appreciate all your work to help me out!Neither was I. You can thank whatever undiagnosed disorder I have. Sometimes my mind and an idea get stuck together like two dogs humping, and the only way to get unstuck is to let it run its course. I would have done all this even if I knew you had died last night and would never see it.
I'm an insomniac and a liar; I wasn't really done for the night.
I gave decoding the controller's response a paltry effort and the results seem somewhat promising.
Here's another script I wrote to play with the data:
and here's what it yielded:Python:import struct """The only values shown on the information page at the time of sniffing is battery V 53.2, Bat total KWH 0.4 Kwh, bat cap 100%. The rest of the values are 0.""" controllerData =bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035') softwareData = bytearray(b'\xff\xaaU\x08\x01\xff\x01\x00\x00%\x01.') # for bt in softwareData: # print(bt) print("len(controllerData) = ",len(controllerData), "and screenshot shows 24 values on the screen") a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData) print("could it be unsigned INTs? :",a) a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData) print("could it be signed INTs? : ",a) print("moving first byte to end") firstByte = controllerData[0] controllerData = controllerData[1:] controllerData.append(firstByte) a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData) print("could it be unsigned INTs? :",a) a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData) print("could it be signed INTs? : ",a)
note these two numbers that jumped out when I shifted the first byte to the end ( I did that just to change the starting point of the conversion from bytes to 16-bit INTs):Code:Z:\G\solarData\venv\Scripts\python.exe Z:/G/solarData/decodeMessages.py len(controllerData) = 50 and screenshot shows 24 values on the screen could it be unsigned INTs? : (43775, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, 51200, 20, 0, 1024, 0, 0, 0, 0, 25737, 13571) could it be signed INTs? : (-21761, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 0, 0, 4, -14336, 20, 0, 1024, 0, 0, 0, 0, 25737, 13571) moving first byte to end could it be unsigned INTs? : (21930, 65326, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 5320, 0, 0, 4, 0, 0, 0, 35072, 868, 65333) could it be signed INTs? : (21930, -210, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 0, 0, 1024, 0, 5320, 0, 0, 4, 0, 0, 0, -30464, 868, -203) Process finished with exit code 0
View attachment 287236
So it seems pretty likely to me that we learned at least one thing:
Because:Python:batV = controllerData[30:32] a = struct.unpack("h" * ((len(batV)) // 2), batV)[0]/100 print("is this battery voltage? : ",a)
and the odds of that being a coincidence are pretty low.Code:is this battery voltage? : 53.2
And since "Bat Total KwH" is 3 spots down in the UI from "Bat V" and there is a 4 in the list 3 spots down from 5320, it is plausible that:
Python:batKwH = controllerData[36:38] a = struct.unpack("h" * ((len(batKwH)) // 2), batKwH)[0]/10 print("is this Bat total KWH? : ",a)
Likely I had to shift bytes to reveal these values because not every piece of data in the controller's response is a 16 bit integer.Code:is this Bat total KWH? : 0.4
The first byte might be an acknowledgement of the software's command.
Things like "bat work status" might be a single byte that only has (ex) 3 possible values: 1(low), 2(med), 3(high)
Things like "first out status" might be a single bit of a byte representing up to 255 individual binary values
Things like "wind average DC I" might be a 32 bit float (context cue: in the UI it's the only value with two decimal points shown: 0.00A - or that could just be random)
In order to figure out what bytes the rest of the UI values correspond to, you'll need to play around with it as I demonstrated, and get familiar with struct. It will be really helpful if you can force some of these numbers to change and observe the difference in the message received from the controller. In my previous post with the packets, every reply from the controller was exactly the same. We have identified ONE variable out of (est) no less than 24. If just one bit associate with a known value could be changed in the controller for even just a second, it would be immediately obvious which part of the reply corresponds to that value.
For example, I think you said you don't have a wind turbine connected, so naturally you probably wouldn't care to identify which bytes correspond to RPM, but if you could simulate a few pulses to the turbine sensor input (just jab that input with a signal wire a few times) it should cause the RPM reading to jump by at least a fraction of an RPM, which would result in one or two bytes you can cross off the list when trying to solve for other variables.
You could force some other numbers too:
and so on.
- Extern input volts (hook up 120V/240V mains to it?)
- Solar input I, V, W, kWK (hook up a solar panel to it, even just a tiny cheap one)
- Wind average DC I, V, W, kWH (connect a bench power supply to the wind turbine power input)
Were you able to find any clues in the decompiled software? That may yet be the fastest option.
import serial
import struct
import time
SerialObj = serial.Serial('COM9') # COMxx format on Windows
SerialObj.baudrate = 2400 # set Baud rate to 2400
SerialObj.bytesize = 8 # Number of data bits = 8
SerialObj.parity ='N' # No parity
SerialObj.stopbits = 1 # Number of Stop bits = 1
time.sleep(.2)
# Requesting data from controller by sending these values:
values = bytearray([0xFF,0xAA,0x55,0x08,0x01,0xFF,0x01,0x00,0x00,0x25,0x01,0x2E])
SerialObj.write(values)
print ("Polling Controller...")
SerialObj.flush() # Flushing buffer for new data
controllerData = SerialObj.read(50)
print(f"Response from controller: {controllerData}")
controllerData = bytearray(controllerData)
# for bt in softwareData:
# print(bt)
#print("len(controllerData) = ",len(controllerData), "and screenshot shows 24 values on the screen")
#a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData)
#print("could it be unsigned INTs? :",a)
#a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData)
#print("could it be signed INTs? : ",a)
print("Formatting data...")
firstByte = controllerData[0]
controllerData = controllerData[1:]
controllerData.append(firstByte)
a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData)
print("could it be unsigned INTs? :",a)
a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData)
print("could it be signed INTs? : ",a)
batKwH = controllerData[36:38]
a = struct.unpack("h" * ((len(batKwH)) // 2), batKwH)[0]/10
print("")
print(f"Bat total KwH : {a} KwH")
print("")
batV = controllerData[30:32]
a = struct.unpack("h" * ((len(batV)) // 2), batV)[0]/100
print(f"Battery Voltage : {a}V")
I see that you found the appropriate alternative to serial.readlines() since it looks for b'\n' (newline) character:I and still working on gathering those other data points. I have used this python script with the part you wrote to now poll the data without using the program. It works using the same USB to serial adapter and not having to use the Serial Sniffer anymore so there's a bit of progress. As I get time to connect those other inputs or have enough wind (during the hours I am awake anyways) I will keep you posted. Here is the script I have so far and thanks again for the help getting me thus far.
Python:import serial import struct import time SerialObj = serial.Serial('COM9') # COMxx format on Windows SerialObj.baudrate = 2400 # set Baud rate to 2400 SerialObj.bytesize = 8 # Number of data bits = 8 SerialObj.parity ='N' # No parity SerialObj.stopbits = 1 # Number of Stop bits = 1 time.sleep(.2) # Requesting data from controller by sending these values: values = bytearray([0xFF,0xAA,0x55,0x08,0x01,0xFF,0x01,0x00,0x00,0x25,0x01,0x2E]) SerialObj.write(values) print ("Polling Controller...") SerialObj.flush() # Flushing buffer for new data controllerData = SerialObj.read(50) print(f"Response from controller: {controllerData}") controllerData = bytearray(controllerData) # for bt in softwareData: # print(bt) #print("len(controllerData) = ",len(controllerData), "and screenshot shows 24 values on the screen") #a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData) #print("could it be unsigned INTs? :",a) #a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData) #print("could it be signed INTs? : ",a) print("Formatting data...") firstByte = controllerData[0] controllerData = controllerData[1:] controllerData.append(firstByte) a = struct.unpack("H" * ((len(controllerData)) // 2), controllerData) print("could it be unsigned INTs? :",a) a = struct.unpack("h" * ((len(controllerData)) // 2), controllerData) print("could it be signed INTs? : ",a) batKwH = controllerData[36:38] a = struct.unpack("h" * ((len(batKwH)) // 2), batKwH)[0]/10 print("") print(f"Bat total KwH : {a} KwH") print("") batV = controllerData[30:32] a = struct.unpack("h" * ((len(batV)) // 2), batV)[0]/100 print(f"Battery Voltage : {a}V")
def serialSendRcv(self, input):
self.markemSerial = serial.Serial(self.comPort,
self.baudRate,
timeout=self.timeOut,
parity=serial.PARITY_NONE,
dsrdtr=1,
write_timeout=self.timeOut)
callingFunc = inspect.stack()[1].function
ret = b''
buf = b''
# The Markem Imaje printer does not transmit any message terminating character so we need this convoluted
# function to decide whether or not a message is complete based on no more characters coming in
waitingForReply = True
# Markem uses DSR/DTR to signify when it is ready to receive a command. If serial is unplugged or printer
# is off, DSR/DTR will be off, and PanelPC serial port will not send. We use write_timeout to catch this
# specific case, and pass a generic timeout along to the calling function.
try:
self.markemSerial.write(input)
if self.printSerialDialog: print(callingFunc, "Sent: ", input)
buf = bytes()
sendTime = time.time()
while waitingForReply == True:
# With a sending timeout already handled, we evaluate for a receiving timeout.
if time.time() > sendTime + self.timeOut:
self.markemSerial.close()
raise Exception ("Connection to printer requested by",callingFunc," timed out on receive")
if self.markemSerial.in_waiting > 0:
waitingForReply = False
if waitingForReply == False:
# once a message starts coming in, we consider a deadspace of 30ms to be the end of a message
tic = time.time()
while (time.time() - tic) < .030:
if self.markemSerial.in_waiting > 0:
tic = time.time() # reset the 30mS timer after every byte that comes in,
buf += self.markemSerial.read()
#self.markemSerial.close()
except serial.serialutil.SerialTimeoutException:
raise Exception("Connection to printer requested by",callingFunc," timed out on transmit")
if buf == b'\x06':
ret = b'ACK'
elif buf == b'\x15':
raise Exception ("connection requested by",callingFunc,"refused by Markem Printer. "
"Printer is locked or message is currently being manually edited by operator")
else:
if len(buf) < 2:
raise Exception ("Connection to printer requested by",callingFunc," established but no data received.")
else:
ret = buf
if self.printSerialDialog: print(callingFunc, "Received: ", ret)
self.markemSerial.close()
return ret
Polling Controller...
Response from controller: b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:\x03\x00\x00\x04\x00\x00\xf8\x14\x00\x00\x00\x00\x04\x00\x00\xb8\x03\x00\x00\x00\x00\x99d\x04m'
len(controllerData) = 50 and screenshot shows 24 values on the screen
could it be unsigned INTs? : (43775, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 826, 0, 4, 63488, 20, 0, 1024, 0, 952, 0, 0, 25753, 27908)
could it be signed INTs? : (-21761, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 826, 0, 4, -2048, 20, 0, 1024, 0, 952, 0, 0, 25753, 27908)
Formatting data...
could it be unsigned INTs? : (21930, 65326, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 14848, 3, 1024, 0, 5368, 0, 0, 4, 47104, 3, 0, 39168, 1124, 65389)
could it be signed INTs? : (21930, -210, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 14848, 3, 1024, 0, 5368, 0, 0, 4, -18432, 3, 0, -26368, 1124, -147)
Bat total KwH : 0.4 KwH
Battery Voltage : 53.68V
So, processing this a little more since you got some variables to change:This was the script output at the time of this screenshot.
View attachment 287578Code:Polling Controller... Response from controller: b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:\x03\x00\x00\x04\x00\x00\xf8\x14\x00\x00\x00\x00\x04\x00\x00\xb8\x03\x00\x00\x00\x00\x99d\x04m' len(controllerData) = 50 and screenshot shows 24 values on the screen could it be unsigned INTs? : (43775, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 826, 0, 4, 63488, 20, 0, 1024, 0, 952, 0, 0, 25753, 27908) could it be signed INTs? : (-21761, 11861, 511, 17, 9472, 0, 0, 0, 0, 0, 0, 0, 826, 0, 4, -2048, 20, 0, 1024, 0, 952, 0, 0, 25753, 27908) Formatting data... could it be unsigned INTs? : (21930, 65326, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 14848, 3, 1024, 0, 5368, 0, 0, 4, 47104, 3, 0, 39168, 1124, 65389) could it be signed INTs? : (21930, -210, 4353, 0, 37, 0, 0, 0, 0, 0, 0, 14848, 3, 1024, 0, 5368, 0, 0, 4, -18432, 3, 0, -26368, 1124, -147) Bat total KwH : 0.4 KwH Battery Voltage : 53.68V
import struct
oldResponse = bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x04\x00\x00\xc8\x14\x00\x00\x00\x00\x04\x00\x00\x00\x00\x00\x00\x00\x00\x89d\x035')
newResponse = bytearray(b'\xff\xaaU.\xff\x01\x11\x00\x00%\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00:\x03\x00\x00\x04\x00\x00\xf8\x14\x00\x00\x00\x00\x04\x00\x00\xb8\x03\x00\x00\x00\x00\x99d\x04m')
print(len(oldResponse),len(newResponse))
batV = oldResponse[31:33]
a = struct.unpack("h" * ((len(batV)) // 2), batV)[0]/100
print("old battery voltage: ",a)
batV = newResponse[31:33]
a = struct.unpack("h" * ((len(batV)) // 2), batV)[0]/100
print("new battery voltage: ",a)
alreadyAccountedFor = [31,32]
for i in range(0,50):
if (newResponse[i] != oldResponse[i]) and (i not in alreadyAccountedFor):
print("byte",i,"old:",oldResponse[i],"new:",newResponse[i])
Z:\G\solarData\venv\Scripts\python.exe Z:/G/solarData/newDecode.py
50 50
old battery voltage: 53.2
new battery voltage: 53.68
byte 24 old: 0 new: 58
byte 25 old: 0 new: 3
byte 40 old: 0 new: 184
byte 41 old: 0 new: 3
byte 46 old: 137 new: 153
byte 48 old: 3 new: 4
byte 49 old: 53 new: 109
Process finished with exit code 0
unknown = newResponse[24:26]
a = struct.unpack("h" * ((len(unknown)) // 2), unknown)[0]
print("unknown #1: ",a)
unknown = newResponse[40:42]
a = struct.unpack("h" * ((len(unknown)) // 2), unknown)[0]
print("unknown #2: ",a)
unknown = newResponse[48:50]
a = struct.unpack("h" * ((len(unknown)) // 2), unknown)[0]
print("unknown #3: ",a)
unknown = newResponse[22:26]
a = struct.unpack("I" * ((len(unknown)) // 4), unknown)[0]
print("unknown #1: ",a)
unknown = newResponse[38:42]
a = struct.unpack("I" * ((len(unknown)) // 4), unknown)[0]
print("unknown #2: ",a)
unknown = newResponse[46:50]
a = struct.unpack("I" * ((len(unknown)) // 4), unknown)[0]
print("unknown #3: ",a)
unknown = newResponse[22:26]
a = struct.unpack("f" * ((len(unknown)) // 4), unknown)[0]
print("unknown #1: ",a)
unknown = newResponse[38:42]
a = struct.unpack("f" * ((len(unknown)) // 4), unknown)[0]
print("unknown #2: ",a)
unknown = newResponse[46:50]
a = struct.unpack("f" * ((len(unknown)) // 4), unknown)[0]
print("unknown #3: ",a)
for i in range(0,46):
unknown = newResponse[i:i+4]
a = struct.unpack("L" * ((len(unknown)) // 4), unknown)[0]
print("unknown #"+str(i)+": ",a)
unknown #0: 777366271
unknown #1: 4281226666
unknown #2: 33500757
unknown #3: 285343534
unknown #4: 1114623
unknown #5: 4353
unknown #6: 620757009
unknown #7: 2424832
unknown #8: 9472
unknown #9: 37
unknown #10: 0
unknown #11: 0
unknown #12: 0
unknown #13: 0
unknown #14: 0
unknown #15: 0
unknown #16: 0
unknown #17: 0
unknown #18: 0
unknown #19: 0
unknown #20: 0
unknown #21: 973078528
unknown #22: 54132736
unknown #23: 211456
unknown #24: 826
unknown #25: 67108867
unknown #26: 262144
unknown #27: 1024
unknown #28: 4160749572
unknown #29: 351797248
unknown #30: 1374208
unknown #31: 5368
unknown #32: 20
unknown #33: 0
unknown #34: 67108864
unknown #35: 262144
unknown #36: 1024
unknown #37: 3087007748
unknown #38: 62390272
unknown #39: 243712
unknown #40: 952
unknown #41: 3
unknown #42: 0
unknown #43: 2566914048
unknown #44: 1687748608
unknown #45: 73701632
unknown #0: 4.85824679097302e-11
unknown #1: -2.3173046450918344e+38
unknown #2: 9.373868928549246e-38
unknown #3: 1.0254938989567953e-28
unknown #4: 1.5619194982011206e-39
unknown #5: 6.099852215205929e-42
unknown #6: 1.1102252745564227e-16
unknown #7: 3.397913357845675e-39
unknown #8: 1.3273099054084667e-41
unknown #9: 5.184804318001823e-44
unknown #10: 0.0
unknown #11: 0.0
unknown #12: 0.0
unknown #13: 0.0
unknown #14: 0.0
unknown #15: 0.0
unknown #16: 0.0
unknown #17: 0.0
unknown #18: 0.0
unknown #19: 0.0
unknown #20: 0.0
unknown #21: 0.00048828125
unknown #22: 5.466048731323637e-37
unknown #23: 2.963129680722685e-40
unknown #24: 1.1574725315322989e-42
unknown #25: 1.5046333071511383e-36
unknown #26: 3.6734198463196485e-40
unknown #27: 1.4349296274686127e-42
unknown #28: -1.0384598668829812e+34
unknown #29: 2.50416005753358e-26
unknown #30: 1.9256755600628782e-39
unknown #31: 7.522170156495618e-42
unknown #32: 2.802596928649634e-44
unknown #33: 0.0
unknown #34: 1.504632769052528e-36
unknown #35: 3.6734198463196485e-40
unknown #36: 1.4349296274686127e-42
unknown #37: -3.051759267691523e-05
unknown #38: 1.0814548027565045e-36
unknown #39: 3.415132513375298e-40
unknown #40: 1.3340361380372259e-42
unknown #41: 4.203895392974451e-45
unknown #42: 0.0
unknown #43: -6.617444900424222e-24
unknown #44: 2.257881474622049e+22
unknown #45: 2.687152535330902e-36
unknown #0: 777366271
unknown #1: -13740630
unknown #2: 33500757
unknown #3: 285343534
unknown #4: 1114623
unknown #5: 4353
unknown #6: 620757009
unknown #7: 2424832
unknown #8: 9472
unknown #9: 37
unknown #10: 0
unknown #11: 0
unknown #12: 0
unknown #13: 0
unknown #14: 0
unknown #15: 0
unknown #16: 0
unknown #17: 0
unknown #18: 0
unknown #19: 0
unknown #20: 0
unknown #21: 973078528
unknown #22: 54132736
unknown #23: 211456
unknown #24: 826
unknown #25: 67108867
unknown #26: 262144
unknown #27: 1024
unknown #28: -134217724
unknown #29: 351797248
unknown #30: 1374208
unknown #31: 5368
unknown #32: 20
unknown #33: 0
unknown #34: 67108864
unknown #35: 262144
unknown #36: 1024
unknown #37: -1207959548
unknown #38: 62390272
unknown #39: 243712
unknown #40: 952
unknown #41: 3
unknown #42: 0
unknown #43: -1728053248
unknown #44: 1687748608
unknown #45: 73701632
by Duane Benson
by Duane Benson
by Jake Hertz