Need Help Decoding Automotive CAN Bus Extended Frame Bits

Thread Starter

SaMirakle

Joined Mar 6, 2012
20
Hi everyone,

I don't know why but I'm having a tough time decoding CAN Bus data frames, even after looking at many sources showing bit field diagrams of how the data is supposed to be structured.

Background on project/goal

My main goal is to connect directly to an electronic turbo actuator from a diesel engine on a bench without the need of the ECU (engine control module) to simulate the data the actuator requires from the ECU in order to control the motor inside of the actuator. Ultimately, I'd like to be able to identify each of the messages in order to determine what type of information the actuator is requesting from the ECU in order to function. I'm using an Arduino with a CAN Bus shield that utilized an MCP2515 CAN controller and a TJA1050 CAN transceiver. I've found and tried about a handful of MCP2515/CAN Bus libraries and have found one that seems to work the best. I'm successfully been able to read the Hex ID and data from a few of the actuators that I have.

Issues

I can't seem to understand exactly how to decode the bits contained in the Hex ID and data messages based on the CAN Bus 2.0B standards. Since this is a diesel engine module I'm sure it falls under the extended CAN format with a 29-bit identifier. I don't have any educational background or training in how bit fields are supposed to be arranged. I've written out the hex values in binary format and have tried multiple times to try to match up the bits with the CAN Bus bit field structures but I'm not sure if I'm doing it correctly.

Can anybody help guide me in how I'm supposed to identify each bit of information according to the bit field structures? I'll leave some of the data below for reference as well as how the bit fields are structured. Thanks in advance!


Extended ID: 0x18FFDC00 DLC: 8 Data: 0x10 0xFF 0xFF 0xFF 0x01 0x9F 0xFF 0xFF
Extended ID: 0x0CFFC600 DLC: 8 Data: 0x32 0x00 0x01 0xFF 0xFF 0xFF 0xFF 0xFF
Extended ID: 0x18FFC502 DLC: 8 Data: 0xFF 0xFF 0xFF 0x51 0x00 0xFF 0xFF 0x00
Extended ID: 0x18FF0A02 DLC: 8 Data: 0x00 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF 0xFF
Extended ID: 0x18EAFFFE DLC: 3 Data: 0x00 0xEE 0x00

References on CAN Bus Messages:

https://www.rpi.edu/dept/ecse/mps/sloa101.pdf

http://www.simmasoftware.com/j1939-presentation.pdf

https://www.mi.fu-berlin.de/inf/groups/ag-tech/projects/ScatterWeb/moduleComponents/CanBus_canover.pdf

https://www.ti.com/lit/an/sloa101b/sloa101b.pdf?ts=1599789609578&ref_url=https%3A%2F%2Fwww.google.com%2F
 

Papabravo

Joined Feb 24, 2006
14,412
I think you have everything you need. The SOF is a single dominant bit to start a frame. It will never show up in any readable data. The identifier field is 29-bits long which is accounted for as 7 hex digits at 4 bits each plus a single bit of the 8th digit (could be 0 or 1) on the left hand end. On the wire I do not think the 29 identifier bits are continuous. The DLC (Data Length Code) is always 8 for 8 byte data frames, and it is always 3 for 3 byte data frames. In document sloa103.pdf refer to Figure 3 which shows a CAN frame with a 29-bit identifer. The 29 bits are broken up into two subfields. The 11 most significant bits come first. This is followed by the SRR and IDE bits. The SRR is a place holder in the extended format and the IDE MUST be a recessive bit to denote that the remainder of the identifier follows. Then you have the remianing 18 bits of the identifier. r1 and r0 are reserved bits, DLC you know all about and the data field you know all about. CRC, ACK, EOF, and IFS will never show up in any interface as readable data so you don't need to worry about them for now. Only if you want to take a trip into the weeds.
 

Thread Starter

SaMirakle

Joined Mar 6, 2012
20
Hi there. Thanks for the response! So is there any special method of identifying each bit individually? I'm a bit confused about the whole Most/Least Significant Byte thing. Based on all of the documentation above, would I need to be doing any of that reverse order bit sequence stuff? Also, would you mind breaking down how you would go about piecing out one of the Hex IDs above and possibly even one of the corresponding data fields to show me as an example (bit by bit) ? Thank you so much in advance!
 

Papabravo

Joined Feb 24, 2006
14,412
Hi there. Thanks for the response! So is there any special method of identifying each bit individually? I'm a bit confused about the whole Most/Least Significant Byte thing. Based on all of the documentation above, would I need to be doing any of that reverse order bit sequence stuff? Also, would you mind breaking down how you would go about piecing out one of the Hex IDs above and possibly even one of the corresponding data fields to show me as an example (bit by bit) ? Thank you so much in advance!
Nope there is none of that "bit reversal" stuff in CAN. Bits on the wire look the same as they do coming out of a register. Lets take the 1st extended identifier:
0x18FFDC00 ≡ 18 | FF | DC | 00
This looks like four bytes, but you have to remember that the 18 represents only 5 bits. If you number the bits in the conventional way, they are labeled 28 through 0 from left to right. You can set the controller up to pass and reject messages with particular identifiers using the "Mask and Match" registers. The data bytes are just the same as if you define an array in memory. So using that analogy

d[0] = 0x10 ;
d[1] = 0xFF ;
d[2] = 0xFF ;
d[3] = 0xFF ;
d[4] = 0x01 ;
d[5] = 0x9F ;
d[6] = 0xFF ;
d[7] = 0xFF ;

Similarly we could represent the identifier as:

id[0] = 0x18 ;
id[1] = 0xFF ;
id[2] = 0xDC ;
id[3] = 0x00 ;

It is true that looking on the wire there are protocol bits which are dropped by the controller (RTR, IDE, r0, r1, ACK, SOF, EOF, and IFS) and never show up in any of the hardware registers and there are also "bit stuffing" bits which are added by the transmitter and removed by the receiver and you never see them. You know the rule about bit stuffing -- Right?
 

Thread Starter

SaMirakle

Joined Mar 6, 2012
20
Nope there is none of that "bit reversal" stuff in CAN. Bits on the wire look the same as they do coming out of a register. Lets take the 1st extended identifier:
0x18FFDC00 ≡ 18 | FF | DC | 00
This looks like four bytes, but you have to remember that the 18 represents only 5 bits. If you number the bits in the conventional way, they are labeled 28 through 0 from left to right. You can set the controller up to pass and reject messages with particular identifiers using the "Mask and Match" registers. The data bytes are just the same as if you define an array in memory. So using that analogy

d[0] = 0x10 ;
d[1] = 0xFF ;
d[2] = 0xFF ;
d[3] = 0xFF ;
d[4] = 0x01 ;
d[5] = 0x9F ;
d[6] = 0xFF ;
d[7] = 0xFF ;

Similarly we could represent the identifier as:

id[0] = 0x18 ;
id[1] = 0xFF ;
id[2] = 0xDC ;
id[3] = 0x00 ;

It is true that looking on the wire there are protocol bits which are dropped by the controller (RTR, IDE, r0, r1, ACK, SOF, EOF, and IFS) and never show up in any of the hardware registers and there are also "bit stuffing" bits which are added by the transmitter and removed by the receiver and you never see them. You know the rule about bit stuffing -- Right?
Hi again! So you're saying that the hex value of 18 (0001 1000) is really only 5 bits (1 1000) because you have to ignore the preceding 0's? Is that a standard defined in CAN communications or is that common knowledge in the realm of binary?

I've come across the term "bit stuffing" in a lot of these documents. From my understanding, bit stuffing is used when there's a repetitive sequence of the same bit (4 in a row or 5 can't remember). It will automatically insert an opposite bit or something. So when bit stuffing occurs, does that mean that the inserted bit is registered as a new bit that we have to consider as part of the overall data field? I was a bit confused about that as well.

Also, I could use a little clarity on the "filler" bits found within the whole message such as SOF, SRR, IDE, RTR. etc. Do you start counting your first bit at the SOF frame? How do you account for the RTR, IDE, DLC, etc?

Part of my end goal is to be able to identify what information this actuator is requesting from the other nodes on the CAN bus (most likely the ECU).
 

Papabravo

Joined Feb 24, 2006
14,412
Hi again! So you're saying that the hex value of 18 (0001 1000) is really only 5 bits (1 1000) because you have to ignore the preceding 0's? Is that a standard defined in CAN communications or is that common knowledge in the realm of binary?

I've come across the term "bit stuffing" in a lot of these documents. From my understanding, bit stuffing is used when there's a repetitive sequence of the same bit (4 in a row or 5 can't remember). It will automatically insert an opposite bit or something. So when bit stuffing occurs, does that mean that the inserted bit is registered as a new bit that we have to consider as part of the overall data field? I was a bit confused about that as well.

Also, I could use a little clarity on the "filler" bits found within the whole message such as SOF, SRR, IDE, RTR. etc. Do you start counting your first bit at the SOF frame? How do you account for the RTR, IDE, DLC, etc?

Part of my end goal is to be able to identify what information this actuator is requesting from the other nodes on the CAN bus (most likely the ECU).
It is because the CAN identifier is 29 bits long and it is mapped into 4 bytes of 8 bits for a total of 32 bits. Those three leading zeros are meant to be ignored. Bits used for bit stuffing are inserted by the transmitter as the bit stream goes out onto the wire, delaying the actual bit stream by one bit time. The stuff bits are "ignored" by the receiver as if they weren't even there. For your purposes you need to focus on the stuff that is "visible" through the interface. If it does not show up in the data registers of the CAN controller you can for all practical purposes ignore it.

SOF - Start of Frame This is a single dominant bit that appears after a minimum length string of recessive bits that belong to the IFS (Inter-frame Space)
SRR - Substitute RTR Bit This is just a place holder. In a Standard Frame this would be the RTR bit
IDE - Identifier Extended This bit is used to distinguish a Standard Identifier (11 bits long) from an Extended Identifier (29 bits long)
RTR - Remote Transmission Request This bit when dominant can be used to gather information from ALL the other nodes on the network. A message with the RTR bit equal 0 (dominant) will generally have a zero length or NULL datafield. This is like a broadcast poll request to all node who recognize the identifier to send their data - whatever it is.
r0, r1 Just reserved bits with no known function
CRC - Cyclic Redundancy Check is 15 bits long. You can find the calculation algorithm in the CAN specification, but don't bother. You can't see it in the interface and you can't do anything about if it flags an error. The controller has you covereed on this one and will throw and ERROR frame (Active or Passive) if it detects a CRC error. In 15 years of working with CAN I never saw one nor was I able to create one. Why you might ask. The answer is that each transmitter is monitoring the bus to determine if it can hear the exact same bits that it is transmitting. If the transmitter detects a bus fault it will throw an ERROR frame long before the CRC can detect something wrong.
ACK - The ACK is used by ALL receivers on the bus who heard the transmitted frame correctly regardless of the intended recipient. The transmitter only requires one node to correctly hear the tranmission. A useful diagnostic is to put a lonely node on a network, try to transmit a frame and watch it re-transmit that frame FOREVER.
EOF - End of Frame is seven recessive bits with bit stuffing disabled
IFS - Inter-frame Space seven more recessive bits


BTW. Without some cooperation from the manufacturer or someone on the inside, your chances of successfully decoding and understanding all of what is going on in a CAN network are grim. These are essentially the family jewels and they usually don't share this information with the public. Good Luck, whatever you are trying to do.
 

bwilliams60

Joined Nov 18, 2012
1,343
@Papabravo I have a question to do with this as well. At what point does the OEM take over and make it so it cant be decoded? I think that is the piece that I am missing. For example, if you sniff the data, you can see that the body controller has sent out a message to turn on the RF turn signal. I am pretty sure we can decipher that much but at what point does it become the part where the family jewels are closely guarded and we cant see any more?
 

Papabravo

Joined Feb 24, 2006
14,412
@Papabravo I have a question to do with this as well. At what point does the OEM take over and make it so it cant be decoded? I think that is the piece that I am missing. For example, if you sniff the data, you can see that the body controller has sent out a message to turn on the RF turn signal. I am pretty sure we can decipher that much but at what point does it become the part where the family jewels are closely guarded and we cant see any more?
With an 11-bit identifier space it is not out of the question to map the identifier space according to which nodes are producers and which nodes are consumers. Knowing if the identifier field contains a MAC address is also a big help. The 29-bit space is more difficult because the data space is larger. Maybe you understand how the identifier space is broken up but maybe there are things you need to know how to do in order to enable other features. This all presumes there is a 1 to 1 correspondence between message identifiers and observable actions or responses. That my not necessarily be the case. Figuring out the protocol which may be involved in establishing and maintaining connections can also be challenging. Over time it may be possible to build a knowledge base to allow certain things to be accomplished, but I would not bet the farm on being able to access all of the functionality. The OEM will try to make decoding by sniffer difficult thing from the get go.

I would like to understand how the Ukranians were able to sidestep John Deere and offer firmware upgrades to their tractors at a fraction of the cost charged by the company. I don't think tractors run on a CAN network, but I would still like to know.

I also have a friend who works at FCA who says the radio connected to the CAN network is the primary attack vector for compromising a vehicle. I'll bet they are spending vast amounts of treasure to keep that from happening.
 
John Deere uses the J1939 standard. Depending on how much of the standard defined messages versus proprietary messages it may not be too hard to sort out. I'd kind of like to tinker with one myself just to see what can be sorted out.
 

Papabravo

Joined Feb 24, 2006
14,412
John Deere uses the J1939 standard. Depending on how much of the standard defined messages versus proprietary messages it may not be too hard to sort out. I'd kind of like to tinker with one myself just to see what can be sorted out.
In the early 1990's J1939 was primarily for buses and large diesel trucks. I did not realize that it was used on tractors. I do remember that they used 29-bit identifiers exclusively. The only hardware we had available at that time was was the Philips 82C200 and the intel 82526 and we stuck with 11-bit frames for DeviceNet. The intel 82527 had just been introduced and did not figure into our plans.
 
@Papabravo I didn't renew my membership and don't have access, but I remember seeing a lot of messages for hydraulics, pumps, and agricultural related parts defined in the standard when I was looking into it a year ago. Of course they could define their own and further complicate and force dependence on the dealers. My motivation was more towards trucks at the time, but many still use the 1708/1587 standard for what I was wanting to do and after a little digging decided it wasn9 worth my time.
 
Last edited:

Papabravo

Joined Feb 24, 2006
14,412
@Papabravo I didn't renew my membership and don't have access, but I remember seeing a lot of messages for hydraulics, pumps, and agricultural related parts defined in the standard when I was looking into it a year ago. Of course they could define their own and further complicate and force dependence on the dealers. My motivation was more towards trucks at the time, but many still use the 1708/1587 standard for what I was wanting to do and after a little digging decided it wasn9 worth my time.
Part of the genius of CAN was defining the MAC and the physical layer without defining how the identifier or the data fields are to be used. The standard defines the syntax of the messages but not the semantics. As a result we have a garden with thousands of flowers blooming. I'm sure your experience is not uncommon. Unless you have a valid reason to sniff out those semantics by observation - it just isn't worth the effort.
 
@SaMirakle what engine does this turbo actuator actually go on? I looked through some information I downloaded at one time. I couldn't seem to arrange the data you posted in any way to match a valid source address, destination address, and PGN at the same time that follows the J1939 standard from the identifiers you posted.
 

Thread Starter

SaMirakle

Joined Mar 6, 2012
20
@geekoftheweek They're supposed to be off of Cummins ISX engines. They seem to belong to the HE451VE models and may possibly even be compatible with the HE400VG and even the HE551VE turbos. I was able to find a spreadsheet made by CSS Electronics that automatically parses the hex ID's to find the correct PGN, Priority, PF, PS, SA, etc. It seems that most of them are categorized as "Proprietary B" which is meant to be hidden by the manufacturer. I think there's a reserved range of PGN numbers specifically for proprietary purposes only.

I can't imagine the actuators requiring much information beyond less than a handful of data such as the exhaust/boost pressure, throttle position, etc. This leads me to think that the data values may easily be determined as pressure, position, or speed.
 

Thread Starter

SaMirakle

Joined Mar 6, 2012
20
@Papabravo Thanks for the info. So you think the proprietary PGN's are going to be tough to identify without getting in touch with an engineer at Holset/Cummins then huh? You think that's how the guys that developed this Banshee VGT Controller got it done? I'm wondering how they were able to figure this stuff out on the smaller HE351VE actuator...

Any thoughts?
 
They're supposed to be off of Cummins ISX engines. It seems that most of them are categorized as "Proprietary B" which is meant to be hidden by the manufacturer. I think there's a reserved range of PGN numbers specifically for proprietary purposes only.
There is a range of proprietary PGNs defined. Even considering them things don't look right. Any chance of listing what the spreadsheet breaks things into? My information I have on hand is over eight years old so maybe there's something I'm missing, but it still seems off.

If I remember right Cummins Insite lists commanded actuator position, actual position, turbine speed, exhaust temp, and exhaust manifold pressure. I'm not sure which ones actually come from the actuator other than position and speed since I think the rest are wired to the ECU itself. I personally don't do too much with Cummins since most of the ones at work are in the yard trucks that get dealt with on another shift other than quick patches to get them through the night so I may be wrong there too. The other ones I do work with usually don't have any problems other than routine services.

I would assume the only information from ECU to actuator would be the commanded position, and maybe a startup sequence of some sort to get things in sync. Everything else would be data from the actuator to the ECU. It would take a lot to make an actuator controller account for engine brakes, regen cycle, filters getting dirty, and whatever other variables come in to play at the moment.

Chances are the Banshee people both had to get in touch with engineers and pay a fair chunk of change to get the needed information. Otherwise they would run in to reverse engineering and intellectual property problems. Another thing I found last week while trying to figure out what to do about wrong parts being sent out is both Cummins and Detroit use the same NOx sensors, same connectors, but different programs / calibration and part numbers. I'm not sure how Dodge and Cummins ECUs work together, but it may be a case of different ECUs that made it possible.
 
I did a little digging through the logs from my experimentation. I programmed an app for my phone that connected to a Nexiq adapter and logged all the messages at the diagnostic port and sorted out a couple (RPM, odometer, battery volts, and fuel level). One thing I found is many messages repeat data that could be sent with a different message, and even though that message also gets sent out the same bits in that message are all '1' or undefined. From your data above I would assume even proprietary messages do the same thing. I couldn't find any messages that actually came from specific parts. The major ECUs seem to separate the network and filter, modify, or whatever it takes to make messages for it's connected parts work. It wold be fun to get your type of project going, but it's probably going to be an uphill battle all the way.
 

Papabravo

Joined Feb 24, 2006
14,412
@Papabravo Thanks for the info. So you think the proprietary PGN's are going to be tough to identify without getting in touch with an engineer at Holset/Cummins then huh? You think that's how the guys that developed this Banshee VGT Controller got it done? I'm wondering how they were able to figure this stuff out on the smaller HE351VE actuator...

Any thoughts?
No, I really don't have anything for you on this subject. I think that the different makers have their reasons for keeping things close to the vest. I understand there must be some sharing to support the diagnosis and repair function, but I always supposed that involved the exchange of substantial sums so there would be a strong motivation to keep things confidential. I never really had to go there because, when I worked on this stuff at Allen-Bradley their spec was open. They begged independent developers to build all manner of compatible products. They even funded a conformance test suite and a lab you could go to, get your product tested for conformance, and get a conformance test certificate.
 

Thread Starter

SaMirakle

Joined Mar 6, 2012
20
@geekoftheweek I think you're on the right track there with the commanded position, actual position, turbine speed, and exhaust sensors. Here's a link to that J1939 to PGN Converter spreadsheet from CSS Electronic's website:

https://docs.google.com/spreadsheets/d/10f7-TFU9oViSQZYGFYVPDia2w1hd5eOPMlgJXmx31Lg/edit?usp=sharing

I also found a guy that had done a DIY VGT controller project with an Arduino back in 2015. He posted all of his source code online but it's so much code that it's beyond my level of understanding at the moment. Take a look at it and see if you can maybe make any sense of how he's going about controlling that turbo actuator there:

https://mopar1973man.com/cummins/articles.html/general-cummins/84_engine/89_air-exhaust/holset-he351-ve-vgt-standalone-turbo-arduino-controller-tuner-r159/

Also, @Papabravo, let me know what your thoughts are on this guy's project above. His code doesn't seem to incorporate any unknown proprietary PGNs or not. I think he's simulating the same data that @geekoftheweek mentioned above.
 
Just to make sure things are what they are in my mind... At this point the only thing connected is the turbo actuator and nothing else. If so and those are your values in the spreadsheet then I don't think they are J1939 messages. The source address for a turbo actuator is 02. The PSU value I would think should only be 00 (Engine 1), 40 (Dash), or 255 (Global) since this is almost always the destination address in a J1939 message. There are a few exceptions. Parts of it look workable, but too much of it doesn't.

I have a feeling since everything seems to be Dodge related you will need to look in to ODB2 type messages. As far as I know J1939 isn't used in passenger vehicles. You'll have to get into industrial, agriculture, buses, middle to large dump trucks, semis, and the like before it switches over.

The Arduino stuff looks interesting and would be fun to play with and figure out. If I had an actuator laying around I'd already have it hooked up and be trying things out. Good luck!
 
Top