Measuring thousands of sensors using a single board

rhydo

Joined Jul 5, 2016
14
Hello everyone,

I initially developed a system based on Arduino where I could measure 6 resistors sequentially and basically ascertain whether they were there or not. So infinite resistance basically meant 'no resistor'. This way I could tract the presence, absence and misplacement of resistors. This was not very hard to do. Now here is the problem -

I have to scale this exact system to one that measures thousands of resistors (50,000 as of now), ascertains whether each is in its place or not and then transmits this acquired data wirelessly to a main computer every 15 minutes.

I have experience in circuit design and basic programming and sensor measurement but I have never worked on such a massive scale. Could any of you guy kindly help me figure out where to start? What would be the cheapest and easiest hardware platform to utilize in order to solve this problem. I went through certain Data Acquisition Devices but there are so many of them and most seem expensive (above $200). I hope I have explained the problem clearly. Any help would be highly appreciated! WBahn Joined Mar 31, 2012 26,398 Actually, no, you haven't explained the problem clearly. How are these thousands of resistors connected and how are you measuring them, for starters? Since you say that you have to use "this exact system", it might be relevant to know how this exact system currently works in order to even begin to hope to be able to offer suggestions on how to scale it up. Thread Starter rhydo Joined Jul 5, 2016 14 I apologise for the confusion. I'll try to be more clear now. When I said 'exact same system' I only meant that the process of measuring resistors in order to ascertain whether they were present on not would be the same. I used an arduino to measure the resistance of 6 different resistors and compare them to predefined values using 'if' 'else' statements. I measure them using a standard potential divider arrangement - one resistor is a standard known resistor and the other is the one to be measured connected in series, an arduino pin gets connected to the node and basically measures potential drop from which I calculate the resistance. So If I measure a resistance value of lets say 1kohm I compare it to the preset value in the code. A match would mean the resistor is in the right position, a mismatch would mean the resistor has been misplaced and a reading of infinity would mean the resistor has been removed. This is what I did with the arduino, basic resistance measurement using analog pins based on ohm's law. Now I need to measure thousands of resistors and basically perform the same 3 tests - missing, misplaced or correctly positioned by comparing the measured values to standard values. Once this is done the data is to be transmitted over to a PC every 15 minutes. NorthGuy Joined Jun 28, 2014 611 This sounds a little bit bizarre. Are you saying that you have a board with 50,000 resistors soldered connected to common ground on it and a 50,000-contact connector with each contact connected to one resistor? Or do you have a 100,000-contact connector, with each resistor connected to two contacts? Or do you have something totally different? WBahn Joined Mar 31, 2012 26,398 I'm still unclear about your present technique. It sounds like you are manually taking a resistor and placing it in series with a known resistor and then measuring the voltage at the junction. How does this correspond to determining if the resistor is in the right position? Could you provide a sketch of your current system? Are you trying to make these measurement in-system using six different input pins on the Arduino? Thread Starter rhydo Joined Jul 5, 2016 14 You are right. That is exactly what I am doing. This corresponds to determining whether the resistor is in the right position by comparing it to the predefined array in my code. So if in my code the first position (where the arduino pin I am reading is connected) is supposed to have 10 kohms but I measure a 100 ohm resistor there then the standard (predefined) value and the measured value don't match and I can ascertain that the resistor on that position is misplaced. Also, yes, I am making these measurements using 6 different input pins of the arduino. dannyf Joined Sep 13, 2015 2,197 Fairly easy. 1. Replicate your current system to cover 50k resistors. If 5 resistors per arduino, you need 10k arduino. 2. Use one arduino to quite those 10k arduino and then submit that data to your upstream system. Basically you are 99percent there without doing much more than you have done so far. WBahn Joined Mar 31, 2012 26,398 You setup image didn't post properly. If it is a reasonably-sized jpg or png file, use the "Upload a File" button. If you still have problems, ask and we'll walk you through it. WBahn Joined Mar 31, 2012 26,398 You are right. That is exactly what I am doing. This corresponds to determining whether the resistor is in the right position by comparing it to the predefined array in my code. So if in my code the first position (where the arduino pin I am reading is connected) is supposed to have 10 kohms but I measure a 100 ohm resistor there then the standard (predefined) value and the measured value don't match and I can ascertain that the resistor on that position is misplaced. Also, yes, I am making these measurements using 6 different input pins of the arduino. If you are making the measurements in-circuit, then the rest of the circuit is going to affect your measurements, perhaps in a dominant way. But I can't say more until I see your actual approach. Thread Starter rhydo Joined Jul 5, 2016 14 I am sure there must be a more elegant solution than chaining 10,000 (and one) arduinos together to measure 50,000 resistors. I cannot be the first person to face the challenge of having to measure thousands of sensors sequentially and transmit the data to a computer. Isn't there a better way to do this? Thread Starter rhydo Joined Jul 5, 2016 14 The setup image was the same image that was present in the link at the end of the post. I am not sure why the image didn't upload properly but I figured that the link I posted had more information regarding the setup than just the image and hence would go further in giving you and idea of what I am doing. Here it is again - http://learningaboutelectronics.com/Articles/Arduino-ohmmeter.php I used this circuit set up to measure resistance for 6 different resistors being read by 6 analog pins of the arduino. Let me make my question really really simple so that if would be easier for you to point me in the right direction - If you had to read 50,000 sensors, process the acquired data (comparing the values etc. within your code) and then transmit this data as a file over to a computer every couple of minutes, HOW would you even begin to approach this implementation? I have not worked on systems on such a big scale so I am not sure how Embedded systems Engineers would solve such a problem in the industry. if this was a matter of a handful of sensors then it would be a piece of cake, but the only problem I have is the quantity of sensors I need to measure. Please tell me there is a better way to do this than daisy chain thousands of microcontrollers together. WBahn Joined Mar 31, 2012 26,398 So, just to be sure I understand, you are presently removing the six resistors from the circuit they are supposed to be in and placing them into six separate ohmmeter circuits that are being read by a single Arduino? If so, then hardware aside, how do you expect such a manual process to scale to such large numbers? How likely is it that all 50,000 resistors would be removed in the right order so as to match the sequence of resistors expected by the code? If not, then how are you compensating for the effect on the measurement that rest of the circuit has? How much time is available to test all 50k resistors? How many of these sets of resistors must be measured (per day, per week, or whatever time unit is reasonable)? How much can this process cost per set? If I were tasked with this, based on what little information I had at this point, I'd be looking at the feasibility of using some kind of robot. dannyf Joined Sep 13, 2015 2,197 Please tell me there is a better way to do this than daisy chain thousands of microcontrollers together. Sure. But what's "better" is highly subjective and in this case undefined, especially within the constraints you have put on yourself here. Thread Starter rhydo Joined Jul 5, 2016 14 If not, then how are you compensating for the effect on the measurement that rest of the circuit has? Could you please clarify this statement? I couldn't follow. Let me try and clarify more. Imagine there are 10 'items' on a shelf. You are tasked with tracking whether each item stays in place or is removed. 10kohm measured at a location (lets say) = 'Item is present' Infinite resistance measured = 'Item is absent' 5kohm measured (on the same location) = 'This item has been misplaced from the 5kohm shelf' This is it. If you follow this logic then you might be able to see that this is fairly straightforward to do using an arduino if the 'item' quantity is low. I am not removing resistors from another circuit. The resistor values being measured ARE part of the circuit as you can see in the link I posted twice above. Time available to measure these values can be anywhere from 1 minute to 1 hour after which the collected data is transmitted to a PC and the process restarts again. There will be one such 'set'. A set containing 50k resistors. Each read sequentially. The cost of the hardware should be less than$250.

How would you approach this problem if you were tasked with it. Feel free to ignore the constraints I set on myself using the arduino environment and the sequential measurement of each resistance value. If only provided with the information above, what would be your thoughts?

NorthGuy

Joined Jun 28, 2014
611
I think the challenge here is to create a mechanical insertion device, so that the person testing resistors could change them fast enough and at the same time ensure good contact to make detection useful. Bad contact will cause wrong resistor values. Achieving a good contact requires time. This would require some sort of debouncing and some quiet period when the actual detection is made. You can detect the insertion events automatically, or you can have an extra button which the worker should press to initiate detection. The button will increase detection quality, but is likely to slow things down. Only a very good button can handle so many presses. Either way, I don't think designing the insertion mechanism is going to be easy.

Let's suppose you come up with marvelous engineering design which allows good insertion within 2 seconds. This means 1800 measurements per hour. If you pay $10/hour to the worker, it will cost you$280 and will take amost four days. It is probably cheaper to order all new resistors, and your new resistors will arrive sooner.

Other than the mechanical insertion part, the electronics is trivial and you probably can do it under $10. I wouldn't try to design the electronics until the mechanical part is done, because the design of the mechanical part may change the requirements for the electronics. Storing the data is not a big deal. Unless insertions are somehow ordered, it's basically 2 numbers - number of resistors tested, and number of resistors passed - 4 bytes overall. It may be the best to display them on LCD connected to your device - entering two numbers into a computer after 4 days of hard work doesn't strike me as a big deal. But you can go with USB or wireless (WiFi or Bluetooth), which might be the most difficult part of your electronics design. Last edited: crutschow Joined Mar 14, 2008 27,734 I am sure there must be a more elegant solution than chaining 10,000 (and one) arduinos together to measure 50,000 resistors. I cannot be the first person to face the challenge of having to measure thousands of sensors sequentially and transmit the data to a computer. Isn't there a better way to do this? So you have 50,000 wires from 50,000 resistors that you want to measure? You may not be the first person to have to measure many sensors but you may be to first to need to do it for 50,000. Are all the resistors approximately the same value? The most straight-forward method is to use a multiplexer to sample each resistor. For common 16-channel multiplexer chips (such as the CD4067) that would require some 3125 chips. To address each channel would require 12-bits decoded into 3125 select lines and a 4-bit common address for each mux. So all you have to do then is interface 50,000 resistor connections to 3125 multiplexers and generate the decode multiplexer logic. Piece of cake. The Digikey price for the CD4067 in 2000-up quantities is$0.532/ea so the total cost for the multiplexers alone would be $1663. And allowing a half-square inch for each mux would require a board of 40 inches on a side for the multiplexers alone. Thus your total system cost is more like a few thousand dollars as opposed to your budget of a few hundred. Thread Starter rhydo Joined Jul 5, 2016 14 I think the challenge here is to create a mechanical insertion device, so that the person testing resistors could change them fast enough and at the same time ensure good contact to make detection useful. Bad contact will cause wrong resistor values. Achieving a good contact requires time. Other than the mechanical insertion part, the electronics is trivial and you probably can do it under$10. I wouldn't try to design the electronics until the mechanical part is done, because the design of the mechanical part may change the requirements for the electronics.

Storing the data is not a big deal. Unless insertions are somehow ordered, it's basically 2 numbers - number of resistors tested, and number of resistors passed - 4 bytes overall.
Trust me I have the insertion problem taken care of. I have been bashing my head over it for a long time and made a break through. I can take care of the removal and replacement of the resistors easily and the errors are theoretically non-existant. The system also worked well when I tested it out on a small scale with an arduino. I have absolutely no idea if it will work on the scale of thousands until I buitd a prototype and test it. I wish I could discuss the mechanical aspect with you but I beleive you will understand that some things I just cannot share online.

You mentioned the ease of the circuit design. So I will have 50,000 wires coming down, each requiring sequential measurement of resistance. How would you go about designing the data logging circuit for this monstrosity? Ignore the issue related to the removal or replacement of resistance values and the contact resistances. The system also doesnt need to be super accurate or overtly fast.

WBahn

Joined Mar 31, 2012
26,398
Could you please clarify this statement? I couldn't follow.

Let me try and clarify more.

Imagine there are 10 'items' on a shelf. You are tasked with tracking whether each item stays in place or is removed.

10kohm measured at a location (lets say) = 'Item is present'
Infinite resistance measured = 'Item is absent'
5kohm measured (on the same location) = 'This item has been misplaced from the 5kohm shelf'

This is it.
So are these 50,000 resistors initially not part of any circuit and the ONLY think they would be connected to would be the ohmmeter?

Time available to measure these values can be anywhere from 1 minute to 1 hour after which the collected data is transmitted to a PC and the process restarts again.
There will be one such 'set'. A set containing 50k resistors. Each read sequentially.
The cost of the hardware should be less than \$250.
Think about the numbers. Even if you allowed a full hour to take the measurements, you need to measure 14 resistors each second. If you only have a minute, then that goes to 833 measurement a second.

If you only have one set and these resistors are not part of some circuit (other than the ohmmeter), then what's the point of restarting the measurement process on that same set as soon as you finished the first round?

John P

Joined Oct 14, 2008
1,909
I think this is initially a hardware design challenge, much more than it's a microprocessor project.

Are all the resistors present at the same time, can the wiring be done so as to suit the measurement circuit, or are you stuck with some existing wiring design? How is connection being made to the resistors, and does insertion/removal need to be an easy process? How accurately must the measurements be made?

And as always, "What is the budget?" This is not going to be something you'll put together on a breadboard with the TV on.

Edited now that I see you're saying the insertion process is already designed. But what if I say you should wire the sockets in a matrix, so a resistor can be installed at every crosspoint? That's an alternative to one wire, or two wires, for every resistor.