I have two Raspberry Pis, connected by an ethernet cable. Can I somehow convince the master Pi that the other Pi is its own right hand? Camera, USB ports, GPIO, SD card files, serial devices, all of it; I want to be able to access all of these resources (or as much as possible) of the 2nd pi, from the first pi, as if they were local to the first pi.

Take the MIPI camera port for example, currently I have on the master Pi, something like (fictitious pseudo code) :
picture1 = myCamera.takePicture(res='64mp')
and for the master to get a picture from the second camera on the other pi (running flask server) it is something like:
picture2 = httpGet.send('http://ip.of.2nd.pi/camera/takePicture?res=64mp')
which will prompt the 2nd pi to take a picture with its camera and send the image back to the master as a payload.

Today I discovered pickle so now I can send actual python objects back and forth and don't need to keep wrapping all my Python functions in HTTP methods, but can it be even simpler than pickle?

Can it be as simple as:
picture1 = pi1.myCamera.takePicture(res='64mp') # this camera exists on this Pi
picture2 = pi2.myCamera.takePicture(res='64mp') # this camera exists on the remote Pi
?
If so, how?

Please don't get focused on the camera thing and start suggesting multi-camera boards. I know they exist (yes even for 64mp sensors) but there is a lot more going on than just two cameras and I have good reasons for using two separate Pis. I'm trying to strike a balance between TL;DR and preemptively defending against the inevitable assertion that I'm making things more complicated than they need to be, so please forego the customary 8 pages of vetting and just trust that the question I'm asking is relevant to the project that birthed it.

 

Per your last paragraph, No.

Possible by virtualization but impractical on a RPI if you want close to normal performance of the device. https://developer.arm.com/documenta.../Raspberry-Pi-4/Quickstart-for-Raspberry-Pi-4


You could VNC into each system for remote display and operation but that's not an extension.

 

Do you know about multiplexers (and synchronizers)?

It is not what you asked for, but depending on what your are doing it may be what you need.

 

Per your last paragraph, No.

If anyone would know, it would be you. So here's me making away with tail tucked. Thanks for the link, I was not aware you can make a virtual Pi. I was actually looking for a way to do that last month and didn't find anything.

 

Do you know about multiplexers (and synchronizers)?

It is not what you asked for, but depending on what your are doing it may be what you need.

Please don't get focused on the camera thing and start suggesting multi-camera boards. I know they exist (yes even for 64mp sensors) but there is a lot more going on than just two cameras and I have good reasons for using two separate Pis.

Sorry to have missed this part—I’ll admit I am not fully up to normal capacity recently.

Hope you find your solution.

 

If anyone would know, it would be you. So here's me making away with tail tucked. Thanks for the link, I was not aware you can make a virtual Pi. I was actually looking for a way to do that last month and didn't find anything.

That's the short answer. The 8 pages answer is sure, it's possible, but it takes a lot more than Python functions in HTTP methods to be actually usable. Most of the time you don't really care about the physical sensor, you care about the data for that sensor. Using the Home Assistant automation system as an example: You abstract the data from Cameras, USB ports, GPIO, SD card files, serial devices, etc ... into client/server accessible server ports from many machines/devices and create overall integration views (Dashboards) of that data.

 

That's the short answer. The 8 pages answer is sure, it's possible, but it takes a lot more than Python functions in HTTP methods to be actually usable. Most of the time you don't really care about the physical sensor, you care about the data for that sensor.

Right, I care about the data.
This is what I care about in order:
1. sharing a heavy load between two CPUs
2. Making things simpler from a programming perspective
Your dashboard is very sleek but I don't need all that and it sounds to me like what I've already got is about as simple as it gets.

 

Right, I care about the data.
This is what I care about in order:
1. sharing a heavy load between two CPUs
2. Making things simpler from a programming perspective
Your dashboard is very sleek but I don't need all that and it sounds to me like what I've already got is about as simple as it gets.

1. What types of loads? If the load is CPU limited number crunching on blocks of data can be processed in parallel there are systems like:
https://www.learnpdc.org/RaspberryPi-mpi/

https://www.suzannejmatthews.com/post/2022-03-03-raspberrypi-cluster-headless/
Likely cheaper and more reliable just to use a single RPi 5.

2. It's just an example. There are more lightweight ways to do the same thing.

 

1. What types of loads? If the load is CPU limited number crunching on blocks of data can be processed in parallel there are systems like:

Mainly, it is the load of the 64MP camera modules. I built a single-camera system running a multithreaded app that cyclically reads and writes to various devices (6+) over UART, USB, SPI, I2C, and reads/writes GPIO, and takes pictures. There is no polling interval, it all just happens at whatever speed is possible. As a threaded task ends, it goes back into the threadpool queue. With one system (one pi, one camera, one set of external devices) the overall speed was just barely acceptable. Then I duplicated the system (second pi, second camera, second set of external devices) but I need those systems to run as a single process. So I copied all the same scripts to the second Pi but instead of running its own multithreaded app, it runs a Flask server. The multithreaded app on the first Pi now has HTTP calls added to the threadpool, which get the needed data from the second pi.

I'm not sure if this kind of application is suited to the type of distributed system you linked to. What do you think? I only dipped my toes into the links and it feels complicated, and like it would be a lot of time invested to find out if it's a good fit.

Likely cheaper and more reliable just to use a single RPi 5.
2. It's just an example. There are more lightweight ways to do the same thing.

Thanks, I think you might be right. I did do some research on the RPi5 back when it came out but I did not see anything that made me think it would offer any advantage over the RPi4 for this application because I assumed the speed would be limited to the MIPI port, and an issue which limits the 64MP Arducam to one sensor per board would still apply, so I would still need two boards to take two pictures. But on your prompt I had a second look and found this:

I just tested RPI5 8GB bookworm LITE with dual arducam-64mp hawkeyes. The capture speed with max res 9152x6944 using libcamera-jpeg is ~3 seconds vs 5-7 seconds on RPI4.

on RPI4 it was not possible to use two cameras on one device at max res due to some IO issue couldnt make some buffer so i had to have two RPI4 one for each camera. RPI5 doesent have such issue.

So it seems that if I upgrade to RPi5 I can use both cameras on one board, and the time to take two pictures could be about the same as taking a single picture on the RPi4. It will have twice as many external devices to communicate with, so the overall speed of the dual camera RPi5 system might be slightly slower than that of my single camera RPi4 system, or I might be again surprised. It is definitely worth a try. I just dropped $260 on a Pi5 plus active cooler, NVMe adapter, WD Black M.2 SSD, and 27W power supply. If it has a chance of matching or beating the speed of my single camera system, it probably only has that chance if it is not trying to operate off an SD card.

 

If you are going to use the network, drop HTTP and do something with UDP and a bare bones RPC protocol. The overhead for an HTTP exchange is way more than you want or need. You could do something very simple since you have a very constrained set of operations.

 

If you are going to use the network, drop HTTP and do something with UDP and a bare bones RPC protocol. The overhead for an HTTP exchange is way more than you want or need. You could do something very simple since you have a very constrained set of operations.

I am working on that as we speak. I don't like Flask anyway; its implementation of threading is... weird. I don't really know what it's doing, something screwy with the timeline. For example I have a serial device that sometimes fails, and when it does, further calls to it result in permissions errors. I handle this in my main threaded app by closing the port, deleting the object, and creating a new one:

import myScript

mySerialThing = myScript.serialThing

def thisGetsCalledCyclically():
    try:
        mySerialThing.getData()
    except PermissionError:
        mySerialThing.ser.close()
        del (mySerialThing)
        mySerialThing = myScript.serialThing

This works just fine with PyQt5 threading, but under flask it gives errors like "mySerialThing referenced before assignment." It's like it just scans the script from top to bottom and sees that a mySerialThing object is created on line 11 but there is a call to mySerialThing higher up on line 7, so it assumes that mySerialThing doesn't exist prior to line 11, even though I already created it on Line 3. And it generates this exception as soon as I start the server, before I ever even make a HTTP call to the function dealing with mySerialThing. So my theory is that flask has some built-in 'idiot proofing' that is checking the server script for potential booboos before execution, but it's just not smart enough to be entrusted with idiot proofing. If I get rid of the last line it works just fine (except if/when that permission error happens, then it never works again until the server is restarted).

EDIT
I only used flask because it has threading built in. So if for example it's waiting the 5-7 seconds to capture a picture when a call to mySerialThing comes in, it can respond to that call without waiting for the picture to complete first. Any idea how to replicate that behavior without Flask? I currently only have an understanding of how threading works within PyQt5 and I don't want to write another PyQt5 script for the second Pi. I am doing the research myself at this moment but if you have any pointers I am all ears.

Thanks.

 

@Ya’akov
see edit to previous post

 

@Ya’akov
see edit to previous post

Well, first—I am not a Python programmer. I know just enough to patch existing programs when I find them broken, etc. But, if I was writing this in perl I would probably use an event loop and forking rather than threads. There are several excellent event loop modules for perl, and I have to think there would be something for Python.

My habit of eschewing threads probably grew out of the fairly lousy thread support in perl. But I didn’t miss them. However, they may have advantages for you in this case, I really don’t know.

There must be a generic threading library for Python that would do what you want…

 

Well, first—I am not a Python programmer. I know just enough to patch existing programs when I find them broken, etc.

Ah, ok sorry.

There must be a generic threading library for Python that would do what you want…

There are many, that's the problem. When I go to a restaurant I've never been to, it is comforting when there are only 2 or 3 things on the menu.