Basic ESP8266 data logging question

Thread Starter

ebeowulf17

Joined Aug 12, 2014
3,307
What I'd like to do is collect data with an ESP8266 and write it into .CSV files on a server hard drive in the same building, on the same network, via WiFi. I'm imagining this process being analogous to writing files to an SD card. I had assumed this would be trivial and well documented, but all of my searches so far have led only to web servers with dedicated hosting, server side data management, etc. I'd really like to have the ESP8266 generate, fill, and manage the files, with the server / hard drive being relatively passive.

What I mean by that is I'm hoping I don't need to have any special code actively running on the server to receive and save data from the ESP8266. I was hoping that with the appropriate network permissions in place that the server would just let the ESP8266 handle the files directly.

If this is possible, could someone share some links to good information on how to do this? Or even just give me some clues as to the correct keywords and search terms to use?

Maybe what I want to do just isn't realistic, but I'm hoping that I've just been doing a bad job of searching so far! Any help you all can provide will be greatly appreciated.
 

Raymond Genovese

Joined Mar 5, 2016
1,653
What I'd like to do is collect data with an ESP8266 and write it into .CSV files on a server hard drive in the same building, on the same network, via WiFi. I'm imagining this process being analogous to writing files to an SD card. I had assumed this would be trivial and well documented, but all of my searches so far have led only to web servers with dedicated hosting, server side data management, etc. I'd really like to have the ESP8266 generate, fill, and manage the files, with the server / hard drive being relatively passive.

What I mean by that is I'm hoping I don't need to have any special code actively running on the server to receive and save data from the ESP8266. I was hoping that with the appropriate network permissions in place that the server would just let the ESP8266 handle the files directly.

If this is possible, could someone share some links to good information on how to do this? Or even just give me some clues as to the correct keywords and search terms to use?

Maybe what I want to do just isn't realistic, but I'm hoping that I've just been doing a bad job of searching so far! Any help you all can provide will be greatly appreciated.
Assuming that I understand what you mean, it may not be realistic.

You can certainly have an ESP8266 connected to an SD card and have it write csv files to that card.

Certainly, the server can get info from an ESP8266 and then write a csv file to its own csv card. This is simply the ESP8266 sending client information that acts as a command to the server, to write something to a csv file - but the server is doing the actual writing because the sd card is the server's peripheral.

I don't see how the esp8266 is going to actually write csv files directly to a server's SD card (or hard drive etc..).

What would you hope to accomplish if it could, that you could not do in a standard way?
 

Thread Starter

ebeowulf17

Joined Aug 12, 2014
3,307
Maybe I've mis-spoken. I'm not trying to bypass the server operating system or anything. I just don't want to need an extra application there. Every example I've seen requires some Java or Python scripts on the server. I'd rather just have the Arduino do the work instead of having to write and install programs on the server as well. All I'm trying to accomplish is not having to write and maintain a dedicated data-receiving program on the server side.

Let me put it another way, when you say commands to the server, what might that include? Can I send a command from the ESP8266 to the server to create a file with a specific name? Can I send a command that writes data into that file? Or can I only send commands to the server if I've already written a special program on the server for receiving those commands?

I suspect (and hope) that my confusion stems largely from me mis-using established terminology. Assume that everything I'm writing could be phrased wrong or using the wrong vocabulary. I've learned a decent amount about electronics, and a little bit about Arduino and C++ code, but I'm the absolute worst when it comes to networking, file structures, operating systems, IT, etc.
 

Raymond Genovese

Joined Mar 5, 2016
1,653
Maybe I've mis-spoken. I'm not trying to bypass the server operating system or anything. I just don't want to need an extra application there. Every example I've seen requires some Java or Python scripts on the server. I'd rather just have the Arduino do the work instead of having to write and install programs on the server as well. All I'm trying to accomplish is not having to write and maintain a dedicated data-receiving program on the server side.

Let me put it another way, when you say commands to the server, what might that include? Can I send a command from the ESP8266 to the server to create a file with a specific name? Can I send a command that writes data into that file? Or can I only send commands to the server if I've already written a special program on the server for receiving those commands?

I suspect (and hope) that my confusion stems largely from me mis-using established terminology. Assume that everything I'm writing could be phrased wrong or using the wrong vocabulary. I've learned a decent amount about electronics, and a little bit about Arduino and C++ code, but I'm the absolute worst when it comes to networking, file structures, operating systems, IT, etc.
It sounds to me, and I may be wrong, that you are misunderstanding the idea of whose peripheral it is, the client's or the server's; or even just computer #1 or computer #2.

So, when I read this:

Let me put it another way, when you say commands to the server, what might that include? Can I send a command from the ESP8266 to the server to create a file with a specific name? Can I send a command that writes data into that file? Or can I only send commands to the server if I've already written a special program on the server for receiving those commands?


Yes, you can send commands AND yes, you must have written the code on the server to process those commands.

If it is the server's peripheral, then the client can't say "give me the use of your peripheral", except by programming on the server that functionally, but not literally, accomplishes the transaction - which, in this case, means that the server still is the processor that writes the information to the server's SD card.

There is something called direct memory access (DMA) that does allow another piece of hardware to "borrow" memory access of a processor, but this is a completely different concept and not terribly relevant to what you are talking about.

Beyond that, I think that the idea to embrace is that a peripheral, like an SD card "belongs" to a piece of hardware that it is connected to and to share the peripheral with another piece of hardware, and to do so wirelessly, is not too realistic at the present time.

To illustrate, an SD card has a set of connections to one processor (say a server) and that is the processor that controls the writing to the SD card (technically, the connections are actually to an embedded processor within the SD card). You can't have another processor (say a client) sharing those connections so that both could access the SD card unless you had a way to enable one set of connections and disable another set of connections.

Not sure I know of a case where that is done with wireless connections or even wired connections - although someone else may know of some examples.
 

Thread Starter

ebeowulf17

Joined Aug 12, 2014
3,307
It sounds to me, and I may be wrong, that you are misunderstanding the idea of whose peripheral it is, the client's or the server's; or even just computer #1 or computer #2.

So, when I read this:

Let me put it another way, when you say commands to the server, what might that include? Can I send a command from the ESP8266 to the server to create a file with a specific name? Can I send a command that writes data into that file? Or can I only send commands to the server if I've already written a special program on the server for receiving those commands?


Yes, you can send commands AND yes, you must have written the code on the server to process those commands.

If it is the server's peripheral, then the client can't say "give me the use of your peripheral", except by programming on the server that functionally, but not literally, accomplishes the transaction - which, in this case, means that the server still is the processor that writes the information to the server's SD card.

There is something called direct memory access (DMA) that does allow another piece of hardware to "borrow" memory access of a processor, but this is a completely different concept and not terribly relevant to what you are talking about.

Beyond that, I think that the idea to embrace is that a peripheral, like an SD card "belongs" to a piece of hardware that it is connected to and to share the peripheral with another piece of hardware, and to do so wirelessly, is not too realistic at the present time.

To illustrate, an SD card has a set of connections to one processor (say a server) and that is the processor that controls the writing to the SD card (technically, the connections are actually to an embedded processor within the SD card). You can't have another processor (say a client) sharing those connections so that both could access the SD card unless you had a way to enable one set of connections and disable another set of connections.

Not sure I know of a case where that is done with wireless connections or even wired connections - although someone else may know of some examples.
Sure, the hard drive belongs to the server, but I can create and modify files on that hard drive from any computer on the server (with the right credentials.) Once I've logged my laptop into the "secure" network, I'm able to create, copy, and delete files on the server just as if they were on my laptop. I don't have to write special programs on the server to accomplish that. The basic networking structure and operating systems handle that.

I don't want the Arduino to take direct, complete control of the hard drive. I don't want to bypass the server completely. I want the operating system on the server to create or modify files on its hard drive in response to what the Arduino says.

So I guess another way to approach the question would be: Can the Arduino send commands that are similar to what another windows computer sends when it asks to create or move files?
 

Raymond Genovese

Joined Mar 5, 2016
1,653
/---/
So I guess another way to approach the question would be: Can the Arduino send commands that are similar to what another windows computer sends when it asks to create or move files?
Oh, ok, I think I have a little better idea of what you mean....get an arduino to mimic a user on a window's network...logging in and sending "keypresses" like a regular human user, so to speak.

Yeah, I guess you could do that, but I don't know how easy it would be.

Now, contrast that with the more standard way: Take a look at a project that I did a few years ago here http://www.esp8266.com/viewtopic.php?f=11&t=6854&p=36831&hilit=sensor+board#p35199

The ESP side sends a packet (a csv line) to a simple windows side listener, which writes the line to a file on the windows machine - eazy peazy.
 
Top