SGSLP function block: Logging to Text File on external drive

rexpeter · 2016-11-10 16:55:24

Hi,
I'm looking for an example how to store some variables into a text file on an external drive (USB stick on RPi, or network drive)
On some topics here on the forum there are references to use the SGSLP function block, but an example makes it normally way more easy to understand the functionality.

Let's say we take 2 pieces of 1-wire temperature sensors and sampling every 5 seconds, but store an average of the temperature reading for each sensor every minute to a logging file (TXT file, or CSV file or something like that).
In order to see WHEN these temperatures occur, a time-stamp should be included in the file.
Is it possible to select the delimiter between the parameters (like ";" or "TAB" etc.)?

I can imagine we need to save data first to RAM drive in order to avoid too many write cycles to USB drive or external drive and do the processing (averaging of data over 1 minute and eventually formatting the output (instead of 24.23019383 degrees C, that is will become 24.2 degrees C) before it will be saved to the logging file.

Maybe there is somewhere already available that I simply have overlooked up to now, but I could not find it.

Thanks in advance,

Regards,
Peter

Last edited by rexpeter (2016-11-10 16:56:18)

scoobsalamander · 2016-11-10 22:59:08

Is a database an option? (MySql) There are plenty NAS drives who are able to run a MySql dB server.

rexpeter · 2016-11-10 23:14:49

Hi Scoobsalamander,

Honestly said I would prefer a straight forward text file / cvs file. But storing it to a NAS drive instead of a USB connected device on the raspberry itself is certainly a nice option.

Regards,
Peter

scoobsalamander · 2016-11-10 23:43:23

Maybe you should take a look at the EPC FB.

This is the 'read me' file from one of the examples....

EPC - Calling external programs 
===============================
 
This is an example on calling external programs on Linux machines. The 
example is based on the EPC function block, which requires a licence for 
advanced function blocks.

The first input (EPC:uVec1) receives sine, sawtooth and binary signals from 
generators. These data is sampled and stored in a temporary file. The data at 
the second input (EPC:uVec2) is not stored because the 2nd input is not in the 
*sl* list (parameter of the EPC block). You trigger the execution of the 
external program by the EXEC input. At that moment the captured data gets 
processed. Also the actual values at the second input get processed.  

In this example the executed command is  
/bin/bash /rex/rexcore/epc_script.sh

The outputs of the EPC block display the results of the external script. 
The results can be static values or a sequence of data. In this example 
the first output displays static values and the second one displays a sequence 
of data. 

Remember that **frequent writes to SD cards or compact flash disks can corrupt 
your storage medium**. Consider using tmpfs (ramdisk) for working files of the 
EPC function block. You will need to define a symlink to e.g. /tmp or /run/tmp 
inside the /rex/rexcore directory.

## Timing of the project ##

The algorithm runs each 500 milliseconds (0.5 s). See the EXEC function block,  
tick x ntick0 = 0.1 x 5 = 0.5 

## Prerequisites ##
- RexCore and AdvBlk modules must be installed and running on the target device.
- A valid licence for advanced function blocks is required.
- Copy the .sh file and the .csv files to /rex/rexcore on the target device.

## Running the example ##
- The **exec.mdl* file is the project main file.
- Open it with *RexDraw*, compile and download it to the target device.
- Switch to online mode and watch the algorithm.
- Enable online monitoring of the VTOR blocks (Target->Monitor selection) and 
the EPC block.
- Change the BSTATE parameter of the MP_EXECUTE block and watch the outputs of 
the VTOR blocks.
- Inspect the working files of the EPC block in /rex/rexcore/ on the target 
device.

## Documentation ##

- **Press F1 for help** on the selected function block in the *RexDraw* program.
- [EPC function block documentation](http://www.rexcontrols.com/media/HTML/DOC/ENGLISH/EPC.html)
- [Function blocks of the REX Control System](http://www.rexcontrols.com/media/HTML/DOC/ENGLISH/index.html)
- [Complete documentation of the REX Control System](http://www.rexcontrols.com/documentation-and-support)

## Additional information ##

- Visit the [REX Controls company webpage](http://www.rexcontrols.com) 
for more information about the example projects and developing advanced 
automation and control solutions using the REX Control System.

Or the REXLANG FB can also be used to write data to a file....

What do you want to do with the data? Graphs? Reports?

rexpeter · 2016-11-11 19:38:26

Thanks for sharing this, appears an nice alternate solution. But it still does not explain to me the SGSLP function block, and I am interested to see how that one should be implemented.
So anyone experience with that block and can help me on the requested example?

Regards,
Peter

rexpeter · 2016-11-15 14:22:14

Hi Scoobsalamander,

Sorry I forgot to mention what I want to do with the data.
The idea is to simply have historical data files (for instance from every month) as backup file, used as evidence for proving the data was good at that period of time. No immediate need to show that data directly in the User Interface, but to be able to send it on request of auditors that want evidence. They can open it up in Excel to see the data eventual in a trend plot, or as numerical data.

Still hoping someone can provide an example of using this SGSLP function block.

Regards,
Peter

scoobsalamander · 2016-11-15 14:37:52

I believe the SGSLP is to be used to save and get parameters from certain function blocks. Like this you should be able to adjust parameters during runtime and save them(tuning parameters from PID FB for example). If you for example have to restart after a power failure you can then load the saved parameters again.

Using the REXLANG FB makes more sence I believe....then you could use something like this :

void SaveValue(fileid, idx, value)

rexpeter · 2016-11-18 22:41:33

Hi all,

This is more or less what I want to achieve. A TXT or CSV file that provides and overview of the historical parameters saved once every minute with date / time stamp and a header information.
Which function block to use for such kind of archive / logging file? If such files automatically can be stored as a single file once every week or month it would be great.

There or various blocks but I could not find a nice example on how to achieve this. And I feel this might be a commonly used thing in industrial applications.

So basic question: Who can provide a basic example that is doing such a job?

Example file below (TAB separated values):

TIME MM DD T01 T02 T03 T04 T05 T06 T07 Operate Cycle RunHrs ActiveHrs Counter1 LatchingAlarm
00:00 02 01 41.16 47.9 52.74 3.07 0.51 7.27 0.23 0 1 678 672 1329 0
00:01 02 01 40.98 49.38 54.48 3.13 0.51 7.27 0.23 0 1 678 672 1329 0
00:02 02 01 40.89 48.53 53.28 3.35 0.51 7.28 0.23 0 1 678 672 1329 0
00:03 02 01 40.83 53.05 54.94 3.13 0.51 7.28 0.22 0 1 678 672 1329 0
00:04 02 01 41.12 45.04 54.85 2.72 0.51 7.27 0.22 0 1 678 672 1329 0
00:05 02 01 40.81 49.18 53.99 3.07 0.51 7.25 0.2 0 1 678 672 1329 0
00:06 02 01 40.54 50.24 53.83 3.26 0.51 7.23 0.22 0 1 678 672 1329 0
00:07 02 01 40.43 49.39 53.47 3.43 0.51 7.25 0.22 0 1 678 672 1329 0
00:08 02 01 40.41 48.81 53.61 3.14 0.51 7.27 0.22 0 1 678 672 1329 0
00:09 02 01 40.35 48.12 53.29 1.99 0.51 7.26 0.22 0 1 678 672 1329 0
00:10 02 01 40.28 49.02 53.38 2.7 0.51 7.24 0.23 0 1 678 672 1329 0
00:11 02 01 40.16 49.39 53.66 3.18 0.51 7.22 0.23 0 1 678 672 1329 0

Regards,
Peter

scoobsalamander · 2016-11-19 01:11:42

I would just use the TREND FB and store the data in an archive.

From within REXview you can export the data fromt the TREND and/or Archive to a CSV file.

Small example (original example + TREND block)

Last edited by scoobsalamander (2016-11-19 01:35:04)

rexpeter · 2016-11-19 13:11:07

Hi Scoobsalamander,

Thanks for your proposal, but i intend to use the device remotely without the using rexview. So the data logging / archieving to a CSV, or TXT file should be done automatically every week or every month. Without having checked your example, I think this is not happening if you speak about the export from the trenplot data inside RexView.
That is the reason I expected that there is another function block like SGSLP available for this purpose.

So I guess still not the solution I am looking for....

jaroslav_sobota · 2016-11-20 22:53:38

Hi Peter and Scoobsalamander,
thanks for all your comments. SGSLP block is not intended for data logging. It is intended for storing the changes in parameters permanently (e.g. user-defined temperature setpoints via web HMI).

I would use the TRND block for averaging and archiving the data and move it to a database afterwards as Scoobsalamander recommended. Since this is not an option for Peter, I would use the AVG block for averaging the temperatures and the EPC block for storing the data in a text file. However, from there, you have to rely on "Linux magic" and use e.g. sed or awk commands. It's also possible to use the REXLANG block, which is very similar to the EPC-based approach.

I'll get back to this topic and create some examples based on your discussion, but not in the next week since I'm at SPS/IPC in Nuernberg/Germany.

Jaroslav

rexpeter · 2016-12-13 23:29:07

Hi Jaroslav,

I'm still very interested in the way HOW to simply log data within RexControl system (without the need of running a separate script where data will be logged independently from the RexControl system. Whenever you have the possibility to come back to this, I looking forward for this.
Thanks in advance,

Regards,
Peter

jaroslav_sobota · 2016-12-19 14:33:41

rexpeter wrote:

Let's say we take 2 pieces of 1-wire temperature sensors and sampling every 5 seconds, but store an average of the temperature reading for each sensor every minute to a logging file (TXT file, or CSV file or something like that).
...
I can imagine we need to save data first to RAM drive in order to avoid too many write cycles to USB drive or external drive and do the processing (averaging of data over 1 minute and eventually formatting the output (instead of 24.23019383 degrees C, that is will become 24.2 degrees C) before it will be saved to the logging file.

Hi guys,
for averaging, you can use:

AVG block - in this case it must be placed inside a task with sampling period 5 seconds and n=12 should be set. This will give you a 1-minute running average. Therefore you can export its output directly, without any need for postprocessing.
TRND block - in this case the sampling period of the task can be arbitrary, you just have to set the pfac parameter so that pfac*Ts=60, where Ts is the sampling time in seconds. It is also necessary to set ptype1=5: Average. By that, you'll get the average temperature in 1-minute intervals at the y1 output.

Both of these approaches are handled in RAM and give you the value which you want to store in a text file.

Hope this helps somehow. I'll add some more hints...

Jaroslav

rexpeter · 2016-12-20 06:44:09

Thanks Jaroslav for your feedback on how to average data and to display it nicely in a trend plot.
But this unfortunately still does not give me the information on how to log to a txt or csv file using the most practical or easiest way.
I know there is archive function and a trendblock tha might capture data, but what is the most easy way to process the original request "logging data to file on an external device (USB drive or NAS)?
Thanks in advance.

Thanks,
Peter

et0 · 2017-01-10 18:18:34

Hi all, I'd be very interested to see an example of this, as I need to do the same kind of thing:

Store a "line" of comma separated temperature and other values, every one minute or so, onto a flash drive or similar for later analysis purposes.

However it is done, code or a module, an example would be really useful!

Thanks
Regards
Evan

rexpeter · 2017-01-11 15:43:36

Hi Evan,

I agree and factually still looking forward for an example of "logging data to a text file" as well.

Regards,
Peter

jaroslav_sobota · 2017-01-12 16:12:38

Hi guys,
it took way too long, but here it comes. See this example on logging data to a text file using the REXLANG block.

The README file should contain all the information you need. I am not writing more about the example on purpose. Your comments are highly welcome.

Regards,
Jaroslav

Last edited by jaroslav_sobota (2017-01-12 19:32:33)

rexpeter · 2017-01-13 13:07:37

Hi Jaroslav,

Thank you very much for this example.
Based on the "relative complexity" of such a basic feature, I strongly suggest to have a function block available for this.

It is not a complaint, but a challenge since I only refer to the features on the website about Rex Control stating:

Features of the REX Control System
Graphical programming without hand-coding

But nevertheless with this example it appears to be easy to create what I would like to have.
Thanks once more!

Best regards,
Peter

jaroslav_sobota · 2017-01-13 16:07:07

Hi Peter,
glad I could help. You are right, the code is way too complicated for such a simple task. We'll add a few more commands to REXLANG, which will simplify the code.

On the other hand, there is an internal archiving system in REX, which is way more efficient than storing data in a text file. The downside is that (as of current version 2.50.1) the data can be extracted only using RexView. If there was a command-line utility to extract data from the internal archive to CSV file, which you could run e.g. every day at midnight, would that fit your needs?

As for the hand-coding, I understand your comment and thank you for that. It is not in human power to have function blocks for everything and completely avoid hand coding. Still I believe there is a lot you can do with REX without typing a line of code...

Moreover, in some scenarios, a few lines of code are worth tens or hundreds of function blocks. It's the possibility to combine these two approaches almost seamlessly which we consider one of the big advantages of REX.

Kind regards,
Jaroslav

rexpeter · 2017-01-13 23:18:32

Hi Jaroslav,

Thanks for your feedback on this.

"On the other hand, there is an internal archiving system in REX, which is way more efficient than storing data in a text file. The downside is that (as of current version 2.50.1) the data can be extracted only using RexView. If there was a command-line utility to extract data from the internal archive to CSV file, which you could run e.g. every day at midnight, would that fit your needs?"

That would be great of course!
Thanks once more for all your effort providing these examples and information!

Regards,
Peter

rexpeter · 2018-12-12 22:38:22

Hi Jaroslav / Tomas,

Just wondering if there is anything like this as you mentioned available meanwhile. A command-line utility to extract data from the internal archive to CSV file.

Or a new function block to store data to a logging file? Maybe in the new version of Rexygen?

Regards,
Peter

jaroslav_sobota · 2018-12-13 08:51:10

Hi Peter,
thanks for your question. The new version is just around the corner. What you ask for is not included, sorry. It's definitely on our todo list for the following one.

Kind regards,
Jaroslav

REXYGEN Community Forum

Announcement

REXYGEN forum has been moved. This one is closed.Head over to the new REXYGEN Community Forum at https://forum.rexygen.com.Looking forward to meeting you there!

#1 2016-11-10 16:55:24

SGSLP function block: Logging to Text File on external drive

#2 2016-11-10 22:59:08

Re: SGSLP function block: Logging to Text File on external drive

#3 2016-11-10 23:14:49

Re: SGSLP function block: Logging to Text File on external drive

#4 2016-11-10 23:43:23

Re: SGSLP function block: Logging to Text File on external drive

#5 2016-11-11 19:38:26

Re: SGSLP function block: Logging to Text File on external drive

#6 2016-11-15 14:22:14

Re: SGSLP function block: Logging to Text File on external drive

#7 2016-11-15 14:37:52

Re: SGSLP function block: Logging to Text File on external drive

#8 2016-11-18 22:41:33

Re: SGSLP function block: Logging to Text File on external drive

#9 2016-11-19 01:11:42

Re: SGSLP function block: Logging to Text File on external drive

#10 2016-11-19 13:11:07

Re: SGSLP function block: Logging to Text File on external drive

#11 2016-11-20 22:53:38

Re: SGSLP function block: Logging to Text File on external drive

#12 2016-12-13 23:29:07

Re: SGSLP function block: Logging to Text File on external drive

#13 2016-12-19 14:33:41

Re: SGSLP function block: Logging to Text File on external drive

#14 2016-12-20 06:44:09

Re: SGSLP function block: Logging to Text File on external drive

#15 2017-01-10 18:18:34

Re: SGSLP function block: Logging to Text File on external drive

#16 2017-01-11 15:43:36

Re: SGSLP function block: Logging to Text File on external drive

#17 2017-01-12 16:12:38

Re: SGSLP function block: Logging to Text File on external drive

#18 2017-01-13 13:07:37

Re: SGSLP function block: Logging to Text File on external drive

#19 2017-01-13 16:07:07

Re: SGSLP function block: Logging to Text File on external drive

#20 2017-01-13 23:18:32

Re: SGSLP function block: Logging to Text File on external drive

#21 2018-12-12 22:38:22

Re: SGSLP function block: Logging to Text File on external drive

#22 2018-12-13 08:51:10

Re: SGSLP function block: Logging to Text File on external drive

Board footer

REXYGEN forum has been moved. This one is closed.
Head over to the new REXYGEN Community Forum at https://forum.rexygen.com.

Looking forward to meeting you there!