Efficient data storage on Ruuvitag

[espruino-discuss]

Posted at 2018-07-23 by Melonator

Hey, I'm not sure if this is the right place to ask but I'm having a problem combining two Espruino/JavaScript concepts on my RuuviTag sensor beacon and I was wondering if anyone here had any insight into the matter.
Basically, I'm trying to use a variation of the program on this web page: [https://lab.ruuvi.com/temp-logger/]

But I'm trying to modify it so that I can store a lot more values by lowering the precision and storing values that take up less space in the RuuviTag memory.
For that purpose I found this Espruino method which supposedly lets you store sensor reading as 8 bit integers instead of 32 or 64 bit floats: [https://www.espruino.com/Data+Collection]
Here is what I have right now:

const period = 1000; // milliseconds (every 30 minutes)
const length = 10; // number of records (for 100 hours / 4 days)

var ruuvi = require("Ruuvitag");
var records = new Int8Array(length);

function log() {
  ruuvi.setEnvOn(true);
  if (records.unshift({time:getTime(),temp:ruuvi.getEnvData().temp}) > length){
    records.pop();
  }
  ruuvi.setEnvOn(false);
}

function getData() {
  records.forEach( function(d,i) {
    console.log(i+"\t"+ d.temp +"\t"+ (getTime()-d.time));
  });
}

ruuvi.setAccelOn(true);
setInterval(log, period);

When I upload and run this code it says this over and over:

in function called from system
Uncaught Error: Function "unshift" not found!
 at line 2 col 15
  if (records.unshift({time:getTime(),temp:ruuvi.getEnvData(...
              ^

And when I type the getData() command into the left side while the program is running I get this:

in function called from system
>getData()
0	undefined	NaN
1	undefined	NaN
2	undefined	NaN
3	undefined	NaN
4	undefined	NaN
5	undefined	NaN
6	undefined	NaN
7	undefined	NaN
8	undefined	NaN
9	undefined	NaN
=undefined

Am I missing something about the code? Or is it harder than I think it is to log values that take up less space in the memory? I don't see any reason why the unshift function would not be found. If unshift won't work for this type of operation, is there something else that will?

[espruino-discuss]

Posted at 2018-07-24 by @gfwilliams

The issue is that Int8Array/etc are 'typed' arrays. They point to a fixed size chunk of memory. As a result, things like shift/unshift/push/pop that alter the length can't be implemented efficiently - in fact they're not even part of normal JavaScript for those arrays: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Int8Array

Normal arrays let you do all that stuff, but then you pay for that flexibility with inefficient storage.

So, the solution is either to keep an index in the array and move around (shown on that Data Collection page), or to actually move all elements of the array backwards each time, and shove your data onto the end.

function log() {
  ruuvi.setEnvOn(true);
  // move data back 2 at a time
  records.set(new Int8Array(records.buffer, 2/*bytes*/)); 
  // add new data
  records[records.length-2] = getTime();
  records[records.length-1] = ru­uvi.getEnvData().temp;
  ruuvi.setEnvOn(false);
}

Also, since you have an array of bytes you can't just push the object {time:getTime(),temp:ru­uvi.getEnvData().temp} into the array - you need to split it down into individual bytes manually as I have above.

At that point, getTime() won't be too useful since you can only store values -128..127, but the temperature will be fine. You could always use new Int32Array(length) to store bigger values though.

---

Posted at 2018-07-24 by Melonator

Thanks a lot, that really helps!

---

Posted at 2018-07-30 by Melonator

On another slightly related track, Once I get values and store them to the RuuviTag, is there a way to move them to the internal storage of the raspberry pie? Or does the Espruino IDE have the capability to store data locally?

---

Posted at 2018-07-31 by @gfwilliams

There's nothing built-in. For simple things I'd write a function on the RuuviTag that outputs a tab or comma-separated list, and then you just copy&paste.

You could write some code on the Pi that connected, executed that command, waited for the result and then disconnected if you wanted to do it automatically.

If you have the Pi running all the time and within radio range, you can advertise the data and then use https://github.com/espruino/EspruinoHub - that'll now log any data that is received automatically.

---

Posted at 2018-07-31 by Melonator

Cool. Thanks for the suggestions!