Move all configuration from BSB_lan_config.h to webinterface, stored on EEPROM

[fredlcore]

In order to be able to distribute compiled binaries of BSB-LAN, the complete configuration should be possible via the webinterface. Most parameters can already be configured during runtime via URL commands, so these could be called in the background via JavaScript etc.
More problematic could be the various #define configurations, but since we now have enough space on the Due, all functional parts (de-)activated via #defines should be enabled by default and could then be disabled on the Mega if necessary.

All data should be stored on EEPROM which is provided for the Due on the V3 BSB-LAN boards and has the same size as the one built-in in the Mega: 4096 bytes.

The EEPROM storage should be divided into two sections:
First section: Fixed length configuration options, such as pin numbers for OneWire bus, DHT22, IP-address, PPS parameters etc.
Second section: Divided into two parts for configuration options of (at compile time) unknown length, such as log_parameters array, passkey, exclude_GPIO. First part is a table layouted like this:
Byte 1+2: Offset to actual data
Byte 3: Type of data (may not be necessary if EEPROM.get() is able to determine this in all cases)
Byte 4: Length of data (or length of entries, if EEPROM.put()/get() can be used at all times)
Byte 5 to Byte 5+length: actual data

Since all configuration data is always written at the same time, the actual distribution of data in the second part of the second section can vary, but due to the lookup-table, data can easily be located.

In order to maintain backward compatibility, variables set via BSB_lan_config.h should be written to EEPROM during setup, and then a warning should be printed in the web-configuration that all changes will be overwritten upon reboot as long as the entries in BSB_lan_config.h remain.

Most of the entries probably do not need default values, but for the network part, it would be best to set the device to DHCP mode instead of setting some kind of fixed IP address (although this would probably also be acceptable).

I hope I've covered all caveats, but I guess I have not ;)...

[dukess]

In order to maintain backward compatibility, variables set via BSB_lan_config.h should be written to EEPROM during setup, and then a warning should be printed in the web-configuration that all changes will be overwritten upon reboot as long as the entries in BSB_lan_config.h remain.

Just make a flag, that will be set to EEPROM at first run and prevent settings implementation from BSB_lan_config.h

[fredlcore]

Good idea!

[fredlcore]

Do you think a struct containing all the settings would be useful? This could probably be written in one go with EEPROM.put and could be read also in one go with EEPROM.read.

The best way would be to also add information about what kind of data an entry consists of. So whether it is Checkbox, number, dropdown... If the code could then evaluate this and build a website with the correct input fields, it would be very easy to add new configuration options later on. Any ideas?

[dukess]

The only thing I don't like about the structure idea is that it will be recorded all at once. But the number of write cycles of EEPROM is very limited, AFAIK. Or am I worrying too much about this for nothing?

[dukess]

We can write a function that, when writing new parameters, compares the contents of the structure byte by byte and writes only the changed data.

[fredlcore]

No, you are right, of course, because some options will be changed more often than others. But a function like you suggested could take care of that. The (literally) bigger problem would be the amount of RAM needed because the whole structure would have to be kept in RAM (possibly all the time). For the Due that's not an issue (4kB of EEPROM size vs. 96kB of Due RAM), but for the Mega it would not be possible. That would either be a point of departure where the Mega code is kept as it is, or the _config.h would be kept and loaded into RAM after the config is read from the EEPROM.

On my other project (robotan.de) I use a struct to access the EEPROM of the lawn robot so that I know where each configuration option is located. Maybe that's a more feasible option?

[dukess]

I do not think that there will be any problem with the availability of free memory, since almost all settings stored in RAM now.
About structure: of course, for parameters whose length is known, the structure is the best solution.

[fredlcore]

Yes, that's true, but if we want to use the whole 4k EEPROM size of the Mega (or the EEPROM on the V3 BSB-LAN board), it would eventually consume all the remaining RAM available. But maybe that's never going to happen...

The other question I have is: Does any kind web-based configuration mena that we have to define a maximum number of entries for one option (such as currently with log_parameters[20]) or if a more flexible approach (such as with ipwe_parameters[]) would be possible. A fixed number of, say, 20 parameters would probably not be a major issue, but the space both in memory and flash would be blocked even if only very few parameters are being used.

Here's a first draft of a structure, in this case just for the generation of the web-interface, but could be added with the concrete (pointer to the) value of the corresponding option:

typedef struct {
	uint8_t id;		// a unique identifier that can be used for the input tag name
	const char* desc;	// pointer to text to be displayed for option - is text length necessary if we just read until NULL?
	uint8_t input_type;	// input type (text, dropdown etc.)
	uint8_t var_type;	// variable type (string, integer, float, boolean etc.), could maybe be derived from input_type or vice versa
	uint8_t category;	// for grouping configuration options
} configuration;

We could then have defines to the configuration rows to make access more human-readable; let's say the IP address is stored in row 0 of the struct:
#define IP_ADDRESS 0
then we could accesss it via
configuration.value[IP_ADDRESS]
or something like that...

[dukess]

More compact but more hard-coded

struct __attribute__((__packed__)) settings { //prevent padding
byte readsettingsfromeeprom; //0 - read from _config, 1 - read from EEPROM
byte mac[6]; //MAC address. If all zeros - init with hardware-defined MAC
byte ipaddr[4];
byte mask[4];
byte dns[4];
byte gateway[4];
uint_16t port; //1-65535
boolean logTelegram;
boolean log_unknown_only;         // should we log only unknown commands when logging bus telegrams?
boolean log_bc_only;              // should we log only broadcast commands (dest = 0x7f) when logging bus telegrams?
...
}

[fredlcore]

Maybe using two structs would then be most effective, with the one from my suggestion getting an additional field with a pointer to the position of your struct where the actual value is stored (if that is even possible)?
Because with your struct, we still would have to hard-code a web-interface, and adding a new configuration option would require adding the respective HTML code as well etc.
With the my struct above, we could just loop through the struct entries and the respective configuration entries would be displayed.

Could/should we just allocate a memory area of X byte which contains all the configuration values in the way you structured it above and then have a (calculated?) pointer that memory area which will then be written to EEPROM?

[hacki11]

I'm not sure if my input is wanted, but here it is :)
byte readsettingsfromeeprom=> byte settingsSource { EEPROM, FLASH }

I don't know about the logs but is it possible to introduce a log level instead of several bools?
uint_8 telegramLogLevel { NONE, UNKNOWN, BROADCAST, ALL } A higher level includes all lower levels, means if BROADCAST is selected, UNKNOWN will also be written.

Would stick to one naming style, lowerCamelCase or snake_case, but i think it is all just a draft for conceptual issues and you would have changed it in your final version.

[hacki11]

We can write a function that, when writing new parameters, compares the contents of the structure byte by byte and writes only the changed data.

Afaik, the EEPROM.update will do exactly that for you automatically, even writing a whole struct.

[fredlcore]

Ah, thanks, that's a helpful hint - but I'll have to check if the (non-standard) EEPROM library for the serial EEPROM we use on the Due also supports that feature.

[fredlcore]

...and it does, yeah!
https://github.com/CombiesGit/I2C_EEPROM#void-updateunit16_t-addressuint8_t-value