New LMIC can be busy after TXCOMPLETE: need to document, fix sample apps for clarity

[terrillmoore]

The behavior of the LMIC changed between 2.3.2 and 3.10.0, which requires that apps be more careful about sending messages. With the upgraded MAC code, the LMIC and the network can get involved in lengthy discussions after an uplink. However, the LMIC correctly notifies the app whenever a transmission completes. We need to:

• document the behavior clearly
• add information about how best to deal with the situation of extended network conversations
• most important, update the sample apps to follow best practices.

Original comment from @DeveloppeurPascal follows.

I found the reason of the problem : automatic MAC response throw TXCOMPLETE event, so I send next message, but the previous payload was pending... a new payload is sent during send process from the previous one, so it disappear.

We must modify the do_send() function like this :

  if (LMIC.opmode & OP_TXRXPEND) {
   Serial.println(F("OP_TXRXPEND, not sending"));
  } else if (LMIC.opmode & OP_TXDATA) {
   Serial.println(F("OP_TXDATA, not sending"));
  } else {
      LMIC_setTxData2(1, LoRaWanMsgToSend, LoRaWanSizeOfMsgToSend, 0);
      // Next TX is scheduled after TX_COMPLETE event.
  }

Please add the OP_TXDATA test in your samples to prevent the problems I add.

Originally posted by @DeveloppeurPascal in #545 (comment)

[cyberman54]

@terrillmoore a question in this context: i added code to my app to print uplink and downlink MAC commands. It's not fully working yet, i am not sure if i grab all MAC commands.

With LMIC 3.1.0, what would be suitable place to process MAC commands by an app?

I currently do it in the lmic_rxmessage_cb_t RX callback function.
I see some downlink MACs, but seem to grab to much data from LMIC.frame whch cannot be resolved in MAC CIDs. This way i see also uplink MACs, but not the first which transmit immediately after join (in my case this is a DevTimReq).

[Rock-N-RollaMann]

First just wanted to say thank you to everyone who contributes(d) to this library. Moreover just wanted to add my experience I had this problem with the update version of the "Lmic" library and added the OP_TXDATA to my code, however I had the same results. Currently I am using the older version of the "Lmic" and everything is working fine.

[altishchenko]

There is a subtle logic issue here. If for some reason the network server does not acknowledge a confirmed uplink, but still sends some commands with port 0, fSuccess flag in the call to txMessageCb will still be set true, because pendTxConf flag gets reset to 0 in the 'open code the logic' block ( lmic.c:2351 ) and is further treated as unconfirmed uplink by TXCOMPLETE event.
Because of this code block, pendTxConf is an unreliable source of confirmed/unconfirmed indication and can't be used in the event logic processing functions.

[pomplesiegel]

I believe this is related:

I am seeing behavioral changes in v3.x (vs 2.x) such that

1. The program used to transmit a full payload right after a join, but this no longer happens
2. The second message has no payload, and this happens occasionally after many transmits as well

These appear on TTN like so:

[Rock-N-RollaMann]

Yea, I agree I've been seeing that behavior as well with my node (feather m0). However, it sends "payload [not provided] once" and then goes back to normal without downlinks confirmations. Which you would see in the older "lmic" libraries. Moreover, I think it is okay, and it will probably help you save battery life with fewer downlinks.

[altishchenko]

It is ADR and other device control stuff like DevStatusReq/Ans etc. CFList and other settings the server pushes down to the device, queries it about battery and signal quality. Current LMIC does not wait for the application to provide some data frame for the reply piggybacking, it just sends the reply asap in an unconfirmed frame, giving you EV_TXCOMPLETE event as a result.
Once, twice or many times you see this depends on your network server, gateway and frequency plan configuration on the device, as well as on signal quality.

[pomplesiegel]

@altishchenko thank you for the context! Do you know an easy way to have the device still send its original intended payload, as right now it's totally unwittingly skipping a sampling interval directly following a downlink. I would rather not have to turn off ADR and link-check to fix this. Thank you!

[altishchenko]

@pomplesiegel Michael, not for the unconfirmed uplink. If your device sends confirmed uplinks, then you could first check for ACK in EV_TXCOMPLETE and also later check the SUCCESS return from the LMIC_setTxData2 (or more hacky for OP_TXDATA) as a signal that there was no extra data queued. Everything is much more complicated for the unconfirmed data - you can't distinguish between your transmissions and library's originated ones. As I said earlier, no mechanism is provided to piggyback control data on application provided data frames. As an another measure, library could use yet another special flag to mark its own transmissions and bypass EV_TXCOMPLETE generation in such case. Terry? (@terrillmoore)

[pomplesiegel]

Hmm indeed I don't do confirmed uplinks, as they are so expensive. The thing I don't understand is why the behavior has changed between LMIC 2.x and 3.x, leading to these empty payloads? That's the only thing that I mind, as if we are sampling infrequently, that could potentially be a lost hour or day of sampled data. I believe that did not used to happen.

[altishchenko]

@pomplesiegel Michael, then the only way out is attempting to send your data and checking the return status of the LMIC_setTxData... If it returns LMIC_ERROR_BUSY you should retry your transmission in a couple of seconds (and keep retrying until it returns SUCCESS), that's it.