Add initial test procedures for WebUSB & BLE flashing.

[microbit-carlos]

No description provided.

[microbit-carlos]

@abchatra is there a way to load MakeCode beta into the electron and the Windows store apps?

[abchatra]

No way for electron as it is packaged in. We will have electron build this week.

For windows app, you can type /@beta in the extension search box which will load beta.

[martinwork]

Could both WebUSB and Bluetooth tests 1 and 2 be combined into one test, with test 2 as a last step, as below?

I had performed Bluetooth test 1 on 8 combinations before I read test 2 must be preformed immediately after test 1!

Should the webUSB tests specify starting hexes, like the Bluetooth tests, or at least starting with a live hex?

WebUSB:

For each browser:

• Open the test MakeCode in the test browser
• Remove all previous webUSB pairings (or use Download ... Disconnect after connecting each micro:bit)
• Create 2 projects: display an icon; write to serial.
• For each micro:bit version:
• • Connect micro:bit to the computer via USB.
• • Drag and drop flash a hex created in the previous MakeCode
• • Use Download ... Disconnect (if it's there) to disconnect webUSB
• • Click Download to flash icon project - confirm there are connection dialogues
• • Click Download to flash serial project - confirm no connection dialogues, and a quick partial flash
• • Click Show data Device to confirm serial data is received

Bluetooth flashing

I guess a V1 and a V2 should be tested (except with the python hex).

Do we need to test so many starting hexes?

After USB flashing each starting hex, previous pairings should be deleted in Bluetooth settings, and in the iOS app Manage Connections page, before trying to Bluetooth flash, rather than deal with the errors that occur otherwise.

I think we can test with a USB powered micro:bit, to rule out hitting problems caused by low batteries.

Open the app
Create a new project to display a letter
Connect micro:bit to the computer via USB.
For each starting hex file:

• Flash the starting hex via USB.
• Delete previous pairings in the Bluetooth section of the Settings app
• Return to the app Hone page, then choose Create Code
• Open the project if necessary
• Tap Download to flash the project (choose "My pattern is different")
• Change the displayed letter
• Tap Download to flash the project again - confirm a quick partial flash

Bluetooth MY_DATA

Suggestions:

• Add a pause in the datalogging project, so there isn't so much data (I guess we're only testing the utility service is present, not how well it performs)
• Add a "During logging" fetch to ensure the service is present in both modes.

With one micro:bit V2, use the Fetch MY_DATA button in My Programs to fetch datalogging results.

• Flash a datalogging project without the Bluetooth extension.
• Do an After logging fetch (as instructed, reset to Bluetooth mode first).
• Flash a datalogging project that includes the Bluetooth extension
• Do a During logging fetch (without resetting to Bluetooth mode).

[martinwork]

The Bluetooth flashing tests should include pairing with and flashing over a hex built in the MakeCode under test, which I have just done for iOS/Android + V1/V2 with MakeCode 8.0.8 partial flashes. And with a Bluetooth program to test full flash.

[microbit-carlos]

Could both WebUSB and Bluetooth tests 1 and 2 be combined into one test, with test 2 as a last step, as below?

Yes, my original intention was to separate the tests to be able to also test them independently, but it's better to combine them and have fewer steps overall.

Should the webUSB tests specify starting hexes, like the Bluetooth tests, or at least starting with a live hex?

Ah, yes, the full-flash step might not be tested correctly if the micro:bit had a MakeCode-under-test programme already. I'll add a step to flash the "meet the microbit" programme or a different hex.

[for Bluetooth] ... Do we need to test so many starting hexes?

Unfortunately I think we want to be safe, as there are classrooms with tablets that need flash the micro:bit with the factory image always with bluetooth.
From all the test files Python is probably the lower priority one, but I think it's worth checking as well, or do you think the "meet the micro:bit" test would definitely Python too?

After USB flashing each starting hex, previous pairings should be deleted in Bluetooth settings, and in the iOS app Manage Connections page, before trying to Bluetooth flash, rather than deal with the errors that occur otherwise.

Is this something that we expect users to do? Would the app do this automatically or provide instructions to the user to do this?

I think we can test with a USB powered micro:bit, to rule out hitting problems caused by low batteries.

Sounds good, I'll add it as well.

Tap Download to flash the project (choose "My pattern is different")

Is this step always needed? Even if the pattern is the same?

[Bluetooth MY_DATA] .. Add a pause in the datalogging project, so there isn't so much data (I guess we're only testing the utility service is present, not how well it performs)

Yeah, I was thinking logging as quickly as possible would always guarantee have quite a bit of data on the log, but it's not really necessary, so I'll add a small delay which ensure it takes a bit longr to fill the storage and still provide plenty of entries by the time it is read via BLE.

[Bluetooth MY_DATA] ... Add a "During logging" fetch to ensure the service is present in both modes.

Sorry, I'm not sure I understood this one.

The Bluetooth flashing tests should include pairing with and flashing over a hex built in the MakeCode under test, which I have just done for iOS/Android + V1/V2 with MakeCode 8.0.8 partial flashes. And with a Bluetooth program to test full flash.

Ah, that's a good catch! I'll add something as well.

[microbit-carlos]

I've pushed the changes I've made so far, still need to add this part, but I'll probably need to leave it to next week, as I should continue doing some beta testing:

The Bluetooth flashing tests should include pairing with and flashing over a hex built in the MakeCode under test,

[martinwork]

Do we need to test so many starting hexes?
Unfortunately I think we want to be safe, as there are classrooms with tablets that need flash the micro:bit with the factory image always with bluetooth.

My thought was mainly: how are these hexes different from one another from the USB flashing point of view?

After USB flashing each starting hex, previous pairings should be deleted in Bluetooth settings, and in the iOS app Manage Connections page, before trying to Bluetooth flash, rather than deal with the errors that occur otherwise.
Is this something that we expect users to do? Would the app do this automatically or provide instructions to the user to do this?

No, but the main flow of the apps is not aimed at users who keep USB flashing hexes., losing the pairing information.

Tap Download to flash the project (choose "My pattern is different")
Is this step always needed? Even if the pattern is the same?

Same reason as above. It makes the app redo pairing rather than hitting an error to then come back and redo pairing

[Bluetooth MY_DATA] .. Add a pause in the datalogging project, so there isn't so much data (I guess we're only testing the utility service is present, not how well it performs)

Yeah, I was thinking logging as quickly as possible would always guarantee have quite a bit of data on the log, but it's not really necessary, so I'll add a small delay which ensure it takes a bit longr to fill the storage and still provide plenty of entries by the time it is read via BLE.

I'm not sure why we need lots of data.

[Bluetooth MY_DATA] ... Add a "During logging" fetch to ensure the service is present in both modes.
Sorry, I'm not sure I understood this one.

This tests that the config flags are set so the utility service can be available in application mode when required, as well as in Bluetooth mode., so that when already paired with a Bluetooth enabled data logging hex, you can fetch data while the data logging continues, without resetting to Bluetooth mode.

[microbit-carlos]

Do we need to test so many starting hexes?
Unfortunately I think we want to be safe, as there are classrooms with tablets that need flash the micro:bit with the factory image always with bluetooth.

My thought was mainly: how are these hexes different from one another from the USB flashing point of view?

My main concern is small changes/tweaks between CODAL releases years apart (for instance the OOB hex file has a CODAL version over 3 years old, without the 3-reset-tap-to-pair).

I think we need to keep the MakeCode Live and MakeCode under test hex files. The Python editor hex file we should probably keep as well as it could have changes in configuration (accidental or on purpose) in future releases. To be fair, that should be caught in the Python Editor release tests, but it's possible the change was introduced in an in-between CODAL release (compared with the CODAL versions in the different MakeCodes). So, this is a good way to ensure it's still working combined with a new MakeCode release as well.

Between the "meet the micro:bit" and "out of box", would you be confident that if it works on one it would work with both?
If that's the case we could drop one these two.

After USB flashing each starting hex, previous pairings should be deleted in Bluetooth settings, and in the iOS app Manage Connections page, before trying to Bluetooth flash, rather than deal with the errors that occur otherwise.
Is this something that we expect users to do? Would the app do this automatically or provide instructions to the user to do this?

No, but the main flow of the apps is not aimed at users who keep USB flashing hexes., losing the pairing information.

Tap Download to flash the project (choose "My pattern is different")
Is this step always needed? Even if the pattern is the same?

Same reason as above. It makes the app redo pairing rather than hitting an error to then come back and redo pairing

Right, I see what you mean now. Could you provide some very simple steps to add them to the instructions? Or is Tap Download to flash the project (choose "My pattern is different") enough for both Android and iOS?

[Bluetooth MY_DATA] .. Add a pause in the datalogging project, so there isn't so much data (I guess we're only testing the utility service is present, not how well it performs)
Yeah, I was thinking logging as quickly as possible would always guarantee have quite a bit of data on the log, but it's not really necessary, so I'll add a small delay which ensure it takes a bit longr to fill the storage and still provide plenty of entries by the time it is read via BLE.

I'm not sure why we need lots of data.

No, we don't need loads of data, but I wanted to make sure there is enough, as I'd expect an experienced tester trying to run through these steps as fast as possible to flash via WebUSB, and immediately after trying to read the MY_DATA file.

[Bluetooth MY_DATA] ... Add a "During logging" fetch to ensure the service is present in both modes.
Sorry, I'm not sure I understood this one.

This tests that the config flags are set so the utility service can be available in application mode when required, as well as in Bluetooth mode., so that when already paired with a Bluetooth enabled data logging hex, you can fetch data while the data logging continues, without resetting to Bluetooth mode.

Ah, yes, I'll add that as well.

[microbit-carlos]

Okay, I think I've added everything discuss, except for this one:

Right, I see what you mean now. Could you provide some very simple steps to add them to the instructions? Or is Tap Download to flash the project (choose "My pattern is different") enough for both Android and iOS?

@martinwork could you check it's all alright? Feel free to add more comments or improvements.