Why ignore cut_point_averages[0] when computing averages time?

[cucrui]

I could not figure out that in benchmarking_server.py line 274, why the averages time computed by sum(cut_point_averages[1:]) / (cut_frequency - 1), and why not include cut_point_averages[0]. The code show as below.

if self.requests_answered != 0 and q % cut_frequency == 0:
        averages.append(sum(cut_point_averages[1:]) / (cut_frequency - 1))
        cut_point_averages = []
        
        edge_averages.append(sum(edge_only_averages[1:]) / (cut_frequency - 1))
        edge_only_averages = []
        
        cloud_averages.append(sum(cloud_only_averages[1:]) / (cut_frequency - 1))
        cloud_only_averages = []
        
        cps.append(cut_point)
        print('cps', cps)

Another question is in benchmarking_server.py line 337, why the denominator is a fixed value, while list cut_point_averages[] is dynamic increasing, the code show as below.

print(cut_point_averages, 'avg:', sum(cut_point_averages[1:]) / (cut_frequency - 1))

Thanks for your reply!

[FrancisMcN]

Your first question about ignoring cut_point_averages[0] is a good one, we excluded that value deliberately.

The first time Keras' predict method is called there is additional overhead which slows it down. So we just skip over the first run when calculating the averages and just count each subsequent run.

In the code it says cut_frequency = 3 because each measurement is taken three times, but when computing the averages for our results we only count runs 2 and 3 and ignore run one because of what I mentioned earlier with Keras' predict method.

Your second question about using a fixed value denominator is a good point. It looks like the average should instead be calculated using the length of cut_point_averages, thankfully this part of the code is just to show output to the user while the benchmarking is running and doesn't affect the results produced.

Thanks for looking over our code!

[cucrui]

Your first question about ignoring cut_point_averages[0] is a good one, we excluded that value deliberately.

The first time Keras' predict method is called there is additional overhead which slows it down. So we just skip over the first run when calculating the averages and just count each subsequent run.

In the code it says cut_frequency = 3 because each measurement is taken three times, but when computing the averages for our results we only count runs 2 and 3 and ignore run one because of what I mentioned earlier with Keras' predict method.

Your second question about using a fixed value denominator is a good point. It looks like the average should instead be calculated using the length of cut_point_averages, thankfully this part of the code is just to show output to the user while the benchmarking is running and doesn't affect the results produced.

Thanks for looking over our code!

You are welcome and thanks for your replay!

I meet another question that when i try to reproduce the expriment with my own setup, one jetson edge server and a cloud server, the optimal cut point percentage of LeNet is different from the paper's result. And I find out that the most time-cost stage is data transimission.

So the question is whether the optimal cut point percentage change when the experiment setup is different.

[FrancisMcN]

It's expected that the optimal cut points will change depending on your setup. We learned in our research that the choice of hardware and environmental conditions all affect the optimal cut point.

We also learned the environmental factor with the most impact tended to be the network.

Thanks for running our work on different hardware, it's good to see!