Comment from Mentor

[changjurhee]

Do you build it? What you are suggesting is that we build the bridge, and then we drive the train across it to see whether it's supports that load or not.
09:46:59
And if not, the bridge breaks, and we have to rebuild it.
09:47:06
So that's one issue, the other issue.
09:47:11
Is that what you're suggesting will not indicate how the system will perform?
09:47:18
In practice. So if we test it a hundred times, and all 100 calls are connected that doesn't mean that it will never drop a call it just means it didn't drop a call.
09:47:34
Then, and the requirement is not actually that you never drop a call.
09:47:39
I think the requirement is related to availability is, what does anyone know?
I mean, it says so, basically, what you want to do is you wanna try to understand?
09:48:00
How are they going to add value to the system?
And in this context, what does adding value mean?
09:48:11
So if we were making a product, adding value might mean that we could sell more products, or we could charge more for the product.
09:48:18
And so that will have a positive business impact on the company that makes that product.
09:48:26
And so that's how we prioritize requirements in this context.
09:48:31
For this project. We wanna think about things that actually matter to you.
09:48:35
And so what I would suggest is that you evaluate how the project is going to be graded.
09:48:43
So we are actually going to evaluate the system that you build.
09:48:48
If you understand how we're going to evaluate it, then you can write requirements that will reflect the evaluation criteria.
And if it's something is worth more points, it should probably be more important or higher priority than something that's worth less points.
09:49:11
You don't want to spend all your time and energy on things that won't have an impact when the project's done right.
09:49:20
So when it comes to availability, what you wanna understand is what we're gonna do to evaluate the system with respect to availability.
09:49:32
you.
And so I don't actually have a concrete answer for
09:49:38
I mean, I know that we're going to do things like introduce jitter and drop packets and things like that.
09:49:47
And I think we're also going to disconnect the call, and the system will have to detect that the call has been disconnected.
09:49:54
But the requirement should reflect that.
09:49:59
Once you have requirements that reflect things that actually matter to you.
09:50:06
Then what you want to do is try to understand what decisions in your design.
Well effect the systems, ability to support those requirements.
09:50:18
So let's say that the requirement is that we need to.
09:50:23
We need to notify participants that the call has been dropped within 1 s.
09:50:30
Let's say, and so now we want to start thinking about.
09:50:33
Well, you know what in the system will affect that.
09:50:38
And so we need something that would detect whether the call is still engaged, or whether it's been dropped.
09:50:46
We need to think about where that's being done and what other work that particular element has to do, and so forth, to make sure that we can detect and notify within our time limit.
09:51:01
So we would want to do that for all of the quality attribute requirements that are important to you that are going to have an impact on your final grade.
09:51:14
And so, and so once you understand what is, what are the decisions that will impact things that are important.
09:52:26
So do you mean, you mean, we need more understand for this software system definitely document, the the designs.
MB Matthew Bass
09:52:26
So good!
09:52:39
Yeah, I mean. So, for example, you have these architectural diagrams here.
09:52:45
But what, specifically, are you trying to communicate in them?
09:52:50
So at this point the requirements aren't really clear and likely.
09:52:57
The decisions that are going to affect those requirements are not very clear.
09:53:03
And so I wonder what you're trying to document.
09:53:07
So I tell you, understand what the key decisions are.
09:53:13
Then then at least in terms of communicating the key design decisions to us.
And so the question then, is, how do you know whether that's sufficient or not?
09:59:25
And the answer to that would be, Do some experiments. See?
09:59:32
See what the bandwidth requirements are. If you turn off video, if you have lower quality video, you have lower quality audio, how many participants will that allow you to deal with what sort of capacity do you still require for your network, and if if that's not good enough there are other things that you
09:59:57
can. So, for example, you could buffer.
10:00:05
And you know buffering will allow you to deal with that packet loss. It costs
10:00:11
With variable network bandwidth, but it comes at a cost, because now you have to fill the buffer, and you have to figure out how large the buffer would be, and so on.
10:00:23
And so so these are the things that you wanna think about and do experiments on you.
10:00:28
Just wanna make sure that you are doing them for the most important requirements.
Let's see. Oh, you just put it in the chat. Okay.
cR changiu Rhee
10:01:19
Actually, we have the experimental plan for the performance. As you commentator, we are going to find the optimal video quality with a different environment.
MB
Matthew Bass
10:01:44
Yeah, so this is good. I mean, it looks like, you guys have thought about this a lot I just, I have some suggestions kind of for each of these.
10:01:54
So if you can send me an email with links to all of your documents, I can then look through them and give you some feedback.
10:02:06
So this is good. You just need to focus the experiments a little bit more concretely.
cR changiu Rhee
10:02:06
Yes.
MB
Matthew Bass
10:02:13
So.
10:02:13
So you have you wanna deal with packet loss and jitter and.
10:02:32
And you are. You have some general effort that says you want to find the optimal quality of the clients.
10:02:40
Video transmission by considering factors that can be set in the system and external environment.
10:02:50
But but these are so. If you step back and you think about this from an architectural perspective, there are architectural strategies for dealing with packet loss or jitter.
10:03:05
The impact of those may not be clear, and so what you don't have here.
10:03:12
So what you say here is that you generally want to try to figure out how to optimize the solution.
10:03:19
But you don't have any specific architectural approaches identified that can help you deal with this more systemically.
10:03:27
What this
10:03:29
I need you to do is try to look at configuration items.
10:03:33
Maybe find a more efficient code or something like that.
10:03:37
But not but not introduce architectural strategies like, you know, buffering, for example.
10:03:46
So what you want is just to focus these a little more and look at what your options are.
10:03:53
And so one strategy that you have, that you've identified is reducing the bandwidth demands.
10:04:02
Oh, I've got! I've got a run in just a minute.
10:04:05
Here, and so you could do an experiment to see what impact that has.
10:04:14
You know, something else you could do is introduce the buffer.
10:04:17
You could do experiments to try to figure out how large of a buffer you would need things like that.
10:04:26
And so if you just, you know. So you have a all the right elements here, you just need to focus them a little bit more.
You need to be more specific with the requirements, the requirements should reflect things that really matter to you as a team, and then the the experiments should be related to the architectural approaches that are that may help you deal with those or or achieve those requirements.
10:05:02
And ideally you sketched. You put a plan together that allows you to do a couple of iterations of this so that you do a little bit of design work you do some experiments.
10:05:15
You from the experiments. Get more information about the design decisions, and how well they will or will not support your requirements.
10:05:25
And you take that information, and you put that back into the design and refine the design.
10:05:30
Accordingly, and if you go through that cycle a couple of times, you should get to the point where you have a design that that has been validated, at least validated enough.
10:05:46
And then you can. Then you can plan for implementation. And at that point you likely will have implemented pieces of the system already.
10:05:56
So implementation will get much easier.
10:06:06
Any other questions?
10:06:36
We use use case diagrams? What's the best way to describe the overall architecture and division of rooms in milestone?
10:06:45
One, so in terms of the best way to discuss the architecture, you want to think about it from the reader's perspective.
10:06:57
So you want to ask yourself, what information are you trying to communicate?
10:07:02
To whom, so if you're trying to communicate to me, what specifically are you trying to communicate to me?
10:07:11
Because in milestone one our expectation is not that you have much done in terms of architecture at this point, so what I would really what I'm really interested in more than the architecture is what does your plan look like?
10:07:28
Do you have a plan that will allow you to validate your architecture before implementation?
10:07:35
For example.
10:07:38
What are the key requirements? Do you understand what the requirement are?
10:07:44
Do you understand what some architectural approaches are that can that might help you achieve the requirements.
10:07:56
Those are the kinds of things that I'd be interested in.
10:08:00
Milestone one the architecture itself is like not gonna evolve all that much until milestone to so milestone one.
10:08:11
You can just describe, say, these are the general approaches, you know, for dealing with Jitter.
10:08:20
We are going to reduce the the image quality.
We're gonna reduce the sound quality.
10:08:26
We're going to introduce a buffer, and you could put that on a slide and bullet points.
10:08:35
You can just communicate that verbally. You could communicate that.
10:08:41
However, you want, you don't need to worry about spending much time
So for the next question, the drivers, Il will review them, offline, and I'll give you more detailed feedback, but they need to be refined
10:09:08
So you need to think about them as test cases for your design.
10:09:13
So could you concretely determine whether a design supports that specific requirement or not.
10:09:21
And so, if it's not clear what the requirement means, if there's no measure, for example, then it certainly needs to be refined.
10:09:33
And so your requirements do need to be refined.
10:09:36
But I can. I can give you concrete feedback.

[changjurhee]

If you understand how we're going to evaluate it, then you can write requirements that will reflect the evaluation criteria.
And if it's something is worth more points, it should probably be more important or higher priority than something that's worth less points.
You don't want to spend all your time and energy on things that won't have an impact when the project's done right.

Architectural Drivers	The criteria for evaluating the demo	points
?	Demonstrate video and call quality by counting for 30 seconds (echo cancel, clear voice)	60 points.
?	Conference calling setup & conference calling with at least 4 participants (echo cancel, clear voice, video)	60 points.
?	Demonstration of robustness in an adverse environment with packet loss, network jitter, and under network load (this includes voice and control data traffic)	60 points.
?	UI	40 points (Best), 20 points (2nd), 10 points (3rd), 5 points (4th) (points will awarded based on client/sponsor judgment)
?	User registration and contact identifier assignment	20 points.
?	Change user information	20 points.
?	Recover password	20 points.
?	Receive call with app in the foreground	20 points.
?	Receive call with another app in foreground	20 points.
?	Receive call with application minimized	20 points.
?	Call initiation, incoming notification, and acceptance, rejection, user busy (answered, busy or rejected)	20 points.
?	Lost connection detection	10 points.
?	Adding and modifying user contacts	10 points
?	Initiate call using contact identifier and contacts list	10 points.
?	Any application crash or improper behavior during the demonstration will result in a	30-point deduction for each occurrence.

[changjurhee]

Hi all,
Thanks for this. I’m not able to edit the document (at least on this device), so I’m putting comments here. Please let me know if you have any other questions.
Thanks!
Matt

Comments:
QA requirements: be careful not to include design decisions in your requirements. You should think about these as test cases for your design (ie criteria you’d use to determine if the design is sufficient). Thus things like “when network conditions change, the system shall reduce the bandwidth required” is a strategy for achieving something. The something is the requirement & is what should be documented. Otherwise the evaluation becomes focused on reducing the bandwidth needs & not the impact of reducing the bandwidth needed (this may or may not be sufficient on its own). The same is true for most of the QA requirements.
Make sure the requirements reflect what is really important (if you achieve all the requirements, I’d expect a perfect grade). Not clear that what you have does this.
The matrix with the tactic, experiment, etc is good. Likewise, the description of the experiments are great. In some cases if feels like maybe you have too many diverse objectives for a single experiment, but they are a great start.
With the plan, I’d allow for more questions to arise that might require experimentation. This means you’d want to have a process for identifying, prioritizing, and conducting experiments on an ongoing basis.
Good start!

[changjurhee]

• be careful not to include design decisions in your requirements. You should think about these as test cases for your design (ie criteria you’d use to determine if the design is sufficient).
• Make sure the requirements reflect what is really important (if you achieve all the requirements, I’d expect a perfect grade). Not clear that what you have does this.

[changjurhee]

• Matt 미팅 요청
  • 화요일 9시 (미팅 요청)
  • 목요일 9시 (정규 미팅)
  • 검토 의견에 대한 문의 (Make sure the requirements reflect what is really important (if you achieve all the requirements, I’d expect a perfect grade). Not clear that what you have does this.)

[changjurhee]

Team,

Max users 10, Bandwidth Wi-Fi Limits it will be less the on the Pi4 but assume internet conditions meaning things can vary widely.

For drop, latency and jitter I would use the following documents as a guide.

https://getvoip.com/blog/voip-jitter-and-latency/

https://citeseerx.ist.psu.edu/viewdoc/download;jsessionid=7BF02CC2312F130154A802D89B423C39?doi=10.1.1.403.3474&rep=rep1&type=pdf

https://vocal.com/voip/jitter-buffer-for-voice-over-ip/

QA looks ok but you may also want to consider verify video quality as well i.e. frame rate / image quality

Best regards,

Dan

[changjurhee]

• Dan 요청
  • QA 재 문의
  • GStreamer library 사용 문의

[changjurhee]

Q1. Can we use the GStreamer library for video transmisson in our project ? Yes you may use GStreamer or raw data transfer as in the sample code

Q2. I'm going to inform some more QA senarios and Test Experiments. Would you please give some comment ? we thought to consider verify video quality as well i.e. frame rate / image quality. see below captured images (image quality can be image size, compression quality and frame rate). You initial tables look good so far.

Q3. for target ranges bit rate and fps for input data, should we find the range by ourselfs ? Yes

Q4. As you commented Max Users 10, can I consider that as a design constraint? Yes, for this exercise

Have you been able to get the initial demo code and PI4 working correctly?

[junsoopark]

Dan에게 추가 요청

• If needed, is it available to use some transport level plugins of gstreamer such as rtp pay/depayloader and rtpjitterbuffer elements for handling RTP packets?
  https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpjitterbuffer.html?gi-language=c

[changjurhee]

• "Make sure the requirements reflect what is really important (if you achieve all the requirements, I’d expect a perfect grade). Not clear that what you have does this." - Matt's comment
  • Can you give more detailed advice in your comments ?
  • Currently, We have crafted 8 QA senarios. Did you mention that we need to craft more QA scenarios to achieve all the requirements which ?
  • Crafting QA scenarios from Dan's project description isn't easy to specify the QA requirement in a measurable/testable way. Do you think we need to meet and discuss with Dan to specify requirements ?

If you look at the following requirement from your document:
"When network condition changed (Jitter / Network Bandwidth) for system operation, system will determine & change bandwidth within 1 second to keep communication quality” it says that you need reduce frame rate, or video size when network jitter occurs.
If you introduce such a mechanism this requirement will have been satisfied. If we look at the grading criteria, however, we will evaluate the call quality (in some way that’s not been specified) under various conditions. It’s certainly possible that introducing this mechanism will not result in adequate call quality (despite the requirement being satisfied).
You want to make sure that the requirement reflects the criteria that we’re going to use to evaluate the system (an example requirement could be something like this depending on how we’ll evaluate call quality: if the capacity of the network reduces, the system will maintain the call connection and there will be no noticeable delay or degradation in the quality of the audio and video connection and no noticeable lag will be introduced).
This says that the outcome is what’s required regardless of the mechanism used to achieve it. Thinking about requirements in this way will force you to find out specifically how the system will be evaluated. This will enable you to use suitable criteria for your design and experimentation efforts.

• Dan's Project Description contains lots of "How" items. Do we have to describe requirements only "What" items and move "How" details to design decision or something even that is described in "Project Description" ? For example, Project Description contains such as "adjust bandwidth", Should we write/modify like this "maintain quality.." ? Please, give some your comment in detail.

The example given above is an example where you’ve included the solution. I’ve given an alternative example above as well

[changjurhee]

• QA comment
  • response : video / audio
  • environments : network reduce / packet loss conditions
• Risk experiments
  • Was a set of experiments considered for evaluating QA?
• role division
  • management
  • experiments
  • design / implementation
  • documentation

[changjurhee]

You know, what you're trying to do is to make sure that you design the most valuable system that you can design. And so you want traceability.
So you want to have the goals that you're trying to achieve.
be reflected by the requirements that you have written. You want the design decisions that you make to be so, to support the requirements that you have written.
And so, if you don't write down all of the requirements. then that increases the likelihood that your system will not support all of the goals.
And so what I'm saying is that is that you are missing some requirements the way you have it written. And I think it's because you've just combined them into one. And and I'm saying, that's not a good idea, because they're really different concerns that require different kinds of solutions. If if the same solution would work equally well for both.
then it would be fine to combine them. but the same solution won't work equally well for both. So you want to make sure that you can deal with both of them

and I don't know if you've covered availability yet in any of your lectures.
But availability is really about
fault tolerance. So some fault occurs. You want the system to be able to tolerate that fault with some defined kind of response. And so the way you think about availability is to identify what's called a fault model, or a list of all the faults that the system should be able to tolerate, and what sort of result or response you have if that fault occurs.
you know.
So so what we have here are 2 different kinds of faults.
One is packet loss. and one is reduced bandwidth network capacity.
And so your question was, well, how do we write the scenarios for these 2? I think with that that was your question. And the reason you said that is because you don't know what the response measure should be. Right. Now that that's okay. You write it as best you can, identifying the missing information.
and then you do something to try to gain that missing information and that something might be, do an experiment. It might be right. Some code, it might be. Go talk to Dan or Paulo, or myself, or whatever it might be. Do some research, I mean that's something could be anything
but the way you would write it in this case is the stimulus. The event that your system should respond to in one case will be packet loss
in the other case will be network capacity reduces.
And the response should be that the system continues to support the video call with, and and the subjective measure would be with no noticeable distortion for the audio or no no noticeable choppy playback. or no no noticeable choppy playback.

and that will need to be refined in the future. but it will be refined by doing some taking some action. So either asking Dan if he knows
what the target is that would cause audio to become distorted.
I mean it. It. It may be a little bit hard to answer concretely it may end up continuing to be subjective, and you know, and maybe you do experiments and just listen to the audio playback. I I don't know.
But I mean there, there are 2 basic strategies for for dealing with packet loss. In this context.
one is to is to resend. another is to is to duplicate packets.

and if you, if you you know you have packet one 2, and you, Miss Packet 3, and then you have packet 4 you can either resend packet 3, or you can just ignore it all together, or you can duplicate packet 2 or packet 4.
Those are your options, basically.
But and they have a different impact on the way that the audio sounds.
So if you listen to voice over IP systems.
sometimes you'll hear things kind of get stretched out like this, and sometime I it's it sort of sounds top up beat like that. So one is, they're just dropping the packets. The other is, they're they're duplicating the packets.
Does that make sense? Did I? Did you understand everything? I said.

I I believe it was with the code that you got originally, but if not, you can just ask Dan how to how to configure the router.
to introduce packet loss or change network capacity.

I'm not sure where the requirements are coming from.
So you're saying it's Chopin they could video, it'd be better than to just I I I don't know. I don't know. That's what I'm asking you. So what I'm saying is that the requirements should reflect the criteria that we're going to use. And so the questions that you should be asking are this should be designed so that you understand what the criteria is, and then you can write requirements that reflect that criteria.
Th. That's what you should be doing. My, what I'm questioning now is whether these requirements actually reflect the criteria that we're going to use when we grade the system.
and I don't think they do. I think that you have some requirements in there that you guys came up with yourselves, or maybe they were assumptions that you made about what something meant.
But I think you need to double check to make sure that these requirements actually reflect the criteria that we're going to use.

don't just ask him what is the requirement, or don't just ask him what he meant by some statement. Ask, how is the system going to be evaluated and make sure that you continue to ask those questions until you actually understand yourselves.
If you don't understand how the system is going to be evaluated, then
the likelihood that you design a system that will be evaluated well is gonna go down.
So the first step is to make sure that you, as a team. You guys are building this system. You need to understand what that criteria is, or you're not going to be able to build a system that will achieve that criteria.

So you should do that with all of these. because when you look at security, I think that all we're looking at is whether you have encryption.
That's it. We're not looking at whether the system is secure. We're not looking at whether the password is protected in all situations. We are just looking at whether you have encryption.
I mean, we're also looking to make sure that you have a that you require a username and a password.
But that's just a functional requirement. Again, that doesn't say how secure your system is. It's just says, does it have this feature?
And so you have a number of modifiability scenarios in here. You want to make sure that you understand going to evaluate that
because you don't want to put a lot of time and energy into things that are not going to benefit you.

So so keep in mind that when we're talking about doing experiments, the goal of doing an experiment is validating the design decision, not figuring out how to develop features. The goal is to make sure that your strategy is is acceptable. Once you have validated that, your strategy or your design decisions will work, then you can actually go back and design your system.

I I get that so that I I I can understand. But I I I'm looking at the things that you're doing, and to me it doesn't seem like it's focused like the experiments are really focused on validating the design decision. It feels like it's focused on figuring out the design. So how do you actually implement? The functionality is what it looks like. You're trying to figure out. Those are 2 2 different things. I'm not saying that you don't need to do that. I'm just saying that you may not need to do that in order to in order to determine whether your your design decision is going to be sufficient or not. or in order to determine at what point playback becomes choppy. or whatever

And the what what you have to decide amongst yourselves is, what questions do we need to answer in order to satisfy us that the design decision will work or not? And so 1 one thing and we've talked about this already. But one thing that you want to figure out is what is your target video quality. And so you are going to do some experiments. This is my understanding is that you're going to do some experiments to figure out what the target video quality is. So you want to see when it starts to look choppy.

But when you're presenting your architecture overview, what are you going to present? You're going to present the key decisions that support your goals. So this is not going to look at a functional decomposition, because that can be achieved with any sort of functionality can be achieved independent of how you decompose your system. you're going to look at exactly what we've been talking about.
So what are the requirements that you're trying to satisfy the quality attribute requirements.
What did you do to satisfy them? How do you know that they work? We're not. That's what you want to try to communicate in your architectural documentation. And so you have to have some of that information in order to be able to communicate. It is why we would not expect you to have much architecture milestone one. by milestone too. You're hopefully going to have some idea about what you're going to build. And so we would like to see that you have actually validated and know what you're going to get?
Are you engineering the solution? Or are you just hacking together something that works?
And so that we're going to look at mostly a milestone, too milestone one. if you have specific decisions. So I would talk about things like
reducing the data needs. so reduce the the video quality in order to try to to try to maintain continuous playback. And if you have other design decisions.
then Then you can talk about those, and we could talk about how to document them. Once we talk about what they are.