Speed & Latency Testing Demo for ISPs

Compliance & Testing with Friendly’s Speed & Latency Tools

Friendly Technologies has launched its fully functional Speed & Latency Testing tool, which is fully compliant with FCC DA 18-710 and is designed to provide a full testing and reporting solution for Internet Service Providers.

This webinar was conducted on April 29th, 2021, by Sean van der Walt of Friendly Technologies, specifically with ISPs in mind.

Friendly’s Speed & Latency Testing tool is an industry-based framework that supports the entire range of testing and reporting needs, as detailed by FCC DA-710 requirements. The tool is being provided as a part of the existing Friendly TR-069 Device Management platform.

Friendly’s Speed & Latency Tool is designed to fully comply with the FCC and ACMA, ACCC requirements while providing the most feature-rich offering to the service providers.

Download the presentation from the webinar here:

Webinar Transcript:

ARIELA ROSS: Hello, everyone, I just want to make a note that the session is being recorded. At the end, we’ll have a Q&A session. We’re trying to keep this fairly brief today, so we won’t take up too much of your time. I am Ariela Ross, Head of Marketing at Friendly Technologies. Today we’re doing a live demo of Friendly Speed and Latency tool, which is designed to fully comply with FCC and ACMA requirements. Also, as a follow-up to the session, we’ll be providing the materials and a recording for your use. Now, I’m going to go ahead and pass the torch on Sean van der Walt was our Product Management Specialist and he’ll go ahead and take you through this. Alright, you got it, Sean?

SEAN VAN DER WALT: Yeah. Thank you, Ariela. Thanks very much for the intro. Maybe also, just to mention, if you have any questions, just drop those in the question box, and Ariela will get around to the questions towards the end of the session. So we’ve only booked 30 minutes, we don’t want to waste your time. And we want to try and do this as quick and as punchy as possible. So there are a few slides I’d like to just go through so that you have a little bit of idea of what the background of this particular performance testing tool is about.

So first of all, we saw some requirements coming out of the North American market in particular, America, there were a few telecommunications roll-outs in underserviced areas and the regulatory body in America, the FCC needed some assurances that the money being spent on these infrastructure projects were being well spent, and consumers were getting a good experience. So the FCC mandated some particular performance testing for the service providers in the underserviced areas. And this actually resulted in the broadband forum coming up with a set of amendments, in particular, the TR-143 technical report, which actually covers the broadband performance testing requirements for the ISPs. And in particular, the ACS platform, the server platform, as well as the devices and how they need to comply to performance benchmarking. So this particular demo goes around showing you how this is achieved.

So the three particular tests we’re interested in is a latency test, so that’s really a ping test, a download test, and an upload text. And the download and upload tests are using either FTP or HTTP for transports. And the idea behind all of this is to have this completely automated. So Australia, obviously, with the NBN roll-out, had some specific requirements, and in particular, all service providers providing NBN wholesale services to retail customers, that the Australian ISPs need to comply with a set of frameworks. So in general, there are essentially the three bodies ACMA – the regulator here in Australia, the Competition Commission, as well as NBN, who have each had a set of their frameworks for broadband speed testing and quality assurance for consumers using the NBN service. So some of those actually include that the Telcos or the service provider must be able to provide some sort of speed testing on the connections that they have with the consumer and be able to prove that, the consumers are able to exit the contract if the service provider is not able to actually achieve the speeds advertised as promised, and also that there shouldn’t be any cost to your consumer moving to a lower speed plan at a lower price. And with the consumer having confidence that they’re actually paying for what they’re getting.

So some of those ISP testing requirements include choosing a sample of devices as a fleet of your deployment, choosing a sample set across different CVCs or CSAs or point of interconnect, or across the different access technologies, whether that’s fiber to the premise, fiber to the curb, node, HFC, or even wireless last-mile deployments. So really just getting a sample set of testing across those, and being able to provide a speed report or test report according to those different speed plans. So of course, as a service provider, you may be offering certain specific speed plans but this is just to give you an idea. And part of the process then is being able to actually do speed and latency testing during some of the peak hour periods. So typically 7 pm until 11 pm at night, those are the peak periods consumers are using the service, and also being able to do some testing during off-peak periods, which probably is not that interesting because it’s during peak periods that one really has to be concerned about. But it gives you a good benchmark between off-peak and peak time. So then, obviously doing some reporting on those averages. So averaging the test reports or test results per hour, per speed plan, and then reporting some of those low and high percentiles for the speed and latency testing. So these are the general high-level kind of speed and latency test requirements.

And really to achieve that, one would need an ICS server to perform and execute automated speed testing and latency testing. So this diagram on the right gives you an idea. So there is an ACS where the administrator is able to set up these kinds of group update tests. So essentially, a whole set of devices or it could be a complete fleet of broadband devices that are registered with this ACS, it could be in a small fleet, or it could be into many, many millions of devices registered. But the tester admin would actually do a filtered selection, it could be a random selection, or it could be a selection based on certain criteria, or as we said, those different access technologies or different points of interconnectivity, and of course, the different speed plans. So what then happens is, basically, the ACS sets up this update group to perform a latency, an upload, and a download test. So these are batched so you take it could be if it’s not hundreds of could be a couple of thousand devices that you may be testing as a sample set. And those are then batched over an hour of testing because you don’t really want to overload your network with a huge number of downloads or uploads all happening more or less at the same time. So you’d set that up to be bashed over an hour-based testing. Then basically, the ACS issues this set of parameter values to the devices, so it really instructs the device to do a download test, then an upload test, and then a latency test.

So it’s important to know, it’s not the server, the ACS that’s actually performing the upload or download test. It is issuing the instruction to the device for the device to perform the actual test. So the physical device at the edge, the router, the gateway, the broadband device, then performs the download or upload test according to the parameters that it’s being configured with, or the ping, or latency test. So the devices then physically perform that test and then report back to the ACS, the results. So typically those results, start time in time amount of bytes, uploaded, downloaded. So those results are reported to the ACS, and the ACS then records that into a database. And of course, here we’re interested in time series data because we’re doing four one-hour peak tests, and we’re doing three-hour off-peak tests as well. So all of those go into a database in a time-series format. And as you can see in the diagram there are various options for reporting those results and I’ll give you a little bit of a live demonstration here.

So I’m just mindful of the time and I don’t want to go over. So on the diagram, we have an ISP server. So it could be the same as the ACS or it could be a separate defined file server. The file server is responsible for posting the download-upload files and actually performing the FTP/HTTP transport. This is a bit of technical detail. I will leave this in the presentation, you will get copies of this, but this is a process flow of a download and an upload diagnostic using HTTP. This does not only show a single thread of HTTP because it’s essentially opening a TCP socket. But this is the flow of events. And if you look at the tables at the bottom, there’s a beginning of Time and End of Time and test bytes received or test bytes sent. So those are the important bits of parameters that we get in an ACS back from the device, that is the results that are reported.

So some of the issues you could experience, and this comes out of many live customers from experience. So of course, are the devices online-offline at the time of the test. So you may not get a result, because the device may be offline for whatever reason. The customers switching it off, power situation at the customer, whatever the case may be. So that’s one issue. Another is, obviously the customer consuming services while you’re doing the speed test. And I guess this is part of the reason to do the speed test is to actually get a good idea of what sort of performance the consumer is experiencing while they’re doing Netflix or YouTubing, or live streaming video, etc. Another issue you would need to look at is the physical broadband device. And the compatibility for this download diagnostics and upload diagnostics, speed testing, and latency testing. Not all devices support that. And in particular, there was an update, an amendment one to the standard to the TR-143 which supports multi-threading. And a lot of devices don’t support that. And it is important because it gives you way more accurate representation of speed testing. So multi-threading really is the ability to have multiple TCP sockets running at the same time downloading or uploading those files instead of having a single TCP socket bottleneck. So that’s quite important to look out and probably, it would be one of the single biggest factors affecting your speed testing. The other is obviously the speed, the ACS, and the file server having sufficient resources to cater for the total number of devices being tested over the hour or instantaneously. And of course, another big one on your network is to look out for network congestion. Just kind of the bottleneck bandwidth, the bottlenecking between all the devices being tested or batched over a period of one hour, and your network being able to handle all those download-upload files. The file sizes are typically like a 32 MB file for a download test. But that is completely configurable, could be any size, but we found +/- 32-megabyte file to be the most efficient from a network utilization point of view as well as getting the best performance results as well.

So I was talking about TR-143, and that amendment one. So supporting multi-threading is very important. And in particular, supporting the maximum number of connections. So there is the device. Depending on the devices, resources, and capabilities, it may not be able to support too many concurrent threads. But if the resources on the router is sufficient, we see devices with up to 32 maximum connections. Number of connections means from you, as the administrator is able to actually configure how many concurrent sessions you want to have running at a time, up to that maximum. So you may only want to run 12 or 10 connections. So that’s configurable. That object needs to be present in the devices that you’re using to get the best efficiency out of that download-upload test procedure.

And this is typically what it looks like in the data model. So if you look at your ACS data model of your device, you will find somewhere in the data tree, a download diagnostics object. And in that example here you see the anecdote of my connections, the download transport. So this device supports both FTP and HTTP. So you could configure it to do either, and the number of connections. So I’ll just show you like, you’ll see the download URLs and other settings, we’ll just show you that in the live demo very shortly. So getting the outputs of this into some reports, you may have thousands and thousands of test results, and those can be summarized into your different speed plans, and looking at them are you achieving the kind of 80% plus results out of your speed testing.

This kind of gives you a bit of an idea of what sort of the percentile summaries could be. And whether you’re meeting — this is like, the third-lowest speed test results out of a whole bunch of 14-day samples. And, yes, it gives you a very good idea of how you’re doing. I’ve left a couple of references here, a copy of this presentation will be provided. And there’s a set of references that you can cross-reference from broadband forum to a couple of our own references as well.

So I’ll quickly jump over to the live demo. And we’ve got 12 minutes, including, probably need to cover some questions. So in our TR-069 platform, our online demo system, we have a couple of portals here. So I will just jump into, for example, the administrator portal. So let’s have a quick look at that.

So as an administrator there are a few things you’d want to look for. So first of all, I’ve just got a view filter set here. So I’ve got two devices I’m busy testing on. One of them is like example, is the SERCOMM box. So what you’re interested on the SERCOMM, or any broadband device is when you look at the data tree, so let’s have a look at the data tree here. In the data tree, we see download diagnostics. In this case, remember, we saw the beginning of time, end of time measurements, and so forth. And then the number of test bytes received. So these are all the results that we’re getting back from the device. So aside from download, we have the upload diagnostics as well. So essentially, the upload or the download speed test is basically the time difference and the number of bytes sent or received. And you divide that by the time and megabits per second.

So the other is the latency test. And it has the object for the latency test results as well. So I think we’re just doing download-upload test here now, but I will show you one of the other devices for the latency test. So when we set up a download-upload test, I’ll just do one manual live test right now. So let’s go to Device Diagnostics and I’m just going to issue a manual download test right now. So if I do a download diagnostics test and I send this off to this device immediately, it’ll go through pending, sent, and then completed. So within seconds, this device has received the diagnostics request, and it has actually done a download diagnostics test. And if we have a look at the data tree for download diagnostics, this is set up in the local timezone, which I think this is in Israel. You’ll see there 7:20, so 20 the past hour, which is now. Anyway, so these results will give us if we take the bytes received divided by the time difference, we’ll get the result. But I’m going to get that into showing you where we’re going to see where it is calculated.

Okay, so that’s one device. To give you another device, this is a TP-link device. You’ll notice that in the data tree, each of these devices kind of place their diagnostics results in different places, it’s not really that standard. But the object which download diagnostics is standard between them and you’ll see the same kind of figures between the two devices. So this is the same thing, download, upload, and we have the UDP Echo, which is the actual latency test. So I showed you doing the individual manual download-upload test.

But what we do when you have maybe many millions of devices registered to your ACS, and you want to do a pre-filtered selection on, let’s say, a few thousand devices, you will set up a group update task. The group update task will consist of the following. You will set up a task. So if you go through new and you go through setting up a task here, you can create an IP ping diagnostic task, you can create a download task and you can create an upload task. So this is set up to run hourly and this started yesterday, this is just set up for the demo purposes and we’re reactivating it every hour. So this is continuously going into reactivation. So it means every one hour, this diagnostic test will run. And so that means if like in this case, the demo, there’s only one device here, but you might have many thousands of devices that you’re doing speed testing on. And again, that could be according to the different speed plans. So that’s setting up the group update.

Sorry, I don’t have time to show you how to set that up. But it’ll go through creating a new task here. So that’s how they’re set up and you can pre-select. It could be a selection of a random selection, it could be like set of serial number, or manifest or serial numbers, or it could be manual selections, or it could be filtered down based on a certain selection criteria. And that’s how you’d set up the automated speed testing.

We do have another process here of monitoring. And in the monitoring, for example, we’re also doing some Wi-Fi neighbor diagnostic testing as well. And we just set this up as a KPI because I’m going to just show you what this looks like when we get to the outputs. So as far as the outputs are concerned, what we’re looking at here is a single device. If I go back and I have a look at — let’s take one of the demo devices, this one up here. And I’ll take the serial number, which is this one, the serial number here. On our monitoring system, this is what I’m busy looking at. Oh sorry, there are two different devices here. This is the TP link. So the TP-link device is giving us an output on upload diagnostics. I can see this is running at a certain set of megabits per second, the download diagnostics as well at a certain set of megabits per second. I’m also looking at the latency testing here. For example, this is 52-millisecond latencies. But in addition, to remember, we did the upload-download and latency task to automate that speed test. But we also set up KPI monitoring to look at, in this case, this device we’re looking at Wi-Fi neighborhood collisions, and we’re looking at 2.4G and 5G Wi-Fi collisions. So I had a look at some of those channel changes, the device is making changes to the channels, and the collisions that you’re experiencing on those particular devices. So just to give you an idea of what’s happening to that individual device. And similarly, we can have a look at the second device which was the SERCOMM device and if I go here, there we go, SERCOMM we will be able to see its results as well. So we’re seeing upload, download, there are 8 Mbps, and so forth. So that’s an example. I think we’ve got a few minutes for questions, and I’m going to jump over to questions. Anyone got any questions?

ARIELA ROSS: Alright, I had some sent to me directly. Let’s see. One of the first ones is do you have a list of certified broadband devices?

SEAN VAN DER WALT: Okay, thanks. Yeah, so that’s always a good question. Basically, from an ACS point of view, we don’t maintain a certification list. However, any device that is certified to the broadband forum device certifications standard will automatically work with our platform. So yeah, we have many, many thousands of devices that are connected, and remember what I said you need to look at whether that particular broadband device supports the TR-143 amendment one which is inclusive of multithreading testing. Okay, next question.

ARIELA ROSS: Let’s see the next one here is can we use our own BI reporting tool?

SEAN VAN DER WALT: Oh yes. So we have a reporting tool that we do supply. And certainly, let me just see if I got my password in. So you can use your own BI. We have many customers using their own enterprise BIs is including Microsoft Power BI. So yes, just an example of an online BI system that we do provide, that customers do use. And the reporting can all be available in this reporting system. Next question.

ARIELA ROSS: Alright, the next one we have is how many devices can we perform speed testing on?

SEAN VAN DER WALT: So probably unlimited, but it is limited by the server resources for the file server as well as your backhaul networking bandwidth, whether you can support that. But we have a lot of customers running between 1000 and 5000, 6000, 7,000 devices for speed testing.

ARIELA ROSS: Alright. I think we have time maybe for one more question, and then we’ll wrap it up there. Anybody who has questions as a follow-up, I’ll go ahead and send an email with this information and the recording and you can get back with me and I’ll make sure to get your question answered. So the last question is, we were thinking of using the TR-369 USP for new set-top boxes, do you support the new protocol?

SEAN VAN DER WALT: Yes, yes, yes. So what I’ve shown you now is the broadband forums specifications around TR-143, and that’s also supported in TR-369 USP. So as an example, I can try and let’s see if we have one of these, but we do have already inclusive support of TR-369 USP.

ARIELA ROSS: Alright, perfect. And I think with that, we’ll go ahead and wrap up the session.

SEAN VAN DER WALT: Okay.

ARIELA ROSS: Yeah. Great.

SEAN VAN DER WALT: Okay. So if there are any more questions, I will happily answer them. And if you could please disseminate them to all the attendees.

ARIELA ROSS: Absolutely. Alright. Well, thank you for joining everyone. We appreciate your time. And again, any questions, if you would like another demo for yourself, you’re always welcome to get in touch. And you’ll be getting an email, so you’ll have my email address. And again, I’ll make sure it gets to the right place.

SEAN VAN DER WALT: Sorry to interrupt Ariela. We can also allow a remote log into our demo system if anyone wanted to test the speed testing. So we’re happy to assist with an online demo and assist with the speed testing as well.