[BUG] Inappropriate calculation of mbpsBandwidth statistic in mobile networks

[lukasmahr]

Hi!

I already created a small comment about this topic, but as I think it might be a bug in SRT, I thought it would be more appropriate to create a new issue about this.

Describe the bug
When I try to transmit a live video stream with SRT 1.4.1 over a mobile network connection the bandwidth estimations of SRT are far too high.

To Reproduce
Steps to reproduce the behavior:

1. Run SRT 1.4.1 on a server listening for an incoming SRT stream
2. Connect another device to a mobile network with good LTE connection (my speedtest: 14 Mbps download, 7.7 Mbps upload and 33 ms ping)
3. Transmit a video stream over the mobile network to the server and log the mbpsBandwidth

Expected behavior
The value of mbpsBandwith is far higher than the expected 7.7 Mbps.
In my test setup the estimated bandwidth was varying between 400 and 800 Mbps.

Screenshots
Here's a wireshark snipped captured on the server side. The SRT sequence numbers that can be evenly devided by 16 are marked bold. Delta is the time difference to the previous packet.

No.	Time	Delta	Sequence-No.	Source	Destination	Protocol	Length	Info
1	0.902803590			{clientIP}	{serverIP}	UDT	106	UDT type: handshake
2	0.903013058	0.000209468		{serverIP}	{clientIP}	UDT	106	UDT type: handshake
3	0.956679062	0.053666004		{clientIP}	{serverIP}	UDT	122	UDT type: handshake
4	0.956994934	0.000315872		{serverIP}	{clientIP}	UDT	122	UDT type: handshake
5	1.536002451	0.307441251	1965305408	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 0 msgno: 1
6	1.536056437	0.000053986	1965305409	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 1 msgno: 2
7	1.536060167	0.000003730	1965305410	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 2 msgno: 3
8	1.536200554	0.000140387		{serverIP}	{clientIP}	UDT	86	UDT type: ack seqno: 1
9	1.537200313	0.000999759	1965305411	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 3 msgno: 4
10	1.537219984	0.000019671	1965305412	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 4 msgno: 5
11	1.537490488	0.000270504	1965305413	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 5 msgno: 6
12	1.537857188	0.000366700	1965305414	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 6 msgno: 7
13	1.538060081	0.000202893	1965305415	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 7 msgno: 8
14	1.538082165	0.000022084	1965305416	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 8 msgno: 9
15	1.538782887	0.000700722	1965305417	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 9 msgno: 10
16	1.539773467	0.000990580	1965305418	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 10 msgno: 11
17	1.542971567	0.003198100	1965305419	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 11 msgno: 12
18	1.545136250	0.002164683	1965305420	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 12 msgno: 13
19	1.545158900	0.000022650	1965305421	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 13 msgno: 14
20	1.545161596	0.000002696	1965305422	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 14 msgno: 15
21	1.545162677	0.000001081	1965305423	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 15 msgno: 16
22	1.545163468	0.000000791	1965305424	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 16 msgno: 17
23	1.556472440	0.011308972	1965305425	{clientIP}	{serverIP}	UDT	246	UDT type: data seqno: 17 msgno: 18

Desktop (please provide the following information):

• OS: Ubuntu 18.04 (Bionic)
• SRT Version: 1.4.1

Additional context
According to the provided wireshark log and the SRT implementation my assumption of the bitrate estimation is the following:
The receiver takes every 16th and 17th packet based on their sequence number and calculates the time difference between those two packets.
Thereafter, the receiver calculates the estimated number of 1500 Byte sized packets that could be transmitted based on the time difference and sends this information to the sender.

The sender converts the number of packets per second into a bitrate.

For simplicity, I intentionally skipped some parts like taking an average value of multiple estimations and throwing away values that are too far away from the median.

Unfortunatly, I'm not sure about the accuracy of the wireshark timestamps, because I could not detect a proper correlation between the bitrate estimation of SRT and the timings in wireshark.

However, I currently have the following theory: this algorithm for bitrate estimation takes the assumption that data packets are equally distributed over time.
Maybe the media access control of mobile networks or something else in the communication path leads to a bundled transmission of data packets. As a consequence, the packet distribution over time would not be even any more.

What do you think of this theory or do you have another idea what might cause this observation?

Thank you very much in advance!

[lukasmahr]

After further looking into this I could partially validate my theory.
Especially in mobile networks (wifi networks as well), packets partially have a strong delay.

In my test setup, the median of the time delta between the 16th and 17th packets was 33 microseconds.
The random occurence of single values over 50,000 or some even over 500,000 lead to an average value of 650.

Those outlier get eliminated during stats processing by removing outliers that are higher than 8 times of the median value of the values that are processed for calculation.
Another issue is the window size that is used for calculation. Those "slow" packets aren't frequent enough to make sure that they are always present in the calculation process.
Therefore it is necessary to increase the current windows size of 16.

[mbakholdina]

Hi @lukasmahr ,

Thank you for submitting the issues and your thoughts on the problem.

The model currently used in SRT for available bandwidth estimation is a heritage of UDT protocol. It’s based on the packet pair technique which accuracy may depend on the network equipment and network congestion. According to UDT documentation, both overestimation and underestimation can happen and the current model can tolerate misestimation to a scale up to 10x or 1/10 of the real link capacity. We have observed this behaviour as well and are going to improve the model in future.

The main difficulty is the nature of network traffic and various network conditions which should be taken into account. According to the latest research papers, the traffic shows high variations at a small aggregation times, which indicates that the traffic is self-similar and has a heavy-tailed characteristics. The reason of self-similarity is the existence of burstiness over a wide range of timescales. Examples of heavy-tailed distributions are Log-normal, Pareto, Weibull, Generalized Extreme Value (GEV) and log-gamma distribution. These distributions are more accurate in representing the Internet traffic. Bursty traffic behaviour is one of the issues in 3G/LTE networks as well, here you will find some research on the topic.

Here is the document which describes the currently used model as well as contains my first thoughts on possible improvements and related research papers. You’ll mention that some of the suggested improvements are inline with your thoughts on the problem.

One small correction, window size N=16 is used for the receiving speed estimation at the slow start period of Congestion Control while for bandwidth estimation this value is equal to N=64. I’ve also found the paper which provides the performance comparison of different window size: 8, 16, 32, 64 and is worth looking into. According to this paper, changing the window size when estimating bandwidth to a greater than 64 value most probably will not lead to a better performance in terms of better estimation vs CPU/memory consumption. However, we assume that changing the window size from N=16 to N=32 or N=64 when estimating packet arrival rate (receiving speed) will definitely lead to a better performance especially together with currently used median filter adjustments.

Regarding your experiments, what would be useful to know and to have for the following analysis:

1. Which application is used to perform the experiment? srt-live-transmit? Could you please share the commands, SRT settings applied?
2. Could you please share the Wireshark .pcapng dumps collected both on the receiver and sender side?
3. If srt-live-transmit application is used for tests, could you please collect SRT core statistics as well in .csv format? Please use the following options to do this

--statsfile [filename].csv --pf csv --s 100

where

-statsout - SRT statistics output: filename. Without this option specified, the statistics will be printed to the standard output.
-pf, -statspf - SRT statistics print format. Values: json, csv, default.
-s, -stats, -stats-report-frequency - The frequency of SRT statistics collection, in milliseconds.

I also would like to share the tools I am using for the analysis with you:

1. lib-tcpdump-processing - a library to process tcpdump trace file and extract SRT packets for further analysis.
2. srt-stats-plotting - script designed to plot graphs out of SRT core statistics.

I hope it will help with the further analysis. Looking forward for further cooperation on the topic.

[lukasmahr]

Hi @mbakholdina,

thank you very much for your response!

I created a GitHub repo with the test setup I used to get the following results:

Speedtest on the sender side using speedtest.net over LTE connection:

• 13.3 Mbps down
• 9.62 Mbps up
• 29 ms ping

The bandwidth estimation went up to 400 Mbps. I provided the corresponding SRT stats as CSV files here.

I hope this helps to reproduce the whole thing.
I think it might be really interesting to check whether there are any differences depending on the ISP used.
For my tests I used the german T-Mobile network.

[mbakholdina]

Hi @lukasmahr ,
Thank you for the details!

What's important for me to have together with SRT statistics from srt-live-transmit is also Wireshark/tshark .pcapng dumps. Could you please before running the experiment start tshark on both machines, repeat the test once again and give me both .pcapng dumps and SRT .csv statistics from both sender and receiver?

The command I use to capture the traffic on a particular interface and port is

tshark -i en0 -f "udp port 4200" -s 1500 -w _results/tshark-trace-file.pcapng

How to setup tshark on CentOS/Ubuntu is written here.

Thank you beforehand!

[lukasmahr]

Hi @mbakholdina ,
I attached the corresponding Wireshark dumps to the previously provided SRT stats.
The sender IP is 192.168.1.1 and the receiver IP is 192.168.1.2.

receiver.zip
sender.zip