Hickups/dropouts/skips on streams

[rhroy]

Issue: A few times per hour we experience issues as in hickups/skips on our stream servers. This issue seems to happen randomly on multiple shoutcast servers.

What are we using:

• VPS on TransIP
  Windows Server 2016 x64
  8GB Ram
  300 GB Disk/60 GB Free

Applications/Software running on the machine:

• ProppFrexx with StereoTool as VST on the output.
  Google Drive to sync our music/mixes when uploaded, files are synced locally.
  Icecast (for our own livestreams only)
  Pira.cz Silence Detector (to notify us when there is no audio)
  Node.js small script to send the current playing track to our API.
  Small tools like 7-zip, mp3tag, putty, treesize, vbcable, winamp, notepad++ and mysql workbench only used occasionally.
  Powershell script running every hour on XX:20 to handle some mp3 tagging and moving of newly uploaded mixes.
  LatencyMon

Average CPU is 30-50% and memory around 45-50% when i’m remote desktop connected.
ProppFrexx uses around 18-20% CPU and per mp3 encoder it’s around 2-3% each.

What have we done to investigate this issue:

• Isolate stream servers to see if it’s only specific servers but it seems to be on all 4 servers.
  Time that it happens is random, there is no pattern to be found
  Upgrade/update Windows/ProppFrexx/StereoTool to latest version
  Check if powershell scripts are running on the time that there are issues
  Check if Google Drive is syncing when issues occur
  Ran LatencyMon and followed advice.
  Read viewtopic.php?p=3240&hilit=dropouts#p3240
  Windows sound: Select 'No Sounds' as the profile/schema and click OK.
  Windows: Choose the option to 'Adjust For Best Performance'.
  Playback buffer 1500ms
  C:\Users\Administrator>bcdedit.exe /set {current} nx AlwaysOff
  The operation completed successfully.
  Power options: High Performance
  a) Uncheck the drive option: 'Compress this drive to save disk space'
  b) Uncheck the drive option: 'Allow files on this drive to have contents indexed..

Report LatencyMon:

_________________________________________________________________________________________________________
CONCLUSION
_________________________________________________________________________________________________________
Your system appears to be having trouble handling real-time audio and other tasks. You are likely to experience buffer underruns appearing as drop outs, clicks or pops. One or more DPC routines that belong to a driver running in your system appear to be executing for too long. At least one detected problem appears to be network related. In case you are using a WLAN adapter, try disabling it to get better results. One problem may be related to power management, disable CPU throttling settings in Control Panel and BIOS setup. Check for BIOS updates.
LatencyMon has been analyzing your system for 0:01:43 (h:mm:ss) on all processors.


_________________________________________________________________________________________________________
SYSTEM INFORMATION
_________________________________________________________________________________________________________
Computer name: WIN-E3AI731H6NL
OS version: Windows Server 2016, 10.0, version 1607, build: 14393 (x64)
Hardware: Standard PC (i440FX + PIIX, 1996), QEMU
BIOS: Default System BIOS
CPU: GenuineIntel Westmere E56xx/L56xx/X56xx (Nehalem-C)
Logical processors: 4
Processor groups: 1
Processor group size: 4
RAM: 8191 MB total


_________________________________________________________________________________________________________
CPU SPEED
_________________________________________________________________________________________________________
Reported CPU speed (WMI): 2594 MHz
Reported CPU speed (registry): 2594 MHz

Note: reported execution times may be calculated based on a fixed reported CPU speed. Disable variable speed settings like Intel Speed Step and AMD Cool N Quiet in the BIOS setup for more accurate results.


_________________________________________________________________________________________________________
MEASURED INTERRUPT TO USER PROCESS LATENCIES
_________________________________________________________________________________________________________
The interrupt to process latency reflects the measured interval that a usermode process needed to respond to a hardware request from the moment the interrupt service routine started execution. This includes the scheduling and execution of a DPC routine, the signaling of an event and the waking up of a usermode thread from an idle wait state in response to that event.

Highest measured interrupt to process latency (µs): 13250.60
Average measured interrupt to process latency (µs): 12.567832

Highest measured interrupt to DPC latency (µs): 1167.50
Average measured interrupt to DPC latency (µs): 4.143904


_________________________________________________________________________________________________________
REPORTED ISRs
_________________________________________________________________________________________________________
Interrupt service routines are routines installed by the OS and device drivers that execute in response to a hardware interrupt signal.

Highest ISR routine execution time (µs): 0.0
Driver with highest ISR routine execution time:

Highest reported total ISR routine time (%): 0.0
Driver with highest ISR total time:

Total time spent in ISRs (%) 0.0

ISR count (execution time <250 µs): 0
ISR count (execution time 250-500 µs): 0
ISR count (execution time 500-1000 µs): 0
ISR count (execution time 1000-2000 µs): 0
ISR count (execution time 2000-4000 µs): 0
ISR count (execution time >=4000 µs): 0


_________________________________________________________________________________________________________
REPORTED DPCs
_________________________________________________________________________________________________________
DPC routines are part of the interrupt servicing dispatch mechanism and disable the possibility for a process to utilize the CPU while it is interrupted until the DPC has finished execution.

Highest DPC routine execution time (µs): 1071.552814
Driver with highest DPC routine execution time: ntoskrnl.exe - NT Kernel & System, Microsoft Corporation

Highest reported total DPC routine time (%): 0.075064
Driver with highest DPC total execution time: portcls.sys - Port Class (Class Driver for Port/Miniport Devices), Microsoft Corporation

Total time spent in DPCs (%) 0.197670

DPC count (execution time <250 µs): 80019
DPC count (execution time 250-500 µs): 0
DPC count (execution time 500-10000 µs): 3
DPC count (execution time 1000-2000 µs): 1
DPC count (execution time 2000-4000 µs): 0
DPC count (execution time >=4000 µs): 0


_________________________________________________________________________________________________________
REPORTED HARD PAGEFAULTS
_________________________________________________________________________________________________________
Hard pagefaults are events that get triggered by making use of virtual memory that is not resident in RAM but backed by a memory mapped file on disk. The process of resolving the hard pagefault requires reading in the memory from disk while the process is interrupted and blocked from execution.

NOTE: some processes were hit by hard pagefaults. If these were programs producing audio, they are likely to interrupt the audio stream resulting in dropouts, clicks and pops. Check the Processes tab to see which programs were hit.

Process with highest pagefault count: latmon.exe

Total number of hard pagefaults 18
Hard pagefault count of hardest hit process: 9
Number of processes hit: 3


_________________________________________________________________________________________________________
PER CPU DATA
_________________________________________________________________________________________________________
CPU 0 Interrupt cycle time (s): 4.302984
CPU 0 ISR highest execution time (µs): 0.0
CPU 0 ISR total execution time (s): 0.0
CPU 0 ISR count: 0
CPU 0 DPC highest execution time (µs): 1071.552814
CPU 0 DPC total execution time (s): 0.431911
CPU 0 DPC count: 63930
_________________________________________________________________________________________________________
CPU 1 Interrupt cycle time (s): 1.956163
CPU 1 ISR highest execution time (µs): 0.0
CPU 1 ISR total execution time (s): 0.0
CPU 1 ISR count: 0
CPU 1 DPC highest execution time (µs): 495.825752
CPU 1 DPC total execution time (s): 0.223267
CPU 1 DPC count: 7480
_________________________________________________________________________________________________________
CPU 2 Interrupt cycle time (s): 1.061561
CPU 2 ISR highest execution time (µs): 0.0
CPU 2 ISR total execution time (s): 0.0
CPU 2 ISR count: 0
CPU 2 DPC highest execution time (µs): 142.494988
CPU 2 DPC total execution time (s): 0.013560
CPU 2 DPC count: 806
_________________________________________________________________________________________________________
CPU 3 Interrupt cycle time (s): 1.455170
CPU 3 ISR highest execution time (µs): 0.0
CPU 3 ISR total execution time (s): 0.0
CPU 3 ISR count: 0
CPU 3 DPC highest execution time (µs): 217.182344
CPU 3 DPC total execution time (s): 0.146034
CPU 3 DPC count: 7807
_________________________________________________________________________________________________________


Is there something else we can check to further get to the root cause of this?

Thanks in advance! Love the software!

[radio42]

Yes, investigate who else is using that VPS.

Note, that you are running PF on a virtual server with the minimum requirements for a dedicated server.
Note, that average CPU or memory doesn’t mean much.
For real time audio procession the IO is much more important. And note, that peak spikes of latency is relevant not an average.

So try to run PF on a dedicated real (not virtual) PC, to compare, if that works.
Try to use Win10 or 11 with a solid WASAPI audio device. Maybe use 16GB of RAM with a decent IO sub-system, eg. audios located on local SSDs. And see, if that makes a difference.

But any Virtual Server is typically running on shared hardware. If for example another VPS on the same physical bare metal runs in parallel and is at certain times also doing some heavy IO task - this might cause spikes you don’t even can monitor on your VPS.

Try to increase your audio buffers to reduce the chance, that the audio might break etc.
And use a dedicated sound card! Only using the No Sound device means no sound card is driving the audio, but your system clock must handle it, which on various virtual servers might be problematic.

But unfortunately I can not give dedicated advices for each specific hardware environment.

[rhroy]

Hi There,

Thank you very much for your (quick!) response on this issue.
We will have to further deep dive into the VPS and perhaps take your comment on using a dedicated server for this purpose.

Should we increase the audio buffers to above 1500? what number do you advise?
Do you also advise a specific PCI sound card or onboard? we don't use MIC's etc from this machine.

[radio42]

Sorry, I do/can not recommend any specific hardware. Any good card recognized under Windows (if you go more professional, take one which natively comes with an ASIO driver) will be fine.
No, 1500ms is already quite high!