And our station went off air....

You have a question or need an advice about how to do something? Ask it here!
Post Reply
User avatar
Havenstad FM
Posts: 104
Joined: 23 Apr 2016 12:36
And our station went off air....

Post by Havenstad FM »

Hi Bernd,

Yesterday morning PFOA just stopped playing music. Playlist was loaded, but somehow the tracks could not be played because a connection to the MLS machine (where all tracks are stored) was suddenly forbidden. However, via the windows network path i could connect without any problem. After a restart of PFOA normal operation resumed. Unfortunately debug logging was off at that time and normal logs give no indication to the error.

This morning i woke up and again the station was off air. This time i have a debug log and it seems that the station was off air between 06:05 until 09:00. Please analyze the accompanying log and see if you can find out what is wrong.

Regards,

Jack Reitsema
ITC Havenstad.FM
Attachments
Debug_20161231.log.zip
(79.64 KiB) Downloaded 359 times
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: And our station went off air....

Post by radio42 »

Hi Jack,

I analyzed your debug log...
There are a few XML issues with certain meta data tag's; which simply indicate, that some files contain a few corrupt tag's; but this is totally uncritical!
Also the Media Library Server connection was NOT returning any error!
BUT...
at 06:05:30 a PlayNext command was internally triggered, because the track "Danyel Gérard - Butterfly" ended (its Next cue-point was reached).
And here the DJ-Player was invoked to start its track; BUT this failed!!!
The reason for this was, that the track in the DJ-Player was not loaded yet (meaning it is still currently being loaded and as such not being ready to play yet).

And from now on all looks very weird, as it seams, that the loading of the next track NEVER finishes!
I have no idea why, as at 6:45 a next program was loaded and and new tracks have been scheduled.
But in between NOTHING happened with the system.

So I can only guess, but it seems, that your file system was at this time completely inaccessible.
Note, that the MLS only transmits the file paths (the UNC paths) to ProppFrexx ONAIR - and this was successful - but ProppFrexx seems then to be unable to access and load those files... and got stuck!

This is what the debug log reveals... so it actually looks like your system IO was completely blocked.
Effectively, this looks like it is NOT caused by ProppFrexx.
Actually I cannot say what causes this, if it is an error in your network card, on your hard drive, an issue with a Anti-Virus software... I am sorry, but this needs further investigation on the OS level, which I can not control - I am sorry.

It is funny, that ONLY the track "HavenstadFM - TOH - Algemeen" was able to successfully load?!
Could it be, that this one is located on a local drive; whereas all other tracks are located on a networked drive (\\PF-MLS\D).
Is it possible, that this networked drive was faulty at that time?
---

I also noticed, that you are ONLY using 2 DJ-Players A and B!
And this is absolutely NOT recommended for automatic playback (I guess we already have discussed this in the past a couple of times).
With only 2 DJ-Players, there might be situations of dead air, especially when playing short jingles followed by another track.
While track A (the previous regular track) is about to finish and reaches its Next cue-point, it triggers track B, which starts to play (the short jingle). However, this track B might now finish, before track A reaches his natural end (cue-out)...

So I can only recommend, that you switch to using 3 DJ-Players!
User avatar
Havenstad FM
Posts: 104
Joined: 23 Apr 2016 12:36
Re: And our station went off air....

Post by Havenstad FM »

Hi Bernd,

First of all let me wish you all the best wishes for the coming year!

Secondly, let's hope to solve this problem as this morning we were again off air. I'm not sure at what time exactly but i include the log again for you. Already some listeners are complaining... (they use our station as their alarmclock)

Strange thing is that the track "Havenstad - TOH - Algemeen" (which is our opening jingle for every hour) is referenced directly through a network mapping Z: which points to \\PF-MLS\D\HFMNEW\
So it seems tracks via Z: load fine, but tracks via \\PF-MLS\D\HFMNEW cannot be loaded, but after a restart of PFOA the problem has disappeared and all tracks are loading again.

I linked all remote libs via 192.168.10.18:8888 which results in the UNC path \\PF-MLS\D\HFMNEW\.... Should i perhaps try to add the remote medialibs via the network map Z: ?

Your concerns regarding just 2 dj-players are duly noted. At the moment we do not have an extra digital output to our external mixer. Besides we have been operating with only 2 players since 2006 which is the reason why we are very keen on the timings for especially the shorter playlist items like jingles, shouts, etc. But we'll discuss it again and i agree, we could add an extra player at least for the unmanned hours.

With regards,

Jack Reitsema
ITC Havenstad.FM
Attachments
Debug_20170101.log.zip
(65.15 KiB) Downloaded 349 times
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: And our station went off air....

Post by radio42 »

Looks exactly the same as the previous one.
I guess I might need to send you a more detailed debug version.
When did the problem start? WhT have you changed since it happens.

What about all my questions above?
Virus software, hard disk issue etc. are you sure you can exclude all this?
User avatar
Havenstad FM
Posts: 104
Joined: 23 Apr 2016 12:36
Re: And our station went off air....

Post by Havenstad FM »

Hi Bernd,

The problem started after our update to v4.0.2.10 But this coincided with our switch to a new PC which is now running the MLS and has all our media on SSD's.

Before we had our media on a NAS and a MLS on an old pc. (NAS 192.168.10.51 -> MLS 192.168.10.17 -> PFOA 192.168.10.20) On both the MLS and PFOA machines i had a network map Z: pointing to the NAS media directory. In the MLS i created folder based medialibs pointing to the Z:\.... and in PFOA i created remote medialibs connecting to 192.168.10.17 which was the MLS and i used MLS setting 'Use Absolute Filepaths'.

That setup worked, but a lot of buffering was going on and also the reading of TAGs and searching were pretty slow.

Now we use a new machine for all our media and for running the MLS. So we do not use the NAS for our media anymore. This new pc has all media on SSD to optimize especially write speeds. This new machine is the PF-MLS 192.168.10.18.
In the MLS I created folder based medialibs which point to the local SSD drive where the media is located. In PFOA i connect to these medialibs via Add Remote Libs and used connection to 192.168.10.18. I still use the setting 'Use Absolute Filepaths'

Since we use the last setup we have a lot of troubles with downtime. Empty playlists because of the album tags which were the same (see my other issue). After working around that issue by disabling history checking we experience these problems where a full playlist is visible in PFOA, auto-play is ON but no player is playing. Manually starting a player does not work because the track cannot be loaded. In windows file manager every track in \\PF-MLS\D\HFMNEW.... is playable in for example the VLC mediaplayer which rules out firewall and authentication issues. Only PFOA will not load the tracks. If i create a Z: drivemap to the same directory on PF-MLS i can load and play the tracks in PFOA without any problem. If i restart PFOA everything works fine again for about a day, but the next morning we are off air again.

It looks like the connection to the remote libs is only valid for a day.

I have now removed the remote libs and instead created my network mapping Z: pointing to the \\PF-MLS\D\HFMNEW and created folder based medialibs to this Z:\... folders in PFOA. Hereby i am not in need of the MLS anymore and we will see tomorrow morning if we are still in the air.

Regards,

Jack Reitsema
ITC Havenstad.FM
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: And our station went off air....

Post by radio42 »

What you can try (til we find the real root cause; as many components seems to have changed) is go back to the previous PF version - if that behaves in the same way, we at least have an indication; else the other way around.
I know many many user how also use the latest version and a MLS, without any issues.

However, I need to add a few more debug log writes, to better understand, why our playback all of a sudden stops.
It could also be a sound card or another driver issue on the new PC causing all that trouble.
But in order to know this, I need to exactly understand, at which command the playback stops.

One more question:
Too me it looks, like playback suddenly stops.
So when you find the PC 'off-air':
- is the still a playlist with 'unplayed track? Or how does the playlist look like?
- Are tracks loaded to the DJ Player? Or are the DJ Players empty?

As said, to me it looks, like tracks are loaded to the DJ Players, as well as there are unplayed tracks in the current playlist; but the DJ Players stop playing the loaded tracks!
If this is the case, this doesn't sound like it is an issue with the MLS?!
User avatar
Havenstad FM
Posts: 104
Joined: 23 Apr 2016 12:36
Re: And our station went off air....

Post by Havenstad FM »

Hi Bernd,

This morning we had no problems.

If the pc is off air, it is brought to my attention by a whatsapp message (as i am on a vacation). I can login remotely and find the playlist full with unplayed tracks. Only our opening jingle has played (Z:\10_Jingles\00_Nieuws_Systeem\01_Uuropeners\TOP_OF_HOUR_ALGEMEEN.mp3) which is using a drive mapping. All other tracks are from remote libs and have a network path as their absolute filename (\\PF-MLS\D\HFMNEW\....) which is probably served by the MLS.
If i remember correctly both players are loaded but not playing and when i manually click the play button nothing happens. When i try to drag one of the tracks to a player it will not load. If i look up the same track with windows filemanager using the same network path and credentials and double click the audio file it loads and plays in VLC mediaplayer (my default audio player). Unfortunately i did not try to drag that track to a proppfrexx player. To restore normal operations i just restart PFOA, but the next day the same happens again. This happened the last 5 days of 2016 and the first day of 2017.

BTW, the PF-MLS machine does not have a soundcard, just the soundchip on the motherboard. I don't think the MLS server needs a soundcard, does it?

Now that i changed the remote medialibs in PFOA to folderbased libs using a network mapping (which i did yesterday evening) it seems the problem is gone. We had no problems today.

You realize that by debugging the same situation again i have to change back to remote libs and we will probably go of air again tomorrow morning - a situation that our station manager is not too happy about. I am willing to do this if it helps You, but not while i am on this slow hotel wifi. I will be back home end of this week and am willing to setup the debug situation over the weekend to get you a debug log on monday 09-01-2017. (I'm already too busy with these problems during my vacation - my wife wants some of my attention too ;-)

Regards,

Jack Reitsema
ITC Havenstad.FM
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: And our station went off air....

Post by radio42 »

There is a new v4.0.2.13-beta available.
(so "Check for Beta-Versions..." to install), which adds a couple of additional Debug-Log messages to hopefully reveal a better picture and root cause...
However, please also see my questions above...
User avatar
Havenstad FM
Posts: 104
Joined: 23 Apr 2016 12:36
Re: And our station went off air....

Post by Havenstad FM »

Hi Bernd,

I am happy to report that we had no problems anymore these first two weeks of january.
It seems that the switch to drive mapped folder libraries did the trick.

Regards,

Jack Reitsema
ITC Havenstad.FM
Ecinmilo0
Posts: 1
Joined: 28 Apr 2017 06:04
Re: And our station went off air....

Post by Ecinmilo0 »

Your question helped me. return man 3

Post Reply