An exception was thrown by a TaskScheduler

[mart3223]

Hi Bernd,

This night our system stopped, the following errors I captured. The Overlay Player was loaded but completely freezed. Even the next scripts didn't start, after 02.00 it was quiet. I see as a first error "An exception was thrown by a TaskScheduler" but I can't bring that error back to a recently change of our scripts / programming.

Overlaylog

Code: Select all

28-5-2014 2:56:33: Overlay:Commercials, Command:Advertising (Hele_uur), Start:28-5-2014 3:00:00 processed
28-5-2014 2:56:44: WARNING: Overlay-Scheduler-Exception:'An exception was thrown by a TaskScheduler.'!
28-5-2014 3:56:39: WARNING: Overlay 'Commercials@2014-05-28T04:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=3).
28-5-2014 3:56:39: WARNING: Overlay 'Nieuws@2014-05-28T04:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=3).
28-5-2014 3:56:39: WARNING: Overlay 'Uuropener@2014-05-28T04:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=3).
28-5-2014 4:56:40: WARNING: Overlay 'Commercials@2014-05-28T05:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=6).
28-5-2014 4:56:40: WARNING: Overlay 'Nieuws@2014-05-28T05:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=6).
28-5-2014 4:56:40: WARNING: Overlay 'Uuropener@2014-05-28T05:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=6).
28-5-2014 5:56:42: WARNING: Overlay 'Commercials@2014-05-28T06:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=9).
28-5-2014 5:56:42: WARNING: Overlay 'Nieuws@2014-05-28T06:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=9).
28-5-2014 5:56:42: WARNING: Overlay 'Uuropener@2014-05-28T06:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=9).
28-5-2014 6:56:43: WARNING: Overlay 'Commercials@2014-05-28T07:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=12).
28-5-2014 6:56:43: WARNING: Overlay 'Nieuws@2014-05-28T07:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=12).
28-5-2014 6:56:43: WARNING: Overlay 'Uuropener@2014-05-28T07:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=12).
28-5-2014 7:56:45: WARNING: Overlay 'Commercials@2014-05-28T08:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=15).
28-5-2014 7:56:45: WARNING: Overlay 'Nieuws@2014-05-28T08:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=15).
28-5-2014 7:56:45: WARNING: Overlay 'Uuropener@2014-05-28T08:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=15).
28-5-2014 7:58:34: Calendar 'D:\_Reclame\Overlay.calendar' updated (last modified: 22-5-2014 23:03:43)

Debug-Log:

Code: Select all

8-5-2014 1:58:46: Processing Script-Line 3 (Retries=60): 2 Random : REMOTE:192.168.1.13:8088//Various Dance Classics Pop Edition (Filter=, Options=SuppressHistoryCheck)
28-5-2014 1:58:46: GetNextTracks processed, 3 script result entries returned.
28-5-2014 2:00:00: PLS-PlayNext.
28-5-2014 2:56:33: Overlay 'Commercials@2014-05-28T03:00:00' starts processing.
28-5-2014 2:56:33: Overlay 'Commercials@2014-05-28T03:00:00' start loading.
28-5-2014 2:56:33: Overlay 'Commercials@2014-05-28T03:00:00' ProgressChanged State=0.
28-5-2014 2:56:33: OverlayScheduler: Overlay:Commercials, Command:Advertising (Hele_uur), Start:28-5-2014 3:00:00 processed
28-5-2014 2:56:33: Overlay 'Commercials@2014-05-28T03:00:00' starting LoadAdvertSlot.
28-5-2014 2:56:33: Overlay 'Commercials@2014-05-28T03:00:00' LoadAdvertSlot 2 Advert-Tracks (4 Total) found.
28-5-2014 2:56:33: Overlay 'Commercials@2014-05-28T03:00:00' LoadAdvertSlot finished 4 Total-Tracks (00:01:17.0640000 Playtime).
28-5-2014 2:56:33: Overlay 'Commercials@2014-05-28T03:00:00' finish loading.
28-5-2014 2:56:33: Overlay 'Commercials@2014-05-28T03:00:00' starting (4 Tracks).
28-5-2014 2:56:44: Overlay 'Nieuws@2014-05-28T03:00:00' enqueued as parallel overlay.
28-5-2014 2:56:44: OverlayScheduler: WARNING: Overlay-Scheduler-Exception:'An exception was thrown by a TaskScheduler.'!
28-5-2014 2:56:45: Overlay-Scheduler-Exception: 28-5-2014 2:56:44 [v3.0.13.24]:********** Error/Exception Information **********
Error Message: An exception was thrown by a TaskScheduler.
Error Type: System.Threading.Tasks.TaskSchedulerException
Error Source: mscorlib.Void ScheduleAndStart(Boolean)
Inner Exception: Exception of type 'System.OutOfMemoryException' was thrown.
********** System Stack Trace **********
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
   at System.Threading.Tasks.Task.InternalStartNew(Task creatingTask, Delegate action, Object state, CancellationToken cancellationToken, TaskScheduler scheduler, TaskCreationOptions options, InternalTaskOptions internalOptions, StackCrawlMark& stackMark)
   at System.Threading.Tasks.TaskFactory.StartNew(Action action, TaskCreationOptions creationOptions)
   at ProppFrexx.Player.b.h(j A_0)
   at ProppFrexx.da.a(j A_0)
   at ProppFrexx.da.a(Object A_0, ReminderEventArgs A_1)
Error Message: Exception of type 'System.OutOfMemoryException' was thrown.
Error Type: System.OutOfMemoryException
Error Source: mscorlib.Void StartInternal(System.Security.Principal.IPrincipal, System.Threading.StackCrawlMark ByRef)
********** System Stack Trace **********
   at System.Threading.Thread.StartInternal(IPrincipal principal, StackCrawlMark& stackMark)
   at System.Threading.Thread.Start(StackCrawlMark& stackMark)
   at System.Threading.Thread.Start(Object parameter)
   at System.Threading.Tasks.ThreadPoolTaskScheduler.QueueTask(Task task)
   at System.Threading.Tasks.Task.ScheduleAndStart(Boolean needsProtection)
********** Environment Stack Trace **********
   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
   at System.Environment.get_StackTrace()
   at ProppFrexx.ab.a(Exception A_0, Boolean A_1)
   at ProppFrexx.da.a(Object A_0, ReminderEventArgs A_1)
   at DevExpress.XtraScheduler.SchedulerStorageBase.RaiseReminderAlert(ReminderEventArgs e)
   at DevExpress.XtraScheduler.SchedulerStorageBase.OnRemindersAlerted(Object sender, ReminderEventArgs e)
   at DevExpress.XtraScheduler.Native.ReminderEngine.OnReminderAlert(Object sender, ReminderBaseEventArgs e)
   at DevExpress.XtraScheduler.Native.ReminderEngineBase.RaiseReminderAlert(ReminderBaseEventArgs e)
   at DevExpress.XtraScheduler.Native.ReminderEngineBase.RaiseRemindersAlertsCore(ReminderBaseEventArgs args)
   at DevExpress.XtraScheduler.Native.ReminderEngineBase.RaiseRemindersAlerts(DateTime currentTime)
   at DevExpress.XtraScheduler.Native.ReminderEngineBase.OnTimerTick(Object sender, EventArgs e)
   at DevExpress.XtraScheduler.Native.LeakSafeTimerTickEventRouter.OnTimerTick(Object sender, EventArgs e)
   at System.Windows.Forms.Timer.OnTick(EventArgs e)
   at System.Windows.Forms.Timer.TimerNativeWindow.WndProc(Message& m)
   at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
   at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
   at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
   at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
   at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
   at System.Windows.Forms.Application.Run(Form mainForm)
   at ProppFrexx.dw.a(String[] A_0)
********** End **********

28-5-2014 2:56:45: Overlay 'Uuropener@2014-05-28T03:00:01' enqueued as parallel overlay.
28-5-2014 2:58:45: ProgramScheduler: Program:Muziekmix (Muziekmix), Start:28-5-2014 3:00:00, End:28-5-2014 4:00:00, Started:True
28-5-2014 2:58:46: Processing Script-Line 1 (Retries=60): 2 Random : REMOTE:192.168.1.13:8088//7-12-inch singles Tipparade (Filter=, Options=SuppressHistoryCheck)
28-5-2014 2:58:46: GetNextTracks processed, 2 script result entries returned.
28-5-2014 2:58:46: Processing Script-Line 2 (Retries=60): 2 Random : REMOTE:192.168.1.13:8088//7-12-inch singles Top 40 (Filter=, Options=SuppressHistoryCheck)
28-5-2014 2:58:46: GetNextTracks processed, 3 script result entries returned.
28-5-2014 2:58:46: Processing Script-Line 3 (Retries=60): 2 Random : REMOTE:192.168.1.13:8088//Various Dance Classics Pop Edition (Filter=, Options=SuppressHistoryCheck)
28-5-2014 2:58:46: GetNextTracks processed, 2 script result entries returned.
28-5-2014 2:58:46: Processing Script-Line 4 (Retries=60): 2 Random : REMOTE:192.168.1.13:8088//Various Top 40 (Filter=, Options=SuppressHistoryCheck)
28-5-2014 2:58:46: GetNextTracks processed, 3 script result entries returned.
28-5-2014 2:58:46: Processing Script-Line 5 (Retries=60): 2 Random : REMOTE:192.168.1.13:8088//Various Hitzone (Filter=, Options=SuppressHistoryCheck)
28-5-2014 2:58:46: GetNextTracks processed, 3 script result entries returned.
28-5-2014 2:58:46: Processing Script-Line 1 (Retries=60): 2 Random : REMOTE:192.168.1.13:8088//7-12-inch singles Tipparade (Filter=, Options=SuppressHistoryCheck)
28-5-2014 2:58:46: GetNextTracks processed, 3 script result entries returned.
28-5-2014 2:58:46: Processing Script-Line 2 (Retries=60): 2 Random : REMOTE:192.168.1.13:8088//7-12-inch singles Top 40 (Filter=, Options=SuppressHistoryCheck)
28-5-2014 2:58:46: GetNextTracks processed, 3 script result entries returned.
28-5-2014 2:58:48: DJPlayerReport: Player=A, Event=TrackLoad: Brennan Heart And Jonathan Mendelsohn - Imaginary
28-5-2014 3:56:39: OverlayScheduler: WARNING: Overlay 'Commercials@2014-05-28T04:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=3).
28-5-2014 3:56:39: OverlayScheduler: WARNING: Overlay 'Nieuws@2014-05-28T04:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=3).
28-5-2014 3:56:39: OverlayScheduler: WARNING: Overlay 'Uuropener@2014-05-28T04:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=3).
28-5-2014 3:58:45: ProgramScheduler: WARNING: Program 'Muziekmix @ 28-5-2014 4:00:00' could not be started (another program is already starting at the same time)!
28-5-2014 4:56:40: OverlayScheduler: WARNING: Overlay 'Commercials@2014-05-28T05:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=6).
28-5-2014 4:56:40: OverlayScheduler: WARNING: Overlay 'Nieuws@2014-05-28T05:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=6).
28-5-2014 4:56:40: OverlayScheduler: WARNING: Overlay 'Uuropener@2014-05-28T05:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=6).
28-5-2014 4:58:46: ProgramScheduler: WARNING: Program 'Piratenmix @ 28-5-2014 5:00:00' could not be started (another program is already starting at the same time)!
28-5-2014 5:56:42: OverlayScheduler: WARNING: Overlay 'Commercials@2014-05-28T06:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=9).
28-5-2014 5:56:42: OverlayScheduler: WARNING: Overlay 'Nieuws@2014-05-28T06:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=9).
28-5-2014 5:56:42: OverlayScheduler: WARNING: Overlay 'Uuropener@2014-05-28T06:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=9).
28-5-2014 5:58:46: ProgramScheduler: WARNING: Program 'Piratenmix @ 28-5-2014 6:00:00' could not be started (another program is already starting at the same time)!
28-5-2014 6:56:43: OverlayScheduler: WARNING: Overlay 'Commercials@2014-05-28T07:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=12).
28-5-2014 6:56:43: OverlayScheduler: WARNING: Overlay 'Nieuws@2014-05-28T07:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=12).
28-5-2014 6:56:43: OverlayScheduler: WARNING: Overlay 'Uuropener@2014-05-28T07:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=12).
28-5-2014 6:58:47: ProgramScheduler: WARNING: Program 'Muziekmix @ 28-5-2014 7:00:00' could not be started (another program is already starting at the same time)!
28-5-2014 7:56:45: OverlayScheduler: WARNING: Overlay 'Commercials@2014-05-28T08:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=15).
28-5-2014 7:56:45: OverlayScheduler: WARNING: Overlay 'Nieuws@2014-05-28T08:00:00' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=15).
28-5-2014 7:56:45: OverlayScheduler: WARNING: Overlay 'Uuropener@2014-05-28T08:00:01' not processed Current 'Commercials@2014-05-28T03:00:00' still running (Status (Shown) background thread still busy, Count=15).

[mart3223]

Hi Bernd,

Memory ... but how? On the system runs only PF en Media Libary. Everything is stored on internal harddisks. Out of memory, in normal state the system uses only 10% of a total of 16GB of memory. The only possibility if ProppFrexx is stable, 2 days it kills himself through a memory failure. Everyday a reboot? How can we fix this, you know how the system is configured by us (Forum user XTended is always talking about the same system) we are using Media Server, System is up to date, Everything is stored on internal drives, 16GB of memory.

MediaLibary has:
Loaded Media Libraries: 91
Total songs / commercials / jingles: +/- 250.000

Is that to much for the system? We are at this moment out of options, we don't know how to get it stable.

[XTended]

Dear Bernd,

We use the Media Library Server since February 2014...
All ideas you gave us, we use. So it must be something else.

[radio42]

Have you may be still added/used your media libs within PF - plus the remote libs?
What is your memory utilization of ProppFrexx within the TaskManager?

When you are only using remote libs and no additional media libs within PF the memory consumption should be at max around 500 MB!
What is yours...

I mean an out-of-memory except is an OS related exception and if it comes, it means out of memory...

Else can we do a TeamViewer session to take a look at?

[XTended]

We only use the Media Library Server, so there aren't any loaded music files anymore within PF; all is set in the MLS.
Our memory consumption looks a little bit higher; see image I created couple minutes ago.

[radio42]

As you might see from the logs.
The exception thrown was a 'System.OutOfMemoryException' - which effectively means: out of memory.
There is not much I can do about this kind of error.

If you are using a lot of media libs resp. a lot of entries within your media libs, it might be a good idea to use the Media Library Server.
See here for details: viewtopic.php?f=9&t=1014

[radio42]

Higher than what?
ProppFrexx OnAir uses less than 400MB!?
So please watch, if that increases over time...

[radio42]

Note, that ProppFrexx is a 32bit application! So it leverages only up to 3GB of main memory!
Where your local files are stored doesn't relate to main memory.
Normal states (10%) also doesn't really matter, as the peak memory usage is what matters.
Your number of media entries are not too much in general, but might be too much for a 32bit process to hold in memory.
How to Fix?
That's why I suggested to use the Media Library Server! That should be the solution in your case.

[radio42]

If you need help to migrate to the Media Library Server let me know...
See the post above...
Basically you define the same media libs as within ProppFrexx now within the MLS. And the you change your regular media libs within PF by Remote Media Lib entries, that's it.

[XTended]

443.620 + 397.888 = higher than your 500 MB memory consumption...
We'll keep in touch.
Debug is still on