Memory problem

You have a question or need an advice about how to do something? Ask it here!
User avatar
Havenstad FM
Posts: 104
Joined: 23 Apr 2016 12:36
Memory problem

Post by Havenstad FM »

Good morning Bernd,

As You might still remember we are using PFOA for years now and mostly without problems.

One problem however has plagued us through the years and that is the memory use of PF. We have 8 Gb of memory in our system and have noticed that PF occupies a slowly increasing amount of memory up to the point where the systems comes to a halt because it is running out of memory. This would normally take about two weeks.

This never was a real problem for us because in our weekly maintenance routine we also restart every windows machine which took care of this problem.

However, at 1-7-2020 12:00hrs, our station changed it's name and with the name change of course also all the jingles, promo's, etc. Where in the past we played all jingles from the local machines (from folder libs) i used the MLS for the new jingles and promo's. Also the frequency for playing jingles and promo's was increased. At that time we were running PF v.4.2.0.7.

On saturday evening at july 4th we experienced the first crash due to memory shortage and then again on sunday morning july 5th. And again on sunday afternoon. I started monitoring the memory use and noticed that the memory was filling up per minute... I kept us on the air by monitoring and restarting in time ;-)

To try and solve the issue i moved all jingles and promo's on the PF machine and created local folder libs for them just as we did before 1-7-2020 with our previous jingles set. This did help a bit, but still, instead of a few hours the leakage slowed down to approx. 10 - 14 hours.

I updated to PF v.4.2.0.8 which made no difference and we increased the memory to 16 Gb to keep the problem manageable and since then i have been analyzing the memory use.

As You might recall we have 2 identical PF machines (the PF-Master and PF-Slave) both with the exact same setup. We use the PF-Master for most of the live broadcasting and the PF-Slave is the backup system for when the Master fails for some reason, or when i take the master down for maintenance etc. then we use the slave for broadcasting.

Only in the master we added the extra memory (up to 16 Gb) the slave still has 8 Gb. That is the only difference. Also, both of the machines experience the same memory leak. During the last few weeks i have constantly switched from master to slave and back again when the memory was filled to around 90% to prevent the crash. I have been experimenting with settings and compared the difference using the second system. I noticed the following: I also include a few screenshots to show the memory details:
  • If i start PF and immediately log in with a non-administrator account the memory leak is barely noticable and i can run for weeks without a problem (see the screenshots of the PF-Master).
  • If i start PF and do *not* log in, the memory leak is filling up the memory at it's fastest and in less than 12 hours the slave system will crash. The master will last longer due to it's larger memory, but will crash too. (see the screenshots of the PF-Slave).
  • If i start PF and immediately log in with an administrator account it is the same as not logging in. Memory fills up rapidly.
  • if i start PF and wait to log in with a non-administrator account until after a minute or so the leak is present as well. Memory is filling up rapidly
So, at this moment, the only way to run PF is by starting it and immediately log in as a non-administrator. This only works during our nonstop hours. As soon as we have had a live program on air and one of our dj's changed the login to his account, i have to restart PF afterwards and login again as a non-administrator, otherwise a memory crash is imminent in a matter of hours.

Included are some screenshots from both the master and the slave system where the slave was running while *not* logged in and the master is running with a non-administrator account logged in. The first screenshots are from 09:05hrs where you can see the slave after a fresh restart and PF is using 390,5 MB and the master has already been running for approx. 11 hours and PF is using 572,5 MB.
I include 3 screenshots of both systems and on the last one at 15:08hrs You can see that on the master the memory use of PF has increased to 586,9 MB whereas on the slave PF's memory use has increased to 6,2 GB !!

So, if i would have run the master while logged out, the increase in memory use would have been the same. Very strange, right?
Another thing to note is that after exiting PF on the slave machine, PF is not closing immediately, but You can see the memory use slowly decreasing and only when it is down to normal values like below 800 MB then PF is really closing and exiting. This will take some time like 20 seconds or so. (i did not time it)

Well Bernd, i hope all this gives you enough clues to get to a solution for the problem. Is it our PF setup? Is it the MLS-setup? Is PF loading the metadata from every played track from a remote library in memory without freeing it again if not logged in? It almost looks that way.

Please let me know if you need more info and i will provide it if i can.

With kind regards,

Jack Reitsema
ICT Omroep Eemsdelta (before: Havenstad.FM)
mailto:ict@omroepeemsdelta.nl

NB. Do we need to update our licenses now that we have a new station name?
Attachments
Screenshots_4.7z
(435.44 KiB) Downloaded 242 times
Screenshots_3.7z
(429.53 KiB) Downloaded 235 times
Screenshots_1.7z
(431.94 KiB) Downloaded 242 times
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: Memory problem

Post by radio42 »

Hi Jack,

First you don’t need to change you license now.
Than thanks for the extensive info giving.
6GB memory use after one day of use is definitely not normal.
So far I am however not aware of any memory leak, as this doesn’t happen on my test systems nor did so far report any other user if such a leak.

So I would need some time to investigate to reproduce , as this might be sometimes tricky.
I hope that this is indeed ProppFrexx causing it, as I have also seen a system driver causing it. But what you describe (use of UAC) doesn’t sound like a system issue.
So give me some time to investigate, esp. as I am currently on holiday.
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: Memory problem

Post by radio42 »

Beside the screenshots, can you also post me the Error.log file to inspect the real error message! When the crash is happening.
You can find the ProppFrexx Error.log file in its AppUserData folder. It should contain an entry about the crash.
If not, this might indicate indeed a system issue and further details might be found in the Windows Event Log,as Windows records every Application crash. All these details are needed to further nail down the root cause of the problem.
User avatar
Havenstad FM
Posts: 104
Joined: 23 Apr 2016 12:36
Re: Memory problem

Post by Havenstad FM »

Hi Bernd,

It was not my intention to disturb your time off. Just take it easy, as i said i am manageing at the moment.

Here are the log entries of the 2 crashes, i hope it helps. FIY, after the first crash i restarted PF, but at the time of the second crash i was sound asleep ;-) I see in the log that we were still running v4.2.0.5 where i thought it was v4.2.0.7 at that time. Anyhow, at this moment we are running v4.2.0.8.

Unfortunately, the windows logs for that date have already been recycled. If you need it i can always let the slave crash once more.

5-7-2020 00:30:05 [v4.2.0.5]:********** Error/Exception Information **********
Error Message: Out of memory.
Error Type: System.OutOfMemoryException
Error Source: System.Drawing.Void CheckErrorStatus(Int32)
********** System Stack Trace **********
at System.Drawing.Graphics.CheckErrorStatus(Int32 status)
at System.Drawing.Graphics.DrawImage(Image image, Rectangle destRect, Int32 srcX, Int32 srcY, Int32 srcWidth, Int32 srcHeight, GraphicsUnit srcUnit, ImageAttributes imageAttrs, DrawImageAbort callback, IntPtr callbackData)
at System.Drawing.Graphics.DrawImage(Image image, Rectangle destRect, Int32 srcX, Int32 srcY, Int32 srcWidth, Int32 srcHeight, GraphicsUnit srcUnit, ImageAttributes imageAttr)
at System.Windows.Forms.ControlPaint.DrawBackgroundImage(Graphics g, Image backgroundImage, Color backColor, ImageLayout backgroundImageLayout, Rectangle bounds, Rectangle clipRect, Point scrollOffset, RightToLeft rightToLeft)
at System.Windows.Forms.Control.PaintBackground(PaintEventArgs e, Rectangle rectangle, Color backColor, Point scrollOffset)
at System.Windows.Forms.Control.PaintBackground(PaintEventArgs e, Rectangle rectangle)
at System.Windows.Forms.Control.OnPaintBackground(PaintEventArgs pevent)
at System.Windows.Forms.Control.PaintWithErrorHandling(PaintEventArgs e, Int16 layer)
at System.Windows.Forms.Control.WmPaint(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
********** Environment Stack Trace **********
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at ProppFrexx.ErrorDialog.GetExceptionInfo(Exception e, Boolean full)
at ProppFrexx.ProppFrexx_ONAIR.pC70(Exception )
at ProppFrexx.ProppFrexx_ONAIR.GC7t(Exception )
at System.Windows.Forms.Application.ThreadContext.OnThreadException(Exception t)
at System.Windows.Forms.Control.WndProcException(Exception e)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Form.ShowDialog(IWin32Window owner)
at ProppFrexx.ProppFrexx_ONAIR.hCOo(Object , ItemClickEventArgs )
at DevExpress.XtraBars.BarItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarButtonItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarButtonItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarItemLink.OnLinkClick()
at DevExpress.XtraBars.BarButtonItemLink.OnLinkAction(BarLinkAction action, Object actionArgs)
at DevExpress.XtraBars.ViewInfo.BarSelectionInfo.ClickLink(BarItemLink link)
at DevExpress.XtraBars.ViewInfo.BarSelectionInfo.UnPressLink(BarItemLink link)
at DevExpress.XtraBars.Ribbon.Handler.BaseRibbonHandler.OnUnPressItem(DXMouseEventArgs e, RibbonHitInfo hitInfo)
at DevExpress.XtraBars.Ribbon.Handler.BaseRibbonHandler.OnMouseUp(DXMouseEventArgs e)
at DevExpress.XtraBars.Ribbon.Handler.RibbonHandler.OnMouseUp(DXMouseEventArgs e)
at DevExpress.XtraBars.Ribbon.RibbonControl.OnMouseUp(MouseEventArgs e)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at DevExpress.Utils.Controls.ControlBase.BaseWndProc(Message& m)
at DevExpress.XtraBars.Ribbon.RibbonControl.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
at DevExpress.Utils.Drawing.Helpers.NativeMethods.UnsafeNativeMethods.DefSubclassProc(IntPtr hWnd, IntPtr Msg, IntPtr wParam, IntPtr lParam)
at DevExpress.Utils.Drawing.Helpers.NativeMethods.UnsafeNativeMethods.DefSubclassProc(IntPtr hWnd, IntPtr Msg, IntPtr wParam, IntPtr lParam)
at DevExpress.Utils.Drawing.Helpers.Win32SubclasserFactory.Win32Subclasser.SubClassProcInner(IntPtr hWnd, IntPtr Msg, IntPtr wParam, IntPtr lParam, IntPtr uIdSubclass, IntPtr dwRefData)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
at aq.mX.Main(String[] args)
********** End **********

6-7-2020 04:58:04 [v4.2.0.5]:********** Error/Exception Information **********
Error Message: Out of memory.
Error Type: System.OutOfMemoryException
Error Source: System.Drawing.Void CheckErrorStatus(Int32)
********** System Stack Trace **********
at System.Drawing.Graphics.CheckErrorStatus(Int32 status)
at System.Drawing.Graphics.DrawImage(Image image, Rectangle destRect, Int32 srcX, Int32 srcY, Int32 srcWidth, Int32 srcHeight, GraphicsUnit srcUnit, ImageAttributes imageAttrs, DrawImageAbort callback, IntPtr callbackData)
at System.Drawing.Graphics.DrawImage(Image image, Rectangle destRect, Int32 srcX, Int32 srcY, Int32 srcWidth, Int32 srcHeight, GraphicsUnit srcUnit, ImageAttributes imageAttr)
at System.Windows.Forms.ControlPaint.DrawBackgroundImage(Graphics g, Image backgroundImage, Color backColor, ImageLayout backgroundImageLayout, Rectangle bounds, Rectangle clipRect, Point scrollOffset, RightToLeft rightToLeft)
at System.Windows.Forms.Control.PaintBackground(PaintEventArgs e, Rectangle rectangle, Color backColor, Point scrollOffset)
at System.Windows.Forms.Control.PaintBackground(PaintEventArgs e, Rectangle rectangle)
at System.Windows.Forms.Control.OnPaintBackground(PaintEventArgs pevent)
at System.Windows.Forms.Control.PaintWithErrorHandling(PaintEventArgs e, Int16 layer)
at System.Windows.Forms.Control.WmPaint(Message& m)
at System.Windows.Forms.Control.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
********** Environment Stack Trace **********
at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)
at System.Environment.get_StackTrace()
at ProppFrexx.ErrorDialog.GetExceptionInfo(Exception e, Boolean full)
at ProppFrexx.ProppFrexx_ONAIR.pC70(Exception )
at ProppFrexx.ProppFrexx_ONAIR.GC7t(Exception )
at System.Windows.Forms.Application.ThreadContext.OnThreadException(Exception t)
at System.Windows.Forms.Control.WndProcException(Exception e)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Form.ShowDialog(IWin32Window owner)
at ProppFrexx.ProppFrexx_ONAIR.hCOo(Object , ItemClickEventArgs )
at DevExpress.XtraBars.BarItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarButtonItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarButtonItem.OnClick(BarItemLink link)
at DevExpress.XtraBars.BarItemLink.OnLinkClick()
at DevExpress.XtraBars.BarButtonItemLink.OnLinkAction(BarLinkAction action, Object actionArgs)
at DevExpress.XtraBars.ViewInfo.BarSelectionInfo.ClickLink(BarItemLink link)
at DevExpress.XtraBars.ViewInfo.BarSelectionInfo.UnPressLink(BarItemLink link)
at DevExpress.XtraBars.Ribbon.Handler.BaseRibbonHandler.OnUnPressItem(DXMouseEventArgs e, RibbonHitInfo hitInfo)
at DevExpress.XtraBars.Ribbon.Handler.BaseRibbonHandler.OnMouseUp(DXMouseEventArgs e)
at DevExpress.XtraBars.Ribbon.Handler.RibbonHandler.OnMouseUp(DXMouseEventArgs e)
at DevExpress.XtraBars.Ribbon.RibbonControl.OnMouseUp(MouseEventArgs e)
at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)
at System.Windows.Forms.Control.WndProc(Message& m)
at DevExpress.Utils.Controls.ControlBase.BaseWndProc(Message& m)
at DevExpress.XtraBars.Ribbon.RibbonControl.WndProc(Message& m)
at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)
at DevExpress.Utils.Drawing.Helpers.NativeMethods.UnsafeNativeMethods.DefSubclassProc(IntPtr hWnd, IntPtr Msg, IntPtr wParam, IntPtr lParam)
at DevExpress.Utils.Drawing.Helpers.NativeMethods.UnsafeNativeMethods.DefSubclassProc(IntPtr hWnd, IntPtr Msg, IntPtr wParam, IntPtr lParam)
at DevExpress.Utils.Drawing.Helpers.Win32SubclasserFactory.Win32Subclasser.SubClassProcInner(IntPtr hWnd, IntPtr Msg, IntPtr wParam, IntPtr lParam, IntPtr uIdSubclass, IntPtr dwRefData)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.UnsafeNativeMethods.DispatchMessageW(MSG& msg)
at System.Windows.Forms.Application.ComponentManager.System.Windows.Forms.UnsafeNativeMethods.IMsoComponentManager.FPushMessageLoop(IntPtr dwComponentID, Int32 reason, Int32 pvLoopData)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoopInner(Int32 reason, ApplicationContext context)
at System.Windows.Forms.Application.ThreadContext.RunMessageLoop(Int32 reason, ApplicationContext context)
at aq.mX.Main(String[] args)
********** End **********

With kind regards,

Jack Reitsema
ICT Omroep Eemsdelta.
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: Memory problem

Post by radio42 »

No problem, it indeed seems to be somehow related to UAC.
It sees to only happen, when you are logged out and the login dialog is then shown...?!
I'll need to inspect... as said, it might take a bit of time, as memory leaks are hard to detect and grasp, but I am at it...

Disabling UAC also seems to be a workaround for now...
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: Memory problem

Post by radio42 »

Just a quick update...
So far, I was not able to reproduce it. So I need to continue with further test scenarios.
To help me with this, can you maybe send me one typical script (.pfs file) you are running when you experience the memory increase.
User avatar
Havenstad FM
Posts: 104
Joined: 23 Apr 2016 12:36
Re: Memory problem

Post by Havenstad FM »

Hi Bernd,

In the attachment 2 *.pfs files that are used a lot during the nonstop hours.
Hope this helps,

Regards,
Jack
Attachments
edr_ns_format.7z
2 zipped format files
(1.16 KiB) Downloaded 241 times
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: Memory problem

Post by radio42 »

I am now testing since many days all possible scenarios with your kind of setup/scripts and all possible other options, esp. around UAC.
However, so far I was not able to reproduce any memory leak. On my 3 Test machines I could not detect any leakage and abnormal memory increase! All was normal and good.

So my assumption so far (also because no other customer has a similar behavior), that it must be some other component?!

Can you do me a favor and temp. disable UAC on your machine on which the Memory increases so quickly and check, if that stops the increase of memory?!
If it would still increase (even if UAC is disabled) this would be a sign, that we must continue searching elsewhere.
User avatar
Havenstad FM
Posts: 104
Joined: 23 Apr 2016 12:36
Re: Memory problem

Post by Havenstad FM »

Hi Bernd,

UAC is off, this does not change a thing. Or do you mean hacking the registry?
Screenshot from 2020-08-09 16-24-23.png
Regards,
Jack.
User avatar
radio42
Site Admin
Posts: 8295
Joined: 05 Apr 2012 16:26
Location: Hamburg, Germany
Contact:
Re: Memory problem

Post by radio42 »

No, I did mean the UAC option within ProppFrexx, which disables the internal user management.

Post Reply