MLS Performance

[Luka C]

Dear,

let's imagine an environment with eight computers. Computer A is a file server running Windows Server 2003 32-bit - it servers all of a radio station's audio content. B is a newer Windows 7 machine intended for the broadcast studio play out. It runs ProppFrexx 24/7 alongside ProppFrexx MediaLibraryServer (64-bit) to feed 12 libraries to itself and computers C - G. Those machines are clients to the MLS meant for content / playlist preparation. All in all, each machines connect to B for track/library information and B for the actual audio files over Windows Shares on the network (one router + "passive" switches). The MLS servers two kinds of libraries: database libraries connect to MySQL views on A to read the legacy automation system's tables (soon to be discontinued) and playlist libraries saved in a shared folder on A. MLS is configured to automatically reload each two hours at 00:00.

When comparing opening a library (main ribbon - Open - Media Library) containing 54.000 entries in 4 different scenarios, the results are:
- opening a DSN library served over MLS takes 2 minutes and 10 seconds,
- opening the library over a local DSN takes 34 seconds,
- opening a playlist library served over MLS takes 1 minute and 47 seconds (MLS access longs iterates over the library for 96 seconds, but actual opening time is longer)
- opening a playlist library added locally but saved on A takes 6 seconds

When searching, getting results:
- from MLS during the automated reload of libraries (per example, between 14:00:00 and 14:07:00) is impossible - MLS retrieves the first couple of entries of a media library regardless of the search query; per example, if Lib1's first three tracks are Track01.mp3, Track02.wav and Track03.flac, those will be displayed as search results for queries "Track", "02" or "Not Matching At All" at 14:03 when the automated MLS rescan takes places
- from MLS-served DSN library takes much longer than from a local DSN connection (Search Also in Filenames disabled)
- from MLS-served PFP library takes much longer than from a locally added PFP library (Search Also in Filenames disabled)

Why are we opening a media library/playlist with 54.000 tracks inside? Our Editorial Staff sometimes wishes to see which are the newest additions to our archives. Respectfully, those are the tracks at the bottom of the library.

Why did we implement MLS and not loading all of the libraries from each client machine manually? It is important for us that any changes or additions to our libraries are "instantly" visible at all workstation. Per example, when a journalist is finished editing a news report, he adds it to the library so it is available for play out. If all workstation would reload libs by themselves, they would never reflect changes at the same time.

How can we optimize MLS?
Do you need additional information to troubleshoot this?

And an off-topic question - you might decide answering in a separate thread: how to synchronize new additions to libraries and changes to the schedulers as soon as possible? I read that such behavior can be done with the RCM/TCP server connections.

Thank you.
Kind regards,
Luka C.

[radio42]

First I would notice, that your setup would be far from optimal resp. recommended.
You might be aware, that real-time audio processing incl. the media library management when accessing database or folder based media libs (especially from many other PCs in parallel) is a heavy I/O task.

So what is happening on your side:

1) your actual audio files are located your (Windows Server 2003 - 32-bit).
I don't know what the I/O sub-system is on that machine, but it should be very very fast, when 8 other machines might access that in parallel (evern if through a single MLS
If that machine uses slow devices (e.g. older SATA II disks or disk with only 5400RPM)...this is a bottleneck.

2) Every request from/to the MLS needs to go through your network via Windows Shares.
This is another bottleneck in your I/O sub-system. As accessing audio files through shared drives is much slower than accessing them directly.
E.g. local storage using SATA III can use a bandwidth of 6 GBit/sec.!
A network share is limited to the current bandwidth available in your Local Area Network.
If this is not a very fast (at least 1 GigaBit network), at best dedicated for those servers, you might get some delays.

3) DNS connection to a MySQL database
Reloading/Rescanning a database entirely is also slow, but that is also due to the speed of the database. During that reload access to that database is limited - mostly due to the concurrency settings of the database. But that is just another 'speed' limit while a reload.
However, in addition, any database access also utilizes the network, limiting your network bandwith further (see 2).

To optimize things, I would recommend:
a) place your MLS directly onto the server where the audio content is located, so that the MLS accesses them locally
b) make sure your I/O sub-system is very fast on that machine!! e.g. use SATA III, SSD disks if possible.
c) use a Win7 or Win8 64-bit operating system! The MLS is a 64-bit process and will leverage the extra memory available.
d) make sure all other PCs are connected via a very fast LAN (at least a Gigabit network with enough bandwidth available at any time; good switches might help as well)

When you are planning to connect 8 PCs to the MLS - that is a heavy I/O thing, requiring a heavy machine; which is obvious, the I/O sub-system on that MLS needs to be fast enough to serve all parallel I/O requests! This can not be achieved with commodity (cheap hardware).

[Luka C]

Thank you for your thoughtful response.
The network is comprised of Mikrotik/Cisco hardware. All machines access it via 1Gbps network cards.
Machine A is an older HP server, filled with 15000 rpm drives. We are planning to upgrade as soon as funds will be available. Referencing my local test, I can agree running MLS on the machines that audio is stored increases speeds - the current A however does not have more than 4GB RAM so a 32-bit OS is not installed.
The database server is optimize for large result sets and it's configuration limits are set high. Note that results noted in my original post were measured while only two connections were established to MySQL and only one to MLS.

Putting the DB aside, I am still in hope for a temporary solution for this results:
- opening a playlist library served over MLS takes 1 minute and 47 seconds (MLS access longs iterates over the library for 96 seconds, but actual opening time is longer)
- opening a playlist library added locally but saved on A takes 6 seconds

We might temporarily revert to manually adding libs to each instance. I seem to have forgotten the flag for manually setting the configuration path when launching the program. Can you post it again, please?

Would you be so kind to also answer how to synchronize new additions to libraries and changes to the schedulers as soon as possible with TCP server (Master/Slave)?

[radio42]

I am afraid, that there is not much to optimize...(if you are really using the latest MLS versions).
54.000 media entries needs to be send from the MLS to the client PC...that might be a couple of MB to send/receive...plus some validation.
Depending on your MLS settings, even all TAGs might need to be read.
What are your settings on the MLS ?
However, why don't you simply open those playlists directly?

Do you mean the "-config" command-line option?
I can also not remember what I once have send you...

[radio42]

And what are your 'Meta Data/WaveForm' settings on the local PFOA instance?
(maybe the TAG reading is forced and takes that long; a quick test here with 60.000 entries loaded a Remote-Lib in a few seconds).

And, are the entries accessible from your local PC with the same UNC path?
If not the physical files even needs to be transferred...you can check this, when going to the general settings, section 'Folders/Libraries' by clicking on the 'Remote Library Settings...' button.
When the 'Allow Remote Data Access' is enabled, check, if there are actually any files present in the specified 'Temp. Directory'.
If yes, your PC can not access the files directly and they need to be transmitted in addition; which would explain why access is so slow!

[radio42]

I made some further tests here...it must be something with your network or other settings as explained above (e.g. no access to the UNC paths)...
Here opening an entire remote lib (in a new playlist window) from a (slow) USB 2.0 device containing 100.000 entries took some like 3-4 seconds...