Is there a way to speed up imms.db and mms-audio-library?

Moderator: Moderator

User avatar
Uatschitchun
Overlord
Posts: 3189
Joined: Tue Dec 06, 2005 6:55 pm
Location: Germany
Contact:

Is there a way to speed up imms.db and mms-audio-library?

Postby Uatschitchun » Tue May 22, 2007 6:23 pm

Hi there ...

I've got one question that matters for 1.0.8.x and 1.1.0 so I posted in general.

Cause of MMS being not able to update tags in imms.db, I rebuilt the db on my livingroom-pc ... so far so good (or not ;)

It took nearly 2 days to fill it up ... and now it's 36 MB ...

Even after rebuilding the db it takes 108min for mms-audio-library to rescan all files (just a rescan, no new files) ...

At a certain point (at filling up) mms-audio-library needs over 1sec per file to add it to the db.

Ok, I must admit, that all this is done over ethernet (100MBit) cause the data is kept by my server.

What do I want to say with that!?

We have two possibilities:
1.) Make imms.db (or audio.db for 1.1.0) updateable, so that one has not to recreate the hole db for just updating to new tags/tracks

2.) Speed up the db creation/filling so that it is just faster ;)

Don't know what it will be in 1.1.0 ... but if it stays like this ... :evil:

What are your impressions?
Is it possible to speed up things?
Is the new db in 1.1.0 able to be updated?
Lg
Roman

User avatar
acmelabs
Overlord
Posts: 2365
Joined: Mon Feb 20, 2006 9:18 pm
Location: Germany
Contact:

Re: Is there a way to speed up imms.db and mms-audio-library

Postby acmelabs » Tue May 22, 2007 9:21 pm

Uatschitchun wrote:Hi there ...
What are your impressions?

Well - I saw you chatting with Anders about this, so could be things at this point in time are sorted for you already, but let me say this:
The db speed in 1.1.0 for audio is breathtaking - but I'm afraid - there ain't an update machnism either (not yet ;-) )

Regards,
Andreas

User avatar
arj
Site Admin
Posts: 2316
Joined: Thu Dec 01, 2005 8:51 pm
Location: Denmark
Contact:

Postby arj » Tue May 22, 2007 9:38 pm

When running mms-audio-library it will try harder than normal to guess the audio properties. This should result in better metadata (like length of track) but it will also take longer time.

The following is the code (line 181 in songinfo.cc):

Code: Select all

#ifndef STANDALONE
     : fileref(filename.c_str(), AudioProperties::Average) {}
#else
     : fileref(filename.c_str(), AudioProperties::Accurate) {}
#endif


You can change Accurate to Average for faster performance. I don't know how much it will be though (if you do please report how much it helps. The idea has always been that mms-audio-library was run in the background / one night so it didn't matter if it was slow. But 2 days is too slow. But I'm pretty sure the network is slowing this down a lot. There's no pipelining, it just one track at a time.

The problem with making it updateable is that you need to parse all the files anyway to know what their metadata are, so why not just it instead of checking everything. It won't be faster to update, it might even be slower, since the slow part is extracting the metadata information.

That said, it should only be needed to remove the old db if you change the metadata on a file. If you just add new files you can just append to the db. So it's a rather rare case IMO.

It will be the same in 1.1.0.

User avatar
arj
Site Admin
Posts: 2316
Joined: Thu Dec 01, 2005 8:51 pm
Location: Denmark
Contact:

Postby arj » Tue May 22, 2007 10:28 pm

I just did a test here on my collection. 88GB, 11283 songs,

real 31m52.281s
user 16m18.772s
sys 3m51.501s

6.0M /home/arj/.mms/imms.db

So it seems it is a problem with the network. I remember when I was at uni we had our homedirs running from an NFS server and compiling was at least 10 times slower than just moving the source to /tmp and recompiling it from there. One just had to remember to copy it back when one was done playing with it :-)

User avatar
Uatschitchun
Overlord
Posts: 3189
Joined: Tue Dec 06, 2005 6:55 pm
Location: Germany
Contact:

Postby Uatschitchun » Wed May 23, 2007 8:18 am

arj wrote:When running mms-audio-library it will try harder than normal to guess the audio properties.

What is "normal" then? The way MMS itself extracts the metadata?

This should result in better metadata (like length of track) but it will also take longer time.

There's nothing wrong with good metadata ;)
And the problem seems to not be the time for extracting the data ...
The first hundreds of tracks are read very fast, but the longer the extraction runs, the bigger imms.db gets, the slower the extraction !? Like if imms.db gets too big for the tool to know (in an acceptable time) where to store the data into the file and if a track is already there ...
Could it speed up things if the db is split up into smaller files? Gues not, cause the amount of data stays the same ;(
What about indexing for faster recognition of: file exists in db already => yes/no

The following is the code (line 181 in songinfo.cc):
...
You can change Accurate to Average for faster performance.

How do I do this? Comment out the "Accurate" line? Or does this line changes from "Accurate" to "Average"?:

Code: Select all

-CXXFLAGS= -I/usr/local/include -I/usr/include/ -I../ -I/usr/include/taglib -pipe -O0 -g -DDEBUG -DSTANDALONE
+CXXFLAGS= -I/usr/local/include -I/usr/include/ -I../ -I/usr/include/taglib -pipe -O0 -g -DDEBUG


if you do please report how much it helps

Sure :wink:

The idea has always been that mms-audio-library was run in the background

There's nothing wrong with that idea ;)
Is it possible (if "Average" is a lot faster than "Accurate") to just scan with "Average" and do "Accurate" scanning later? But that will require the ability to update ... :?

There's no pipelining, it just one track at a time.

If it would pipeline, the cpu-usage of mms-audio-library would be a lot higher then ... and it really is high ...

it should only be needed to remove the old db if you change the metadata on a file. If you just add new files you can just append to the db. So it's a rather rare case IMO.

For me it's a rather usual procedure to listen to music that is not correctly tagged ... and tag the files correctly if I have time ...
So a solution would be to move newly tagged files, so MMS recognizes them as 'new' and appends them!? But isn't a tool designed to help and make things easier? With moving files I have to help the db cause updating is done outside :roll:

What I will test is the creation of imms.db on the server ...
It uses the same mountpoints as the client, so the db should be exchangeable.

@acmelabs:
The db speed in 1.1.0 for audio is breathtaking

For sure!
Entering a new album is a hell lot faster in 1.1.0 ... don't know why :roll:
Lg

Roman

esprit
master
Posts: 503
Joined: Tue Dec 06, 2005 5:50 pm
Location: France
Contact:

Postby esprit » Wed May 23, 2007 8:50 am

Isn't it possible to store the last modification time of indexed files in the base ?

So you don't have to load every file and parse metadata, only those which are not indexed or have been changed ?

User avatar
arj
Site Admin
Posts: 2316
Joined: Thu Dec 01, 2005 8:51 pm
Location: Denmark
Contact:

Postby arj » Wed May 23, 2007 1:25 pm

esprit wrote:Isn't it possible to store the last modification time of indexed files in the base ?

So you don't have to load every file and parse metadata, only those which are not indexed or have been changed ?


Yes, that's an excellent idea. Thanks!

User avatar
Uatschitchun
Overlord
Posts: 3189
Joined: Tue Dec 06, 2005 6:55 pm
Location: Germany
Contact:

Postby Uatschitchun » Wed May 23, 2007 3:06 pm

Cool :P
Lg

Roman

esprit
master
Posts: 503
Joined: Tue Dec 06, 2005 5:50 pm
Location: France
Contact:

Postby esprit » Wed May 23, 2007 5:22 pm

great ! :)

Another feature could be a "purge" function, wich remove old information about files that are not there anymore. Could (should) be a command-line option, so database is purged only when wanted, not automatically, cause one could have media on a removable disk and doesn't want to loose information about it.
Just an idea :idea:

User avatar
Uatschitchun
Overlord
Posts: 3189
Joined: Tue Dec 06, 2005 6:55 pm
Location: Germany
Contact:

Postby Uatschitchun » Thu May 24, 2007 6:56 am

esprit wrote:Another feature could be a "purge" function, wich remove old information about files that are not there anymore.

Yeah ... housekeeping :P
Lg

Roman

User avatar
Uatschitchun
Overlord
Posts: 3189
Joined: Tue Dec 06, 2005 6:55 pm
Location: Germany
Contact:

Postby Uatschitchun » Thu May 24, 2007 1:21 pm

Btw ... after running 'mms-audio-library' a good time now (15h) locally (to check performance against networked), I noticed that one in top:

2705 uatschit 25 0 909m 2624 403m R 15.2 31.9 777:15.53 mms-audio-libra


ps aux | grep mms
1000 2705 58.1 31.7 929716 411516 pts/13 R+ May23 777:41 mms-audio-library /media/mp3/


Pheah ... isn't that quite a lot of memory it uses!? :roll:
Lg

Roman

esprit
master
Posts: 503
Joined: Tue Dec 06, 2005 5:50 pm
Location: France
Contact:

Postby esprit » Thu May 24, 2007 1:35 pm

Isn't it only fs cache data ?

free will tell you more about (global) real memory usage

User avatar
Uatschitchun
Overlord
Posts: 3189
Joined: Tue Dec 06, 2005 6:55 pm
Location: Germany
Contact:

Postby Uatschitchun » Thu May 24, 2007 4:04 pm

Is it?

ps aux's first line reads:
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

with:
rss RSS resident set size, the non-swapped physical memory that a task has used (in kiloBytes).

and
vsz VSZ virtual memory size of the process in KiB (1024-byte units). Device mappings are currently excluded; this is subject to change. (alias vsize).
Lg

Roman

esprit
master
Posts: 503
Joined: Tue Dec 06, 2005 5:50 pm
Location: France
Contact:

Postby esprit » Thu May 24, 2007 5:12 pm

Yeah you're right.

You can try gexmap (which need a kernel module : exmap) it will give you great information on what part (lib, code or data) of the process is so hungry.

User avatar
Uatschitchun
Overlord
Posts: 3189
Joined: Tue Dec 06, 2005 6:55 pm
Location: Germany
Contact:

Postby Uatschitchun » Thu May 24, 2007 6:15 pm

Ok, running ... and what exactly is of interest?
There seems to be no possibility to dump the output somehow ... have found 'exmap-console' on the net, but it has a daemon, server, client ... puh ;)

Will see, have mms-audio-library running again and wait a while to give room to collect momory ...
Lg

Roman


Return to “feature requests”

Who is online

Users browsing this forum: No registered users and 1 guest