Image i20230620 + Update 20230630: Local database update very slow

[klassikmann]

An old problem appeared again, see #966.
An "Update database" of a single album or a small sub-tree lasts for a very long time. Observing the update I see short calls to mpd, mpc and awk, then a cmd-list.sh using most of the time, and after that the search for .mpdignore files.

[hjfisch]

No problem for me .
Rebuild database ( 60000 songs) takes about 15 minutes . Adding an album 1-2 minutes .
I've connected a 4tb SSD with USB to a RPI 3b+ .
B.t.w. I boot from USB-Stick .
Till now no probs with latest Build .

[klassikmann]

@hjfisch

Adding an album 1-2 minutes

That's much longer than before: 10 to 15 seconds are usual for adding an album.

[klassikmann]

After your work on camilladsp has been brought to a conclusion, (at least for the time being) I would like to recall the problem of very slow database updates.

[rern]

@klassikmann,

All right.

rAudio database update consists of:

• MPD mpc rescan (or update) - Default database
• rAudio cmd-list.sh - Database for UI

Let's start with time cmd-list.sh to see how long it takes for rAudio part.

rern

[klassikmann]

rAudio:/srv/http/bash  #  time cmd-list.sh
Wrong scontrol identifier:

real    0m57.204s
user    0m14.900s
sys     0m44.093s

[klassikmann]

@rern
About 90% of the elapsed time are spent in the "album^^artist^^date^^file" handling from line 91 to 105 in cmd-list.sh.

[rern]

Try sed -i '/albumignore/ s/^/#/' $dirbash/cmd-list.sh then time cmd-list.sh again.

[klassikmann]

@rern

Try sed -i '/albumignore/ s/^/#/' $dirbash/cmd-list.sh then time cmd-list.sh again.

No change.

Addition:
FWIW
My $dirmpd/albumignore doesn't exist, and $dirmpd/mpdignorelist contains one entry which points to an empty file.

[ceejayemm]

@klassikmann

FWIW - my $dirmpd/albumignore also does not exist, however, like you $dirmpd/mpdignorelist DOES exist but in my case contains valid entries pointing to hidden files (.mpdignore) in the directories where I want files ignored eg:

/mnt/MPD/NAS/NAS_Music/Pink Floyd/Wish You Were Here/.mpdignore

The .mpdignore files then contain additional names of the files/folders I want excluded relative to the folder containing the .mpdignore file eg:

Backups
Wish You Were Here - iTunes LP (v2..itlp

This was created some while ago via Library / NAS and then selecting the Exclude (File /) Directory menu option (as was) which originally allowed you to exclude both files and directories but currently only allows directories (?) to be excluded. You can also list the excluded files/directories via Player / Lists / Ignored Directories.

Chris

[klassikmann]

@ceejayemm
Thank you for your explanation. My comment

... $dirmpd/mpdignorelist contains one entry which points to an empty file ...

describes an intended configuration. The single and empty .mpdignore in the root of the music directory is a leftover from tests long ago.

@rern
Could it be that cmd-list.sh is treating all albums in the music directory regardless a full update or a local update (with only a few albums) has to be performed.
Observation: In cmd_list.sh lines 49 til 52 you find "mpc listall" without a location parameter:

albumList() {
	mpclistall=$( mpc -f '%album%^^[%albumartist%|%artist%]^^%date%^^%file%' listall 2> /dev/null )        # include no album tag
	[[ $mpclistall ]] && albumlist=$( awk -F'/[^/]*$' 'NF && !/^\^/ {print $1|"sort -u"}'<<< $mpclistall ) # exclude no album tag, strip filename, sort unique
}

EDIT:
Earlier releases (e.g. rAudio-20220808 just after my report #966) contain an extra parameter "albums=$1" in "listAlbums".

[klassikmann]

rAudio-20230828: Same behaviour (no surprise: cmd-list.sh is unchanged).

[klassikmann]

@rern
20230916, I see you prepare an update release: Some news about the slow local database update problem and my observations described in #1377 (comment) ?