Repository navigation
[BUG] File name too long to be handled #380
Description
File name too long
When mpd plays a youtube video, all open ncmpcpp instances crash (and cannot be oppened again while that video is being streammed) as it is not able to handle a title so long as the one that's being fed into it.
Error:
terminate called after throwing an instance of 'boost::filesystem::filesystem_error'
what(): boost::filesystem::status: File name too long: "/home/marcosrdac/.cache/ncmpcpp/lyrics//videoplaybackexpire=1587969835&ei=yyqmXvuiMZaixgSFh7P4Cw&ip=2804%3A7f7%3Aa78a%3A2223%3Af021%3Ae3c%3A74ad%3Ac7ae&id=o-ADoKdC6bh8ChfPQfgsKOfTDbQpw1Ro_0w2GGNTtC0xt6&itag=251&source=youtube&requiressl=yes&mh=0T&mm=31%2C29&mn=sn-b8u-nn5s%2Csn-gpv7dn7k&ms=au%2Crdu&mv=m&mvi=3&pl=38&initcwndbps=480000&vprv=1&mime=audio%2Fwebm&gir=yes&clen=2759856&dur=158.txt"
zsh: abort (core dumped) ncmpcpp
I perceive it happens because the title in question is a very big url. Anyway, mpd can play it, so we got a problem in our best mpd player client hahaha
Steps to reproduce the behavior:
# add video stream to mpd playlist
youtube-dl --add-metadata --prefer-insecure -g "ytsearch1:rammstein" | mpc add
# play last song in playlist (the stream)
mpc play $(mpc playlist | wc -l)
Expected behavior
ncmpcpp run normally without crashing.
Desktop
- OS: Arch Linux x86_64
- Terminal: st (running zsh or bash)
- Version 0.21.11
Additional context
I believe this bug is the same reported at #55.
Activity
lenerd commentedon Jun 30, 2020
Hi @marcosrdac,
I can reproduce the crash when ncmpcpp is configured to download the lyrics in the background, e.g., with the following setting:
It looks like ncmpcpp tries to save the lyrics to file and uses the "song title" -- in this case a long part of the url -- as filename. This is too long for the filesystem (e.g., ext4 supports filenames up to 255 bytes).
So a workaround would be to disable automatic fetching of lyrics.
A possible fix would be to hash the song title to some smaller string and use that as filename.
marcosrdac commentedon Jul 29, 2020
Sorry for taking so much to answer you, @lenerd . I'm much appreciated about your workaround. I would not get to the conclusion that ncmpcpp's problem was at getting lyrics for the "song". Anyway, it's still a bug for us to keep one eye at.
m040601 commentedon Nov 21, 2020
UPDATE:
I'm now thinking, that my comments might not be "that" related to this issue. Please correct me or instruct me to move my comments to a new different issue, should this be the case.
I have in my config, the default:
# fetch_lyrics_for_current_song_in_background = "no"ORIGINAL:
Just chiming in to report
the same issuea very suspiciously similar issue and try to help pinpoint the problem.In my case the messages are:
terminate called after throwing an instance of 'MPD::ClientError'or
Response line too largeI also believe this is related to, #55
My system:
I strongly suspect it is related to the metadata containing something "too big" or "illegal" for ncmpcpp.
Other clients with mpd play the files/url without problem. Mpc can still play the file. It's just the user interface of ncmpcpp that doesnt load. Mpc add/update works solid and my mpd database is solid as a rock, no matter how many gigabytes of new music I throw at it.The reason of my suspicion is that I've been for the last years, adding huge amounts of music "scrapped" from youtube with youtube-dl.
Typically something like this:
With this, by default, I get the standard XYZ-youtubeid.m4a (aac codec) audio file.
The XYZ file name is derived from the name of the video was on youtube. It may contain whatever garbage people might put in there, lots of unicode emoticons, "dangerous" characters etc. But usually my ext4 linux file system copes well with that (puts single quotes around it) and I never actually bother to cleanup or rename the file names.
Than it fills the tags "Title" and "Artist". The "Title" tag has the same content as the "filename" except for the "-youtubeid". The "Artist" tag is very unreliable, most of the time ends up with the youtube user name. Not the "artist" name
Another thing that --add-metadata option from youtube-dl does is, to take the text written as description by the youtube user who uploaded the video. Thats the text that shows up in the video page.
This text is then put in both metadata tags
and
That's right, it fills the same content in both tags. I think these are called "itunes style tags" or "mp4 tags" or whatever. These are not ID3 tags.
I'm not sure if youtube-dl does any kind of sanitation, before writing th tags. So you can imagine that whatever garbage or size the original text contained, it will end up there.
I also notice that this crash happens more frequent with files containing lots of text with CJK (chinese japanese or korean) . Maybe because their size/encoding is bigger ?
I have "Comments" enabled in my mpd.conf. So the content of the "Comments" tag also ends up in the mpd database. With no problem.
It really is just ncmpcpp crashing.
My crashes usually happen when I press "2", go to "browser" and then press space to add a huge folder containing lots of subfolders and files. If it crashes, I then have to do, mpc clear.
And only then can I restart ncmpcpp. I then avoid adding that huge folder at once. Instead I navigate to its subfolders and add them one by one, to find out which one is the subfolder containing files that will trigger the crash.
Once I have ncmpcpp started I then can use the tag editor to edit the metadata zillions of files I have in my collection, no matter how big or long their tag contents might be. Without any crash or problem.
Will try to pinpoint exactly which ones and select some specific files triggering this crash, to bring it here.
m040601 commentedon Nov 28, 2020
UPDATE from my previous comment.
As a recap, the messages I'm seeing are either
terminate called after throwing an instance of 'MPD::ClientError'or
Response line too largeWhich I am 99% sure is also related with issue, #55
I have now noticed this happens not only with ncmpcpp, but also ncmpc and even web based clients like ympd and mympd. So, not sure if this is an issue with ncmpcpp or with the mpd database itself.
I have now started to identify files that trigger this behaviour.
I had previously removed their containing folders from the mpd music directory. I then went one by one, adding the file individually to the mpd music directory. Trying to pinpoint which file was triggering this behaviour
As I suspected such files have a huge amount of text in the metadata tag "Comment".
My music collection has ~5000 files.
I have this enabled on my mpd.conf,
metadata_to_use "+comment"Steps to reproduce:
Go to for example, https://www.youtube.com/watch?v=hmmpeehs_RE
Notice that this video has a "huge" description text.
Download, only the audio file, and the metadata.
youtube-dl -f m4a --add-metadata https://www.youtube.com/watch?v=hmmpeehs_RENotice that will get a file named "Masami ....".m4a. It's not a big file, only 2.2 MB. It's aac encoded in a m4a (mp4) container.
Examine ALL of this file's metadata audio tags. You can use any of:
You'll notice the two tags "Description" and "Comment" contain that huge amount of text.
Let's change the filename to "m.m4a" . It doesnt matter it still triggers the crash.
Remove it from the music directory and update the database with mpc update. Ncmpcpp starts again.
Let's change the metadata. For example:
exiftool -Comment="my short comment" m.m4aAdd it to the music directory and voila no crashes anymore.
Notice that the file metadata tag "Description" is still there containing a huge amount of text. But that doesnt matter, because neither mpd nor ncmpcpp make use of it.
Question : What is exactly this size limit ? What is and who defines the maximum amount of data that you can "stuff" in such a metadata tag like "Comment" ?
m040601 commentedon Nov 28, 2020
Another example, https://www.youtube.com/watch?v=63WIYqACGr4
This one besides containing a huge amount of text, it is also an example of files containing some ?problematic ?dangerous characters that end up in a metadata tag. In my case, it was the "Comment" tag. But who knows, it could be in the "Title", "Artist" etc , more important tags.
Notice where it says:
You dont see anything abnormal in the webpage displayed in the browser.
But, I noticed in vim that it contains this character, "<200e>", that also ends up in the metadata tag "Comment".
It's right here in this line
This as far as I noticed seems harmless.
Seems to be the Unicode mark, 'LEFT-TO-RIGHT MARK' (U+200E) - <200e>.
The left-to-right mark character is an invisible control character. If you open up a text file in VIM containing such a character, you will see it as <200e> in Unicode.
https://en.wikipedia.org/wiki/Left-to-right_mark
But, anyway, lesson learned, be ware of what junk characters might end up in metadata tags inside music files in your database.
RANT:
And, stretching this issue a little bit, the question, relevant for ncmpcpp and/or mpd. What kind of that "junk" might/would hypotetically ever cause trouble ?
And more paranoic thinking, what kind of vector attack could this be for my system ?
What if I add a music file to my collection containing "sudo rm -rf /" somewhere in the "Title" metadata tag?
What kind of input data "sanitation" could ncmpcpp/mpd do, if this could ever be considered a risk ? Or would this have to be handled by taglib, https://github.com/taglib/taglib ?
If I recall it correctly:
arybczak commentedon Dec 21, 2020
youtube-dl --add-metadata --prefer-insecure -g "ytsearch1:rammstein"doesn't return anything here.Any other way to reproduce?
JaanVr commentedon Jan 8, 2021
@arybczak
This video and its metadata causes similar issues on my setup.
Interestingly it causes a similar issue in mpc...
marcosrdac commentedon Jan 8, 2021
This youtube-dl command searches youtube for the word rammstein and returns the first video as two urls: one for video and another for audio. Here you can see it:
Fix crash when background-fetching lyrics for streams
Fix crash when background-fetching lyrics for streams
Fix crash when background-fetching lyrics for streams (#561)
arybczak commentedon Jul 24, 2024
Should be fixed by #561.