It seems like the strange characters are gone from the website I provided. But it still may cause crash if the website has the characters that the specific content-type cannot handle

@congsun can you show me how you were attempting to create the document that was causing the crash? The initialString shouldn't be nil by the time it gets to HTMLParser, so I've left that crash in to loudly point out where we're doing something unexpected.

Also, the ASCII string encoding expects clean 7-bit ASCII, and decoding will fail if any byte has the high bit set, so I'm not sure the solution you suggest here is generally applicable.

Hi. I was just use the regular way to create the document and somehow the app just crashed totally. The website that cause a crash was the one I provided before, and somehow they fixed it. However, before they fix that problem and when I try to open the webpage with chrome, there are several question mark inside a triangle shows up. My solution is not perfect, I just want you to notice this annoying problem and maybe we can fix it :)

Oh, I was misreading your change, my apologies! I agree we shouldn't be crashing there.

My best guess is that the Content-Type header specified a known charset, but the data was not actually encoded using that charset. We should make a unit test for this, then decide how to fix it.

I don't think falling back to ASCII makes sense as that'll refuse to decode if any byte has its high bit set (NSASCIIStringEncoding is strictly 7-bit ASCII). If you look at the DeterminedStringEncodingForData() function in HTMLEncoding.m, we try to fall back to win1252 (which is the fallback in the spec), and finally we fall back to iso8859-1 as our Encoding of Last Resort (any set of bytes decodes as iso8859-1).

I'm having a hard time finding any suggestions in the spec about how to deal with a failed decoding. I'll check again to see if it has a suggestion, otherwise we should just do whatever browsers do.

Ultimately you've pointed out that an assumption we make in HTMLReader – that the Content-Type's charset will always work – is hilariously flawed.

@congsun I decided to try treating an incorrect Content-Type's charset as if it didn't specify any charset at all. I just pushed my efforts to the master branch, in commit 5056ddd. What do you think?

Great, let me look at that 👍

@nolanw Hi, I checked out your code, it seem like it would still crash when the content-type is still correct, say UTF8, but there are several unidentifiable characters in the content of that data. I don't know if my understanding is wrong, but it seems like we would need a higher error-tolerant encoding way to deal with that if the encoding string returns nil.

I will try to find a web page that has those tricky characters :)

I think the unit test I added in that commit covers that particular case? If there are unidentifiable characters in the content then I'd say the Content-Type wasn't correct.

I guess we do still crash if the data starts with a BOM but isn't actually encoded as the BOM suggests. Not sure what to do there. It's the same problem really: being "certain" about the wrong encoding.

I agree, nice work!!!

Cool, I'll make a new release in a day or two. I wanna think more about tricky BOMs.

Yep, I'm happy with this.

HTMLReader 0.8.1 includes this fix as commit 5056ddd. Thanks again @congsun!