No apostrophes allowed!

[cyisfor]

writeln(createDocument!(DOMCreateOptions.None)("It's").root.html);
=>
I'ts

:/

Instead of encoding text as entities when stored in the document, it also escapes those entities again when printing the document out. writeHTMLEscaped doesn't have any support for not doing that, so there's no way to stop characters from getting encoded as numeric entities. You have to escape the apostrophe if it's inside an attribute value, but there's no accounting for that. Plus there's smart quotes, and innumerable utf-8 characters that make perfectly valid HTML, but are forced into ugly numeric escapes instead, with this strategy.

It's confusing terminology too. "decode entities" seems to mean turning " " into " ". I don't think anyone would want to do that by default, and that's actually _trans_coding named entities into numeric entities, not decoding anything. "decoding entities" should at least mean turning the utf-8 encoded " " into " " Strictly speaking, that's transcoding too, since " " is still just a sequence of undecoded bytes, not a complex data structure. Decoding is where you take " " or " " and turn it into a call to onEntity() or something. It looks like the code tries to do that, but... doesn't really do it right.

I like to adopt the strategy of "never decode, until you are ready to display." Which is fancy way of writing "never decode." I only decode stuff when I need meaningful data inside it, otherwise I run the risk of double decoding it, or being unable to specify what encoding it forces everything into.

Really, entities are entirely separate from the structure of the HTML document. Entities aren't separate nodes in the DOM for any major web browser, they're just more bytes within the text node. There are also two entirely separate categories of entity that get conflated together: 7-byte characters that would mess up the HTML if unescaped, and 8-byte characters. I'll want to escape some HTML by replacing < and >, and maybe " with entities, but no other ones. Conversely, if someone wants to turn their HTML into proper english schoolteacher HTML, they will want all codepoints escaped, but leave < and > intact.

Here's how I think it should work:

writeln(createDocument("It’s").root.html);
=>
It’s

writeln(escapeEntities(createDocument("It’s <e/>").root.html));
=>
It&apos;ts <e/>

writeln(escapeEntities!(named: false)(createDocument("It’s <e/>").root.html));
=>
It&#2019;ts <e/>

writeln(escapeHTML(createDocument("It’s "<e/>"").root.html));
=>
It’s "<e/&gt"

writeln(escapeAttribute("this attribute has a \" in it, as well as > and <"));
=>
this attribute has a " in it, as well as > and <

node.html("<p>");
writeln(node.html);

=>
<p>

node.html(escapeHTML("<p>"));
writeln(node.html);

=>
<p>

I'm honestly starting to think that a HTML parser should not deal with entities at all. They have to be dealt with using a separate character-by-character parser since they're on the level of characters, and not part of the document structure.

[marcioapm]

Yes, I see the problem here...

The DecodeEntities flag is for the parser, not for the DOM. It is very valuable to have the parser decode, because if you are simply using the parser you don't want to have to care about decoding yourself. e.g. You just want to convert HTML into plain text.

What's happening is the parser feeding everything decoded to the DOM. The DOM then encodes it again before output. The encoder only encodes as numeric, as that is the fastest, and it's encoding anything that is not ASCII printable. Perhaps this is too strict.

I think encoding/decoding/escaping should happen automatically, and internally everything should be stored unencoded/unescaped, so that you can manipulate everything as normal text. The document structure is already implicit in the document tree.
What I mean by this is that you want to be able to look for a node, and grab it's text and display it in the console or manipulate it, and it should just work. i.e. doc.root.text.replaceAll("\'", "moo") should always work, independent of how it looks in the source HTML.

When valid HTML is needed as output, which is not always the case, then we re-encode everything back into valid HTML and output it. If all you care is the structure, then you can also create the DOM without the DecodeEntities flag. At this point, the DOM won't try to re-encode anything when outputting (or at least it shouldn't).

I think at the moment this is not very clear, and perhaps is not working 100% as I describe, but that's where I would like to go.

What do you think?

P.S. Better (any!) documentation is also in order, but I have no bandwidth for this project at the moment :(

[cyisfor]

I think that "automatically" is a dangerous prospect, because it limits what people can do with encoding, especially if there is no lower level option that doesn't do all that magic transcoding. It also increases the complexity of the parser (has to handle entities, utf-8, coding state, etc). I still maintain that a HTML parser has no place in parsing HTML entities. Entities were created specifically for the purpose of having characters that will not be parsed, so parsing them sort of defeats the purpose of their existence. But on the level of the DOM, you can wrap the accessors in escape/unescape to get at least most people's use-cases for implicit entity escaping.

What you can do is have no entity encoding or decoding at all internally, but the createTextNode() function transparently calls escapeEntities when passed text, with no option to selectively escape <> or & and no option to avoid escaping anything at all, nor to use existing entities rather than double escaping them. When text() is invoked as a getter, it can call unescapeEntities on the result, before returning it. With, again, no option to avoid unescaping entities. The lack of these options will pretty much force the user to use utf-8, and will not allow them to embed raw < characters inside their text, which of course could never possibly have any valid application. And it would prevent them from using entities, with all entity usage completely invisible and out of their hands.

I suppose html()/setter could also guard escaping of entities, except not < and >. You'd also have to instrument attr() to escape/unescape all & characters and all quote and double quote characters.

I am pretty lukewarm to the idea of trying to guess when the user "really" wants escaped text or not. You could add an option to the DOM object itself, to either retain multibyte characters, or escape them when setting text() or html().

Honestly it would just be easier to say "escape your text with escapeEntities before inserting it, or we'll point and laugh."