Some problem caused by "escape_if_needed"

[CliffHan]

In this case, the function "escape_if_needed" in dom_walker.rs will change the text and made it unrenderable.

This html part contains some math formula, wrapped with $$.

<p>Mathematically, the model's outputs are judged according to <strong>cross-entropy loss</strong>,
    which measures the difference between the model's output probabilities and the real next token.
    One possible formula in the case of language modeling is
    $$ \mathcal{L} = -\log p_y $$
    where \( p_y \) is the probability given by the language model for the correct next token.[<a data-cite="xent"></a>]
    (This formula is greatly simplified from the general cross-entropy formula, but is still valid in the
    special case of language modeling.) So if the model gives a probability 1 to the actual next token, the loss is zero.
    Lower probabilities (if the model thinks some other token is likely to come next) lead to higher losses.
  </p>  

after htmd::convert(), I got

Mathematically, the model's outputs are judged according to **cross-entropy loss**, which measures the difference between the model's output probabilities and the real next token. One possible formula in the case of language modeling is $$ \\mathcal{L} = -\\log p\_y $$ where \\( p\_y \\) is the probability given by the language model for the correct next token.\[\] (This formula is greatly simplified from the general cross-entropy formula, but is still valid in the special case of language modeling.) So if the model gives a probability 1 to the actual next token, the loss is zero. Lower probabilities (if the model thinks some other token is likely to come next) lead to higher losses.

Obvious the "\" -> "\\" changed the formula, made it unable to render

Maybe some configurable options could be add for this case?

[letmutex]

Thanks for the feedback, I've left Rust for a while. Will check this once I get some free time, PRs are also welcome.

[bjones1]

CommonMark doesn't provide any official support for math. For example, $a*b + c*d$ renders as $ab + cd$ here! So, either some escaping or a better parser is needed in the general case. Pandoc has a math mode that handles this, but it's a bit messy: $a*b + c*d$ becomes [\\(a\*b + c\*d\\)]{.math .inline} when converted from markdown to HTML back to markdown. Ick.

Here's what renders when I copy and paste the output you provided above here, remove spaces before and after $$, and convert \( and \) to $ (which github understands).

Mathematically, the model's outputs are judged according to cross-entropy loss, which measures the difference between the model's output probabilities and the real next token. One possible formula in the case of language modeling is $$\mathcal{L} = -\log p_y$$ where $p_y$ is the probability given by the language model for the correct next token.[] (This formula is greatly simplified from the general cross-entropy formula, but is still valid in the special case of language modeling.) So if the model gives a probability 1 to the actual next token, the loss is zero. Lower probabilities (if the model thinks some other token is likely to come next) lead to higher losses.

So, it's a bit ugly, but the conversion is actually correct!