fix(markdown): improve CommonMark conformance and parser structure

[jfmcdowell]

Note

AI Assistance Disclosure: This PR was developed with assistance from Claude Code.

Summary

This PR significantly improves the markdown parser's CommonMark spec conformance and reorganizes the parser module structure for better maintainability.

Conformance improvements:

• Fixed link/reference parsing edge cases (nested brackets, escaped characters, parenthesis depth limits)
• Enhanced emphasis parsing with proper delimiter stack algorithm and flanking rules
• Improved block structure handling (setext headings, list interruption rules, lazy continuation)
• Added Unicode case folding for link reference matching (CommonMark requirement)
• Fixed HTML block detection and inline HTML parsing

Structural improvements:

• Split monolithic inline.rs (2600+ lines) into focused submodules:
  • inline/code_span.rs — backtick-delimited code
  • inline/emphasis.rs — bold/italic with delimiter stack
  • inline/links.rs — links and images
  • inline/html.rs — autolinks and inline HTML
  • inline/entities.rs — character references
• Adopted syntax/mod.rs convention (consistent with HTML/CSS parsers)
• Extracted helpers to simplify the main parsing loop (handle_inline_newline, classify_quote_break_after_newline)

Dependency:
CommonMark requires full Unicode case folding for reference label matching (e.g., ß → ss, ẞ → ss).
caseless (v0.2.2) is:

• Pure Rust, WASM-compatible
• Provides full case folding via default_case_fold_str()
• MIT licensed, maintained by unicode-rs (same org as unicode-normalization)

Test Plan

- just test-crate biome_markdown_parser
- just test-markdown-conformance
- just f && just l

• Snapshot tests updated for improved parsing

#3718

[changeset-bot]

⚠️ No Changeset found

Latest commit: e63d4f8

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Walkthrough

The PR restructures the Markdown parser: it splits and reintroduces inline parsing into modular submodules (code spans, emphasis, links, HTML, entities), adds lexer contexts CodeSpan and EmphasisInline, and implements a CommonMark-style delimiter/emphasis engine. HTML block detection/termination logic was rewritten with explicit kinds. List, header and quote parsing gained virtual-line/indent handling and emphasis scoping. The HTML renderer was replaced by a visitor-buffered HtmlRenderer. Utilities added: reference-label normalization, Unicode punctuation table, and case-folding support; small Cargo deps updated.

Possibly related PRs

• biomejs/biome PR 8525: Large, overlapping refactor touching the same lexer, parser and inline parsing modules (lexer/mod.rs, parser.rs, syntax/inline.rs, fenced_code_block.rs, header.rs, list.rs, quote.rs).
• biomejs/biome PR 8857: Related work moving CommonMark conformance tests and test-suite handling (coverage/xtask integration).

Suggested reviewers

• dyc3
• ematipico

🚥 Pre-merge checks | ✅ 2

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately summarises the main objective: improving CommonMark conformance and restructuring the parser, which aligns with the substantial changes across the codebase.
Description check	✅ Passed	The description comprehensively covers the conformance improvements, structural refactoring, dependency additions, and test plan, clearly relating to the extensive changeset across parser modules and dependencies.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (3)

crates/biome_markdown_parser/src/syntax/list.rs (2)

307-440: Indent-aware blank-line lookahead is still missing for nested lists.
skip_blank_lines_between_items uses a non-indent-aware predicate, so nested lists with blank lines can terminate early when the next marker is indented >3 spaces. Consider threading the marker indent into the predicate (and using the new indent-aware helper).

🔧 Suggested fix

-fn parse_list_element_common<M, FMarker, FParse>(
+fn parse_list_element_common<M, FMarker, FParse, FHas>(
     p: &mut MarkdownParser,
     marker_state: &mut Option<M>,
     current_marker: FMarker,
     parse_item: FParse,
-    has_item_after_blank_lines: fn(&mut MarkdownParser) -> bool,
+    has_item_after_blank_lines: FHas,
     is_tight: &mut bool,
     last_item_ends_with_blank: &mut bool,
 ) -> ParsedSyntax
 where
     FMarker: Fn(&mut MarkdownParser) -> Option<M>,
     FParse: Fn(&mut MarkdownParser) -> (ParsedSyntax, ListItemBlankInfo),
+    FHas: Fn(&mut MarkdownParser) -> bool,
 {

-        parse_list_element_common(
+        parse_list_element_common(
             p,
             &mut self.marker_kind,
             current_bullet_marker,
             parse_bullet,
-            has_bullet_item_after_blank_lines,
+            |p| has_bullet_item_after_blank_lines_at_indent(p, self.marker_indent),
             &mut self.is_tight,
             &mut self.last_item_ends_with_blank,
         )

---

1478-1683: Guard first-line block parsing when the item should be an indented code block.
When spaces_after_marker >= 4, CommonMark treats the line as an indented code block; the current first-line checks can still parse ATX headings/quotes/thematic breaks. Skipping those checks in that case prevents misclassification (e.g., - # heading should be code, not a heading).

🛠️ Suggested guard

-        if first_line {
+        if first_line && spaces_after_marker <= INDENT_CODE_BLOCK_SPACES {
             enum NestedListMarker {
                 Bullet,
                 Ordered,
             }
             ...
         }

crates/biome_markdown_parser/src/syntax/fenced_code_block.rs (1)

266-339: Closing fences inside block quotes won’t be recognised.
Quote prefixes are stripped before at_closing_fence, but line_has_closing_fence insists the prefix is whitespace‑only. That means > markers make the check fail, so quoted fences never close and will emit unterminated diagnostics. Please allow block‑quote markers in the prefix or skip them in line_has_closing_fence.

💡 One way to adjust `line_has_closing_fence` (below this block)

 fn line_has_closing_fence(p: &MarkdownParser, is_tilde_fence: bool, fence_len: usize) -> bool {
     let Some((start, source)) = get_source_context(p) else {
         return false;
     };

     let line_start = find_line_start(&source[..start]);

-    if !is_whitespace_prefix(source, start, line_start) {
+    let quote_depth = p.state().block_quote_depth;
+    if quote_depth == 0 && !is_whitespace_prefix(source, start, line_start) {
         return false;
     }

     let list_indent = p.state().list_item_required_indent;

     // Skip required list indent (must have enough whitespace)
     let Some(idx) = consume_indent(source.as_bytes(), line_start, list_indent, true) else {
         return false;
     };

     // Skip optional extra indent (up to 3 spaces per CommonMark)
     // This always succeeds since required=false
-    let idx = consume_indent(source.as_bytes(), idx, 3, false).unwrap();
+    let mut idx = consume_indent(source.as_bytes(), idx, 3, false).unwrap();

+    // Skip block-quote markers
+    let bytes = source.as_bytes();
+    for _ in 0..quote_depth {
+        if bytes.get(idx) != Some(&b'>') {
+            return false;
+        }
+        idx += 1;
+        if matches!(bytes.get(idx), Some(b' ' | b'\t')) {
+            idx += 1;
+        }
+    }

     let fence_char = if is_tilde_fence { b'~' } else { b'`' };

🤖 Fix all issues with AI agents

In `@Cargo.toml`:
- Line 164: The utf8proc dependency breaks wasm32-unknown-unknown because it
wraps C; make it optional and feature-gated or replace it with a pure-Rust
crate: in Cargo.toml mark utf8proc = { version = "0.1.2", optional = true } and
add a feature like "with-utf8proc", or move utf8proc under a non-wasm target
section (e.g. [target.'cfg(not(target_arch = "wasm32"))'.dependencies]); then
update any code that directly calls into utf8proc to be behind cfg(feature =
"with-utf8proc")/cfg(not(target_arch = "wasm32")) guards and provide an
implementation using a pure-Rust alternative (e.g. unicode-normalization or
unicode-width) when the feature is disabled or on wasm. Ensure Cargo features
and conditional compilation are consistent so wasm builds never require the
utf8proc C library.

In `@crates/biome_markdown_parser/src/syntax/thematic_break_block.rs`:
- Around line 41-109: is_thematic_break_pattern currently only accepts
MD_TEXTUAL_LITERAL when its text length is exactly 1, which rejects tokens like
"***" or "* * *"; update is_thematic_break_pattern to treat multi-char
MD_TEXTUAL_LITERAL tokens as sequences of break characters and optional spaces:
when the initial token is MD_TEXTUAL_LITERAL with length>1, verify its
characters are only the candidate break char or whitespace and increment the
count by the number of break characters it contains (treating internal spaces as
skipped), and in the main loop when encountering MD_TEXTUAL_LITERAL with
length>1, scan its chars to accumulate more break characters (and ensure no
invalid chars appear) before calling p.bump_any(); keep existing checks for
T![*], T![-], UNDERSCORE and ensure final condition still requires count >= 3
and line end (NEWLINE or EOF).

🧹 Nitpick comments (14)

crates/biome_markdown_parser/tests/md_test_suite/ok/paren_depth_limit.md (1)

1-1: Nice stress test for parenthesis nesting.

This effectively exercises the depth limit (32 levels) mentioned in the PR objectives. Consider adding a brief comment at the top explaining the test's intent, and perhaps a companion test in the err/ folder that exceeds the limit to validate the boundary behaviour.

crates/biome_string_case/Cargo.toml (1)

16-16: Consider feature-gating utf8proc.

Since utf8proc adds a native C library dependency, you might want to make it optional behind a feature flag (e.g., casefold) so consumers who don't need Unicode case folding can avoid the dependency.

♻️ Suggested feature-gating

 [dependencies]
 biome_rowan = { workspace = true, optional = true }
-utf8proc    = { workspace = true }
+utf8proc    = { workspace = true, optional = true }

 [features]
 biome_rowan = ["dep:biome_rowan"]
+casefold = ["dep:utf8proc"]

crates/biome_string_case/src/lib.rs (1)

668-669: Add a doctest for to_casefold_cow.

The trait method would benefit from inline documentation with a doctest example, consistent with other methods in this trait.

📝 Suggested documentation

+    /// Returns Unicode case-folded text as a Cow, allocating only when needed.
+    ///
+    /// ```
+    /// use biome_string_case::StrOnlyExtension;
+    ///
+    /// assert_eq!("ss", "ß".to_casefold_cow());
+    /// assert_eq!("test", "test".to_casefold_cow());
+    /// ```
     fn to_casefold_cow(&self) -> Cow<'_, Self>;

crates/biome_unicode_table/src/punctuation.rs (1)

24-374: Add a reference comment for table verification.

Per the PR discussion, this table would benefit from a comment indicating how to regenerate or verify it against the source (e.g., Unicode version, markdown-rs commit, or a script).

📝 Suggested comment

+// Generated from Unicode 15.1 General_Category P* and S* ranges.
+// Source: markdown-rs punctuation table (commit: <hash>)
+// To regenerate, see: <link or instructions>
 const PUNCTUATION_RANGES: &[(char, char)] = &[

crates/biome_markdown_parser/src/syntax/reference.rs (1)

23-38: Avoid an extra allocation when case‑folding.
to_casefold_cow().into_owned() clones even when the case‑fold is unchanged. You can keep the existing out when it’s already folded.

♻️ Suggested tweak

-    out.as_str().to_casefold_cow().into_owned()
+    match out.as_str().to_casefold_cow() {
+        std::borrow::Cow::Borrowed(_) => out,
+        std::borrow::Cow::Owned(folded) => folded,
+    }

crates/biome_markdown_parser/src/syntax/html_block.rs (4)

40-49: Consider adding brief doc comments to enum variants.

Whilst the type names (Type1–Type7) map to CommonMark §4.6 directly, a one-liner per variant would help readers unfamiliar with the spec. Optional, but helpful.

---

151-153: Minor: split_once returns the separator-stripped halves.

This is correct for your use case. Just noting that CRLF (\r\n) lines will leave a trailing \r on line because \n is matched first. If you want to strip that too, consider adding .trim_end_matches('\r') or splitting on ['\n', '\r'] more carefully. Likely a non-issue for the block-kind check, but worth a thought.

---

325-349: line is rebuilt per-token, allocating repeatedly.

Per reviewer feedback in the PR summary, storing owned Strings in parser state or loops is a concern. Here, the loop accumulates tokens into line and clears it at each newline. For large HTML blocks, this can be allocation-heavy.

A lighter alternative is to track byte ranges and slice the source directly when checking for the terminator. Not blocking, but worth considering for a follow-up optimisation.

---

351-373: Manual case-insensitive search is correct but could be simplified.

Rust's str::to_ascii_lowercase() or using the unicase crate would make this more readable. The current byte-level iteration is fine for ASCII terminators, but if you ever need Unicode awareness, this will need revisiting.

crates/biome_markdown_parser/src/syntax/inline/code_span.rs (1)

93-161: Potential code duplication with list detection in syntax/mod.rs.

at_list_marker_after_newline replicates logic from at_bullet_list_item, at_order_list_item, and textual_looks_like_list_marker. Consider extracting a shared helper to avoid drift between these implementations.

crates/biome_markdown_parser/src/syntax/inline/html.rs (1)

147-159: skip_spaces closure always returns Some(bool) — the Option wrapper is unnecessary.

The closure never returns None; it always yields Some(skipped). You could simplify the signature to just return bool and remove the ? operator at call sites.

Suggested simplification

-    let skip_spaces = |i: &mut usize| -> Option<bool> {
+    let skip_spaces = |i: &mut usize| -> bool {
         let mut skipped = false;
         while *i < bytes.len() {
             match bytes[*i] {
                 b' ' | b'\t' | b'\n' | b'\r' | b'\x0c' => {
                     skipped = true;
                     *i += 1;
                 }
                 _ => break,
             }
         }
-        Some(skipped)
+        skipped
     };

crates/biome_markdown_parser/src/syntax/header.rs (1)

191-263: header_content_source_len lookahead is thorough but dense.

The function handles trailing hashes, whitespace, and hard-line literals. Consider adding a brief inline comment at the start of each major branch to aid readability.

crates/biome_markdown_parser/src/syntax/mod.rs (1)

1123-1182: is_quote_only_blank_line_from_source scans raw source for quote markers.

This is necessary because token-based checks can't see consumed trivia. The byte-level iteration is correct, though somewhat verbose. Consider a brief comment explaining why source scanning is required here.

crates/biome_markdown_parser/src/to_html.rs (1)

1520-1562: percent_encode_uri preserves existing percent-encoded sequences.

The logic at lines 1525–1537 detects %XX patterns and passes them through unmodified. This prevents double-encoding. However, the check bytes[i + 1].is_ascii_alphanumeric() is slightly loose — valid hex digits are [0-9A-Fa-f], not all alphanumerics. You could use bytes[i + 1].is_ascii_hexdigit() for stricter validation.

Stricter hex digit check

-                && bytes[i + 1].is_ascii_alphanumeric()
-                && bytes[i + 2].is_ascii_alphanumeric()
+                && bytes[i + 1].is_ascii_hexdigit()
+                && bytes[i + 2].is_ascii_hexdigit()

[ematipico]

to_html.rs: This exists specifically as a test harness for CommonMark spec compliance. Commonmark provides test cases as pairs (markdown, expected_html), so that's how we validate correctness. Would you rather we move this to tests/ or gate it behind a feature flag?

That's insightful. I didn't know that and thank you for letting me know. I mean, eventually we will need to away to transform markdown to HTML, so maybe for now we can gate it behind a feature flag of the crate biome_markdown_parser, so that in the future when we will implement proper transformation, we can still validate it against the conformance.

An alternative could be to store TextRange and normalize on lookup. Do you have a preference here?

I think yes, we can normalise on lookup. We will need to set up some benchmarks, so we can see if these optimisations have an actual impact. Unfortunatel,y our current benchmarks don't test memory used.

We could feature-gate it for non-WASM, write a pure Rust crate, or make a simplified inline table but that may break on edge cases. Do you have a preference?

Coderabbit here #8908 (comment) suggests the use of crates that are WASM friendly. We already use unicode-width internally, however it's possible we will need the other crate.

[coderabbitai]

Actionable comments posted: 4

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

crates/biome_markdown_parser/src/syntax/link_block.rs (1)

238-336: ⚠️ Potential issue | 🟠 Major

Bare destination lookahead currently allows whitespace inside raw URLs.
When a whitespace token appears after content in the bare-destination path, the scan continues and can swallow trailing text. That makes inputs like [label]: foo bar look valid in lookahead, while parse_link_destination stops at the first space — the two paths diverge.

A safer approach is to stop the destination scan after the first separator whitespace and let the outer logic decide whether a title starts or trailing text invalidates the definition.

🛠️ Suggested fix

-            if text.chars().all(|c| c == ' ' || c == '\t') && !text.is_empty() {
-                if has_content {
-                    saw_separator = true;
-                }
-                p.bump_link_definition();
-                continue;
-            }
+            if text.chars().all(|c| c == ' ' || c == '\t') && !text.is_empty() {
+                if has_content {
+                    saw_separator = true;
+                    // Consume separator whitespace, then stop scanning destination
+                    p.bump_link_definition();
+                    while !p.at(EOF)
+                        && !p.at(NEWLINE)
+                        && p.cur_text().chars().all(|c| c == ' ' || c == '\t')
+                        && !p.cur_text().is_empty()
+                    {
+                        p.bump_link_definition();
+                    }
+                    break;
+                }
+                p.bump_link_definition();
+                continue;
+            }

🤖 Fix all issues with AI agents

In `@crates/biome_markdown_parser/src/syntax/html_block.rs`:
- Around line 249-265: at_html_block_interrupt currently lacks the ≤3-space
indent guard, so indented code can be misdetected as an HTML block; update
at_html_block_interrupt to early-return false when the current line is indented
more than 3 spaces (mirror the guard used in at_html_block) before calling
html_block_kind, i.e. add the same indentation check used in at_html_block
(using the parser's indentation/column API) so html_block_kind is only consulted
for lines with ≤3 leading spaces.

In `@crates/biome_markdown_parser/src/syntax/mod.rs`:
- Around line 763-784: The function real_line_indent_from_source currently finds
the line start by rfind('\n') only, which fails for CR-only or CRLF line
endings; update the logic that computes line_start to look for the last line
break char (either '\n' or '\r')—e.g., rfind on a predicate c == '\n' || c ==
'\r' (or compute the last index of either and take the greater), and ensure CRLF
is treated as a single break when computing the start (if you find '\n' and the
preceding byte is '\r', use the '\r' index + 1 consistently); keep the remaining
column counting over source[line_start..].chars() unchanged and reference
p.source().source_text() and p.cur_range().start() to locate the position.

In `@crates/biome_markdown_parser/src/syntax/thematic_break_block.rs`:
- Around line 59-83: The fast-path that handles a single MD_TEXTUAL_LITERAL
currently returns true whenever the token text contains only break
chars/whitespace, but it must also verify the token ends the line; update the
logic in the thematic break detector (the block around the
p.at(MD_TEXTUAL_LITERAL) check) to perform a lookahead: consume the current
MD_TEXTUAL_LITERAL token, skip any following SPACE/TAB tokens, and assert the
next token is NEWLINE or EOF before returning true, mirroring the same
end-of-line guard used in the remap fast path (also update the analogous
fast-path at the later block around lines ~168-176).

In `@crates/biome_markdown_parser/src/to_html.rs`:
- Around line 1531-1546: The percent-encoding validator erroneously accepts
non-hex chars because it uses bytes[i + 1].is_ascii_alphanumeric() and bytes[i +
2].is_ascii_alphanumeric(); update both checks to bytes[i +
1].is_ascii_hexdigit() and bytes[i + 2].is_ascii_hexdigit() in the loop that
iterates value.char_indices(), keeping the surrounding logic that pushes the
preserved 3-byte slice (result.push_str(&value[i..i + 3])) and updates last = i
+ 3; (refer to variables value, bytes, i, last, and result) so only valid %HH
sequences are preserved.

🧹 Nitpick comments (6)

crates/biome_markdown_parser/src/to_html.rs (4)

237-240: Consider adding a clarifying comment for this edge case.

The condition quote_indent > 1 && idx > 0 && remaining.len() > 1 followed by stripping a single character is non-obvious. A brief comment explaining when this path is taken (presumably for lazy continuation lines in nested quotes?) would aid future maintainers.

---

500-509: Suppressed node check is O(n) per enter call.

The linear scan through suppressed_inline_nodes on every enter() could become expensive for documents with many inline links. For a test harness this is acceptable, but production use would benefit from a HashSet<SyntaxNode> for O(1) lookups.

---

1866-1871: Test doesn't catch the hex digit validation bug.

The test uses %20 which contains valid hex digits, so it passes despite the is_ascii_alphanumeric() bug. Consider adding a test case with an invalid percent sequence like %GH to verify correct encoding.

🧪 Suggested additional test case

     #[test]
     fn test_percent_encode_uri() {
         let input = format!("https://a{}b.c/%20/%", '\u{1F44D}');
         let encoded = percent_encode_uri(&input);
         assert_eq!(encoded, "https://a%F0%9F%91%8Db.c/%20/%25");
+
+        // Invalid percent sequence should have % encoded
+        let invalid = "https://example.com/%GH";
+        let encoded = percent_encode_uri(invalid);
+        assert_eq!(encoded, "https://example.com/%25GH");
     }

---

1314-1318: Minor: Redundant clone.

text.clone() at line 1317 could be avoided since text is already owned. You could restructure to return (text.clone(), Some(text)) or use (text, None) patterns, but this is a minor nit.

crates/biome_markdown_parser/src/syntax/inline/emphasis.rs (1)

238-277: Potential quadratic scan in the link‑bracket prepass.
The per‑position check_bracket_pattern plus linear contains lookups can go O(n²) on long inputs. Consider a single‑pass bracket matcher with O(1) start lookups to keep this linear.

crates/biome_markdown_parser/src/syntax/inline/links.rs (1)

494-847: Consider de‑duplicating link helper routines.
bump_textual_link_def, is_whitespace_token, get_title_close_char, and parse_title_content mirror link_block.rs. A shared helper module would reduce drift long‑term.

[coderabbitai]

⚠️ Potential issue | 🟡 Minor

Handle CR-only line endings when computing real indent.
real_line_indent_from_source only searches for \n, so files using \r as line breaks will treat the whole file as one line and miscompute indentation.

🔧 Suggested fix

-    let line_start = source[..pos].rfind('\n').map_or(0, |i| i + 1);
+    let line_start = source[..pos]
+        .rfind(|c| c == '\n' || c == '\r')
+        .map_or(0, |i| i + 1);

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	/// Compute the real leading indent of the current line from source text.
	/// This is needed because leading whitespace may have been consumed as trivia
	/// in list item context, making `line_start_leading_indent()` return 0.
	/// Token-based lookahead cannot recover the original column once trivia is skipped.
	fn real_line_indent_from_source(p: &MarkdownParser) -> usize {
	let source = p.source().source_text();
	let pos: usize = p.cur_range().start().into();
	// Find the start of the current line
	let line_start = source[..pos].rfind('\n').map_or(0, |i| i + 1);
	// Count leading whitespace columns on this line
	let mut column = 0;
	for c in source[line_start..].chars() {
	match c {
	' ' => column += 1,
	'\t' => column += 4 - (column % 4),
	_ => break,
	}
	}
	column
	}
	/// Compute the real leading indent of the current line from source text.
	/// This is needed because leading whitespace may have been consumed as trivia
	/// in list item context, making `line_start_leading_indent()` return 0.
	/// Token-based lookahead cannot recover the original column once trivia is skipped.
	fn real_line_indent_from_source(p: &MarkdownParser) -> usize {
	let source = p.source().source_text();
	let pos: usize = p.cur_range().start().into();
	// Find the start of the current line
	let line_start = source[..pos]
	.rfind(|c| c == '\n' || c == '\r')
	.map_or(0, |i| i + 1);
	// Count leading whitespace columns on this line
	let mut column = 0;
	for c in source[line_start..].chars() {
	match c {
	' ' => column += 1,
	'\t' => column += 4 - (column % 4),
	_ => break,
	}
	}
	column
	}

🤖 Prompt for AI Agents

In `@crates/biome_markdown_parser/src/syntax/mod.rs` around lines 763 - 784, The
function real_line_indent_from_source currently finds the line start by
rfind('\n') only, which fails for CR-only or CRLF line endings; update the logic
that computes line_start to look for the last line break char (either '\n' or
'\r')—e.g., rfind on a predicate c == '\n' || c == '\r' (or compute the last
index of either and take the greater), and ensure CRLF is treated as a single
break when computing the start (if you find '\n' and the preceding byte is '\r',
use the '\r' index + 1 consistently); keep the remaining column counting over
source[line_start..].chars() unchanged and reference p.source().source_text()
and p.cur_range().start() to locate the position.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@crates/biome_markdown_parser/src/syntax/html_block.rs`:
- Around line 260-268: The pattern in the matches! call incorrectly uses struct
syntax for the tuple variant HtmlBlockKind::Type1; update the pattern to use
tuple syntax (e.g., HtmlBlockKind::Type1(_) or HtmlBlockKind::Type1(..)) so the
match compiles, leaving the other variants (Type2..Type6) unchanged; this change
should be applied where the matches!(kind, ...) invocation is defined.

🧹 Nitpick comments (5)

crates/biome_markdown_parser/src/syntax/thematic_break_block.rs (3)

44-146: Consider consolidating with list.rs::is_thematic_break_pattern.

This file and crates/biome_markdown_parser/src/syntax/list.rs (lines 166-193) both define is_thematic_break_pattern with identical signatures but different implementations—this one operates on tokens while the other operates on raw source text. Given the PR's goal of improving maintainability, consolidating these into a single shared utility (or clearly documenting why both are needed) would reduce duplication and potential drift.

---

189-191: Silent early return could mask upstream bugs.

If has_eol is false here, the function returns without emitting any token, leaving the AST potentially incomplete. While this "shouldn't happen" if at_thematic_break_block is correct, a debug_assert! would help catch any future regressions.

🛡️ Suggested defensive assertion

         if !has_eol {
+            debug_assert!(false, "parse_thematic_break_tokens called but EOL check failed");
             return;
         }

---

168-250: Optional: extract shared logic with is_thematic_break_pattern.

This function duplicates much of the whitespace-skipping, break-character detection, and EOL-checking logic from is_thematic_break_pattern. If you find yourself modifying one, you'll likely need to mirror changes in the other. A small internal helper for the shared patterns could reduce this burden—though given the complexity, deferring this is reasonable.

crates/biome_markdown_parser/src/syntax/html_block.rs (1)

8-9: Module documentation is now outdated.

The note claims all HTML blocks terminate at blank lines, but the implementation below correctly handles specific terminators for Type1–Type5 (e.g., </script>, -->, ?>, >, ]]>). Worth updating to reflect the improved CommonMark compliance.

Suggested update

-//! **Note:** Not 100% CommonMark §4.6 compliant. All HTML blocks terminate at
-//! blank lines, whereas CommonMark types 1-5 have specific terminators (`-->`, `?>`, etc.).
+//! Implements CommonMark §4.6 HTML block types with appropriate terminators:
+//! Type 1-5 use specific terminators, Type 6-7 terminate at blank lines.

crates/biome_markdown_parser/src/to_html.rs (1)

254-326: Consider feature-gating this test harness API.
The module says “test harness”, but HtmlRenderContext/document_to_html are public in the main crate. A feature flag or moving the harness under tests would avoid shipping a non‑production API.

[coderabbitai]

⚠️ Potential issue | 🟠 Major

🧩 Analysis chain

🏁 Script executed:

# Find HtmlBlockKind definition
fd "\.rs$" crates/biome_markdown_parser/src | head -20

Repository: biomejs/biome

Length of output: 1098

---

🏁 Script executed:

# Search for HtmlBlockKind definition
rg "enum HtmlBlockKind|Type1" crates/biome_markdown_parser/src --type rs -B 2 -A 10

Repository: biomejs/biome

Length of output: 83

---

🏁 Script executed:

# Check if the file compiles
cd crates/biome_markdown_parser && cargo check 2>&1 | grep -A 5 "error\|warning" | head -50

Repository: biomejs/biome

Length of output: 143

---

🏁 Script executed:

# Search for HtmlBlockKind enum definition without type flag
rg "enum HtmlBlockKind" crates/biome_markdown_parser/src -A 15

Repository: biomejs/biome

Length of output: 1546

---

🏁 Script executed:

# Read the html_block.rs file to see context around line 260
cat -n crates/biome_markdown_parser/src/syntax/html_block.rs | sed -n '1,50p'

Repository: biomejs/biome

Length of output: 2217

---

🏁 Script executed:

# Check the specific lines mentioned in the review (260-268)
cat -n crates/biome_markdown_parser/src/syntax/html_block.rs | sed -n '250,270p'

Repository: biomejs/biome

Length of output: 760

---

Fix pattern syntax for tuple variant Type1.

HtmlBlockKind::Type1 is a tuple variant Type1(Type1Tag), but line 262 uses struct pattern syntax { .. }. For tuple variants, use Type1(_) instead.

Fix

         matches!(
             kind,
-            HtmlBlockKind::Type1 { .. }
+            HtmlBlockKind::Type1(_)
                 | HtmlBlockKind::Type2
                 | HtmlBlockKind::Type3
                 | HtmlBlockKind::Type4
                 | HtmlBlockKind::Type5
                 | HtmlBlockKind::Type6
         )

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change

	matches!(
	kind,
	HtmlBlockKind::Type1 { .. }
	| HtmlBlockKind::Type2
	| HtmlBlockKind::Type3
	| HtmlBlockKind::Type4
	| HtmlBlockKind::Type5
	| HtmlBlockKind::Type6
	)
	matches!(
	kind,
	HtmlBlockKind::Type1(_)
	| HtmlBlockKind::Type2
	| HtmlBlockKind::Type3
	| HtmlBlockKind::Type4
	| HtmlBlockKind::Type5
	| HtmlBlockKind::Type6
	)

🤖 Prompt for AI Agents

In `@crates/biome_markdown_parser/src/syntax/html_block.rs` around lines 260 -
268, The pattern in the matches! call incorrectly uses struct syntax for the
tuple variant HtmlBlockKind::Type1; update the pattern to use tuple syntax
(e.g., HtmlBlockKind::Type1(_) or HtmlBlockKind::Type1(..)) so the match
compiles, leaving the other variants (Type2..Type6) unchanged; this change
should be applied where the matches!(kind, ...) invocation is defined.

[coderabbitai]

Actionable comments posted: 1

🤖 Fix all issues with AI agents

In `@crates/biome_markdown_parser/src/to_html.rs`:
- Around line 1595-1663: The alt-text extractor is double-escaping characters
because extract_alt_text_inline calls render_textual and also uses
escape_html/escape_html_attribute later; update extract_alt_text_inline so it
collects raw text (use collect_raw_inline_text or equivalent) instead of calling
render_textual in the AnyMdInline::MdTextual branch, and stop pre-escaping in
the MdInlineCode and MdAutolink branches (append the raw collected content after
any trimming instead of calling escape_html); leave final escaping to the code
that writes the alt attribute (escape_html_attribute) so alt text is escaped
exactly once.

🧹 Nitpick comments (1)

crates/biome_markdown_parser/src/syntax/inline/emphasis.rs (1)

274-274: Consider using a HashSet for link_bracket_starts lookups.

The contains(&i) call on a Vec is O(n) per lookup. For documents with many brackets, this could degrade performance. A HashSet<usize> would provide O(1) lookups.

♻️ Suggested refactor

-    let mut link_bracket_starts = Vec::new();
+    let mut link_bracket_starts = std::collections::HashSet::new();
     for pos in 0..bytes.len() {
         if bytes[pos] == b'['
             && let Some(result) = check_bracket_pattern(bytes, pos)
         {
             if result.is_inline_or_full_ref {
                 // Inline link or full reference link
-                link_bracket_starts.push(pos);
+                link_bracket_starts.insert(pos);
             } else {
                 // Could be a shortcut reference - check if definition exists
                 let label = extract_label_text(source, pos, result.close_pos);
                 let normalized = normalize_reference_label(label);
                 if !normalized.is_empty() && reference_checker(normalized.as_ref()) {
-                    link_bracket_starts.push(pos);
+                    link_bracket_starts.insert(pos);
                 }
             }
         }
     }

[ematipico]

@jfmcdowell

Will merge this, however please note that myself and @tidefield found an issue here

biome/xtask/codegen/markdown.ungram

Lines 54 to 68 in b5dd9ce

	AnyLeafBlock =
	MdThematicBreakBlock
	| MdHeader
	| MdSetextHeader
	| AnyCodeBlock
	| MdHtmlBlock
	| MdLinkReferenceDefinition
	| MdLinkBlock
	| MdParagraph
	| MdNewline
	AnyContainerBlock =
	MdQuote
	| MdBulletListItem
	| MdOrderedListItem

These two nodes don't follow the internal naming convention, and they fail in some part ofthe codegen of the formatter. Can you send a PR to fix it? Or @tidefield will do in their PR

[tidefield]

@ematipico I'll do it and include it in my current PR.

[jfmcdowell]

@ematipico I'll do it and include it in my current PR.

@ematipico / @tidefield thanks for the call out and assist. Let me know if you need any help @tidefield