[DevHub] Add function page for wp_scrub_utf8

[andreawetzel]

What is the new page you are requesting?

The function wp_scrub_utf8() is mentioned on the following 6.9 article: Modernizing UTF-8 support in WordPress 6.9 but it is not available in the developers reference for functions: https://developer.wordpress.org/reference/functions/

How will this new page help you?

Documentation for this function would help developers of custom blocks and those working on importing content into WordPress.

[github-actions]

Heads up @WordPress/docs-issues-coordinators, we have a new issue open. Time to use 'em labels.

[Anuj-Rathore24]

/assign

[github-actions]

Hey @Anuj-Rathore24, thanks for your interest in this issue! 🍪🍪🍪
If you have any questions, do not hesitate to ask them in our #docs Slack channel.
Enjoy and happy contributing ❤️

[Anuj-Rathore24]

wp_scrub_utf8 ( string $text ): string

Replaces ill-formed UTF-8 byte sequences with the Unicode Replacement Character.

Description

This function identifies and replaces invalid UTF-8 byte sequences in a string with the Unicode Replacement Character (�). It uses the "maximal subpart" algorithm to ensure secure and interoperable text handling, which means corrupted byte sequences are replaced individually, potentially resulting in multiple replacement characters in a row.

While it's sometimes preferable to preserve invalid bytes for downstream processing, this function is useful when you need guaranteed valid UTF-8 output, such as when generating XML documents, inserting data into databases with strict encoding requirements, or passing text to systems that cannot handle malformed UTF-8. Valid UTF-8 strings pass through unchanged.

Parameters

$text (string) (required)
String which is assumed to be UTF-8 but may contain invalid sequences of bytes.

Return

(string) Input text with invalid sequences of bytes replaced with the Unicode replacement character.

Source

File: /wp-includes/utf8.php

function wp_scrub_utf8( $text ) {
    /*
     * While it looks like setting the substitute character could fail,
     * the internal PHP code will never fail when provided a valid
     * code point as a number. In this case, there's no need to check
     * its return value to see if it succeeded.
     */
    $prev_replacement_character = mb_substitute_character();
    mb_substitute_character( 0xFFFD );
    $scrubbed = mb_scrub( $text, 'UTF-8' );
    mb_substitute_character( $prev_replacement_character );
    return $scrubbed;
}

(https://core.trac.wordpress.org/browser/tags/6.9/src/wp-includes/utf8.php#L61) View on GitHub

Related

Uses:

mb_substitute_character() - PHP function to set/get the substitution character for invalid byte sequences

mb_scrub() - PHP function to scrub/replace invalid multibyte sequences in a string

Used By:

sanitize_text_field() - wp-includes/formatting.php - Sanitizes a string from user input or from the database

wp_kses() - wp-includes/kses.php - Filters text content and strips out disallowed HTML

wp_strip_all_tags() - wp-includes/formatting.php - Properly strips all HTML tags including script and style

See Also:

seems_utf8() - wp-includes/formatting.php - Checks to see if a string is UTF-8 encoded

utf8_uri_encode() - wp-includes/formatting.php - Encodes the Unicode values to be used in the URI

remove_accents() - wp-includes/formatting.php - Converts all accent characters to ASCII characters

sanitize_textarea_field() - wp-includes/formatting.php - Sanitizes a multiline string from user input or from the database

wp_check_invalid_utf8() - wp-includes/formatting.php - Checks for invalid UTF-8 in a string

mb_convert_encoding() - PHP function to convert character encoding

Changelog

Version	Description
6.9	Introduced

[andreawetzel]

This issue can be closed since the documentation from new functions added in 6.9 will be added to the developer documentation when the parser is next run.

[andreawetzel]

Linking to PR to fix PHP parser here: WordPress/phpdoc-parser#247