Optimize performance of blocklist filtering and checking by using Regex

[GromNaN]

A single call to preg_match can replace a lot of lines of code, and is executed in optimized C code instead of PHP.

1. Blocklist filter

The regex /^[abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789]+$/i tell if all the characters of the blocked work are in the alphabet.

(cancelled because I removed the blocklist filter in 3).

2. Check if the ID contains a blocked word

A regex is generated with all the blocked words /(1d10t|b0ob)/i and run with in case-insensitive mode.

3. Apply leet transformation to blocked word

The blocklist if full of leet variations of the same words. Using regex, we can check directly for alternative way of writting the same word. /(ahole)/i becomes /(ah[oO][l1]e)/i to check ah0le, aho1e, ah01e and all other case variations.

Benchmark

PHPBench code

composer req --dev phpbench/phpbench

In phpbench.json

{
    "$schema": "./vendor/phpbench/phpbench/phpbench.schema.json",
    "runner.bootstrap": "vendor/autoload.php",
    "runner.file_pattern": "*Bench.php",
    "runner.path": "tests",
    "runner.iterations": 3
}

In tests/SqidsBench.php

<?php

namespace Sqids\Tests;

use PhpBench\Attributes\ParamProviders;
use PhpBench\Attributes\Revs;
use PhpBench\Attributes\Warmup;
use Sqids\Sqids;

#[Warmup(1)]
final class SqidsBench
{
    #[Revs(1_000)]
    public static function benchInit(): void
    {
        new Sqids();
    }

    #[Revs(1_000)]
    #[ParamProviders('provideSqids')]
    public static function benchEncode(array $params): void
    {
        $params[0]->encode([1_000_000, 2_000_000]);
    }

    public static function provideSqids(): \Generator
    {
        yield 'default' => [
            new Sqids()
        ];
        yield 'custom blocklist' => [
            new Sqids(blocklist: [
                'JSwXFaosAN',
                'OCjV9JK64o',
                'rBHf',
                '79SM',
                '7tE6',
            ])
        ];
    }
}

Before

    benchInit...............................I2 - Mo1.617ms (±0.23%)
    benchEncode # default...................I2 - Mo280.402μs (±0.16%)
    benchEncode # custom blocklist..........I2 - Mo194.539μs (±0.34%)

After 2

    benchInit...............................I2 - Mo164.020μs (±0.44%)
    benchEncode # default...................I2 - Mo34.207μs (±0.46%)
    benchEncode # custom blocklist..........I2 - Mo183.985μs (±0.22%)

After 3

    benchInit...............................I2 - Mo77.698μs (±0.70%)
    benchEncode # default...................I2 - Mo32.270μs (±0.44%)
    benchEncode # custom blocklist..........I2 - Mo183.796μs (±0.63%)

After rebase on #18

    benchInit...............................I2 - Mo67.920μs (±0.22%)
    benchEncode # default...................I2 - Mo23.862μs (±0.52%)
    benchEncode # custom blocklist..........I2 - Mo136.598μs (±0.70%)
    benchEncode # no blocklist..............I2 - Mo22.825μs (±0.39%)

[vinkla]

LGTM! @4kimov will take a look who is the blocklist expert.

[GromNaN]

This regex is not matching anything as the words never contain the tilde ~ char.

[stof]

changing the type of a protected property (from array to ?string is a BC break. I don't know what is the Backward Compatiblity policy of this project (I got here because of your toot about performance improvements, by curiosity) but it might make sense to be care about this.

[GromNaN]

You are right, I reverted to use an other property name and leave this one. Even if I don't see any reason this class would be extended. It should be final, it has an interface.

[vinkla]

You're right @stof, we don't want to introduce any breaking changes.

[GromNaN]

If anyone extends the class and updates $this->blocklist after calling the constructor, then this change will not be used later in the id generator.
But this is already a misuse, as it bypasses the filter.

[4kimov]

@GromNaN I gotta admit, I'm a bit confused by this PR. A few questions:

1. You don't worry about checking if blocklist words contain chars that might not be in alphabet?
2. Does the new isBlockedId take care of the scenario of blocking short ids that match short blocklist words exactly?
3. Same question as number 2, but for ids that start with or end with blocked word?
4. Finally, did I mess up this PR by merging Performance optimizations #18 first? Looks like there's a conflict :[

[GromNaN]

1. You don't worry about checking if blocklist words contain chars that might not be in alphabet?

Removing the words invalid chars from the blocklist is an optimization, not a feature. Even after optimization (step 1), this task is more costly during class initialization, for an insignificant benefit when an ID is generated.

2. Does the new isBlockedId take care of the scenario of blocking short ids that match short blocklist words exactly?

3. Same question as number 2, but for ids that start with or end with blocked word?

If I've missed anything, it's that it's not covered by the tests.

My understanding is that you block any generated short ID that contains a blocked word. Whether the id "starts with", "ends with" or "is equals to" the word is covered by the "contains" verification done using the regex.

4. Finally, did I mess up this PR by merging Performance optimizations #18 first? Looks like there's a conflict :[

That was expected. I rebased the PR.

[4kimov]

You're right, the tests for blocklist logic should cover more scenarios.

I'm not sure just the contains logic would mimic what the spec does.

Here's one recent discussion about this: sqids/sqids-javascript#30

Another example is: only if the word is bigger than 3 chars and it contains numbers and it starts with or ends with blocked word, then we block it. So id like abcd1efgh2ijkl would be blocked if blocklist contains abcd, but not blocked if blocklist contains efgh.

[GromNaN]

Currently, there is a str_contains that returns true as soon as the ShortId contains one of the blocked words. The previous conditions are irrelevant.

sqids-php/src/Sqids.php

Lines 818 to 819 in 9390a85

	} elseif (str_contains($id, (string) $word)) {
	return true;

The regex match is always false, it requires the number to be surrounded by ~.

sqids-php/src/Sqids.php

Lines 814 to 816 in 9390a85

	} elseif (preg_match('/~[0-9]+~/', (string) $word)) {
	if (str_starts_with($id, (string) $word) || strrpos($id, (string) $word) === strlen($id) - strlen((string) $word)) {
	return true;

This condition on id or word smaller that 3 is not tested.

If you find a test, I could fix the code.

[4kimov]

Thanks for pointing out the ~. Looks like a few todo items [for me]. Putting this PR on ice for now.

PS: Updating tests for PHP is also not as straightforward since they're the same across all implementations.

[4kimov]

Added more tests here (explanation for the tests is in the spec). Also, changed the regex for ints as discussed previously. If anybody sees anything else worth testing in isBlockedId, please do tell.

@GromNaN Could you run this PR against the new tests?

[GromNaN]

Thanks for the tests. I updated the regex to use start ^ and end $ matching conditions for words that contains a digit. The init time is slightly impacted (from 66μs to 108μs), but the encoding time remains the same.

    benchInit...............................I2 - Mo108.140μs (±0.29%)
    benchEncode # default...................I2 - Mo24.918μs (±0.17%)
    benchEncode # custom blocklist..........I2 - Mo136.935μs (±0.13%)
    benchEncode # no blocklist..............I2 - Mo22.871μs (±0.09%)

[4kimov]

Thank you @GromNaN, that was fast work :) Merged!

[GromNaN]

It was a good collaboration. I think this implementation using regex can improve performance in other languages too, but I wouldn't do it.

[4kimov]

Indeed. I'm going to wait a few days before publishing the new version. I know we have a breaking change with that ~ regex, but since we're still pre-1.0, I will probably just bump up the minor version (along with a changelog explanation).

Thanks for all the contributions, it was a lot and much appreciated. I see you're a PHP expert. I gave you write access to the repo in case you notice something, and I'm too slow to respond (that way I'm not blocking you).