Standardisation does not work if the number contains non-breaking spaces

[andylaw]

I just had a bug report come in from one of my systems where the users were sure the NHS number was a genuine number because they "had copied the number from the NHS system screen". A brief debugging session later showed the presence of "\xa" non-breaking spaces.

We can fix this (possibly also emitting a warning?) by updating the standardise_format() method

[andylaw]

Now not sure that we should. Allowing non-breaking spaces potentially pollutes the data for downstream processing because a system that allows such characters would then need to sanitise every time that it exported to a third-party system in case that receiver didn't permit them

[pacharanero]

TL;DR

standardise_format() should remove non-breaking spaces and return the numeric portion of the NHS number as a string

Rationale

I guess it depends on where the nhs_number library is being used - if it's sanitising the input of another system then it may receive an NHS number that contains these non-breaking space characters, as in this case. If copy-pasted by a user in good faith, from another system (which we don't necessarily have control over) which has decided to show the NHS number formatted in the "3,3,4" grouping but using non-breaking spaces, we probably want it to still work. End users will find it frustrating if it doesn't, and most won't know what a non-breaking space is, or how to find out why it's not working.

Therefore I think it would be good for standardise_format() to handle any non-breaking space characters, and possibly other types of non-visible character, by removing them. This allows the function to consistently return the 10-digit, spaceless, stringified version of the NHS number, which I think we have regarded as the preferred/recommended format for storage of the number as it is the least susceptible to downstream data corruption.

The NHS Common User Interface guidelines (NPfIT era, deprecated 2019 but never actually superseded, replaced or updated) on input and display of the NHS number here on the National Archives mention that it is mandatory to display the NHS Number on a single line and that it is mandatory to format the NHS Number with 2 spaces: XXX XXX XXXX, so it is quite possible and not illogical that a developer implementing this guidance would decide to use non-breaking space characters, as this would assist in achieving the mandatory outcome.

There is plenty of mention of the characters which are not allowed:

But they don't appear to have considered the case of non-breaking spaces.

(The lack of NHS Common User Interface guidance is a national scandal, but like most similar NHS IT cockups, it is one which nobody seems to care about any more).

So there could well be systems out there which had implemented the NHS CUI, using non-breaking spaces. It makes sense to me that this library should handle them.

@andylaw are there any downsides to doing this in your view? I didn't understand how it would pollute downstream data, because for all other characters (eg. - or normal spaces) standardise_format() removes them, so it would presumably do the same with a non-breaking space character.

[andylaw]

Downside for me is that I use the code to check for invalid records in a separate database. Invalid data (including about 30 separate checks of different fields within the database) is flagged for correction by the contributing sites (of which there are north of 100). This is a "side process", which is simply monitoring data quality and doesn't sit in the processing pipelines. If standardise_format() makes non-breaking spaces valid by effectively hiding them from the checks, then those non-breaking spaces will appear in data exports in the analysis space where they may trip up code that is not expecting them. When I say "pollute downstream data", I really mean "allow potentially code-breaking data to flow downstream".

I don't particularly want to revisit all of the export pipeline code, although that is an option. It certainly wouldn't be right for me to "fix in place".

We've decided locally to continue to flag this as an error and to ask the submitting site to retype the number manually if it is flagged. For context, this is the first time that we have seen this issue in more than 30,000 records entered.

[andylaw]

Of course, the problem stems from the combination of NUM-0001 and NUM-0003, particularly the former

[pacharanero]

For context, this is the first time that we have seen this issue in more than 30,000 records entered.

OK, well if this is a comparatively rare occurrence we can probably afford to ignore it for now. We can leave this issue open as a place to discuss it, for any other users who are affected.

Of course, the problem stems from the combination of NUM-0001 and NUM-0003, particularly the former

It is maddening that the NHS never bothered to build the 'Twitter Bootstrap' of NHS CUI - these problems could have been solved once and for all in an NHS CUI toolkit which would have implemented the entire CUI properly, and it would have been adopted for the same reasons people adopt other UI frameworks - it's quick and easy and things look good, gradually it becomes economically unviable not to use one. Instead we have a PDF. The only software implementation was done in Microsoft Silverlight I believe - because of course NHS paid Microsoft to create the CUI, and they used it as an opportunity to try to nudge people into using their half-baked Silverlight caca.