Tracking Down Characters You Cannot See

Guide · Published September 2026

There is a specific kind of bug that eats an afternoon: two values that are obviously identical refuse to match. Same length on screen, same spelling, same case — and the comparison still says they are different. The culprit is usually a character that renders as nothing at all.

What counts as an invisible character

These are legitimate Unicode characters with legitimate uses that happen to have no visible glyph, documented in the General Punctuation block of the Unicode Character Database:

Why you cannot find them by eye or by search

A find-and-replace for a space does not match a non-breaking space, because they are different characters. Your editor’s “show whitespace” mode usually marks tabs and ordinary spaces but not zero-width characters, which have no width to mark. And a visual scan cannot work by definition. The only reliable approach is to check each character against a known list, which is what a detector does.

Finding them

1. Paste the text that is misbehaving into the invisible character detector.
2. Run the scan.
3. Read the results — each hit names the character and gives its exact line and column.
4. Copy the cleaned version with every detected character stripped.

The line and column matter more than the count. Knowing that row 47 of your export has a zero-width space at column 12 tells you which record is broken and often which system produced it.

The list comparison connection

This is the single most common cause of “these two lists should match and don’t.” A trailing zero-width space on one export and not the other makes every affected row show up as only-in-List-1 and only-in-List-2 simultaneously — present on both sides, matching on neither. If a comparison returns a suspiciously large symmetric difference on data you expected to mostly overlap, clean both sides before investigating anything else. The normal culprits to rule out first are covered in the duplicates guide (case and ordinary whitespace); invisible characters are what remains when those are handled.

Preventing them at the source

A recurring BOM at the start of every CSV from the same system is not bad luck — it is an export setting. Most tools offer “UTF-8” and “UTF-8 with BOM” as separate options, and picking the former stops the problem permanently. Similarly, if pasted CMS content always arrives with non-breaking spaces, pasting as plain text rather than rich text avoids them. Fixing the export beats cleaning the output every time.

What the detector deliberately ignores

It flags invisible formatting characters, not visible-but-unusual ones. Curly quotes, em dashes and accented letters are all visible and rarely the cause of a matching mystery — if those are your problem, it is a normalization question, handled with case and whitespace settings in the comparison tool rather than by stripping characters. Similarly, for CSV files specifically the encoding issues described in CSV vs TSV often travel together with BOM problems.