Skip to content
FrAIFlow

Solutions

Why identical-looking text fails to match

Two copies of café can look identical and still compare unequal. One may end with U+00E9; the other with U+0065 followed by combining U+0301. The accent is not wrong. The stored Unicode sequences differ, and the comparison rule needs to acknowledge that distinction.

A 2019 Stack Overflow question described visually similar Vietnamese text that failed equality checks. The answer used a common normalization form. Our saved browser test uses a different, synthetic café pair: direct equality was false, while comparing both NFC forms returned true. It is not a reproduction of the reporter’s application.

Work through one mismatching pair first

Before changing a whole spreadsheet or database, copy just the two values that ought to match into a small local script. Use harmless sample text if the real values identify customers. Seeing the code points side by side is much more useful than repeatedly retyping an accent.

const typed = "caf\u00E9";
const pasted = "cafe\u0301";
const points = s => Array.from(s, c =>
  "U+" + c.codePointAt(0).toString(16).toUpperCase().padStart(4, "0")
).join(" ");
console.log(points(typed));  // U+0063 U+0061 U+0066 U+00E9
console.log(points(pasted)); // U+0063 U+0061 U+0066 U+0065 U+0301
console.log(typed === pasted); // false
const matches = typed.normalize("NFC") === pasted.normalize("NFC");
console.log(matches); // true

Read the two sequences from left to right. Here the first three letters agree; only the representation of the accented final letter differs. That is the situation NFC is designed to handle. The illustration below makes the two storage forms visible; the extra mark is a combining accent, not an unwanted extra letter.

For a search or import comparison, create a separate normalized key and keep the original value for display. Normalize both sides of a comparison, including values already in your database. Normalizing only the new input leaves old records inconsistent. Before applying the rule to all records, check a few pairs that should match and a few that must remain different.

If the result is still false, inspect for a trailing space, a non-breaking space or a different alphabet. Do not respond by stripping all accents or punctuation. Those are additional matching policies that can merge different names. If a proposed duplicate group affects accounts, payments or identity, show it for review instead of treating visual similarity as proof.

Keep comparison separate from identity

Looks like café / ... U+00E9 / 4 UTF-16 code units
Author-created explanation, not an application capture.

A match under NFC does not prove that two accounts belong to one person. Preview any proposed duplicate groups and decide what a match means for the data. Do not add normalization to passwords, signatures or opaque tokens as incidental cleanup: those systems may require exact byte or character contracts.

Why not use NFKC everywhere?

NFKC adds compatibility mappings. In the recorded test, the ligature U+FB00 stayed unchanged under NFC but became ff under NFKC. That can help a deliberately permissive search key, yet lose a distinction required in the original writing. Choose the policy before choosing the transform.

Normalization also does not solve lookalikes: Latin a and Cyrillic U+0430 remained unequal in the example. Nor does it define a website’s character limit. The two café forms measured 4 and 5 JavaScript UTF-16 code units; grapheme segmentation of the decomposed word produced 4 reader-facing groups. Those are different measurements.

Two forms of café use 4 and 5 UTF-16 code units and compare equal after both are normalized to NFC.
Supporting explanation, not an application screenshot.
Recorded Chrome output comparing the code points in café: raw equality is false, while NFC equality is true.
Saved local test output. Open the image to inspect the original evidence.
Sources, examples and further reading