Handle the first-cell byte-order mark

The invisible character before the header

Many programs, including Excel's UTF-8 CSV option, start a text file with a byte-order mark: the character U+FEFF, which tells the reader that the file is UTF-8. It is invisible in most editors, but it is still a character. When a tool does not remove it, the first header becomes the mark followed by id, and any lookup or join on a column called id fails without an obvious reason.

The inspector removes one byte-order mark at the very start of the pasted text before it reads any cells. The first header is then exactly id.

Example: the id,name header

The page example is a header id,name that starts with a byte-order mark, followed by the record 1,Mira, with comma as the Delimiter. The inspector reports 2 rows and 2 columns, and row 1, column 1 reads id, with no hidden prefix.

Nothing else in the text is changed: other cells keep every character they had.

Without that step, the first header would look identical on screen but compare as different, which is exactly the kind of mismatch that is hard to see.

Checking a file that may carry a mark

• Paste the whole file, from its first character, into Table text.
• Press Calculate and look at the first header in the preview.
• If you compare headers with another file, compare them after both have been through the inspector.
• Do not treat the removal as general whitespace cleanup. Only one mark at the very beginning is removed.
• If you join several files into one text, any mark that sat at the start of the second or third file is now in the middle of the text and stays in the cell where it landed.

Marks in the middle and in exports

A mark in the middle of the text stays part of that cell. Repeated header cleanup trims spaces and similar characters at the edges of cells before comparing, and that trimming also covers a byte-order mark at the start of a cell, so a repeated header from a second joined file can still be recognized. The cells in your output, though, keep the character.

The Tablora app's CSV export starts with a byte-order mark on purpose, so that spreadsheets open accented, Turkish, Japanese and other non-Latin text correctly. If you paste such a file here, the inspector removes that mark, and the first header comes through clean.

Other invisible characters

Zero-width spaces and similar characters are not removed. If a header still fails to match after this step, look for them in the source.

Original synthetic example

Input

id,name 1,Mira

Delimiter

,

Expected records

2

Columns in the first record

2

Cell to inspect

Row 1, column 1: id

Tools

• delimited-inspector

Sources

• RFC4180: Common Format and MIME Type for CSV Files