The engine function file_str_readtow does not take into account the BOM of a UCS-2 LE encoded text file and adds a non-visible character (0xFEFF) at the beginning of the first string. If files should not contain a BOM, could you please write it explicitly?(Although I would prefer if it handled the BOM as notepad++ does not let me create UCS-2 files without a BOM)

file_str_readtow adds no non-visible character 0xFEFF.

I have isolated the code and reproduced the situation. Code and the file is here: http://talemon.com/share/bugtest.zip

I've written "test" in two different lines in a text file and saved it in notepad++ as UCS-2 LE. I've read the file line by line with file_str_readtow. I then use diag_var to output the results of str_len calls on two strings I've read to. First call returns 6 whereas second one returns 4.

I've also looked the file with a hex-editor and it starts with a 0xFEFF which is the BOM.

If you could look at it and tell me why I'm getting different results I would really appreciate it.
Thanks.

Thanks for your demo. No function adds some invisible character, the character was already in your text file. String functions just manipulate character strings. They do not care about the coding or what a certain character means.

BTW, why did you start a level for reading strings?

But file_str_readtow deals with files and it must and certainly does care about coding of a file.(Otherwise my sample program would not break when I change the encoding.)


That 'character' is the byte-order mark and it is essential in recognizing UCS-2 text: (http://en.wikipedia.org/wiki/UCS-2#Byte_order_encoding_schemes)


I just used the code template SED comes with and removed some lines.

Yes, I understand that 0xFEFF serves as a byte-order mark in the coding that you use, but for string functions it's just 2 bytes. A string has no coding. The coding is up to the programmer. The only special character in a string is the 0 that marks the end.

For finding characters with a special meaning in strings you can use the str_chr function. This way you can remove those characters for displaying the content of the string.

Maybe I should re-write my original problem:

The function "file_str_readtow" which reads text files doesn't properly processes those files and adds the file's metadata to the content. I think you should fix file_str_readtow so that it discards that BOM. Even better, it can use it so the engine can process both little-endian and big-endian text files.

Users of a programming language expect string functions to do only what was specified. They are not supposed to process codes or discard characters. This would cause lots of complains and hard to find errors.

The obvious solution to your problem is writing a function to be applied to strings, which processes your UCS-2 code and deletes its control characters.