---

Appendix A The wchar_t mess

The ISO C and POSIX standard creators made an attempt to fix the first problem mentioned in the section ‘char *’ strings. They introduced

• a type ‘wchar_t’, designed to encapsulate an entire character,
• a “wide string” type ‘wchar_t *’, with some API functions declared in <wchar.h>, and
• functions declared in <wctype.h> that were meant to supplant the ones in <ctype.h>.

Unfortunately, this API and its implementation has numerous problems:

• On Windows platforms and on AIX in 32-bit mode, wchar_t is a 16-bit type. This means that it can never accommodate an entire Unicode character. Either the wchar_t * strings are limited to characters in UCS-2 (the “Basic Multilingual Plane” of Unicode), or — if wchar_t * strings are encoded in UTF-16 — a wchar_t represents only half of a character in the worst case, making the <wctype.h> functions pointless.
• On Solaris and FreeBSD, the wchar_t encoding is locale dependent and undocumented. This means, if you want to know any property of a wchar_t character, other than the properties defined by <wctype.h> — such as whether it’s a dash, currency symbol, paragraph separator, or similar —, you have to convert it to char * encoding first, by use of the function wctomb.
• When you read a stream of wide characters, through the functions fgetwc and fgetws, and when the input stream/file is not in the expected encoding, you have no way to determine the invalid byte sequence and do some corrective action. If you use these functions, your program becomes “garbage in - more garbage out” or “garbage in - abort”.

As a consequence, it is better to use multibyte strings, as explained in the section ‘char *’ strings. Such multibyte strings can bypass limitations of the wchar_t type, if you use functions defined in gnulib and libunistring for text processing. They can also faithfully transport malformed characters that were present in the input, without requiring the program to produce garbage or abort.