=head1 NAME perlunicode - Unicode support in Perl =head1 DESCRIPTION =head2 Important Caveats Unicode support is an extensive requirement. While Perl does not implement the Unicode standard or the accompanying technical reports from cover to cover, Perl does support many Unicode features. People who want to learn to use Unicode in Perl, should probably read the L and L, before reading this reference document. Also, the use of Unicode may present security issues that aren't obvious. Read L. =over 4 =item Safest if you "use feature 'unicode_strings'" In order to preserve backward compatibility, Perl does not turn on full internal Unicode support unless the pragma C is specified. (This is automatically selected if you use C or higher.) Failure to do this can trigger unexpected surprises. See L below. This pragma doesn't affect I/O, and there are still several places where Unicode isn't fully supported, such as in filenames. =item Input and Output Layers Perl knows when a filehandle uses Perl's internal Unicode encodings (UTF-8, or UTF-EBCDIC if in EBCDIC) if the filehandle is opened with the ":encoding(utf8)" layer. Other encodings can be converted to Perl's encoding on input or from Perl's encoding on output by use of the ":encoding(...)" layer. See L. To indicate that Perl source itself is in UTF-8, use C. =item C still needed to enable UTF-8/UTF-EBCDIC in scripts As a compatibility measure, the C pragma must be explicitly included to enable recognition of UTF-8 in the Perl scripts themselves (in string or regular expression literals, or in identifier names) on ASCII-based machines or to recognize UTF-EBCDIC on EBCDIC-based machines. B is needed.> See L. =item BOM-marked scripts and UTF-16 scripts autodetected If a Perl script begins marked with the Unicode BOM (UTF-16LE, UTF16-BE, or UTF-8), or if the script looks like non-BOM-marked UTF-16 of either endianness, Perl will correctly read in the script as Unicode. (BOMless UTF-8 cannot be effectively recognized or differentiated from ISO 8859-1 or other eight-bit encodings.) =item C needed to upgrade non-Latin-1 byte strings By default, there is a fundamental asymmetry in Perl's Unicode model: implicit upgrading from byte strings to Unicode strings assumes that they were encoded in I, but Unicode strings are downgraded with UTF-8 encoding. This happens because the first 256 codepoints in Unicode happens to agree with Latin-1. See L"Byte and Character Semantics"> for more details. =back =head2 Byte and Character Semantics Beginning with version 5.6, Perl uses logically-wide characters to represent strings internally. Starting in Perl 5.14, Perl-level operations work with characters rather than bytes within the scope of a C> (or equivalently C or higher). (This is not true if bytes have been explicitly requested by C>, nor necessarily true for interactions with the platform's operating system.) For earlier Perls, and when C is not in effect, Perl provides a fairly safe environment that can handle both types of semantics in programs. For operations where Perl can unambiguously decide that the input data are characters, Perl switches to character semantics. For operations where this determination cannot be made without additional information from the user, Perl decides in favor of compatibility and chooses to use byte semantics. When C (but not C) is in effect, Perl uses the semantics associated with the current locale. (C overrides C in the same scope; while C effectively also selects C in its scope; see L.) Otherwise, Perl uses the platform's native byte semantics for characters whose code points are less than 256, and Unicode semantics for those greater than 255. On EBCDIC platforms, this is almost seamless, as the EBCDIC code pages that Perl handles are equivalent to Unicode's first 256 code points. (The exception is that EBCDIC regular expression case-insensitive matching rules are not as as robust as Unicode's.) But on ASCII platforms, Perl uses US-ASCII (or Basic Latin in Unicode terminology) byte semantics, meaning that characters whose ordinal numbers are in the range 128 - 255 are undefined except for their ordinal numbers. This means that none have case (upper and lower), nor are any a member of character classes, like C or C. (But all do belong to the C class or the Perl regular expression extension C.) This behavior preserves compatibility with earlier versions of Perl, which allowed byte semantics in Perl operations only if none of the program's inputs were marked as being a source of Unicode character data. Such data may come from filehandles, from calls to external programs, from information provided by the system (such as %ENV), or from literals and constants in the source text. The C pragma is primarily a compatibility device that enables recognition of UTF-(8|EBCDIC) in literals encountered by the parser. Note that this pragma is only required while Perl defaults to byte semantics; when character semantics become the default, this pragma may become a no-op. See L. If strings operating under byte semantics and strings with Unicode character data are concatenated, the new string will have character semantics. This can cause surprises: See L, below. You can choose to be warned when this happens. See L<:warnings>. Under character semantics, many operations that formerly operated on bytes now operate on characters. A character in Perl is logically just a number ranging from 0 to 2**31 or so. Larger characters may encode into longer sequences of bytes internally, but this internal detail is mostly hidden for Perl code. See L for more. =head2 Effects of Character Semantics Character semantics have the following effects: =over 4 =item * Strings--including hash keys--and regular expression patterns may contain characters that have an ordinal value larger than 255. If you use a Unicode editor to edit your program, Unicode characters may occur directly within the literal strings in UTF-8 encoding, or UTF-16. (The former requires a BOM or C, the latter requires a BOM.) Unicode characters can also be added to a string by using the C notation. The Unicode code for the desired character, in hexadecimal, should be placed in the braces, after the C. For instance, a smiley face is C. Alternatively, you can use the C notation for characters 0x100 and above. For characters below 0x100 you may get byte semantics instead of character semantics; see L. On EBCDIC machines there is the additional problem that the value for such characters gives the EBCDIC character rather than the Unicode one, thus it is more portable to use C instead. Additionally, you can use the C notation and put the official Unicode character name within the braces, such as C. This automatically loads the L module with the C<:full> and C<:short> options. If you prefer different options for this module, you can instead, before the C, explicitly load it with your desired options; for example, use charnames ':loose'; =item * If an appropriate L is specified, identifiers within the Perl script may contain Unicode alphanumeric characters, including ideographs. Perl does not currently attempt to canonicalize variable names. =item * Regular expressions match characters instead of bytes. "." matches a character instead of a byte. =item * Bracketed character classes in regular expressions match characters instead of bytes and match against the character properties specified in the Unicode properties database. C can be used to match a Japanese ideograph, for instance. =item * Named Unicode properties, scripts, and block ranges may be used (like bracketed character classes) by using the C "matches property" construct and the C negation, "doesn't match property". See L"Unicode Character Properties"> for more details. You can define your own character properties and use them in the regular expression with the C or C construct. See L"User-Defined Character Properties"> for more details. =item * The special pattern C matches a logical character, an "extended grapheme cluster" in Standardese. In Unicode what appears to the user to be a single character, for example an accented C, may in fact be composed of a sequence of characters, in this case a C followed by an accent character. C will match the entire sequence. =item * The C operator translates characters instead of bytes. Note that the C functionality has been removed. For similar functionality see pack('U0', ...) and pack('C0', ...). =item * Case translation operators use the Unicode case translation tables when character input is provided. Note that C, or C in interpolated strings, translates to uppercase, while C, or C in interpolated strings, translates to titlecase in languages that make the distinction (which is equivalent to uppercase in languages without the distinction). =item * Most operators that deal with positions or lengths in a string will automatically switch to using character positions, including C, C, C, C, C, C, C, C, and C. An operator that specifically does not switch is C. Operators that really don't care include operators that treat strings as a bucket of bits such as C, and operators dealing with filenames. =item * The C/C letter C does I change, since it is often used for byte-oriented formats. Again, think C in the C language. There is a new C specifier that converts between Unicode characters and code points. There is also a C specifier that is the equivalent of C/C and properly handles character values even if they are above 255. =item * The C and C functions work on characters, similar to C and C, I C and C. C and C are methods for emulating byte-oriented C and C on Unicode strings. While these methods reveal the internal encoding of Unicode strings, that is not something one normally needs to care about at all. =item * The bit string operators, C, can operate on character data. However, for backward compatibility, such as when using bit string operations when characters are all less than 256 in ordinal value, one should not use C (the bit complement) with characters of both values less than 256 and values greater than 256. Most importantly, DeMorgan's laws (C and C) will not hold. The reason for this mathematical I is that the complement cannot return B the 8-bit (byte-wide) bit complement B the full character-wide bit complement. =item * There is a CPAN module, L<:casing>, which allows you to define your own mappings to be used in C, C, C, C, and C (or their double-quoted string inlined versions such as C). (Prior to Perl 5.16, this functionality was partially provided in the Perl core, but suffered from a number of insurmountable drawbacks, so the CPAN module was written instead.) =back =over 4 =item * And finally, C reverses by character rather than by byte. =back =head2 Unicode Character Properties (The only time that Perl considers a sequence of individual code points as a single logical character is in the C construct, already mentioned above. Therefore "character" in this discussion means a single Unicode code point.) Very nearly all Unicode character properties are accessible through regular expressions by using the C "matches property" construct and the C "doesn't match property" for its negation. For instance, C matches any single character with the Unicode "Uppercase" property, while C matches any character with a General_Category of "L" (letter) property. Brackets are not required for single letter property names, so C is equivalent to C. More formally, C matches any single character whose Unicode Uppercase property value is True, and C matches any character whose Uppercase property value is False, and they could have been written as C and C, respectively. This formality is needed when properties are not binary; that is, if they can take on more values than just True and False. For example, the Bidi_Class (see L"Bidirectional Character Types"> below), can take on several different values, such as Left, Right, Whitespace, and others. To match these, one needs to specify both the property name (Bidi_Class), AND the value being matched against (Left, Right, etc.). This is done, as in the examples above, by having the two components separated by an equal sign (or interchangeably, a colon), like C. All Unicode-defined character properties may be written in these compound forms of C or C, but Perl provides some additional properties that are written only in the single form, as well as single-form short-cuts for all binary properties and certain others described below, in which you may omit the property name and the equals or colon separator. Most Unicode character properties have at least two synonyms (or aliases if you prefer): a short one that is easier to type and a longer one that is more descriptive and hence easier to understand. Thus the "L" and "Letter" properties above are equivalent and can be used interchangeably. Likewise, "Upper" is a synonym for "Uppercase", and we could have written C equivalently as C. Also, there are typically various synonyms for the values the property can be. For binary properties, "True" has 3 synonyms: "T", "Yes", and "Y"; and "False has correspondingly "F", "No", and "N". But be careful. A short form of a value for one property may not mean the same thing as the same short form for another. Thus, for the General_Category property, "L" means "Letter", but for the Bidi_Class property, "L" means "Left". A complete list of properties and synonyms is in L. Upper/lower case differences in property names and values are irrelevant; thus C means the same thing as C or even C. Similarly, you can add or subtract underscores anywhere in the middle of a word, so that these are also equivalent to C. And white space is irrelevant adjacent to non-word characters, such as the braces and the equals or colon separators, so C and C are equivalent to these as well. In fact, white space and even hyphens can usually be added or deleted anywhere. So even C is equivalent. All this is called "loose-matching" by Unicode. The few places where stricter matching is used is in the middle of numbers, and in the Perl extension properties that begin or end with an underscore. Stricter matching cares about white space (except adjacent to non-word characters), hyphens, and non-interior underscores. You can also use negation in both C and C by introducing a caret (^) between the first brace and the property name: C is equal to C. Almost all properties are immune to case-insensitive matching. That is, adding a C regular expression modifier does not change what they match. There are two sets that are affected. The first set is C, C, and C, all of which match C under C matching. And the second set is C, C, and C, all of which match C under C matching. This set also includes its subsets C and C both of which under C matching match C. (The difference between these sets is that some things, such as Roman numerals, come in both upper and lower case so they are C, but aren't considered letters, so they aren't Cs.) The result is undefined if you try to match a non-Unicode code point (that is, one above 0x10FFFF) against a Unicode property. Currently, a warning is raised, and the match will fail. In some cases, this is counterintuitive, as both these fail: chr(0x110000) =~ \p{ASCII_Hex_Digit=True} # Fails. chr(0x110000) =~ \p{ASCII_Hex_Digit=False} # Fails! =head3 B Every Unicode character is assigned a general category, which is the "most usual categorization of a character" (from L). The compound way of writing these is like C (short, C). But Perl furnishes shortcuts in which everything up through the equal or colon separator is omitted. So you can instead just write C. Here are the short and long forms of the General Category properties: Short Long L Letter LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}]) Lu Uppercase_Letter Ll Lowercase_Letter Lt Titlecase_Letter Lm Modifier_Letter Lo Other_Letter M Mark Mn Nonspacing_Mark Mc Spacing_Mark Me Enclosing_Mark N Number Nd Decimal_Number (also Digit) Nl Letter_Number No Other_Number P Punctuation (also Punct) Pc Connector_Punctuation Pd Dash_Punctuation Ps Open_Punctuation Pe Close_Punctuation Pi Initial_Punctuation (may behave like Ps or Pe depending on usage) Pf Final_Punctuation (may behave like Ps or Pe depending on usage) Po Other_Punctuation S Symbol Sm Math_Symbol Sc Currency_Symbol Sk Modifier_Symbol So Other_Symbol Z Separator Zs Space_Separator Zl Line_Separator Zp Paragraph_Separator C Other Cc Control (also Cntrl) Cf Format Cs Surrogate Co Private_Use Cn Unassigned Single-letter properties match all characters in any of the two-letter sub-properties starting with the same letter. C and C are special: both are aliases for the set consisting of everything matched by C, C, and C. =head3 B Because scripts differ in their directionality (Hebrew and Arabic are written right to left, for example) Unicode supplies these properties in the Bidi_Class class: Property Meaning L Left-to-Right LRE Left-to-Right Embedding LRO Left-to-Right Override R Right-to-Left AL Arabic Letter RLE Right-to-Left Embedding RLO Right-to-Left Override PDF Pop Directional Format EN European Number ES European Separator ET European Terminator AN Arabic Number CS Common Separator NSM Non-Spacing Mark BN Boundary Neutral B Paragraph Separator S Segment Separator WS Whitespace ON Other Neutrals This property is always written in the compound form. For example, C matches characters that are normally written right to left. =head3 B The world's languages are written in many different scripts. This sentence (unless you're reading it in translation) is written in Latin, while Russian is written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in Hiragana or Katakana. There are many more. The Unicode Script and Script_Extensions properties give what script a given character is in. Either property can be specified with the compound form like C (short: C), or C (short: C). In addition, Perl furnishes shortcuts for all C