localedata: Fix several issues with the set of characters considered 0-width [BZ #31370]

= `Default_Ignorable_Code_Point`s should have width 0 =

Unicode specifies (https://www.unicode.org/faq/unsup_char.html#3) that characters
with the `Default_Ignorable_Code_Point` property

> should be rendered as completely invisible (and non advancing, i.e. “zero width”),
if not explicitly supported in rendering.

Hence, `wcwidth()` should give them all a width of 0, with two exceptions:

- the soft hyphen (U+00AD SOFT HYPHEN) is assigned width 1 by longstanding precedent
- U+115F HANGUL CHOSEONG FILLER needs a carveout
  due to the unique behavior of the conjoining Korean jamo characters.
  One composed Hangul "syllable block" like 퓛
  is made up of two to three individual component characters, or "jamo".
  These are all assigned an `East_Asian_Width` of `Wide`
  by Unicode, which would normally mean they would all be assigned
  width 2 by glibc; a combination of (leading choseong jamo) +
  (medial jungseong jamo) + (trailing jongseong jamo) would then have width 2 + 2 + 2 = 6.
  However, glibc (and other wcwidth implementations) special-cases jungseong and jongseong,
  assigning them all width 0,
  to ensure that the complete block has width 2 + 0 + 0 = 2 as it should.
  U+115F is meant for use in syllable blocks
  that are intentionally missing a leading jamo;
  it must be assigned a width of 2 even though it has no visible display
  to ensure that the complete block has width 2.

However, `wcwidth()` currently (before this patch)
incorrectly assigns non-zero width to
U+3164 HANGUL FILLER and U+FFA0 HALFWIDTH HANGUL FILLER;
this commit fixes that.

Unicode spec references:
- Hangul:  §3.12 https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf#G24646 and
  §18.6 https://www.unicode.org/versions/Unicode15.0.0/ch18.pdf#G31028
- `Default_Ignorable_Code_Point`: §5.21 https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf#G40095.

= Non-`Default_Ignorable_Code_Point` format controls should be visible =

The Unicode Standard, §5.21 - Characters Ignored for Display
(https://www.unicode.org/versions/Unicode15.0.0/ch05.pdf#G40095)
says the following:

> A small number of format characters (General_Category = Cf )
> are also not given the Default_Ignorable_Code_Point property.
> This may surprise implementers, who often assume
> that all format characters are generally ignored in fallback display.
> The exact list of these exceptional format characters
> can be found in the Unicode Character Database.
> There are, however, three important sets of such format characters to note:
>
> - prepended concatenation marks
> - interlinear annotation characters
> - Egyptian hieroglyph format controls
>
> The prepended concatenation marks always have a visible display.
> See “Prepended Concatenation Marks” in [*Section 23.2, Layout Controls*](https://www.unicode.org/versions/Unicode15.1.0/ch23.pdf#M9.35858.HeadingBreak.132.Layout.Controls)
> for more discussion of the use and display of these signs.
>
> The other two notable sets of format characters that exceptionally are not ignored
> in fallback display consist of the interlinear annotation characters,
> U+FFF9 INTERLINEAR ANNOTATION ANCHOR through
> U+FFFB INTERLINEAR ANNOTATION TERMINATOR,
> and the Egyptian hieroglyph format controls,
> U+13430 EGYPTIAN HIEROGLYPH VERTICAL JOINER through
> U+1343F EGYPTIAN HIEROGLYPH END WALLED ENCLOSURE.
> These characters should have a visible glyph display for fallback rendering,
> because if they are not displayed,
> it is too easy to misread the resulting displayed text.
> See “Annotation Characters” in [*Section 23.8, Specials*](https://www.unicode.org/versions/Unicode15.1.0/ch23.pdf#M9.21335.Heading.133.Specials),
> as well as [*Section 11.4, Egyptian Hieroglyphs*](https://www.unicode.org/versions/Unicode15.1.0/ch11.pdf#M9.73291.Heading.1418.Egyptian.Hieroglyphs)
> for more discussion of the use and display of these characters.

glibc currently correctly assigns non-zero width to the prepended concatenation marks,
but it incorrectly gives zero width to the interlinear annotation characters
(which a generic terminal cannot interpret)
and the Egyptian hieroglyph format controls
(which are not widely supported in rendering implementations at present).
This commit fixes both these issues as well.

= Derive Hangul syllable type from Unicode data =

Previosuly, the jungseong and jongseong jamo ranges
were hard-coded into the script. With this commit, they are instead parsed
from the HangulSyllableType.txt data file published by Unicode.
This does not affect the end result.

Signed-off-by: Jules Bertholet <julesbertholet@quoi.xyz>

1 files changed, 25 insertions, 9 deletions

diff --git a/localedata/charmaps/UTF-8 b/localedata/charmaps/UTF-8
index c9e89f03da..bc694ec4c8 100644
--- a/localedata/charmaps/UTF-8
+++ b/localedata/charmaps/UTF-8
@@ -49856,13 +49856,27 @@ CHARMAP
 END CHARMAP
 
 % Character width according to Unicode 15.1.0.
-% - Default width is 1.
+% Width is determined by the following rules, in order of decreasing precedence:
+% - U+00AD SOFT HYPHEN has width 1, as a special case for compatibility (https://archive.is/b5Ck).
+% - U+115F HANGUL CHOSEONG FILLER has width 2.
+%   This character stands in for an intentionally omitted leading consonant
+%   in a Hangul syllable block; as such it must be assigned width 2 despite its lack
+%   of visible display to ensure that the complete block has the correct width.
+%   (See below for more information on Hangul syllables.)
+% - Combining jungseong and jongseong Hangul jamo have width 0; generated from
+%   "grep '^[^;]*;[VT]' HangulSyllableType.txt".
+%   One composed Hangul "syllable block" like 퓛 is made up of
+%   two to three individual component characters called "jamo".
+%   The complete block must have total width 2;
+%   to achieve this, we assign a width of 2 to leading "choseong" jamo,
+%   and of 0 to medial vowel "jungseong" and trailing "jongseong" jamo.
+% - Non-spacing and enclosing marks have width 0; generated from
+%   "grep -E '^[^;]*;[^;]*;(Mn|Me);' UnicodeData.txt".
+% - "Default_Ignorable_Code_Point"s have width 0; generated from
+%   "grep '^[^;]*;\s*Default_Ignorable_Code_Point' DerivedCoreProperties.txt".
 % - Double-width characters have width 2; generated from
-%        "grep '^[^;]*;\s*[WF]' EastAsianWidth.txt"
-% - Non-spacing characters have width 0; generated from PropList.txt or
-%   "grep '^[^;]*;[^;]*;[^;]*;[^;]*;NSM;' UnicodeData.txt"
-% - Format control characters have width 0; generated from
-%   "grep '^[^;]*;[^;]*;Cf;' UnicodeData.txt"
+%   "grep '^[^;]*;[WF]' EastAsianWidth.txt".
+% - Default width for all other characters is 1.
 WIDTH
 <U0300>...<U036F>	0
 <U0483>...<U0489>	0
@@ -50083,7 +50097,9 @@ WIDTH
 <U3099>...<U309A>	0
 <U309B>...<U30FF>	2
 <U3105>...<U312F>	2
-<U3131>...<U318E>	2
+<U3131>...<U3163>	2
+<U3164>	0
+<U3165>...<U318E>	2
 <U3190>...<U31E3>	2
 <U31EF>...<U321E>	2
 <U3220>...<UA48C>	2
@@ -50138,8 +50154,8 @@ WIDTH
 <UFE68>...<UFE6B>	2
 <UFEFF>	0
 <UFF01>...<UFF60>	2
+<UFFA0>	0
 <UFFE0>...<UFFE6>	2
-<UFFF9>...<UFFFB>	0
 <U000101FD>	0
 <U000102E0>	0
 <U00010376>...<U0001037A>	0
@@ -50240,7 +50256,7 @@ WIDTH
 <U00011F36>...<U00011F3A>	0
 <U00011F40>	0
 <U00011F42>	0
-<U00013430>...<U00013440>	0
+<U00013440>	0
 <U00013447>...<U00013455>	0
 <U00016AF0>...<U00016AF4>	0
 <U00016B30>...<U00016B36>	0