View source for Tamil All Character Encoding

{{Short description|Alternative encoding for Tamil in the Unicode Private Use Area, modelled as a syllabary}}
'''Tamil All Character Encoding''' ('''TACE16''') is a scheme for [[character encoding|encoding]] the [[Tamil script]] in the [[Private Use Areas|Private Use Area]] of [[Unicode]], implementing a [[syllabary]]-based character model differing from the modified-[[ISCII]] model used by [[Tamil (Unicode block)|Unicode's existing Tamil implementation]].<ref name="TACE16Report">{{Cite report |url=https://www.tamilvu.org/coresite/download/TACE16_Report_English.pdf |title=REPORT ON THE FINAL RECOMMENDATIONS OF THE TASK FORCE ON TACE16 |date=}}</ref><ref name="TNGovernmentTenderDocument">{{Cite web |title=TENDER DOCUMENT for Development of Tamil Fonts and Tamil Keyboard driver for 16-bit encodings (Unicode and TACE16) |url=https://www.tamilvu.org/coresite/download/Teder_Document_for_Tamil_fonts_and_kbd_driver.pdf |website=Tamil Virtual Academy}}</ref>

==Keyboard drivers and fonts==
The keyboard driver for this encoding scheme is available on the [[Tamil Virtual Academy]] website for free.<ref name="KBDFonts">{{Cite web |title=தமிழ் எழுத்துருக்கள் |url=https://www.tamilvu.org/ta/tkbd-index-341488 |website=தமிழ் இணையக் கல்விக்கழகம் TAMIL VIRTUAL ACADEMY}}</ref><ref name="GO">[https://www.tamilvu.org/tkbd/doc_file/it_e_5_2013.pdf Tamil Nadu Government's Order(G.O.), Keyboard Drivers and Fonts] {{webarchive |url=https://archive.today/20231227083124/https://www.tamilvu.org/tkbd/doc_file/it_e_5_2013.pdf |date=27 December 2023}}</ref> It uses [[Tamil 99]] and Tamil Typewriter [[keyboard layout]]s, which are approved by the [[Government of Tamil Nadu]], and maps the input keystrokes to its corresponding characters of the TACE16 scheme.<ref name="TNGovernmentTenderDocument"/> To read files created using TACE16, the corresponding Unicode Tamil fonts are also available on the same website.<ref name="KBDFonts"/><ref name="GO" /> These fonts map glyphs for characters of TACE16 format, but also for the [[Unicode block]] for both [[ASCII]] and [[Tamil script|Tamil characters]], so that they can provide [[backward compatibility]] for reading existing files which are created using the [[Tamil (Unicode block)|Tamil Unicode block]].

==Character set==
All the characters of this encoding scheme are located in the [[Private Use Areas|private use area]] of the [[Basic Multilingual Plane]] of [[Unicode]]'s [[Universal Coded Character Set]].

{{TACE16 Character Set}}

==Comparison of TACE16 to present Tamil Unicode==
{{POV section|date=January 2015}}

===Criticism of the standard Unicode character model for Tamil===
{{Criticism section|date=August 2024}}
[[File:Zero Width (Non)Joiner.svg|right|thumb|Unicode's encoding models for [[Devanagari]], Tamil, [[Kannada script|Kannada]], [[Sinhala script|Sinhala]] and [[emoji]] require use of the invisible [[zero-width joiner]] and [[zero-width non-joiner]] characters.]]
The [[Tamil (Unicode block)|existing Unicode character model for Tamil]] is, like most of [[Indic Unicode]],{{efn|Except for [[Tibetan (Unicode block)|Tibetan]], which uses a different model, and for [[Thai (Unicode block)|Thai]] and related scripts, which use a model derived from [[TIS-620]].}} an [[abugida]]-based model derived from [[ISCII]]. It been criticized for several reasons.<ref name="TACE16Report" />

Unicode represents only 31 Tamil base characters as single [[code point]]s, out of 247 [[grapheme cluster]]s. These include stand-alone vowels, and 23 basic consonant glyphs (which, due to not bearing a [[virama]], nonetheless denote a syllable with both a consonant and a vowel when used on their own). The others are represented as sequences of code points, requiring software support for advanced typography features (such as [[Apple Advanced Typography]], [[Graphite (smart font technology)|Graphite]], or [[OpenType#Advanced typography|OpenType advanced typography]]) to render correctly. This also requires the use of invisible [[zero-width joiner]] and [[zero-width non-joiner]] characters in places where the desired grapheme cluster would otherwise be ambiguous. This complexity can result in security vulnerabilities and ambiguous combinations, can require the use of an exception table to forbid invalid combinations of code points, and can necessitate the use of [[Unicode normalization|string normalization]] to compare two [[string (computer science)|strings]] for equality.

Additionally, since syllables with both a consonant and a vowel form 64 to 70% of Tamil text, an abugida-based model which encodes the consonant and vowel parts as separate code points is inefficient, in terms of how long a string needs to be to contain a given piece of text, in comparison with a syllabary-based model.

Furthermore, ISCII is primarily an encoding of [[Devanagari]], and the ISCII encodings of other [[Brahmic scripts]] (including Tamil) encode characters over the code points of the corresponding characters in Devanagari ISCII. Although Unicode encodes the Brahmic scripts separately from one another, the Tamil block mirrors the ISCII layout (with Devanagari-style character ordering, and reserved space in positions corresponding to Devanagari characters with no Tamil equivalent); consequently, the characters are not in the natural sequence order, and strings [[collation|collated]] by code point (analogous to "[[ASCIIbetical]]" sorting of English text) will not produce the expected sorting order. It requires [[Unicode Collation Algorithm|a complex collation algorithm]] for arranging them in the natural order.

===TACE16 in comparison===
{{Prose|date=January 2024}}
The following data provides a comparison of current Unicode Tamil vs. TACE16 on e-governance and browsing:<ref name="TACE16Report" />{{Better source needed|date=August 2024}}
* TACE16 is efficient over Unicode Tamil by about 5.46 to 11.94 percent for [[Computer data storage|data storage]]{{Clarify|reason=compressed or uncompressed?|date=August 2024}}.
* TACE16 is efficient over Unicode Tamil by about 18.69 to 22.99 percent for sorting index data.
* TACE16 is efficient over Unicode Tamil by about 25.39% when the entire data is Tamil. The default collation sequence followed (binary) while using the code-space values in TACE16 is not as per Tamil dictionary order.
* TACE16 is faster in sorting over Unicode Tamil by about 0.31 to 16.96 percent.
* Index creation on TACE16 data is faster by 36.7% than Unicode.
* For full key search on indexed fields, TACE16 performs better than Unicode Tamil by up to 24.07%. In the case of non-indexed fields, TACE16 performs better than Unicode Tamil by up to 20.9%.
* Rendering of static Tamil data works with TACE16.
TACE16 provides performance improvements in processing time and processing space. It encompasses all of the general Tamil text; it is sequential; and it is unambiguous, with any point corresponding to only one character.<ref name="TACE16Report"/>{{Better source needed|date=August 2024}} The TACE16 system takes fewer [[instruction cycle|instruction cycles]] than Unicode Tamil, and also allows programming based on Tamil grammar{{Clarify|date=August 2024}}, which needs extra framework development in Unicode Tamil.

===Responses by the Unicode Consortium===
The [[Unicode Consortium]] publishes a dedicated [[FAQ]] page on the Tamil script which responds to some of the criticisms. In defence of the ISCII model, the Consortium notes that expert [[linguist]]s, [[typographer]]s and programmers were involved in its development, but acknowledges that compromises were made due to ISCII being constrained to single-byte [[extended ASCII]]. The Consortium points out that Unicode Tamil is now implemented by all major [[operating system]]s and [[web browser]]s, and maintains that it should be used in open interchange contexts, such as online, since tools such as [[search engine]]s would not necessarily be able to identify or interpret a sequence of Unicode private-use code points as Tamil text. However, the Consortium does not object to the use of Private-Use Area schemes, including TACE16, internally to particular processes for which they are useful. In particular, it highlights that both [[markup language|markup]] schemes and alternative encoding schemes may be used by researchers for specialised purposes such as [[natural-language processing]].<ref name="unicodefaq">{{cite web |url=https://www.unicode.org/faq/tamil.html |title=FAQ - Tamil Language and Script |publisher=[[Unicode Consortium]]}}</ref>

Unicode defines normative named-sequences for all Tamil pure consonants and syllables which are represented with sequences of more than one code point, and a dedicated table is published as part of the Unicode Standard listing all of these sequences, in their traditional order, along with their correct glyphs. The Consortium points out that it has been open to accepting proposals for characters for which {{em|no}} existing Unicode representation exists: for example, adding several historical fractions and other symbols as the [[Tamil Supplement]] block in version 12.0 in 2019.<ref name="unicodefaq"/>

Regarding collation, the Consortium argues that obtaining the correct result from sorting by code point is the exception rather than the rule, highlighting that, in unmodified [[ASCIIbetical]] ordering, the uppercase Latin letter ''Z'' sorts before the lowercase letter ''a'', and also highlighting that collation rules often differ by language (see e.g. [[ö]]). Regarding space efficiency, the Consortium argues that storage space and bandwidth taken up by text is usually far overshadowed by other accompanying media such as images and video, and that text content performs well under general-purpose compression methods such as [[Deflate]] (originally from the [[ZIP (file format)|ZIP file format]], standardized in RFC 1951 and integrated in the HTTP protocol as a generic encoding scheme).<ref name="unicodefaq"/>

===Unicode Stability Policy===
When first published (version 1.0.0), Unicode made only limited stability guarantees. As such, the [[Tibetan (obsolete Unicode block)|original Tibetan block]] was deleted in version 1.0.1 (and its space has since been occupied by the [[Myanmar (Unicode block)|Myanmar block]]), and the [[Hangul (obsolete Unicode block)|original block for Korean syllables]] was deleted in version 2.0 (and is now occupied by [[CJK Unified Ideographs Extension A]]). Both the current [[Hangul Syllables (Unicode block)|Hangul Syllables block]] for Korean syllables, and the current [[Tibetan (Unicode block)|Tibetan block]], date back to Unicode 2.0. This was done on the assumption that little or no existing content using Unicode for those writing systems existed,<ref name="rfc2279"/> since it would break compatibility with all existing Unicode content in, and [[input method]]s for, those writing systems. After this so-dubbed "Korean mess", the responsible committees pledged not to make such a compatibility-breaking change ever again,<ref name="rfc2279">{{cite IETF |rfc=2279 |url=https://tools.ietf.org/html/rfc2279 |title=UTF-8, a transformation format of ISO 10646 |first=F. |last=Yergeau |date=1998 |id=RFC 2279 |publisher=[[IETF]] |doi=10.17487/rfc2279|doi-access=free }}</ref> which now forms part of the Unicode Stability Policy.<ref>{{cite web |url=https://www.unicode.org/policies/stability_policy.html |title=Unicode Character Encoding Stability Policies |publisher=Unicode Consortium}}</ref>

This stability policy has been upheld ever since, in spite of demands to re-encode or change the character model for both Tibetan and Korean a second time, made by [[China]] and [[North Korea]] respectively.<ref>{{cite web |url=https://babelstone.co.uk/Blog/2006/09/precomposed-tibetan-part-1-brdarten.html |last=West |first=Andrew |author-link=Andrew West (linguist) |title=Precomposed Tibetan Part 1 : BrdaRten |website=BabelStone |date=2006-09-14}}</ref><ref>{{cite web |url=http://www.unicode.org/wg2/docs/n2674.doc |title=China's Statement of BrdaRten ad hoc |id=[[ISO/IEC JTC 1/SC 2|ISO/IEC JTC1/SC2]]/WG2 N2674 |date=2003-10-20 |author=China National Body}}</ref><ref name="wg2-n2167">{{cite web |url=http://unicode.org/wg2/docs/n2167.doc |first=Kent |last=Karlsson |date=2000-03-02 |title=Comments on DPRK New Work Item proposal on Korean characters |id=[[ISO/IEC JTC 1/SC 2|ISO/IEC JTC1/SC2]]/WG2 N2167}}</ref><ref name="wg2-n2231">{{cite web |url=https://unicode.org/wg2/docs/n2231.pdf |title=DPRK letter on character names and ordering in 10646-1: 2000 |date=2000-07-05 |last=Cho |first=Chun-Hui |id=[[ISO/IEC JTC 1/SC 2|ISO/IEC JTC1/SC2]]/WG2 N2231}}</ref> Likewise in relation to Tamil, the Consortium emphasises the "crucial issue of maintaining the stability of the standard for existing implementations", and argues that "the resulting costs and impact of destabilizing the standard" would substantially outweigh any efficiency benefits in processing speed or storage space.<ref name="unicodefaq"/>

There was a proposal to re-encode Tamil<ref>{{Cite web |last=Anantham |first=A.R.Amaithi |date=2012-01-26 |title=Fresh Encoding Proposals |url=https://www.unicode.org/L2/L2012/12033-tamil-presentation.pdf |website=Unicode}}</ref> that was rejected by Unicode, who said that the re-encoding would be damaging and that there was no convincing evidence that Unicode Tamil encoding is deficient.<ref>{{Cite web |date=2012-03-05 |title=Archive of Notices of Non-Approval |url=http://unicode.org/alloc/nonapprovals.html |website=Unicode}}</ref>

==Alternatives==
===Open-Tamil===
The Open-Tamil project<ref>{{Citation |last1=Annamalai |first1=M. |title=Open-Tamil: Tamil language text processing tools for Python v3 |url=https://pypi.org/project/Open-Tamil/ |access-date=2023-12-31 |last2=Arulalan |first2=T.}}</ref> provides many of the common operations. It claims Level-1 compliance of Tamil text processing without using TACE16, but is written on top of extra programming logic which is needed for Unicode Tamil.
==See also==
* [[Clip font]]
* [[Tamil Script Code for Information Interchange]]
* [https://github.com/ThaniThamizhAkarathiKalanjiyam/AnyTaFont2UTF8 AnyTaFont2UTF8] – an [[open source]] project for all Tamil encoding/font mapping characters.

==Footnotes==
{{notelist}}

==References==
{{Reflist}}
{{Character encoding}}

[[Category:Tamil character-encoding standards]]
[[Category:Character sets]]