zoukankan      html  css  js  c++  java
  • How many characters can UTF-8 encode?

    How many characters can UTF-8 encode?

    If UTF-8 is 8 bits, does it not mean that there can be only maximum of 256 different characters?

    The first 128 code points are the same as in ASCII. But it says UTF-8 can support up to million of characters?

    How does this work? 

    回答1

    UTF-8 does not use one byte all the time, it's 1 to 4 bytes.

    The first 128 characters (US-ASCII) need one byte.

    The next 1,920 characters need two bytes to encode. This covers the remainder of almost all Latin alphabets, and also Greek, Cyrillic, Coptic, Armenian, Hebrew, Arabic, Syriac and Tāna alphabets, as well as Combining Diacritical Marks.

    Three bytes are needed for characters in the rest of the Basic Multilingual Plane, which contains virtually all characters in common use[12] including most Chinese, Japanese and Korean [CJK] characters.

    Four bytes are needed for characters in the other planes of Unicode, which include less common CJK characters, various historic scripts, mathematical symbols, and emoji (pictographic symbols).

    source: Wikipedia

    回答2

    2,164,864 “characters” can be potentially coded by UTF-8.

    This number is 27 + 211 + 216 + 221, which comes from the way the encoding works:

    • 1-byte chars have 7 bits for encoding 0xxxxxxx (0x00-0x7F)

    • 2-byte chars have 11 bits for encoding 110xxxxx 10xxxxxx (0xC0-0xDF for the first byte; 0x80-0xBF for the second)

    • 3-byte chars have 16 bits for encoding 1110xxxx 10xxxxxx 10xxxxxx (0xE0-0xEF for the first byte; 0x80-0xBF for continuation bytes)

    • 4-byte chars have 21 bits for encoding 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx (0xF0-0xF7 for the first byte; 0x80-0xBF for continuation bytes)

    As you can see this is significantly larger than current Unicode (1,112,064 characters).

    UPDATE

    My initial calculation is wrong because it doesn't consider additional rules. See comments to this answer for more details.

  • 相关阅读:
    通过模板类简单实现Spark的JobServer
    aggregate 和 treeAggregate 的对比
    IntelliJ Idea 常用快捷键列表
    dataframe 数据统计可视化---spark scala 应用
    用java api读取HDFS文件
    .net Core 简单中间件使用
    .Net Core Ocelot网关使用熔断、限流 二
    .Net Core Ocelot网关使用 一
    Docker 问题处理
    CentOS 创建用户
  • 原文地址:https://www.cnblogs.com/chucklu/p/14658296.html
Copyright © 2011-2022 走看看