IntroPython2016a/supplements/unicode.html at gh-pages · UWPCE-PythonCert/IntroPython2016a

656 lines (547 loc) · 35.3 KB
<!DOCTYPE html>
<!--[if IE 8]><html class="no-js lt-ie9" lang="en" > <![endif]-->
<!--[if gt IE 8]><!--> <html class="no-js" lang="en" > <!--<![endif]-->
  <meta charset="utf-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>Unicode &mdash; Introduction To Python 1.3 documentation</title>
    <link rel="stylesheet" href="../_static/css/theme.css" type="text/css" />
    <link rel="top" title="Introduction To Python 1.3 documentation" href="../index.html"/>
        <link rel="up" title="Supplemental Materials" href="index.html"/>
        <link rel="next" title="Packages and Packaging" href="packaging.html"/>
        <link rel="prev" title="Installing Nano on Windows" href="install_nano_win.html"/> 
  <script src="../_static/js/modernizr.min.js"></script>
<body class="wy-body-for-nav" role="document">
  <div class="wy-grid-for-nav">
    <nav data-toggle="wy-nav-shift" class="wy-nav-side">
      <div class="wy-side-scroll">
        <div class="wy-side-nav-search">
            <a href="../index.html" class="icon icon-home"> Introduction To Python
          </a>
              <div class="version">
              </div>
<div role="search">
  <form id="rtd-search-form" class="wy-form" action="../search.html" method="get">
    <input type="text" name="q" placeholder="Search docs" />
    <input type="hidden" name="check_keywords" value="yes" />
    <input type="hidden" name="area" value="default" />
        </div>
        <div class="wy-menu wy-menu-vertical" data-spy="affix" role="navigation" aria-label="main navigation">
<li class="toctree-l1"><a class="reference internal" href="../session01.html">Session One: Introductions</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session02.html">Session Two: gitHub, Functions, Booleans and Modules</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session03.html">Session Three: Sequences, Iteration and String Formatting</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session04.html">Session Four: Lists, Iteration, Strings, Dictionaries, Sets</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session05.html">Session Five: Files, Streams &amp; String IO</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session06.html">Session Six: Exceptions, Testing and Advanced Argument Passing</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session07.html">Session Seven: Object Oriented Programming</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session08.html">Session Eight: Object Oriented Programming 2</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session09.html">Session Nine: Object Oriented Programming 3</a></li>
<li class="toctree-l1"><a class="reference internal" href="../session10.html">Session Ten: Functional Programming</a></li>
<ul class="current">
<li class="toctree-l1"><a class="reference internal" href="../exercises/index.html">Exercises</a></li>
<li class="toctree-l1 current"><a class="reference internal" href="index.html">Supplemental Materials</a><ul class="current">
<li class="toctree-l2"><a class="reference internal" href="python_learning_resources.html">Useful Python Learning Resources</a></li>
<li class="toctree-l2"><a class="reference internal" href="python_for_mac.html">Setting up your Mac for Python</a></li>
<li class="toctree-l2"><a class="reference internal" href="python_for_windows.html">Setting up Windows for Python</a></li>
<li class="toctree-l2"><a class="reference internal" href="python_for_linux.html">Setting Up Python For Linux</a></li>
<li class="toctree-l2"><a class="reference internal" href="virtualenv.html">Working with Virtualenv</a></li>
<li class="toctree-l2"><a class="reference internal" href="sublime_as_ide.html">Turning Sublime Text Into a Lightweight Python IDE</a></li>
<li class="toctree-l2"><a class="reference internal" href="shell.html">Shell Customizations for Python Development</a></li>
<li class="toctree-l2"><a class="reference internal" href="git_overview.html">git Overview</a></li>
<li class="toctree-l2"><a class="reference internal" href="install_nano_win.html">Installing Nano on Windows</a></li>
<li class="toctree-l2 current"><a class="current reference internal" href="">Unicode</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#fact-number-1">Fact number 1:</a></li>
<li class="toctree-l3"><a class="reference internal" href="#what-the-heck-is-unicode-anyway">What the heck is Unicode anyway?</a></li>
<li class="toctree-l3"><a class="reference internal" href="#fact-number-2">Fact number 2:</a></li>
<li class="toctree-l3"><a class="reference internal" href="#enter-unicode">Enter Unicode</a></li>
<li class="toctree-l2"><a class="reference internal" href="#mechanics">Mechanics</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#what-are-strings">What are strings?</a></li>
<li class="toctree-l3"><a class="reference internal" href="#str-vs-unicode">str vs unicode</a></li>
<li class="toctree-l3"><a class="reference internal" href="#id1">Unicode</a></li>
<li class="toctree-l3"><a class="reference internal" href="#using-unicode-in-py2">Using unicode in Py2</a></li>
<li class="toctree-l3"><a class="reference internal" href="#encoding-and-decoding">Encoding and Decoding</a></li>
<li class="toctree-l3"><a class="reference internal" href="#unicode-literals">Unicode Literals</a></li>
<li class="toctree-l3"><a class="reference internal" href="#using-unicode">Using Unicode</a></li>
<li class="toctree-l3"><a class="reference internal" href="#using-unicode-everywhere">Using unicode everywhere</a></li>
<li class="toctree-l3"><a class="reference internal" href="#encodings">Encodings</a></li>
<li class="toctree-l3"><a class="reference internal" href="#utf-8">UTF-8</a></li>
<li class="toctree-l3"><a class="reference internal" href="#utf-16">UTF-16</a></li>
<li class="toctree-l3"><a class="reference internal" href="#utf-16-criticism">UTF-16 criticism</a></li>
<li class="toctree-l3"><a class="reference internal" href="#latin-1">Latin-1</a></li>
<li class="toctree-l3"><a class="reference internal" href="#unicode-docs">Unicode Docs</a></li>
<li class="toctree-l3"><a class="reference internal" href="#gotchas-in-python-2">Gotchas in Python 2</a></li>
<li class="toctree-l3"><a class="reference internal" href="#unicode-in-python-3">Unicode in Python 3</a></li>
<li class="toctree-l2"><a class="reference internal" href="#basic-unicode-lab">Basic Unicode LAB</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#challenge-unicode-lab">Challenge Unicode LAB</a></li>
<li class="toctree-l2"><a class="reference internal" href="packaging.html">Packages and Packaging</a></li>
<li class="toctree-l2"><a class="reference internal" href="packaging.html#distributing">Distributing</a></li>
        </div>
      </div>
    <section data-toggle="wy-nav-shift" class="wy-nav-content-wrap">
      <nav class="wy-nav-top" role="navigation" aria-label="top navigation">
        <i data-toggle="wy-nav-top" class="fa fa-bars"></i>
        <a href="../index.html">Introduction To Python</a>
      </nav>
      <div class="wy-nav-content">
        <div class="rst-content">
<div role="navigation" aria-label="breadcrumbs navigation">
  <ul class="wy-breadcrumbs">
    <li><a href="../index.html">Docs</a> &raquo;</li>
          <li><a href="index.html">Supplemental Materials</a> &raquo;</li>
    <li>Unicode</li>
      <li class="wy-breadcrumbs-aside">
            <a href="../_sources/supplements/unicode.txt" rel="nofollow"> View page source</a>
      </li>
          <div role="main" class="document" itemscope="itemscope" itemtype="http://schema.org/Article">
           <div itemprop="articleBody">
  <p>Antoher verison os the Unicode discussion &#8211; I htink there is a bit in here that we&#8217;ll want to use.</p>
<div class="section" id="unicode">
<h1>Unicode<a class="headerlink" href="#unicode" title="Permalink to this headline">¶</a></h1>
<p class="left">I hope you all read this:</p>
<p class="left">The Absolute Minimum Every Software Developer Absolutely,
Positively Must Know About Unicode and Character Sets (No Excuses!)</p>
<p class="left"><a class="reference external" href="http://www.joelonsoftware.com/articles/Unicode.html">http://www.joelonsoftware.com/articles/Unicode.html</a></p>
<p class="left">If not &#8211; go read it!</p>
<div class="section" id="fact-number-1">
<h2>Fact number 1:<a class="headerlink" href="#fact-number-1" title="Permalink to this headline">¶</a></h2>
<p class="centered medium">Everything is made up of bytes</p>
<p>If it&#8217;s on disk or transmitted over a network, it&#8217;s bytes</p>
<p>Python provides some abstractions to make it easier to deal with bytes</p>
<p>Unicode is a biggie</p>
<p>Actually, dealing with numbers rather than bytes is big &#8211; but we take that for granted</p>
<div class="section" id="what-the-heck-is-unicode-anyway">
<h2>What the heck is Unicode anyway?<a class="headerlink" href="#what-the-heck-is-unicode-anyway" title="Permalink to this headline">¶</a></h2>
<ul class="simple">
<li>First there was chaos...<ul>
<li>Different machines used different encodings</li>
<li>Then there was ASCII &#8211; and all was good (7 bit), 127 characters<ul>
<li>(for English speakers, anyway)</li>
<li>But each vendor used the top half (127-255) for different things.<ul>
<li>macroman, Windows 1252, etc...</li>
<li>There is now &#8220;latin-1&#8221;, but still a lot of old files around</li>
<li>Non Western-European languages required totally incompatible 1-byte
encodings</li>
<li>No way to mix languages with different alphabets.</li>
<div class="section" id="fact-number-2">
<h2>Fact number 2:<a class="headerlink" href="#fact-number-2" title="Permalink to this headline">¶</a></h2>
<p class="centered medium">The world needs more than 255 charactors.</p>
<p class="centered">Hello, world!   •   Здравствуй, мир!</p>
<p class="centered">Բարեւ, աշխարհի!   •   !مرحبا ، العالم</p>
<p class="centered">!שלום, עולם   •   여보세요 세계!</p>
<p class="centered">नमस्ते, दुनिया!   •   你好，世界！</p>
<div class="section" id="enter-unicode">
<h2>Enter Unicode<a class="headerlink" href="#enter-unicode" title="Permalink to this headline">¶</a></h2>
<p>The Unicode idea is pretty simple:</p>
<blockquote>
<div><ul class="simple">
<li>one &#8220;code point&#8221; for all characters in all languages</li>
</div></blockquote>
<dl class="docutils">
<dt>But how do you express that in bytes?</dt>
<dd><ul class="first last simple">
<li>Early days: we can fit all the code points in a two byte integer (65536 characters)</li>
<li>Turns out that didn&#8217;t work &#8211; we now need 32 bit integer to hold all of unicode
&#8220;raw&#8221; (UTC-4) &#8211; well we dopnt need that many, but common machines don&#8217;t have
24 bit integers.</li>
<dt>Enter &#8220;encodings&#8221;:</dt>
<dd><ul class="first last simple">
<li>An encoding is a way to map specific bytes to a code point.</li>
<li>Each code point can have one or more bytes.</li>
<div class="section" id="mechanics">
<h1>Mechanics<a class="headerlink" href="#mechanics" title="Permalink to this headline">¶</a></h1>
<div class="section" id="what-are-strings">
<h2>What are strings?<a class="headerlink" href="#what-are-strings" title="Permalink to this headline">¶</a></h2>
<p>Py2 strings are sequences of bytes</p>
<p>Unicode strings are sequences of platonic characters</p>
<p>It&#8217;s almost one code point per character &#8211; but there are complications
with combined characters: accents, etc. (we can ignore those most of the time)</p>
<p>Platonic characters cannot be written to disk or network!</p>
<p>(ANSI: one character == one byte &#8211; so easy!)</p>
<div class="section" id="str-vs-unicode">
<h2>str vs unicode<a class="headerlink" href="#str-vs-unicode" title="Permalink to this headline">¶</a></h2>
<p>Python 2 has two types that let you work with text:</p>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">str</span></code></li>
<li><code class="docutils literal"><span class="pre">unicode</span></code></li>
<p>And two ways to work with binary data:</p>
<ul class="simple">
<li><code class="docutils literal"><span class="pre">str</span></code></li>
<li><code class="docutils literal"><span class="pre">bytes()</span></code>  (and <code class="docutils literal"><span class="pre">bytearray</span></code>)</li>
<p><strong>but:</strong></p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [86]: </span><span class="nb">str</span> <span class="ow">is</span> <span class="nb">bytes</span>
<span class="gh">Out[86]: </span><span class="go">True</span>
</pre></div>
<p><code class="docutils literal"><span class="pre">bytes</span></code> is there for py3 compatibility &#8211; but it&#8217;s good for making your
intentions clear, too.</p>
<div class="section" id="id1">
<h2>Unicode<a class="headerlink" href="#id1" title="Permalink to this headline">¶</a></h2>
<p>The <code class="docutils literal"><span class="pre">unicode</span></code> object lets you work with characters</p>
<p>It has all the same methods as the string object.</p>
<p>&#8220;encoding&#8221; is converting from a unicode object to bytes</p>
<p>&#8220;decoding&#8221; is converting from bytes to a unicode object</p>
<p>(sometimes this feels backwards...)</p>
<div class="section" id="using-unicode-in-py2">
<h2>Using unicode in Py2<a class="headerlink" href="#using-unicode-in-py2" title="Permalink to this headline">¶</a></h2>
<p>Built in functions</p>
<div class="highlight-python"><div class="highlight"><pre><span class="nb">ord</span><span class="p">()</span>
<span class="nb">chr</span><span class="p">()</span>
<span class="nb">unichr</span><span class="p">()</span>
<span class="nb">str</span><span class="p">()</span>
<span class="nb">unicode</span><span class="p">()</span>
</pre></div>
<p>The codecs module</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">codecs</span>
<span class="n">codecs</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span>
<span class="n">codecs</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="n">codecs</span><span class="o">.</span><span class="n">open</span><span class="p">()</span> <span class="c"># better to use ``io.open``</span>
</pre></div>
<div class="section" id="encoding-and-decoding">
<h2>Encoding and Decoding<a class="headerlink" href="#encoding-and-decoding" title="Permalink to this headline">¶</a></h2>
<p>Encoding</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [17]: </span><span class="s">u&quot;this&quot;</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="gh">Out[17]: </span><span class="go">&#39;this&#39;</span>
<span class="gp">In [18]: </span><span class="s">u&quot;this&quot;</span><span class="o">.</span><span class="n">encode</span><span class="p">(</span><span class="s">&#39;utf-16&#39;</span><span class="p">)</span>
<span class="gh">Out[18]: </span><span class="go">&#39;\xff\xfet\x00h\x00i\x00s\x00&#39;</span>
</pre></div>
<p>Decoding</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [99]: </span><span class="k">print</span> <span class="s">&#39;</span><span class="se">\xff\xfe</span><span class="s">.&quot;+&quot;x</span><span class="se">\x00\xb2\x00</span><span class="s">&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">&#39;utf-16&#39;</span><span class="p">)</span>
<span class="go">∮∫x²</span>
</pre></div>
<div class="section" id="unicode-literals">
<h2>Unicode Literals<a class="headerlink" href="#unicode-literals" title="Permalink to this headline">¶</a></h2>
<ol class="arabic simple">
<li>Use unicode in your source files:</li>
<div class="highlight-python"><div class="highlight"><pre><span class="c"># -*- coding: utf-8 -*-</span>
</pre></div>
<ol class="arabic simple" start="2">
<li>escape the unicode characters:</li>
<div class="highlight-python"><div class="highlight"><pre><span class="k">print</span> <span class="s">u&quot;The integral sign: </span><span class="se">\u222B</span><span class="s">&quot;</span>
<span class="k">print</span> <span class="s">u&quot;The integral sign: </span><span class="se">\N{integral}</span><span class="s">&quot;</span>
</pre></div>
<p>Lots of tables of code points online:</p>
<dl class="docutils">
<dt>One example:</dt>
<dd><a class="reference external" href="http://inamidst.com/stuff/unidata/">http://inamidst.com/stuff/unidata/</a></dd>
<p><a class="reference download internal" href="../_downloads/hello_unicode.py"><code class="xref download docutils literal"><span class="pre">hello_unicode.py</span></code></a>.</p>
<div class="section" id="using-unicode">
<h2>Using Unicode<a class="headerlink" href="#using-unicode" title="Permalink to this headline">¶</a></h2>
<p>Use <code class="docutils literal"><span class="pre">unicode</span></code> objects in all your code</p>
<p>Decode on input</p>
<p>Encode on output</p>
<p>Many packages do this for you: <em>XML processing, databases, ...</em></p>
<p><strong>Gotcha:</strong></p>
<p>Python has a default encoding (usually ascii)</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [2]: </span><span class="n">sys</span><span class="o">.</span><span class="n">getdefaultencoding</span><span class="p">()</span>
<span class="gh">Out[2]: </span><span class="go">&#39;ascii&#39;</span>
</pre></div>
<p>The default encoding will get used in unexpected places!</p>
<div class="section" id="using-unicode-everywhere">
<h2>Using unicode everywhere<a class="headerlink" href="#using-unicode-everywhere" title="Permalink to this headline">¶</a></h2>
<p>Python 2.6 and above have a nice feature to make it easier to use unicode everywhere</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">unicode_literals</span>
</pre></div>
<p>After running that line, the <code class="docutils literal"><span class="pre">u''</span></code> is assumed</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [1]: </span><span class="n">s</span> <span class="o">=</span> <span class="s">&quot;this is a regular py2 string&quot;</span>
<span class="gp">In [2]: </span><span class="k">print</span> <span class="nb">type</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="go">&lt;type &#39;str&#39;&gt;</span>
<span class="gp">In [3]: </span><span class="kn">from</span> <span class="nn">__future__</span> <span class="kn">import</span> <span class="n">unicode_literals</span>
<span class="gp">In [4]: </span><span class="n">s</span> <span class="o">=</span> <span class="s">&quot;this is now a unicode string&quot;</span>
<span class="gp">In [5]: </span><span class="nb">type</span><span class="p">(</span><span class="n">s</span><span class="p">)</span>
<span class="gh">Out[5]: </span><span class="go">unicode</span>
</pre></div>
<p>NOTE: You can still get py2 strings from other sources!</p>
<div class="section" id="encodings">
<h2>Encodings<a class="headerlink" href="#encodings" title="Permalink to this headline">¶</a></h2>
<p>What encoding should I use???</p>
<p>There are a lot:</p>
<p><a class="reference external" href="http://en.wikipedia.org/wiki/Comparison_of_Unicode_encodings">http://en.wikipedia.org/wiki/Comparison_of_Unicode_encodings</a></p>
<p>But only a couple you are likely to need:</p>
<ul class="simple">
<li>utf-8  (<code class="docutils literal"><span class="pre">*nix</span></code>)</li>
<li>utf-16  (Windows)</li>
<p>And of course, still the one-bytes ones.</p>
<ul class="simple">
<li>ASCII</li>
<li>Latin-1</li>
<div class="section" id="utf-8">
<h2>UTF-8<a class="headerlink" href="#utf-8" title="Permalink to this headline">¶</a></h2>
<p>Probably the one you&#8217;ll use most &#8211; most common in Internet protocols (xml, JSON, etc.)</p>
<p>Nice properties:</p>
<ul class="simple">
<li>ASCII compatible: first 127 characters are the same</li>
<li>Any ascii string is a utf-8 string</li>
<li>compact for mostly-english text.</li>
<p>Gotchas:</p>
<ul class="simple">
<li>&#8220;higher&#8221; code points may use more than one byte: up to 4 for one character</li>
<li>ASCII compatible means in may work with default encoding in tests &#8211; but then blow up with real data...</li>
<div class="section" id="utf-16">
<h2>UTF-16<a class="headerlink" href="#utf-16" title="Permalink to this headline">¶</a></h2>
<p>Kind of like UTF-8, except it uses at least 16bits (2 bytes) for each character: not ASCII compatible.</p>
<p>But it still needs more than two bytes for some code points, so you still can&#8217;t assume two byte per character.</p>
<p>In C/C++ held in a &#8220;wide char&#8221; or &#8220;wide string&#8221;.</p>
<p>MS Windows uses UTF-16, as does (I think) Java.</p>
<div class="section" id="utf-16-criticism">
<h2>UTF-16 criticism<a class="headerlink" href="#utf-16-criticism" title="Permalink to this headline">¶</a></h2>
<p>There is a lot of criticism on the net about UTF-16 &#8211; it&#8217;s kind of the worst of both worlds:</p>
<ul class="simple">
<li>You can&#8217;t assume every character is the same number of bytes</li>
<li>It takes up more memory than UTF-8</li>
<p><a class="reference external" href="http://programmers.stackexchange.com/questions/102205/should-utf-16-be-considered-harmful">UTF-16 Considered Harmful</a></p>
<p>But to be fair:</p>
<p>Early versions of Unicode: everything fit into two bytes (65536 code points).</p>
<p>MS and Java were fairly early adopters, and it seemed simple enough to just use 2 bytes per character.</p>
<p>When it turned out that 4 bytes were really needed, they were kind of stuck in the middle.</p>
<div class="section" id="latin-1">
<h2>Latin-1<a class="headerlink" href="#latin-1" title="Permalink to this headline">¶</a></h2>
<p><strong>NOT Unicode</strong>:</p>
<p>a 1-byte per char encoding.</p>
<ul class="simple">
<li>Superset of ASCII suitable for Western European languages.</li>
<li>The most common one-byte per char encoding for European text.</li>
<li>Nice property &#8211; every byte value from 0 to 255 is a valid character ( at least in Python )</li>
<ul class="simple">
<li>You will never get an UnicodeDecodeError if you try to decode arbitrary bytes with latin-1.</li>
<li>And it can &#8220;round-trip&#8221; through a unicode object.</li>
<li>Useful if you don&#8217;t know the encoding &#8211; at least it won&#8217;t raise an Exception</li>
<li>Useful if you need to work with combined text+binary data.</li>
<p><a class="reference download internal" href="../_downloads/latin1_test.py"><code class="xref download docutils literal"><span class="pre">latin1_test.py</span></code></a>.</p>
<div class="section" id="unicode-docs">
<h2>Unicode Docs<a class="headerlink" href="#unicode-docs" title="Permalink to this headline">¶</a></h2>
<p>Python Docs Unicode HowTo:</p>
<p><a class="reference external" href="http://docs.python.org/howto/unicode.html">http://docs.python.org/howto/unicode.html</a></p>
<p>&#8220;Reading Unicode from a file is therefore simple&#8221;</p>
<div class="highlight-python"><div class="highlight"><pre><span class="kn">import</span> <span class="nn">io</span>
<span class="n">f</span> <span class="o">=</span> <span class="n">io</span><span class="o">.</span><span class="n">open</span><span class="p">(</span><span class="s">&#39;hello_unicode.py&#39;</span><span class="p">,</span> <span class="n">encoding</span><span class="o">=</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="k">for</span> <span class="n">line</span> <span class="ow">in</span> <span class="n">f</span><span class="p">:</span>
    <span class="k">print</span> <span class="nb">repr</span><span class="p">(</span><span class="n">line</span><span class="p">)</span>
</pre></div>
<dl class="docutils">
<dt>Encodings Built-in to Python:</dt>
<dd><a class="reference external" href="http://docs.python.org/2/library/codecs.html#standard-encodings">http://docs.python.org/2/library/codecs.html#standard-encodings</a></dd>
<div class="section" id="gotchas-in-python-2">
<h2>Gotchas in Python 2<a class="headerlink" href="#gotchas-in-python-2" title="Permalink to this headline">¶</a></h2>
<p>file names, etc:</p>
<p>If you pass in unicode, you get unicode</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [9]: </span><span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="s">&#39;./&#39;</span><span class="p">)</span>
<span class="gh">Out[9]: </span><span class="go">[&#39;hello_unicode.py&#39;, &#39;text.utf16&#39;, &#39;text.utf32&#39;]</span>
<span class="gp">In [10]: </span><span class="n">os</span><span class="o">.</span><span class="n">listdir</span><span class="p">(</span><span class="s">u&#39;./&#39;</span><span class="p">)</span>
<span class="gh">Out[10]: </span><span class="go">[u&#39;hello_unicode.py&#39;, u&#39;text.utf16&#39;, u&#39;text.utf32&#39;]</span>
</pre></div>
<p>Python deals with the file system encoding for you...</p>
<p>But: some more obscure calls don&#8217;t support unicode filenames:</p>
<p><code class="docutils literal"><span class="pre">os.statvfs()</span></code> (<a class="reference external" href="http://bugs.python.org/issue18695">http://bugs.python.org/issue18695</a>)</p>
<p>Exception messages:</p>
<blockquote>
<div><ul class="simple">
<li>Py2 Exceptions use str when they print messages.</li>
<li>But what if you pass in a unicode object?<ul>
<li>It is encoded with the default encoding.</li>
<li><code class="docutils literal"><span class="pre">UnicodeDecodeError</span></code> Inside an Exception????</li>
<p>NOPE: it swallows it instead.</p>
</div></blockquote>
<p><a class="reference download internal" href="../_downloads/unicode_exception_test.py"><code class="xref download docutils literal"><span class="pre">unicode_exception_test.py</span></code></a>.</p>
<div class="section" id="unicode-in-python-3">
<h2>Unicode in Python 3<a class="headerlink" href="#unicode-in-python-3" title="Permalink to this headline">¶</a></h2>
<p>The &#8220;string&#8221; object is unicode.</p>
<p>Py3 has two distinct concepts:</p>
<ul class="simple">
<li>&#8220;text&#8221; &#8211; uses the str object (which is always unicode!)</li>
<li>&#8220;binary data&#8221; &#8211; uses bytes or bytearray</li>
<p>Everything that&#8217;s about text is unicode.</p>
<p>Everything that requires binary data uses bytes.</p>
<p>It&#8217;s all much cleaner.</p>
<p>(by the way, the recent implementations are very efficient...)</p>
<div class="section" id="basic-unicode-lab">
<h1>Basic Unicode LAB<a class="headerlink" href="#basic-unicode-lab" title="Permalink to this headline">¶</a></h1>
<ul class="simple">
<li>Find some nifty non-ascii characters you might use.<ul>
<li>Create a unicode object with them in two different ways.</li>
<li><a class="reference download internal" href="../_downloads/hello_unicode.py"><code class="xref download docutils literal"><span class="pre">here</span></code></a> is one example</li>
<li>Read the contents into unicode objects:</li>
<blockquote>
<div><ul class="simple">
<li><a class="reference download internal" href="../_downloads/ICanEatGlass.utf8.txt"><code class="xref download docutils literal"><span class="pre">ICanEatGlass.utf8.txt</span></code></a></li>
<li><a class="reference download internal" href="../_downloads/ICanEatGlass.utf16.txt"><code class="xref download docutils literal"><span class="pre">ICanEatGlass.utf16.txt</span></code></a></li>
</div></blockquote>
<p>and/ or</p>
<blockquote>
<div><ul class="simple">
<li><a class="reference download internal" href="../_downloads/text.utf8"><code class="xref download docutils literal"><span class="pre">text.utf8</span></code></a></li>
<li><a class="reference download internal" href="../_downloads/text.utf16"><code class="xref download docutils literal"><span class="pre">text.utf16</span></code></a></li>
<li><a class="reference download internal" href="../_downloads/text.utf32"><code class="xref download docutils literal"><span class="pre">text.utf32</span></code></a></li>
</div></blockquote>
<ul class="simple">
<li>write some of the text from the first exercise to file &#8211; read that
file back in.</li>
<p class="left">Reference: <a class="reference external" href="http://inamidst.com/stuff/unidata/">http://inamidst.com/stuff/unidata/</a></p>
<p>NOTE: if your terminal does not support unicode &#8211; you&#8217;ll get an error trying
to print. Try a different terminal or IDE, or google for a solution.</p>
<div class="section" id="challenge-unicode-lab">
<h2>Challenge Unicode LAB<a class="headerlink" href="#challenge-unicode-lab" title="Permalink to this headline">¶</a></h2>
<p>We saw this earlier</p>
<div class="highlight-ipython"><div class="highlight"><pre><span class="gp">In [38]: </span><span class="s">u&#39;to </span><span class="se">\N{INFINITY}</span><span class="s"> and beyond!&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="gt">---------------------------------------------------------------------------</span>
<span class="ne">UnicodeEncodeError</span><span class="g-Whitespace">                        </span>Traceback (most recent call last)
<span class="nn">&lt;ipython-input-38-7f87d44dfcfa&gt;</span> in <span class="ni">&lt;module&gt;</span><span class="nt">()</span>
<span class="ne">----&gt; </span><span class="mi">1</span> <span class="s">u&#39;to </span><span class="se">\N{INFINITY}</span><span class="s"> and beyond!&#39;</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span>
<span class="nn">/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/encodings/utf_8.pyc</span> in <span class="ni">decode</span><span class="nt">(input, errors)</span>
<span class="g-Whitespace">     </span><span class="mi">14</span>
<span class="g-Whitespace">     </span><span class="mi">15</span> <span class="k">def</span> <span class="nf">decode</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">errors</span><span class="o">=</span><span class="s">&#39;strict&#39;</span><span class="p">):</span>
<span class="ne">---&gt; </span><span class="mi">16</span>     <span class="k">return</span> <span class="n">codecs</span><span class="o">.</span><span class="n">utf_8_decode</span><span class="p">(</span><span class="nb">input</span><span class="p">,</span> <span class="n">errors</span><span class="p">,</span> <span class="bp">True</span><span class="p">)</span>
<span class="g-Whitespace">     </span><span class="mi">17</span>
<span class="g-Whitespace">     </span><span class="mi">18</span> <span class="k">class</span> <span class="nc">IncrementalEncoder</span><span class="p">(</span><span class="n">codecs</span><span class="o">.</span><span class="n">IncrementalEncoder</span><span class="p">):</span>
<span class="nn">UnicodeEncodeError: &#39;ascii&#39; codec can&#39;t encode character u&#39;\u221e&#39; in position 3: ordinal not</span> in <span class="ni">range</span><span class="nt">(128)</span>
</pre></div>
<p>But why would you <strong>decode</strong> a unicode object?</p>
<p>And it should be a no-op &#8211; why the exception?</p>
<p>And why &#8216;ascii&#8217;? I specified &#8216;utf-8&#8217;!</p>
<p>It&#8217;s there for backward compatibility</p>
<p>What&#8217;s happening under the hood:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="s">u&#39;to </span><span class="se">\N{INFINITY}</span><span class="s"> and beyond!&#39;</span><span class="o">.</span><span class="n">encode</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">(</span><span class="s">&#39;utf-8&#39;</span><span class="p">)</span>
</pre></div>
<p>It encodes with the default encoding (ascii), then decodes</p>
<p>In this case, it barfs on attempting to encode to &#8216;ascii&#8217;</p>
<p>So never call decode on a unicode object!</p>
<p>But what if someone passes one into a function of yours that&#8217;s expecting
a py2 string?</p>
<p>Type checking and converting &#8211; yeach!</p>
<p>Read:</p>
<p><a class="reference external" href="http://axialcorps.com/2014/03/20/unicode-str/">http://axialcorps.com/2014/03/20/unicode-str/</a></p>
<p>See if you can figure out the decorators:</p>
<p><a class="reference download internal" href="../_downloads/unicodify.py"><code class="xref download docutils literal"><span class="pre">unicodify.py</span></code></a>.</p>
<p>(This is advanced Python JuJu: Aren&#8217;t you glad I didn&#8217;t ask you to write
that yourself?)</p>
           </div>
          </div>
          <footer>
    <div class="rst-footer-buttons" role="navigation" aria-label="footer navigation">
        <a href="packaging.html" class="btn btn-neutral float-right" title="Packages and Packaging" accesskey="n">Next <span class="fa fa-arrow-circle-right"></span></a>
        <a href="install_nano_win.html" class="btn btn-neutral" title="Installing Nano on Windows" accesskey="p"><span class="fa fa-arrow-circle-left"></span> Previous</a>
  <div role="contentinfo">
        &copy; Copyright 2014, Christopher Barker, Cris Ewing, .
  Built with <a href="http://sphinx-doc.org/">Sphinx</a> using a <a href="https://github.com/snide/sphinx_rtd_theme">theme</a> provided by <a href="https://readthedocs.org">Read the Docs</a>. 
        </div>
      </div>
    </section>
    <script type="text/javascript">
        var DOCUMENTATION_OPTIONS = {
            URL_ROOT:'../',
            VERSION:'1.3',
            COLLAPSE_INDEX:false,
            FILE_SUFFIX:'.html',
            HAS_SOURCE:  true
    </script>
      <script type="text/javascript" src="../_static/jquery.js"></script>
      <script type="text/javascript" src="../_static/underscore.js"></script>
      <script type="text/javascript" src="../_static/doctools.js"></script>
      <script type="text/javascript" src="https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML"></script>
    <script type="text/javascript" src="../_static/js/theme.js"></script>
  <script type="text/javascript">
      jQuery(function () {
          SphinxRtdTheme.StickyNav.enable();
  </script>
Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

FilesExpand file tree

unicode.html

Latest commit

History

unicode.html

File metadata and controls
