Python's `str.__repr__()`
Sometimes software is written using whatever built-ins you find in your programming language of choice.
This is usually great!
However, it can happen that you depend on the precise semantics of those built-ins.
This can be a problem if those semantics become important to your software and you need to port it to another programming language.
This story is about Python and its str.__repr()__ function.
The piece of software I was helping port to OCaml was constructing a hash from the string representation of a tuple. The gist of it was basically this:
def get_id(x):
id = (x.get_unique_string(), x.path, x.name)
return myhash(str(id))
In other words it's a Python tuple consisting of mostly strings but also a PosixPath object.
The way str() works is it calls the __str__() method on the argument objects (or otherwise repr(x)).
For Python tuples the __str__() method seems to print the result of repr() on each elemenet separated by a comma and a space and surrounded by parenthesis.
So good so far.
If we can precisely emulate repr() on strings and PosixPath it's easy.
In the case of PosixPath it's really just 'PosixPath('+repr(str(path))+')';
so in that case it's down to repr() on strings - which is str.__repr__(),
There had been a previous attempt at this that would use OCaml's string escape functions and surround the string with single quotes (').
This works for some cases, but not if the string has a double quote (").
In that case OCaml would escape the double quote with a backslash (\") while python would not escape it.
So a regular expression substitution was added to replace the escape sequence with just a double quote.
This pattern of finding small differences between Python and OCaml escaping had been repeated,
and eventually I decided to take a more rigorous approach to it.
What is a string?
First of all, what is a string? In Python? And in OCaml?
In OCaml a string is just a sequence of bytes.
Any bytes, even NUL bytes.
There is no concept of unicode in OCaml strings.
In Python there is the str type which is a sequence of Unicode code points[1].
I can recommend reading Daniel Bünzli's minimal introduction to Unicode.
Already here there is a significant gap in semantics between Python and OCaml.
For many practical purposes we can get away with using the OCaml string type and treating it as a UTF-8 encoded Unicode string.
This is what I will do as in both the Python code and the OCaml code the data being read is a UTF-8 (or often only the US ASCII subset) encoded string.
What does a string literal look like?
OCaml
I will not dive too deep into the details of OCaml string literals, and focus mostly on how they are escaped by the language built-ins (String.escaped, Printf.printf "%S").
Normal printable ASCII is printed as-is.
That is, letters, numbers and other symbols except for backslash and double quote.
There are the usual escape sequences \n, \t, \r, \" and \\.
Any byte value can be represented with decimal notation \032 or octal notation '\o040' or hexadecimal notation \x20.
The escape functions in OCaml has a preference for the decimal notation over the hexadecimal notation.
Finally I also want to mention the Unicode code point escape sequence \u{3bb} which represents the UTF-8 encoding of U+3BB.
While the escape functions do not use it, it will become handy later on.
Illegal escape sequences (escape sequences that are not recognized) will emit a warning but otherwise result in the escape sequence as-is.
It is common to compile OCaml programs with warnings-as-errors, however.
Python
Python has a number of different string literals and string-like literals.
They all use single quote or double quote to delimit the string (or string-like) literals.
There is a preference towards single quotes in str.__repr__().
You can also triple the quotes if you like to write a string that uses a lot of both quote characters.
This format is not used by str.__repr__() so I will not cover it further, but you can read about it in the Python reference manual.
The string literal can optionally have a prefix character that modifies what type the string literal is and how its content is interpreted.
The r-prefixed strings are called raw strings.
That means backslash escape sequences are not interpreted.
In my experiments they seem to be quasi-interpreted, however!
The string r"\" is considered unterminated!
But r"\"" is fine as is interpreted as '\\"'[2].
Why this is the case I have not found a good explanation for.
The b-prefixed strings are bytes literals.
This is close to OCaml strings.
Finally there are the unprefixed strings which are str literals.
These are the ones we are most interested in.
They use the usual escape \[ntr"] we know from OCaml as well as \'.
\032 is octal notation and \x20 is hexadecimal notation.
There is as far as I know no decimal notation.
The output of str.__repr__() uses the hexadecimal notation over the octal notation.
As Python strings are Unicode code point sequences we need more than two hexadecimal digits to be able to represent all valid "characters".
Thus there are the longer \u0032 and the longest \U00000032.
Intermezzo
While studying Python string literals I discovered several odd corners of the syntax and semantics besides the raw string quasi-escape sequence mentioned earlier.
One fact is that Python doesn't have a separate character or Unicode code point type.
Instead, a character is a one element string.
This leads to some interesting indexing shenanigans: "a"[0][0][0] == "a".
Furthermore, strings separated by spaces only are treated as one single concatenated string: "a" "b" "c" == "abc".
These two combined makes it possible to write this unusual snippet: "a" "b" "c"[0] == "a"!
For byte sequences, or b-prefixed strings, things are different.
Indexing a bytes object returns the integer value of that byte (or character):
>>> b"a"[0]
97
>>> b"a"[0][0]
<stdin>:1: SyntaxWarning: 'int' object is not subscriptable; perhaps you missed a comma?
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: 'int' object is not subscriptable
For strings \x32 can be said to be shorthand for "\u0032" (or "\u00000032").
But for bytes "\x32" != "\u0032"!
Why is this?!
Well, bytes is a byte sequence and b"\u0032" is not interpreted as an escape sequence and is instead silently treated as b"\\u0032"!
Writing "\xff".encode() which encodes the string "\xff" to UTF-8 is not the same as b"\xff".
The bytes "\xff" consist of a single byte with decimal value 255,
and the Unicode wizards reading will know that the Unicode code point 255 (or U+FF) is encoded in two bytes in UTF-8.
Where is the Python code?
Finding the implementation of str.__repr__() turned out to not be so easy.
In the end I asked on the Internet and got a link to cpython's Objects/unicodeobject.c.
And holy cow!
That's some 160 lines of C code with two loops, a switch statement and I don't know how many chained and nested if statements!
Meanwhile the OCaml implementation is a much less daunting 52 lines of which about a fifth is a long comment.
It also has two loops which each contain one much more tame match expression (roughly a C switch statement).
In both cases they first loop over the string to compute the size of the output string.
The Python implementation also counts the number of double quotes and single quotes as well as the highest code point value.
The latter I'm not sure why they do, but my guess it's so they can choose an efficient internal representation.
Then the Python code decides what quote character to use with the following algorithm:
Does the string contain single quotes but no double quotes? Then use double quotes. Otherwise use single quotes.
Then the output size estimate is adjusted with the number of backslashes to escape the quote character chosen and the two quotes surrounding the string.
Already here it's clear that a regular expression substitution is not enough by itself to fix OCaml escaping to be Python escaping.
My first step then was to implement the algorithm only for US ASCII.
This is simpler as we don't have to worry much about Unicode, and I could implement it relatively quickly.
The first 32 characters and the last US ASCII character (DEL or \x7f) are considered non-printable and must be escaped.
I then wrote some simple tests by hand.
Then I discovered the OCaml py library which provides bindings to Python from OCaml.
Great! This I can use to test my implementation against Python!
How about Unicode?
For the non-ascii characters (or code points rather) they are either considered printable or non-printable.
For now let's look at what that means for the output.
A printable character is copied as-is.
That is, there is no escaping done.
Non-printable characters must be escaped, and python wil use \xHH, \uHHHH or \UHHHHHHHH depending on how many hexadecimal digits are necessary to represent the code point.
That is, the latin-1 subset of ASCII (0x80-0xff) can be represented using \xHH and neither \u00HH nor \U000000HH will be used etc.
What is a printable Unicode character?
In the cpython function mentioned earlier they use the function Py_UNICODE_ISPRINTABLE.
I had a local clone of the cpython git repository where I ran git grep Py_UNICODE_ISPRINTABLE to find information about it.
In unicode.rst I found a documentation string for the function that describes it to return false if the character is nonprintable with the definition of nonprintable as the code point being in the categories "Other" or "Separator" in the Unicode character database with the exception of ASCII space (U+20 or ).
What are those "Other" and "Separator" categories?
Further searching for the function definition we find in Include/cpython/unicodeobject.h the definition.
Well, we find #define Py_UNICODE_ISPRINTABLE(ch) _PyUnicode_IsPrintable(ch).
On to git grep _PyUnicode_IsPrintable then.
That function is defined in Objects/unicodectype.c.
/* Returns 1 for Unicode characters to be hex-escaped when repr()ed,
0 otherwise.
All characters except those characters defined in the Unicode character
database as following categories are considered printable.
* Cc (Other, Control)
* Cf (Other, Format)
* Cs (Other, Surrogate)
* Co (Other, Private Use)
* Cn (Other, Not Assigned)
* Zl Separator, Line ('\u2028', LINE SEPARATOR)
* Zp Separator, Paragraph ('\u2029', PARAGRAPH SEPARATOR)
* Zs (Separator, Space) other than ASCII space('\x20').
*/
int _PyUnicode_IsPrintable(Py_UCS4 ch)
{
const _PyUnicode_TypeRecord *ctype = gettyperecord(ch);
return (ctype->flags & PRINTABLE_MASK) != 0;
}
Ok, now we're getting close to something.
Searching for PRINTABLE_MASK we find in Tools/unicode/makeunicodedata.py the following line of code:
if char == ord(" ") or category[0] not in ("C", "Z"):
flags |= PRINTABLE_MASK
So the algorithm is really if the character is a space character or if its Unicode general category doesn't start with a C or Z.
This can be implemented in OCaml using the uucp library as follows:
let py_unicode_isprintable uchar =
(* {[if char == ord(" ") or category[0] not in ("C", "Z"):
flags |= PRINTABLE_MASK]} *)
Uchar.equal uchar (Uchar.of_char ' ')
||
let gc = Uucp.Gc.general_category uchar in
(* Not those categories starting with 'C' or 'Z' *)
match gc with
| `Cc | `Cf | `Cn | `Co | `Cs | `Zl | `Zp | `Zs -> false
| `Ll | `Lm | `Lo | `Lt | `Lu | `Mc | `Me | `Mn | `Nd | `Nl | `No | `Pc | `Pd
| `Pe | `Pf | `Pi | `Po | `Ps | `Sc | `Sk | `Sm | `So ->
true
After implementing unicode I expanded the tests to generate arbitrary OCaml strings and compare the results of calling my function and Python's str.__repr__() on the string.
Well, that didn't go quite well.
OCaml strings are just any byte sequence, and ocaml-py expects it to be a UTF-8 encoded string and fails on invalid UTF-8.
Then in qcheck you can "assume" a predicate which means if a predicate doesn't hold on the generated value then the test is skipped for that input.
So I implement a simple verification of UTF-8.
This is far from optimal because qcheck will generate a lot of invalid utf-8 strings.
The next test failure is some unassigned code point.
So I add to py_unicode_isprintable a check that the code point is assigned using Uucp.Age.age uchar <> `Unassigned.
Still, qcheck found a case I hadn't considered: U+61D.
My python version (Python 3.9.2 (default, Feb 28 2021, 17:03:44)) renders this as '\u061' while my OCaml function prints it as-is.
In other words my implementation considers it printable while python does not.
I try to enter this Unicode character in my terminal, but nothing shows up.
Then I look it up and its name is ARABIC END OF TEXT MARKER.
The general category according to uucp is `Po.
So this should be a printable character‽
After being stumped by this for a while I get the suspicion it may be dependent on the Python version.
I am still on Debian 11 and my Python version is far from being the latest and greatest.
I ask someone with a newer Python version to write '\u061d' in a python session.
And 'lo! It prints something that looks like ''!
Online I figure out how to get the unicode version compiled into Python:
>>> import unicodedata
>>> unicodedata.unidata_version
'13.0.0'
Aha! And with uucp we find that the unicode version that introduced U+61D to be 14.0:
# Uucp.Age.age (Uchar.of_int 0x61D);;
- : Uucp.Age.t = `Version (14, 0)
My reaction is this is seriously some ungodly mess we are in. Not only is the code that instigated this journey highly dependent on Python-specifics - it's also dependent on the specific version of unicode and thus the version of Python!
I modify our py_unicode_isprintable function to take an optional ?unicode_version argument and replace the "is this unassigned?" check with the following snippet:
let age = Uucp.Age.age uchar in
(match (age, unicode_version) with
| `Unassigned, _ -> false
| `Version _, None -> true
| `Version (major, minor), Some (major', minor') ->
major < major' || (major = major' && minor <= minor'))
Great! I modify the test suite to first detect the unicode version python uses and then pass that version to the OCaml function. Now I can't find anymore failing test cases!
Epilogue
What can we learn from this?
It is easy to say in hindsight that a different representation should have been chosen.
However, arriving at this insight takes time.
The exact behavior of str.__repr__() is poorly documented.
Reaching my understanding of str.__repr__() took hours of research and reading the C implementation.
It often doesn't seem to be worth it to spend so much time on research for a small function.
Technical debt is a real thing and often hard to predict.
Below is the output of help(str.__repr__):
__repr__(self, /)
Return repr(self)
Language and (standard) library designers could consider whether the slightly nicer looking strings are worth the added complexity users eventually are going to rely on - inadvertently or not. I do think strings and bytes in Python are a bit too complex. It is not easy to get a language lawyer[3] level understanding. In my opinion it is a mistake to not at least print a warning if there are illegal escape sequences - especially considering there are escape sequences that are valid in one string literal but not another.
Unfortunately it is often the case that to get a precise specification it is necessary to look at the implementation. For testing your implementation hand-written tests are good. Testing against the original implementation is great, and if combined with property-based testing or fuzzing you may find failing test cases you couldn't dream up! I certainly didn't see it coming that the output depends on the Unicode version. As is said, testing can only show the presence of bugs, but with a, in a sense, limited domain like this function you can get pretty close to showing absence of bugs.
I enjoyed working on this.
Sure, it was frustrating and at times I discovered some ungodly properties, but it's a great feeling to study and understand something at a deeper level.
It may be the last time I need to understand Python's str.__repr__() this well, but if I do I now have the OCaml code and this blog post to reread.
If you are curious to read the resulting code you may find it on github at github.com/reynir/python-str-repr. I have documented the code to make it more approachable and maintainable by others. Hopefully it is not something that you need, but in case it is useful to you it is licensed under a permissive license.
If you have a project in OCaml or want to port something to OCaml and would like help from me and my colleagues at Robur please get in touch with us and we will figure something out.