Implement PEP 498: Literal String Formatting

@warsaw

BPO	24965
Nosy	@warsaw, @ericvsmith, @Rosuav, @vadmium, @1st1, @JelleZijlstra
Files	pep-498-3.diff pep-498-4.diff pep-498-5.diff pep-498-6.diff pep-498-7.diff pep-498-8.diff pep-498-9.diff pep-498-10.diff

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = <Date 2015-09-19.18:57:44.348>
created_at = <Date 2015-08-30.17:47:00.728>
labels = ['interpreter-core', 'type-feature']
title = 'Implement PEP 498: Literal String Formatting'
updated_at = <Date 2015-09-19.18:57:44.346>
user = 'https://github.com/ericvsmith'

bugs.python.org fields:

activity = <Date 2015-09-19.18:57:44.346>
actor = 'eric.smith'
assignee = 'eric.smith'
closed = True
closed_date = <Date 2015-09-19.18:57:44.348>
closer = 'eric.smith'
components = ['Interpreter Core']
creation = <Date 2015-08-30.17:47:00.728>
creator = 'eric.smith'
dependencies = []
files = ['40421', '40430', '40447', '40480', '40484', '40495', '40502', '40515']
hgrepos = []
issue_num = 24965
keywords = ['patch']
message_count = 39.0
messages = ['249362', '249364', '249365', '249475', '249481', '249810', '250343', '250344', '250354', '250420', '250422', '250465', '250467', '250485', '250493', '250525', '250527', '250528', '250529', '250530', '250531', '250532', '250538', '250541', '250544', '250546', '250548', '250557', '250598', '250821', '250849', '250926', '250927', '250929', '250931', '250970', '251073', '251100', '251102']
nosy_count = 8.0
nosy_names = ['barry', 'eric.smith', 'python-dev', 'Rosuav', 'martin.panter', 'yselivanov', 'JelleZijlstra', 'elvis']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = None
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue24965'
versions = ['Python 3.6']

See PEP-498.

>>> f'New for Python {sys.version.split()[0]}'
'New for Python 3.6.0a0'

One thing I've done in this implementation is to build up a string to pass to str.format(), instead of using the original string. This new string uses positional parameters instead of named parameters.

I had originally proposed to add a string.interpolate() to do the heavy lifting here, which would have meant I could use the original string (as seen in the source code), and not build up a new string and pass it to str.format(). I still might do that, but for now, the approach using str.format() is good enough.

Oops, I didn't really mean to include imporlib.h. Oh, well.

Fixed validate_exprs bug.

Make sure f-strings are identified as literals in error messages.

New changeset a0194ec4195c by Eric V. Smith in branch 'default':
Removed Implementation Limitations section. While the version of the code on http://bugs.python.org/issue24965 has the 255 expression limitation, I'm going to remove this limit. The i18n section was purely speculative. We can worry about it if/when we add i18n and i-strings.
https://hg.python.org/peps/rev/a0194ec4195c

This implements the accepted PEP-498. The only other real change I plan on making is to do dynamic memory allocation when building the expressions that make up a JoinedStr AST node. The code has all of the places to do that already laid out, it's just a matter of hooking it up.

There's one nit where I accept 'f' and 'F', but the PEP just says 'f'. I'm not sure if we should accept the upper case version. I'd think not, but all of the other ones (b, r, and u) do.

I need to do one more scan for memory leaks. I've rearranged some code since the last time I checked for leaks, and that's always a recipe for some sneaking in.

And I need to write some more tests, mostly for syntax errors, but also for a few edge conditions.

Comments welcome.

On Sep 09, 2015, at 11:57 PM, Eric V. Smith wrote:

There's one nit where I accept 'f' and 'F', but the PEP just says 'f'. I'm
not sure if we should accept the upper case version. I'd think not, but all
of the other ones (b, r, and u) do.

I think it should be consistent with the other prefixes. Shouldn't be a big
deal to amend the PEP to describe this.

I discussed it with Guido and added 'F' to the PEP.

This version does dynamic allocation for the expression list, and fixes some memory leaks and early decrefs.

I think it's complete, but I'll take some more passes through it checking for leaks.

The good news is that the performance is pretty good, and finally I have a case where I can beat %-formatting:

$ ./python.bat -mtimeit -s 'a=2' "'%s' % a"
1000000 loops, best of 3: 0.883 usec per loop

$ ./python.bat -mtimeit -s 'a=2' '"{}".format(a)'
1000000 loops, best of 3: 1.16 usec per loop

$ ./python.bat -mtimeit -s 'a=2' 'f"{a}"'
1000000 loops, best of 3: 0.792 usec per loop

This example is mildly contrived, and the performance of f-strings is slightly worse than %-formatting once the f-strings contains both expressions and literals.

I could speed it up significantly (I think) by adding opcodes for 2 things: calling __format__ and joining the strings together. Calling __format__ in an opcode could be a win because I could optimize for known types (str, int, float). Having a join opcode would be a win because I could use _PyUnicodeWriter instead of ''.join.

I'm inclined to check this code in as-is, then optimize it later, if we think it's needed and if I get motivated.

For reference, here's the ast and opcodes for f'a={a}':

>>> ast.dump(ast.parse("f'a={a}'"))
"Module(body=[Expr(value=JoinedStr(values=[Str(s='a='), FormattedValue(value=Name(id='a', ctx=Load()), conversion=0, format_spec=None)]))])"

>>> dis.dis("f'a={a}'")
  1           0 LOAD_CONST               0 ('')
              3 LOAD_ATTR                0 (join)
              6 LOAD_CONST               1 ('a=')
              9 LOAD_NAME                1 (a)
             12 LOAD_ATTR                2 (__format__)
             15 LOAD_CONST               0 ('')
             18 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             21 BUILD_LIST               2
             24 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             27 RETURN_VALUE

Another version of that AST that is better for my digestion:

f'a={a}'

Module(body=[Expr(
value=JoinedStr(values=[
Str(s='a='),
FormattedValue(
value=Name(id='a', ctx=Load()),
conversion=0,
format_spec=None,
),
]),
)])

I have been reading over the test cases, and left a bunch of suggestions for more edge cases etc. Some of them might reflect that I haven’t completely learnt how the inner Python expression syntax, outer string escaping syntax, {{curly bracket}} escaping, automatic concatenation, etc, are all meant to fit together.

I’m actually trying out your patch now. A couple strange errors and observations:

>>> f"{'{'}"  # Why is this allowed in an outer format expression--
'{'
>>> f"{3:{'{'}>10}"  # --but not inside a format specifier?
SyntaxError: nesting of '{' in format specifier is not allowed
>>> opening = "{"; f"{3:{opening}>10}"  # Workaround
'{{{{{{{{{3'
>>> f"{3:{'}'}<10}"  # Error message is very strange!
SyntaxError: missing '}' in format string expression
>>> f"{\x00}"  # It seems this is treated as a null terminator
  File "<fstring>", line 1
    (
    ^
SyntaxError: unexpected EOF while parsing
>>> f"{'s'!\x00:.<10}"  # Default conversion is the null character?
's.........'

On 9/13/2015 12:21 AM, Martin Panter wrote:
>>>> f"{'{'}"  # Why is this allowed in an outer format expression--
> '{'
>>>> f"{3:{'{'}>10}"  # --but not inside a format specifier?

This is me being lazy about detecting recursion. I'll fix it.

>>> f"{\x00}" # It seems this is treated as a null terminator
File "<fstring>", line 1
(
^
SyntaxError: unexpected EOF while parsing

This is a byproduct of using PyParser_ASTFromString. I'm not particularly included to do anything about it. Is there any practical use case?

>>> f"{'s'!\x00:.<10}" # Default conversion is the null character?
's.........'

Yes, that's the default. I'll switch to -1, which I think won't have this issue.

Thanks for the review.

Regarding the null terminator, I was mainly smoke testing your code. :) Maybe it would be too hard to support properly. Although I could imagine someone doing things like this:

>>> d = {b"key\x00": "value"}
>>> f"key={d[b'key\x00']}"  # Oops, escape code at wrong level
  File "<fstring>", line 1
    (d[b'key
           ^
SyntaxError: EOL while scanning string literal
>>> rf"key={d[b'key\x00']}"  # Corrected
'key=value'

I also finished quickly reading over the C code, with a couple more review comments. But I am not familiar with the files involved to thoroughly review.

I rewrote the format_spec parser to recursively call the f-string parser, so any oddness in what's allowed in a format_spec is gone.

It took way longer than I thought, but the code is better for it.

Simplified error handling, fixed 2 memory leaks.

All tests now pass with no leaks.

This should be the final version.

Another strange error message (though maybe the new test changes you mentioned caught this):

>>> f'{3:{10}'  # Actually missing a closing bracket '}'
  File "<stdin>", line 1
SyntaxError: f-string: unexpected '}'

Martin Panter added the comment:

Another strange error message (though maybe the new test changes you mentioned caught this):

>>> f'{3:{10}' # Actually missing a closing bracket '}'
File "<stdin>", line 1
SyntaxError: f-string: unexpected '}'

Yes, I found that one, too. Sorry to waste your time on this, but I literally just finished the test changes 15 minutes ago.

Hopefully the last version.

I left a few more comments on Reitveld. Checking the error messages does make me feel a lot more comfortable though.

Cleaned up the error handling in fstring_expression_compile so it's easier to verify and more robust in the face of future changes.

Added a test for an un-doubled '}', which is an error in a top-level literal (and ends a nested expression). Modified existing tests to match.

I changed the generated code to call:
format(x [, spec])

instead of:
x.__format__(spec)

The reason is that the correct way to call __format__ is actually:
type(x).__format__(x, spec)

That is, the __format__ lookup is done on the type, not the instance. From the earlier example, the disassembled code is now:

>>> dis.dis("f'a={a}'")
  1           0 LOAD_CONST               0 ('')
              3 LOAD_ATTR                0 (join)
              6 LOAD_CONST               1 ('a=')
              9 LOAD_GLOBAL              1 (format)
             12 LOAD_NAME                2 (a)
             15 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             18 BUILD_LIST               2
             21 CALL_FUNCTION            1 (1 positional, 0 keyword pair)
             24 RETURN_VALUE

The simplest way to make the lookup correctly is just to call format() itself, which does the right thing.

I still have a concept of adding opcodes to handle FormattedValue and JoinedStr nodes, but that's an optimization for later, if ever.

New changeset a10d37f04569 by Eric V. Smith in branch 'default':
Issue bpo-24965: Implement PEP-498 "Literal String Interpolation". Documentation is still needed, I'll open an issue for that.
https://hg.python.org/cpython/rev/a10d37f04569

Documentation task added as issue bpo-25179.

Thanks to Martin for the great code reviews.

Navigation Menu

Repository navigation

Implement PEP 498: Literal String Formatting #69153

14 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Implement PEP 498: Literal String Formatting #69153

Description

Activity

ericvsmith commented on Aug 30, 2015

ericvsmith commented on Aug 30, 2015

ericvsmith commented on Aug 30, 2015

ericvsmith commented on Sep 1, 2015

ericvsmith commented on Sep 1, 2015

python-dev commented on Sep 4, 2015

ericvsmith commented on Sep 9, 2015

warsaw commented on Sep 10, 2015

ericvsmith commented on Sep 10, 2015

ericvsmith commented on Sep 10, 2015

ericvsmith commented on Sep 10, 2015

vadmium commented on Sep 11, 2015

14 remaining items

vadmium commented on Sep 13, 2015

ericvsmith commented on Sep 13, 2015

vadmium commented on Sep 14, 2015

ericvsmith commented on Sep 16, 2015

ericvsmith commented on Sep 16, 2015

vadmium commented on Sep 18, 2015

ericvsmith commented on Sep 18, 2015

ericvsmith commented on Sep 18, 2015

vadmium commented on Sep 18, 2015

ericvsmith commented on Sep 18, 2015

ericvsmith commented on Sep 19, 2015

python-dev commented on Sep 19, 2015

ericvsmith commented on Sep 19, 2015

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions