Expressions with format specifiers in f-strings give wrong code position in AST

@blueyed

BPO	35212
Nosy	@blueyed, @ericvsmith, @asmeurer, @numirias, @yangyangxcf, @DanielNoord
PRs	bpo-35212: fix col_offset in f-strings #19398 bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions #27729
Superseder	bpo-44885: Incorrect exception highlighting for fstring format

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/ericvsmith'
closed_at = <Date 2022-03-16.12:23:28.718>
created_at = <Date 2018-11-11.12:34:00.842>
labels = ['interpreter-core', 'type-bug', '3.7']
title = 'Expressions with format specifiers in f-strings give wrong code position in AST'
updated_at = <Date 2022-03-16.12:23:28.718>
user = 'https://github.com/numirias'

bugs.python.org fields:

activity = <Date 2022-03-16.12:23:28.718>
actor = 'eric.smith'
assignee = 'eric.smith'
closed = True
closed_date = <Date 2022-03-16.12:23:28.718>
closer = 'eric.smith'
components = ['Interpreter Core']
creation = <Date 2018-11-11.12:34:00.842>
creator = 'arminius'
dependencies = []
files = []
hgrepos = []
issue_num = 35212
keywords = ['patch']
message_count = 4.0
messages = ['329672', '353908', '376163', '415337']
nosy_count = 6.0
nosy_names = ['blueyed', 'eric.smith', 'asmeurer', 'arminius', 'fhsxfhsx', 'danielnoord']
pr_nums = ['19398', '27729']
priority = 'normal'
resolution = 'duplicate'
stage = 'resolved'
status = 'closed'
superseder = '44885'
type = 'behavior'
url = 'https://bugs.python.org/issue35212'
versions = ['Python 3.7']

ast.parse() will give a wrong code position for an expression inside an f-string when the expression is using a format specifier.

Compare the trees of f'{a}' and f'{a:b}' :

>>> ast.parse("f'{a}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=3, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=None,
                    ),
                ],
            ),
        ),
    ],
)

>>> ast.parse("f'{a:b}'")
Module(
    body=[
        Expr(
            lineno=1,
            col_offset=0,
            value=JoinedStr(                        col_offset=0,
                lineno=1,
                col_offset=0,
                values=[
                    FormattedValue(
                        lineno=1,
                        col_offset=0,
                        value=Name(lineno=1, col_offset=1, id='a', ctx=Load()),
                        conversion=-1,
                        format_spec=JoinedStr(
                            lineno=1,
                            col_offset=0,
                            values=[Str(lineno=1, col_offset=0, s='b')],
                        ),
                    ),
                ],
            ),
        ),
    ],
)

In both examples, the name "a" is at the same position in the source code, however the parsed ASTs yield two different offsets,

Name(lineno=1, col_offset=3, id='a', ctx=Load())

and

Name(lineno=1, col_offset=1, id='a', ctx=Load())

.

Expected behavior: col_offset=3 for name "a" in both f-strings.

(In my specific use case, this breaks a semantic highlighting plugin for vim.)

There is another instance of incorrectly reported offsets in f-strings, in this case affecting multi-line strings.

Example:

(
f'aaa{foo}bbb'
f'ccc{bar}ddd'
f'eee{baz}fff'
)

Relevant part of the syntax tree:

JoinedStr(
lineno=2,
col_offset=4,
values=[
Str(lineno=2, col_offset=4, s='aaa'),
FormattedValue(
lineno=2,
col_offset=4,
value=Name(lineno=2, col_offset=10, id='foo', ctx=Load()),
conversion=-1,
format_spec=None,
),
Str(lineno=2, col_offset=4, s='bbbccc'),
FormattedValue(
lineno=2,
col_offset=4,
value=Name(lineno=2, col_offset=5, id='bar', ctx=Load()),
conversion=-1,
format_spec=None,
),
Str(lineno=2, col_offset=4, s='dddcccaaaaaaaaaaaaaa'),
FormattedValue(
lineno=2,
col_offset=4,
value=Name(lineno=2, col_offset=5, id='baz', ctx=Load()),
conversion=-1,
format_spec=None,
),
Str(lineno=2, col_offset=4, s='ddd'),
],
),

The reported code position for "foo" is correct (2,10), but not for "bar" (2,5) and "baz" (2,5).

Could you give an estimate as to whether this issue will be looked at any time soon?

The same thing occurs with specifiers like {a!r}.

This was fixed by #27729.

On 3.10:

>>> ast.parse("f'{a}'").body[0].value.values[0].value.col_offset
3
>>> ast.parse("f'{a:b}'").body[0].value.values[0].value.col_offset
3

The other issue also no longer reproduces.

Navigation Menu

Repository navigation

Expressions with format specifiers in f-strings give wrong code position in AST #79393

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Expressions with format specifiers in f-strings give wrong code position in AST #79393

Description

Activity

numirias commented on Nov 11, 2018

numirias commented on Oct 4, 2019

asmeurer commented on Aug 31, 2020

DanielNoord commented on Mar 16, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions