Incorrect exception highlighting for fstring format

@gvanrossum

BPO	44885
Nosy	@gvanrossum, @ammaraskar, @lysnikolaou, @pablogsal, @isidentical
PRs	bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions #27729 [3.10] bpo-44885: Correct the ast locations of f-strings with format … #27743 [3.9] bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) #27744

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2021-08-12.16:42:19.299>
created_at = <Date 2021-08-11.07:59:02.051>
labels = ['interpreter-core', 'type-bug', '3.11']
title = 'Incorrect exception highlighting for fstring format'
updated_at = <Date 2021-08-12.17:46:49.800>
user = 'https://github.com/pablogsal'

bugs.python.org fields:

activity = <Date 2021-08-12.17:46:49.800>
actor = 'pablogsal'
assignee = 'none'
closed = True
closed_date = <Date 2021-08-12.16:42:19.299>
closer = 'pablogsal'
components = ['Interpreter Core']
creation = <Date 2021-08-11.07:59:02.051>
creator = 'pablogsal'
dependencies = []
files = []
hgrepos = []
issue_num = 44885
keywords = ['patch']
message_count = 8.0
messages = ['399372', '399392', '399397', '399399', '399400', '399469', '399471', '399475']
nosy_count = 5.0
nosy_names = ['gvanrossum', 'ammar2', 'lys.nikolaou', 'pablogsal', 'BTaskaya']
pr_nums = ['27729', '27743', '27744']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue44885'
versions = ['Python 3.11']

Given this code:

print(f"Here is that pesky {xxx/2:.3f} again")

The traceback prints:

Traceback (most recent call last):
  File "/home/pablogsal/github/python/main/lel.py", line 1, in <module>
    print(f"Here is that pesky {xxx/2:.3f} again")
           ^^^
NameError: name 'xxx' is not defined

Removing the formatting part ":.3f" makes it work as expected

(I originally reported this.)

The problem lies here:

cpython/Parser/string_parser.c

Lines 374 to 389 in f66d00f

    
           // The call to fstring_find_expr_location is responsible for finding the column offset 
        
           // the generated AST nodes need to be shifted to the right, which is equal to the number 
        
           // of the f-string characters before the expression starts. In order to correctly compute 
        
           // this offset, strstr gets called in fstring_find_expr_location which only succeeds 
        
           // if curly braces appear before and after the f-string expression (exactly like they do 
        
           // in the f-string itself), hence the following lines. 
        
           str[0] = '{'; 
        
           memcpy(str+1, expr_start, len); 
        
           str[len+1] = '}'; 
        
           str[len+2] = 0; 
        
           int lines, cols; 
        
           if (!fstring_find_expr_location(t, str, &lines, &cols)) { 
        
               PyMem_Free(str); 
        
               return NULL; 
        
           }

The problem is that the strstr call will fail because the string containing the expression doesn't have the formatting part in it, so the offsets never get corrected.

Actually, this has even more problems. Because we are using strstr to find the start of the expression in the parent string, if the expression is repeated the offsets are incorrectly generated:

For example:

print(f"Here is that {xxx} pesky {xxx} again")

This produces:

...
FormattedValue(
value=Name(
id='xxx',
ctx=Load(),
lineno=1,
col_offset=22,
end_lineno=1,
end_col_offset=25),
...
FormattedValue(
value=Name(
id='xxxx',
ctx=Load(),
lineno=1,
col_offset=22,
end_lineno=1,
end_col_offset=25),
...

while

print(f"Here is that {xxx} pesky {xxxx} again")

(different variables) produces:

...
FormattedValue(
value=Name(
id='xxx',
ctx=Load(),
lineno=1,
col_offset=22,
end_lineno=1,
end_col_offset=25),
...
FormattedValue(
value=Name(
id='xxxx',
ctx=Load(),
lineno=1,
col_offset=34,
end_lineno=1,
end_col_offset=38),
...

Ha, we have test that knows is broken:

cpython/Lib/test/test_fstring.py

Line 217 in f66d00f

NOTE: this is currently broken, always sets location of the first

New changeset 8e832fb by Pablo Galindo Salgado in branch 'main':
bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729)
8e832fb

New changeset c28c2e1 by Pablo Galindo Salgado in branch '3.10':
[3.10] bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) (GH-27743)
c28c2e1

New changeset 4b86c9c by Pablo Galindo Salgado in branch '3.9':
[3.9] bpo-44885: Correct the ast locations of f-strings with format specs and repeated expressions (GH-27729) (GH-27744)
4b86c9c

Navigation Menu

Repository navigation

Incorrect exception highlighting for fstring format #89048

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

	// The call to fstring_find_expr_location is responsible for finding the column offset
	// the generated AST nodes need to be shifted to the right, which is equal to the number
	// of the f-string characters before the expression starts. In order to correctly compute
	// this offset, strstr gets called in fstring_find_expr_location which only succeeds
	// if curly braces appear before and after the f-string expression (exactly like they do
	// in the f-string itself), hence the following lines.
	str[0] = '{';
	memcpy(str+1, expr_start, len);
	str[len+1] = '}';
	str[len+2] = 0;

	int lines, cols;
	if (!fstring_find_expr_location(t, str, &lines, &cols)) {
	PyMem_Free(str);
	return NULL;
	}

Incorrect exception highlighting for fstring format #89048

Description

Activity

pablogsal commented on Aug 11, 2021

gvanrossum commented on Aug 11, 2021

pablogsal commented on Aug 11, 2021

pablogsal commented on Aug 11, 2021

pablogsal commented on Aug 11, 2021

pablogsal commented on Aug 12, 2021

pablogsal commented on Aug 12, 2021

pablogsal commented on Aug 12, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions