ast.unparse is not working correctly for strings with multiple quotes

Bug report

CPython versions tested on:

3.11

Operating systems tested on:

macOS

A clear and concise description of the bug:

During the implementation of #108553, I discovered this bug in the original implementation.

Due to the new syntax, the fix for 3.12 above cannot be reused for 3.11 and below.

import ast

s = "f\"'''{1}\\\"\\\"\\\"\" "
print(s)
eval(s)

s = ast.parse(s)
s1 = ast.unparse(s)
print(s1)
eval(s1)

output:

f"'''{1}\"\"\"" 
f''''{1}"""'
Traceback (most recent call last):
  File "/Users/xxx/work/cpython/a.py", line 10, in <module>
    eval(s1)
  File "<string>", line 1
    f''''{1}"""'
    ^
SyntaxError: unterminated triple-quoted string literal (detected at line 1)

Linked PRs

[3.11] pythongh-108843: fix ast.unparse for f-string with many quotes

Nice spot, I've opened a PR that fixes on 3.11 over here: #108980

@sunmy2019

@sunmy2019 oh hmm, I got a little confused by the history here, looks like this isn't actually fixed on main?

A similar patch should make it work on main, something like the following:

Details

diff --git a/Lib/ast.py b/Lib/ast.py
index 17ec7ff6f8..f706d1f282 100644
--- a/Lib/ast.py
+++ b/Lib/ast.py
@@ -1236,17 +1236,35 @@ def visit_JoinedStr(self, node):
 
         new_fstring_parts = []
         quote_types = list(_ALL_QUOTES)
+        fallback_to_repr = False
         for value, is_constant in fstring_parts:
             if is_constant:
-                value, quote_types = self._str_literal_helper(
+                value, new_quote_types = self._str_literal_helper(
                     value,
                     quote_types=quote_types,
                     escape_special_whitespace=True,
                 )
+                if set(new_quote_types).isdisjoint(quote_types):
+                    fallback_to_repr = True
+                    break
+                quote_types = new_quote_types
             elif "\n" in value:
                 quote_types = [q for q in quote_types if q in _MULTI_QUOTES]
             new_fstring_parts.append(value)
 
+        if fallback_to_repr:
+            # If we weren't able to find a quote type that works for all parts
+            # of the JoinedStr, fallback to using repr and triple single quotes.
+            quote_types = ["'''"]
+            new_fstring_parts.clear()
+            for value, is_constant in fstring_parts:
+                if is_constant:
+                    value = repr("'" + '"' + value)  # force repr to escape all quotes
+                    expected_prefix = r"'\'" + '"'
+                    assert value.startswith(expected_prefix), repr(value)
+                    value = value[len(expected_prefix):-1]
+                new_fstring_parts.append(value)
+
         value = "".join(new_fstring_parts)
         quote_type = quote_types[0]
         self.write(f"{quote_type}{value}{quote_type}")
diff --git a/Lib/test/test_unparse.py b/Lib/test/test_unparse.py
index 38c59e6d43..c09865894d 100644
--- a/Lib/test/test_unparse.py
+++ b/Lib/test/test_unparse.py
@@ -635,6 +635,16 @@ def test_star_expr_assign_target_multiple(self):
         self.check_src_roundtrip("[a, b] = [c, d] = [e, f] = g")
         self.check_src_roundtrip("a, b = [c, d] = e, f = g")
 
+    def test_multiquote_joined_string(self):
+        self.check_ast_roundtrip("f\"'''{1}\\\"\\\"\\\"\" ")
+        self.check_ast_roundtrip("""f"'''{1}""\\"" """)
+        self.check_ast_roundtrip("""f'""\"{1}''' """)
+        self.check_ast_roundtrip("""f'""\"{1}""\\"' """)
+
+        self.check_ast_roundtrip("""f"'''{"\\n"}""\\"" """)
+        self.check_ast_roundtrip("""f'""\"{"\\n"}''' """)
+        self.check_ast_roundtrip("""f'""\"{"\\n"}""\\"' """)
+
 
 class ManualASTCreationTestCase(unittest.TestCase):
     """Test that AST nodes created without a type_params field unparse correctly."""

Should I open a PR?

I got a little confused by the history here, looks like this isn't actually fixed on main?

Yes. I got it reverted as requested to separate the fix out.

Should I open a PR?

Go ahead!

pythongh-108843: fix ast.unparse for f-string with many quotes

gh-108843: fix ast.unparse for f-string with many quotes (#108981)

pythongh-108843: fix ast.unparse for f-string with many quotes (pytho…

[3.11] gh-108843: fix ast.unparse for f-string with many quotes (#108980

[3.12] gh-108843: fix ast.unparse for f-string with many quotes (GH-1…

pythongh-108843: fix ast.unparse for f-string with many quotes (pytho…

Navigation Menu

Repository navigation

ast.unparse is not working correctly for strings with multiple quotes #108843

Bug report

CPython versions tested on:

Operating systems tested on:

A clear and concise description of the bug:

Linked PRs

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

ast.unparse is not working correctly for strings with multiple quotes #108843

Description

Bug report

CPython versions tested on:

Operating systems tested on:

A clear and concise description of the bug:

Linked PRs

Activity

hauntsaninja commented on Sep 6, 2023

hauntsaninja commented on Sep 6, 2023

sunmy2019 commented on Sep 6, 2023

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions