Bug: DynamoDB numbers longer than 38 characters are deserialized to the wrong value #8512

Expected Behaviour

TypeDeserializer (used by DynamoDBStreamEvent / StreamRecord.new_image and old_image) returns a Decimal equal to the stored number for any value DynamoDB accepts.

Current Behaviour

_deserialize_n decides whether a number is too long by counting characters after stripping leading zeros, then slices the string. The sign and the decimal point count as characters, and slicing off trailing zeros changes the magnitude. Valid numbers come back wrong, with no error:

All four are within DynamoDB's 38 digits of precision, since trailing zeros don't count.

Code snippet

from decimal import Decimal

from aws_lambda_powertools.shared.dynamodb_deserializer import TypeDeserializer

for value in [
    "-12345678901234567890123456789012345678",
    "1.2345678901234567890123456789012345678",
    "110111111111111110000000000000000000000",
    "12345678901234567890123456789012345678000",
]:
    result = TypeDeserializer().deserialize({"N": value})
    print(result, result == Decimal(value))  # False for every one

Possible Solution

Parse with Decimal(value) and, only when the coefficient has more than 38 digits, normalize() it with a 38-digit context so trailing zeros move into the exponent. The value stays the same and DYNAMODB_CONTEXT.create_decimal no longer raises Rounded, which was the original problem in #4845.

The two test_dynamodb_stream_record_deserialization_large_int* tests currently assert the truncated values and would need their expected values corrected. I have this ready with tests and can open a PR.

Steps to Reproduce

Run the snippet above.

Powertools for AWS Lambda (Python) version

latest (3.35.0, develop at b1af7e8)

AWS Lambda function runtime

3.12

Packaging format used

PyPi

Debugging logs

N/A, no exception is raised.