Skip to content

Delete handler callback context breaks after multiple invocations #83

Closed

Description

@iann0036

It seems that when passing and retrieving data into the callbackContext for the delete handler, after consecutive invocations, it suddenly passes nothing.

I created the following handler:

@resource.handler(Action.DELETE)
def delete_handler(
    session: Optional[SessionProxy],
    request: ResourceHandlerRequest,
    callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
    model = request.desiredResourceState
    progress: ProgressEvent = ProgressEvent(
        status=OperationStatus.IN_PROGRESS,
        resourceModel=model,
    )

    count = 1
    if callback_context.get('count'):
        count = callback_context.get('count') + 1
        LOG.warn("DELETE invocation #{}".format(str(count)))
    else:
        LOG.warn("First DELETE invocation")
    
    if count < 15:
        progress.callbackDelaySeconds = 20
        progress.callbackContext = {
            'count': count,
        }
    else:
        progress.status = OperationStatus.SUCCESS

    return progress

which produces the following logs:

06:40:06
First DELETE invocation
06:40:26
DELETE invocation #2
06:40:46
DELETE invocation #3
06:41:07
DELETE invocation #4
06:41:27
DELETE invocation #5
06:41:47
DELETE invocation #6
06:43:04
Invalid request
Traceback (most recent call last):
  File "/var/task/cloudformation_cli_python_lib/resource.py", line 180, in _parse_request
    event = HandlerRequest.deserialize(event_data)
  File "/var/task/cloudformation_cli_python_lib/utils.py", line 89, in deserialize
    event.requestData = RequestData.deserialize(json_data.get("requestData", 
{}
))
  File "/var/task/cloudformation_cli_python_lib/utils.py", line 54, in deserialize
    req_data = RequestData(**json_data)
TypeError: __init__() missing 1 required positional argument: 'systemTags'

The above timing seems to be fairly consistent. I believe the first 3 messages are actually under the same Lambda invocation per local callback rules and that the next 3 messages would be under a second invocation, then the error. I haven't tested with other provider languages so unsure if this is a Go provider issue or Core CLI (service) issue.

Full handler source with example:
cfntypeexample.zip

Activity

  1. iann0036 commented on Mar 24, 2020

    @iann0036
    Author

    The above also occurs for UPDATE, but not for CREATE

  2. iann0036 commented on Mar 24, 2020

    @iann0036
    Author

    Unsure if related, but I also can get the following after a long (~5 minute) callback cycle on CREATE:

    An error occurred while publishing metrics: An error occurred (AccessDenied) when calling the PutMetricData operation: User: arn:aws:sts::21<snipped>84:assumed-role/uluru-managed-execution-role/24bede68-8<snipped>9e is not authorized to perform: cloudwatch:PutMetricData
    
  3. jaymccon commented on Mar 24, 2020

    @jaymccon
    Contributor

    Unsure if related, but I also can get the following after a long (~5 minute) callback cycle on CREATE:

    An error occurred while publishing metrics: An error occurred (AccessDenied) when calling the PutMetricData operation: User: arn:aws:sts::21<snipped>84:assumed-role/uluru-managed-execution-role/24bede68-8<snipped>9e is not authorized to perform: cloudwatch:PutMetricData
    

    I think this is a separate issue, fortunately any errors with publishing metrics are handled without interrupting handler operation

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't working

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions