Skip to content

Attribute checking on span to prevent AttributeError: 'DroppedSpan' object has no attribute 'id'#626

Closed
sondrelg wants to merge 1 commit into
elastic:masterfrom
sondrelg:master
Closed

Attribute checking on span to prevent AttributeError: 'DroppedSpan' object has no attribute 'id'#626
sondrelg wants to merge 1 commit into
elastic:masterfrom
sondrelg:master

Conversation

@sondrelg

@sondrelg sondrelg commented Nov 6, 2019

Copy link
Copy Markdown

This issue has been tough to debug properly, but briefly explained, I'm running into an error in my project where span is missing the attribute id.

This seemingly happens during the handling of a 401 response, in a series of iterated requests. In about 20 requests, the first 6 run smoothly returning 200s, before the 7th throws the error shown below - this pattern holds up when changing the order of requests and suggests to me that this error might be a symptom of another issue.

The suggested fix solved my problem. Oddly, once fixed, there are no longer any 401s returned.

Stack trace:

File "/app/src/shared/helpers/kerberos.py", line 27, in get
--
  | response = requests.get(url, headers=headers, auth=HTTPKerberosAuth())
  | File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 75, in get
  | return request('get', url, params=params, **kwargs)
  | File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 60, in request
  | return session.request(method=method, url=url, **kwargs)
  | File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
  | resp = self.send(prep, **send_kwargs)
  | File "/usr/local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/base.py", line 141, in call_if_sampling
  | return self.call(module, method, wrapped, instance, args, kwargs)
  | File "/usr/local/lib/python3.7/site-packages/elasticapm/instrumentation/packages/requests.py", line 65, in call
  | return wrapped(*args, **kwargs)
  | File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 653, in send
  | r = dispatch_hook('response', hooks, r, **kwargs)
  | File "/usr/local/lib/python3.7/site-packages/requests/hooks.py", line 31, in dispatch_hook
  | _hook_data = hook(hook_data, **kwargs)
  | File "/usr/local/lib/python3.7/site-packages/requests_kerberos/kerberos_.py", line 400, in handle_response
  | _r = self.handle_401(response, **kwargs)
  | File "/usr/local/lib/python3.7/site-packages/requests_kerberos/kerberos_.py", line 284, in handle_401
  | log.debug("handle_401(): Handling: 401")
  | File "/usr/local/lib/python3.7/logging/__init__.py", line 1366, in debug
  | self._log(DEBUG, msg, args, **kwargs)
  | File "/usr/local/lib/python3.7/logging/__init__.py", line 1513, in _log
  | exc_info, func, extra, sinfo)
  | File "/usr/local/lib/python3.7/logging/__init__.py", line 1483, in makeRecord
  | sinfo)
  | File "/usr/local/lib/python3.7/site-packages/elasticapm/handlers/logging.py", line 201, in log_record_factory
  | return _add_attributes_to_log_record(record)
  | File "/usr/local/lib/python3.7/site-packages/elasticapm/handlers/logging.py", line 220, in _add_attributes_to_log_record
  | span_id = span.id if span else None
  | AttributeError: 'DroppedSpan' object has no attribute 'id'

@beniwohli

beniwohli commented Nov 12, 2019

Copy link
Copy Markdown
Contributor

Hi @sondrelg

thanks for opening the PR. We did release a fix for an issue that sounds very similar to this in version 5.2.1, can you confirm that you run that version? The fix was #602, which should ensure that DroppedSpan instances have an id attribute.

@sondrelg

sondrelg commented Nov 15, 2019

Copy link
Copy Markdown
Author

We've identified our issue as being caused by a 504 Gateway Timeout error. Our app is hosted with Openshift, and the haproxy ingress router times out after 30 seconds by default 🤦‍♂️

To answer your previous question, we did test using v5.2.2 before forking the repo and applying the fix above, and it still logged the error as being caused by a missing id, but the root cause is clearly the timeout.

Feel free to close the request and thanks for the assistance 👍

@beniwohli

beniwohli commented Nov 15, 2019

Copy link
Copy Markdown
Contributor

@sondrelg thanks for the update! It's still weird that you'd get and AttributeError on DroppedSpan.id, since both Span and DroppedSpan objects should have the id attribute since version 5.2.2

I'll close the PR for now, but will investigate what could cause this a bit further some time in the coming weeks.

@beniwohli beniwohli closed this Nov 15, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@sondrelg @beniwohli