Skip to content

tutorial - getting started doc - smoke test job fail with 502 Server Error: Bad Gateway #157

@CloudBreadPaPa

Description

@CloudBreadPaPa

Subject : tutorial - getting started doc - smoke test job fail with 502 Server Error: Bad Gateway

Symptom : pipeline fail with 502 Server Error: Bad Gateway

Detail error message :

...
502 Server Error: Bad Gateway for url: http://<ACI-HOST>.centralus.azurecontainer.io/score
Retrying...
502 Server Error: Bad Gateway for url: http://<ACI-HOST>.centralus.azurecontainer.io/score
Retrying...
url : http://<ACI-HOST>.centralus.azurecontainer.io/score
input : {'data': [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10], [10, 9, 8, 7, 6, 5, 4, 3, 2, 1]]}
headers : {'traceparent': '00-722be4574da391d0a6894d190ddc4a25-dc4ead82e44ac27a-00'}
Traceback (most recent call last):
  File "ml_service/util/smoke_test_scoring_service.py", line 97, in <module>
    main()
  File "ml_service/util/smoke_test_scoring_service.py", line 88, in main
    output = call_web_service(e, args.type, args.service)
  File "ml_service/util/smoke_test_scoring_service.py", line 35, in call_web_service
    output = call_web_app(service.scoring_uri, headers)
  File "ml_service/util/smoke_test_scoring_service.py", line 59, in call_web_app
    raise e
  File "ml_service/util/smoke_test_scoring_service.py", line 55, in call_web_app
    response.raise_for_status()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: http://<ACI-HOST>.centralus.azurecontainer.io/score
##[error]Script failed with error: Error: The process '/bin/bash' failed with exit code 1
/usr/local/bin/az account clear
Finishing: Smoke test

Approach :
"/ml_service/util/smoke_test_scoring_service.py" might issue on it.
Built test env for this smoke test score and tested.
The detail error message was "502 Server Error: Bad Gateway error" with below error message

ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0, with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 11 is different from 10)

Resolution :
Replaced "/ml_service/util/smoke_test_scoring_service.py" file input data shape

input = {"data": [[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
                  [11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1]]}

Replace "/diabetes_regression/scoring/score.py" file input input data

input_sample = numpy.array([
    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
    [11, 10, 9, 8, 7, 6, 5, 4, 3, 2, 1]])

and

test_row = '{"data":[[1,2,3,4,5,6,7,8,9,10,11],[11,10,9,8,7,6,5,4,3,2,1]]}'
  • removed error pipeline Azure DevOps pipeline
  • removed deployment and model on Azure Machine Learning Workspace
  • recreated pipeline

The smoke test passed and pipeline works.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions