Skip to content

Implement Server.serve_forever and corresponding APIs #76843

New issue
New issue
Closed
Closed
Implement Server.serve_forever and corresponding APIs#76843
Labels
3.7 (EOL)end of lifetopic-asynciotype-featureA feature request or enhancement

Description

@1st1
1st1
opened on Jan 25, 2018
BPO 32662
Nosy @vstinner, @asvetlov, @1st1
PRs
  • bpo-32662: Implement Server.start_serving() and Server.serve_forever() #5312
  • bpo-32662: Try making test_asyncio.test_server more reliable #5338
    		•

    Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
closed_at = <Date 2018-05-29.17:03:12.778>
created_at = <Date 2018-01-25.06:44:24.572>
labels = ['type-feature', '3.7', 'expert-asyncio']
title = 'Implement Server.serve_forever and corresponding APIs'
updated_at = <Date 2018-05-29.17:03:12.777>
user = 'https://github.com/1st1'

    bugs.python.org fields:

    activity = <Date 2018-05-29.17:03:12.777>
actor = 'yselivanov'
assignee = 'none'
closed = True
closed_date = <Date 2018-05-29.17:03:12.778>
closer = 'yselivanov'
components = ['asyncio']
creation = <Date 2018-01-25.06:44:24.572>
creator = 'yselivanov'
dependencies = []
files = []
hgrepos = []
issue_num = 32662
keywords = ['patch']
message_count = 7.0
messages = ['310657', '310717', '310740', '310748', '310749', '310750', '310786']
nosy_count = 3.0
nosy_names = ['vstinner', 'asvetlov', 'yselivanov']
pr_nums = ['5312', '5338']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue32662'
versions = ['Python 3.7']

    Activity

    1. 1st1 commented on Jan 25, 2018

      @1st1
      1st1
      on Jan 25, 2018
      MemberAuthor

      As discussed, we want to make Server objects more usable in async/await code and more compatible with asyncio.run.

      This is also needed to handle a use case when two or more servers are created and need to start listening at the same time.

      We propose to:

      1. Add a new bool flag defaulting to True to loop.create_server and loop.create_unix_server: start_serving. By default, loop will return a server that is already accepting connections. When start_serving is set to False, create_server and create_unix_server will return a server that will not listen on its sockets.

      2. A new idempotent Server.start_serving() method can be used to make server listen on its sockets (useful when a server object was created with start_serving=False).

      3. A new Server.serve_forever() method that calls start_serving() and blocks forever, until cancelled. When cancelled, it closes its server object.

      4. A new Server.is_serving() method. This is useful to introspect a server object in unittests.

      5. Server objects should be async context managers. Server.__aexit__ should close the server and await on Server.wait_closed().

      With these new APIs, the following pattern becomes possible:

          async def main():
        srv = await asyncio.start_server(...)
        async with srv:
            await srv.serve_forever()

    asyncio.run(main())
    2. 1st1
      added
      3.7 (EOL)end of life
      topic-asyncio
      type-featureA feature request or enhancement
      on Jan 25, 2018

    3. 1st1 commented on Jan 25, 2018

      @1st1
      1st1
      on Jan 25, 2018
      MemberAuthor

      New changeset c9070d0 by Yury Selivanov in branch 'master':
      bpo-32662: Implement Server.start_serving() and Server.serve_forever() (bpo-5312)
      c9070d0

    4. 1st1
      closed this as completedon Jan 25, 2018

    5. 1st1 commented on Jan 26, 2018

      @1st1
      1st1
      on Jan 26, 2018
      MemberAuthor

      New changeset 4112c5b by Yury Selivanov in branch 'master':
      bpo-32662: Try making test_asyncio.test_server more reliable (bpo-5338)
      4112c5b

    6. vstinner commented on Jan 26, 2018

      @vstinner
      vstinner
      on Jan 26, 2018
      Member

      test_asyncio.test_start_server_1() hangs on "s390x Debian 3.x" buildbot:
      http://buildbot.python.org/all/#builders/13/builds/579

      test_set_nodelay (test.test_asyncio.test_selector_events.TestSelectorUtils) ... ok
test_start_server_1 (test.test_asyncio.test_server.ProactorStartServerTests) ... skipped 'Windows only'
Exception in thread test-client:
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/functional.py", line 193, in run
    self._prog(TestSocketWrapper(self._sock))
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/test_server.py", line 44, in <lambda>
    with self.tcp_client(lambda sock: client(sock, addr)):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/test_server.py", line 19, in client
    sock.connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/functional.py", line 195, in run
    self._test._abort_socket_test(ex)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/functional.py", line 122, in _abort_socket_test
    self.fail(ex)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 680, in fail
    raise self.failureException(msg)
AssertionError: [Errno 111] Connection refused

      Timeout (0:15:00)!
      Thread 0x000003fffcdcd710 (most recent call first):
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/selectors.py", line 468 in select
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/asyncio/base_events.py", line 1555 in _run_once
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/asyncio/base_events.py", line 450 in run_forever
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/asyncio/base_events.py", line 482 in run_until_complete
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/test_asyncio/test_server.py", line 45 in test_start_server_1
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 615 in run
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/case.py", line 663 in __call__
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 122 in run
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/suite.py", line 84 in __call__
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/unittest/runner.py", line 176 in run
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/init.py", line 1861 in _run_suite
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/support/init.py", line 1951 in run_unittest
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 175 in test_runner
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 176 in runtest_inner
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/runtest.py", line 140 in runtest
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 291 in rerun_failed_tests
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 540 in _main
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 510 in main
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/libregrtest/main.py", line 585 in main
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/test/main.py", line 2 in <module>
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 85 in _run_code
      File "/home/dje/cpython-buildarea/3.x.edelsohn-debian-z/build/Lib/runpy.py", line 193 in _run_module_as_main
      test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... Makefile:1090: recipe for target 'buildbottest' failed
      make: *** [buildbottest] Error 1
      program finished with exit code 2
      elapsedTime=2050.270461

    7. vstinner
      reopened this on Jan 26, 2018

    8. vstinner commented on Jan 26, 2018

      @vstinner
      vstinner
      on Jan 26, 2018
      Member

      The test fails differently on s390x SLES 3.x:
      http://buildbot.python.org/all/#/builders/16/builds/576

      ...
test_force_close (test.test_asyncio.test_selector_events.SelectorTransportTests) ... ok
test_set_nodelay (test.test_asyncio.test_selector_events.TestSelectorUtils) ... ok
test_start_server_1 (test.test_asyncio.test_server.ProactorStartServerTests) ... skipped 'Windows only'
test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... Exception in thread test-client:
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 193, in run
    self._prog(TestSocketWrapper(self._sock))
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py", line 44, in <lambda>
    with self.tcp_client(lambda sock: client(sock, addr)):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py", line 19, in client
    sock.connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 195, in run
    self._test._abort_socket_test(ex)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 122, in _abort_socket_test
    self.fail(ex)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/unittest/case.py", line 680, in fail
    raise self.failureException(msg)
AssertionError: [Errno 111] Connection refused

/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py:45: ResourceWarning: unclosed <socket.socket fd=8, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=0, laddr=('127.0.0.1', 56340)>
  self.loop.run_until_complete(main_task)
ERROR
test_start_unix_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... ok
test_start_tls_client_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests) ... skipped 'Windows only'
test_start_tls_server_1 (test.test_asyncio.test_sslproto.ProactorStartTLSTests) ... skipped 'Windows only'
test_start_tls_wrong_args (test.test_asyncio.test_sslproto.ProactorStartTLSTests) ... skipped 'Windows only'
test_start_tls_client_1 (test.test_asyncio.test_sslproto.SelectorStartTLSTests) ... /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/collections/__init__.py:404: ResourceWarning: unclosed <socket.socket fd=7, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('127.0.0.1', 42302)>
  result = _self._make(map(kwds.pop, field_names, _self))
Unhandled error in exception handler
context: {'message': 'Task was destroyed but it is pending!', 'task': <Task pending coro=<BaseStartServer.test_start_server_1.<locals>.main() done, defined at /home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py:31> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x3fff979bbf8>()]>>}
Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 1459, in call_exception_handler
    self._exception_handler(self, context)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/functional.py", line 22, in loop_exception_handler
    self.loop.default_exception_handler(context)
AttributeError: 'NoneType' object has no attribute 'default_exception_handler'
ok
test_start_tls_server_1 (test.test_asyncio.test_sslproto.SelectorStartTLSTests) ... ok
test_start_tls_wrong_args (test.test_asyncio.test_sslproto.SelectorStartTLSTests) ... ok
test_cancel_handshake (test.test_asyncio.test_sslproto.SslProtoHandshakeTests) ... ok
test_close_during_handshake (test.test_asyncio.test_sslproto.SslProtoHandshakeTests) .
...

      ======================================================================
      ERROR: test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests)
      ----------------------------------------------------------------------

      Traceback (most recent call last):
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/test/test_asyncio/test_server.py", line 45, in test_start_server_1
    self.loop.run_until_complete(main_task)
  File "/home/dje/cpython-buildarea/3.x.edelsohn-sles-z/build/Lib/asyncio/base_events.py", line 493, in run_until_complete
    raise RuntimeError('Event loop stopped before Future completed.')
RuntimeError: Event loop stopped before Future completed.

    9. vstinner commented on Jan 26, 2018

      @vstinner
      vstinner
      on Jan 26, 2018
      Member

      Just to be complete, the test also fails on x86 Gentoo Non-Debug with X 3.x:
      http://buildbot.python.org/all/#/builders/99/builds/564

      ...
test_start_server_1 (test.test_asyncio.test_server.SelectorStartServerTests) ... Exception in thread test-client:
Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/functional.py", line 193, in run
    self._prog(TestSocketWrapper(self._sock))
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/test_server.py", line 44, in <lambda>
    with self.tcp_client(lambda sock: client(sock, addr)):
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/test_server.py", line 19, in client
    sock.connect(addr)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/functional.py", line 195, in run
    self._test._abort_socket_test(ex)
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/functional.py", line 122, in _abort_socket_test
    self.fail(ex)
  File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/unittest/case.py", line 680, in fail
    raise self.failureException(msg)
AssertionError: [Errno 111] Connection refused

      Timeout (0:15:00)!
      Thread 0xb7d69700 (most recent call first):
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/selectors.py", line 468 in select
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/asyncio/base_events.py", line 1555 in _run_once
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/asyncio/base_events.py", line 450 in run_forever
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/asyncio/base_events.py", line 482 in run_until_complete
      File "/buildbot/buildarea/3.x.ware-gentoo-x86.nondebug/build/Lib/test/test_asyncio/test_server.py", line 45 in test_start_server_1
      ...

    10. 1st1 commented on Jan 26, 2018

      @1st1
      1st1
      on Jan 26, 2018
      MemberAuthor

      I think these failures were on the previous version of the test; i've fixed it last night and it shouldn't timeout this way anymore.

    11. 1st1
      closed this as completedon May 29, 2018
    12. ezio-melotti
      transferred this issue fromon Apr 10, 2022
    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

    Metadata

    Metadata

    Assignees

    No one assigned

      Labels

      3.7 (EOL)end of lifetopic-asynciotype-featureA feature request or enhancement

      Projects

      No projects

        Milestone

        No milestone

        Relationships

        None yet

        Development

        No branches or pull requests

        Issue actions