can not make calls to Python on a separate thread

The following (copies from Example) worked fine
using (Py.GIL())
{
dynamic test = Py.Import("programs.test_program");
Console.WriteLine(test.func(args));
}

However, the below ends up hanging forever even though it's the same one line program as above. Without the ability to run on a separate thread how can I kill an execution if it takes too long. We deal with massive amount of python scripts at runtime it's gonna be a pain to modify them to terminated the costly computation.

using (Py.GIL())
{
      dynamic test = Py.Import("programs.test_program");

      Thread t = new Thread(delegate () {
              Console.WriteLine(test.func(args));
    });
}

what happens if thread is launched outside of Py.GIL() block?

This is what happens:

An unhandled exception of type 'System.AccessViolationException' occurred in Python.Runtime.dll

Additional information: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.

Try to do something inside the python script by following https://stackoverflow.com/questions/492519/timeout-on-a-function-call. It's running fine in python but once again pythonnet hangs forever. I wonder what is the solution.

import sys
import threading
import _thread as thread 

def quit_function(fn_name):
    print('{0} took too long'.format(fn_name), file=sys.stderr)
    sys.stderr.flush()
    thread.interrupt_main() # raises KeyboardInterrupt

def exit_after(s):
    def outer(fn):
        def inner(*args, **kwargs):
            timer = threading.Timer(s, quit_function, args=[fn.__name__])
            timer.start()
            try:
                result = fn(*args, **kwargs)
            finally:
                timer.cancel()
            return result
        return inner
    return outer

@exit_after(.00001)
def testWrapper():
    while True : 
        i = 0

Hi can I get an update, It's a blocker on our end. I would appreciate if you can advice. Thanks a lot!

The reason why interrupt does not work in pythonnet is due to this issue and corresponding WIP pull request:

#449
#450

The code in pull request is actually OK, but the tests for it are not good.

so what's the fix?

The fix is in the pull request, just look at the diff:

https://github.com/pythonnet/pythonnet/pull/450/files

In short pythonnet should call Py_InitializeEx() instead of Py_Initialize().

Thanks for the pointers. Does this mean I have to get the source (after the fix) and then build it myself is there a binary I can just directly embed in my .Net application.

Or if this is easier. Would you mind advising me how to create a infinity loop detection when I am calling python code.

@keowang

@keowang I tried with master branch and I don't see any issue:

scriptcs (ctrl-c to exit or :help for help)

> #r "C:\Python\Python35_64b\Lib\site-packages\Python.Runtime.dll"
> using Python.Runtime;
> dynamic intr;
> dynamic sys;
> using (Py.GIL()) {sys=Py.Import("sys");}
> sys.path.ToString()
['C:\\Python\\Python35_64b\\python35.zip', 'C:\\Python\\Python35_64b\\Lib', 'C:\\Python\\Python35_64b\\DLLs', 'C:\\ProgramData\\chocolatey\\lib\\ScriptCs\\tools', 'C:\\Python\\Python35_64b', 'C:\\Python\\Python35_64b\\lib\\site-packages', 'C:\\Python\\Python35_64b\\lib\\site-packages\\win32', 'C:\\Python\\Python35_64b\\lib\\site-packages\\win32\\lib', 'C:\\Python\\Python35_64b\\lib\\site-packages\\Pythonwin', 'C:\\Windows\\Microsoft.NET\\Framework64\\v4.0.30319\\']
> sys.path.append(@"C:\Users\denis\Dropbox\coding\pythonnet\issue511");
> using (Py.GIL()) {intr=Py.Import("interrupt_main");}
> intr.testWrapper()
start inf loop
testWrapper took too long
KeyboardInterrupt :

@rmadsen-ks

@rmadsen-ks do you see any issue?

@denfromufa

This is my code running the python script above and it hangs every time.
using (Py.GIL())
{
dynamic testModule = Py.Import("programs.test_program");
output = testModule.testWrapper();
}

Pythonnet installed using following wheel:
pythonnet-2.3.0-cp36-cp36m-win_amd64.whl

Python:3.6 64-bit

Would you please let me know if you can run this code

@keowang

@keowang ok, i did some testing with pythonnet 2.3.0, but I still don't see your exact issue here. Show me the problem again. Here is my testing:

//#r "C:\Python\Python35_64b\Lib\site-packages\Python.Runtime.dll"
using Python.Runtime;
using System;

namespace interrupt
{
    class Program
    {
        static void Main(string[] args)
        {
            dynamic intr;
            dynamic sys;
            PythonEngine.Initialize(true); //, true); second argument if for @rmadsen-ks branch
            using (Py.GIL()) {sys=Py.Import("sys");}
            sys.path.append(System.AppDomain.CurrentDomain.BaseDirectory);
            using (Py.GIL()) 
            {
                intr=Py.Import("interrupt_main");
            }
            using (Py.GIL()) 
            {
                try 
                {
                	intr.testWrapper();
            	}
            	catch {
            		Console.WriteLine("some exception here");
            	}
            }
        }
    }
}

import sys
import threading
import _thread as thread 

def quit_function(fn_name):
    print('{0} took too long'.format(fn_name), file=sys.stderr)
    sys.stderr.flush()
    thread.interrupt_main() # raises KeyboardInterrupt

def exit_after(s):
    def outer(fn):
        def inner(*args, **kwargs):
            timer = threading.Timer(s, quit_function, args=[fn.__name__])
            timer.start()
            try:
                result = fn(*args, **kwargs)
            finally:
                timer.cancel()
            return result
        return inner
    return outer

@exit_after(0.1)
def testWrapper():
    print('start inf loop')
    while True : 
        i = 0

if __name__=="__main__":
    print('before threads launched')
    try:
        testWrapper()
    except KeyboardInterrupt:
        print("forced to interrupt")

Then if I call from python:

python interrupt_main.py
before threads launched
start inf loop
testWrapper took too long
forced to interrupt

If I call from C# compiled app:

interrupt.exe
start inf loop
testWrapper took too long
some exception here

@denfromufa I appreciate your response. The issues is you are running the binary and surprisingly it worked. However, in Visual Studio debug mode intr.testWrapper() will hang every time. There is no way we can get around this by skipping the debug mode and directly monitoring the program behavior via logs which is hugely inconvenient. Would you mind looking into that see if there is some solution that we can have the consistent behavior in both debug and release mode. Thanks

This is how it hangs

Can I please ask what's your exception setting? Thanks!

you can see my exception settings in my screenshot

yes I tried and it's the same thing

did you try break all: ctrl+alt+break

press the key manually?

no, break in the debugger

how do I do that?

debug -> break all

I do not see it here

Hey I fixed it by enabling the native code which makes the debugging very slow. Can I do it without the native code?

"break all" has to be triggered during debugging. i do not have native code debugger enabled. please contact VS team @ msft about debugger issues.

Navigation Menu

Repository navigation

can not make calls to Python on a separate thread #511

10 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

can not make calls to Python on a separate thread #511

Description

Activity

den-run-ai commented on Jul 14, 2017

keowang commented on Jul 14, 2017

keowang commented on Jul 14, 2017

keowang commented on Jul 14, 2017

den-run-ai commented on Jul 14, 2017

den-run-ai commented on Jul 14, 2017

keowang commented on Jul 14, 2017

den-run-ai commented on Jul 14, 2017

keowang commented on Jul 14, 2017

keowang commented on Jul 14, 2017

den-run-ai commented on Jul 15, 2017

den-run-ai commented on Jul 15, 2017

keowang commented on Jul 15, 2017

den-run-ai commented on Jul 16, 2017

keowang commented on Jul 16, 2017

10 remaining items

keowang commented on Jul 19, 2017

keowang commented on Jul 19, 2017

keowang commented on Jul 19, 2017

den-run-ai commented on Jul 19, 2017

keowang commented on Jul 19, 2017

den-run-ai commented on Jul 19, 2017

keowang commented on Jul 19, 2017

den-run-ai commented on Jul 19, 2017

keowang commented on Jul 19, 2017

den-run-ai commented on Jul 19, 2017

keowang commented on Jul 19, 2017

keowang commented on Jul 19, 2017

keowang commented on Jul 19, 2017

den-run-ai commented on Jul 19, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions