embedding docs - expose all available API

@yagweb

Initialize
Py.GIL and Threads
RunSimpleString, Exec, Eval
Py.Import
Dynamic vs static declaration
Py.kw
Setting paths
Type conversions
Shutdown

Here's a list of whats currently publicly exposed but with no xml docs.

Python.Runtime.PythonDerivedType.InvokeMethodVoid(Python.Runtime.IPythonDerivedType, string, string, object[])
Python.Runtime.PythonDerivedType.InvokeGetProperty<T>(Python.Runtime.IPythonDerivedType, string)
Python.Runtime.PythonDerivedType.InvokeSetProperty<T>(Python.Runtime.IPythonDerivedType, string, T)
Python.Runtime.PythonDerivedType.InvokeCtor(Python.Runtime.IPythonDerivedType, string, object[])
Python.Runtime.PythonDerivedType.Finalize(Python.Runtime.IPythonDerivedType)
Python.Runtime.ConverterExtension
Python.Runtime.ConverterExtension.ToPython(this object)
Python.Runtime.Dispatcher
Python.Runtime.Dispatcher.target
Python.Runtime.Dispatcher.dtype
Python.Runtime.Dispatcher.Dispatcher(System.IntPtr, System.Type)
Python.Runtime.Dispatcher.~Dispatcher()
Python.Runtime.Dispatcher.Dispatch(System.Collections.ArrayList)
Python.Runtime.Dispatcher.TrueDispatch(System.Collections.ArrayList)
Python.Runtime.ConversionException
Python.Runtime.ConversionException.ConversionException()
Python.Runtime.ConversionException.ConversionException(string)
Python.Runtime.Exceptions.warn(string, System.IntPtr)
Python.Runtime.Exceptions.deprecation(string, int)
Python.Runtime.Exceptions.deprecation(string)
Python.Runtime.Exceptions.BaseException
Python.Runtime.Exceptions.Exception
Python.Runtime.Exceptions.StopIteration
Python.Runtime.Exceptions.GeneratorExit
Python.Runtime.Exceptions.StandardError
Python.Runtime.Exceptions.ArithmeticError
Python.Runtime.Exceptions.LookupError
Python.Runtime.Exceptions.AssertionError
Python.Runtime.Exceptions.AttributeError
Python.Runtime.Exceptions.EOFError
Python.Runtime.Exceptions.FloatingPointError
Python.Runtime.Exceptions.EnvironmentError
Python.Runtime.Exceptions.IOError
Python.Runtime.Exceptions.OSError
Python.Runtime.Exceptions.ImportError
Python.Runtime.Exceptions.IndexError
Python.Runtime.Exceptions.KeyError
Python.Runtime.Exceptions.KeyboardInterrupt
Python.Runtime.Exceptions.MemoryError
Python.Runtime.Exceptions.NameError
Python.Runtime.Exceptions.OverflowError
Python.Runtime.Exceptions.RuntimeError
Python.Runtime.Exceptions.NotImplementedError
Python.Runtime.Exceptions.SyntaxError
Python.Runtime.Exceptions.IndentationError
Python.Runtime.Exceptions.TabError
Python.Runtime.Exceptions.ReferenceError
Python.Runtime.Exceptions.SystemError
Python.Runtime.Exceptions.SystemExit
Python.Runtime.Exceptions.TypeError
Python.Runtime.Exceptions.UnboundLocalError
Python.Runtime.Exceptions.UnicodeError
Python.Runtime.Exceptions.UnicodeEncodeError
Python.Runtime.Exceptions.UnicodeDecodeError
Python.Runtime.Exceptions.UnicodeTranslateError
Python.Runtime.Exceptions.ValueError
Python.Runtime.Exceptions.ZeroDivisionError
Python.Runtime.Exceptions.Warning
Python.Runtime.Exceptions.UserWarning
Python.Runtime.Exceptions.DeprecationWarning
Python.Runtime.Exceptions.PendingDeprecationWarning
Python.Runtime.Exceptions.SyntaxWarning
Python.Runtime.Exceptions.RuntimeWarning
Python.Runtime.Exceptions.FutureWarning
Python.Runtime.Exceptions.ImportWarning
Python.Runtime.Exceptions.UnicodeWarning
Python.Runtime.DocStringAttribute.DocStringAttribute(string)
Python.Runtime.DocStringAttribute.DocString
Python.Runtime.INativeCall.Void_Call_0(System.IntPtr)
Python.Runtime.INativeCall.Void_Call_1(System.IntPtr, System.IntPtr)
Python.Runtime.INativeCall.Int_Call_3(System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr)
Python.Runtime.INativeCall.Call_3(System.IntPtr, System.IntPtr, System.IntPtr, System.IntPtr)
Python.Runtime.PyAnsiString
Python.Runtime.PyIter.Dispose(bool)
Python.Runtime.PyIter.MoveNext()
Python.Runtime.PyIter.Reset()
Python.Runtime.PyIter.Current
Python.Runtime.PyNumber.PyNumber(System.IntPtr)
Python.Runtime.PyNumber.PyNumber()
Python.Runtime.PyObject.obj
Python.Runtime.PyObject.PyObject()
Python.Runtime.PyObject.~PyObject()
Python.Runtime.PyObject.Dispose()
Python.Runtime.PyObject.Refcount
Python.Runtime.PyObject.TryGetMember(System.Dynamic.GetMemberBinder, out object)
Python.Runtime.PyObject.TrySetMember(System.Dynamic.SetMemberBinder, object)
Python.Runtime.PyObject.TryInvokeMember(System.Dynamic.InvokeMemberBinder, object[], out object)
Python.Runtime.PyObject.TryInvoke(System.Dynamic.InvokeBinder, object[], out object)
Python.Runtime.PyObject.TryConvert(System.Dynamic.ConvertBinder, out object)
Python.Runtime.PyObject.TryBinaryOperation(System.Dynamic.BinaryOperationBinder, object, out object)
Python.Runtime.PyObject.TryUnaryOperation(System.Dynamic.UnaryOperationBinder, out object)
Python.Runtime.PySequence.PySequence(System.IntPtr)
Python.Runtime.PySequence.PySequence()
Python.Runtime.PythonEngine.PythonEngine()
Python.Runtime.PythonEngine.PythonEngine(params string[])
Python.Runtime.PythonEngine.PythonEngine(System.Collections.Generic.IEnumerable<string>)
Python.Runtime.PythonEngine.Dispose()
Python.Runtime.PythonEngine.IsInitialized
Python.Runtime.PythonEngine.ProgramName
Python.Runtime.PythonEngine.PythonHome
Python.Runtime.PythonEngine.PythonPath
Python.Runtime.PythonEngine.Version
Python.Runtime.PythonEngine.BuildInfo
Python.Runtime.PythonEngine.Platform
Python.Runtime.PythonEngine.Copyright
Python.Runtime.PythonEngine.RunSimpleString(string)
Python.Runtime.PythonEngine.Initialize()
Python.Runtime.PythonEngine.Initialize(bool)
Python.Runtime.RunFlagType
Python.Runtime.RunFlagType.Single
Python.Runtime.RunFlagType.File
Python.Runtime.RunFlagType.Eval
Python.Runtime.Py
Python.Runtime.Py.GIL()
Python.Runtime.Py.GILState
Python.Runtime.Py.GILState.Dispose()
Python.Runtime.Py.GILState.~GILState()
Python.Runtime.Py.KeywordArguments
Python.Runtime.Py.kw(params object[])
Python.Runtime.Py.Import(string)
Python.Runtime.Py.SetArgv()
Python.Runtime.Py.SetArgv(params string[])
Python.Runtime.Py.SetArgv(System.Collections.Generic.IEnumerable<string>)
Python.Runtime.PythonException.PythonException()
Python.Runtime.PythonException.~PythonException()
Python.Runtime.Runtime.UCS
Python.Runtime.Runtime.pyversion
Python.Runtime.Runtime.pyversionnumber
Python.Runtime.Runtime.dll
Python.Runtime.Runtime.Py_Main(int, string[])

The easier approach might to write a few more example based on the unittests and limit those examples to use only API that will remain public.

@filmor

@denfromufa I'm not going very far in pythonnet. currently, I have used these methods,

PythonEngine.PythonPath: to embed CPython into .NET as a independ module
Initialize
Py.GIL method:
PyScope class: I use it instead of RunSimpleString, Exec and Eval
PythonEngine.AcquireLock and PythonEngine.ReleaseLock: used in C# callback funtion passed to CPython
PyIter class: to analyse the OrderedDict object return from CPython
dynamic type: used for visiting Python object, field and method.
KeywordArguments class: but did not use Py.kw method

In the future, I will dig deep, such as using threading and asynchronous callback, and share my experience here.

Before writing the docs, I think we need to do some work on the embedding API design.

For the embedding API design, I hold the same view as @filmor,

All functions that operate on pointers directly should be internal. one of the reasons is security.

According to this principle, some functions and properties can be removed from the current API list, such as the Handle property of PyObject, the BeginAllowThreads method, the EndAllowThreads.

In order to remove these APIs related to the IntPtr, some new APIs need to be designed and provided. Use the PythonEngine.AcquireLock and PythonEngine.ReleaseLock as example, these two methods should be removed according to this principle. Their problems include, programe will crash if the ReleaseLock is called redundantly, and CPython will be blocked if forge to call the ReleaseLock method. The Py.GIL class can be extended to cover their usage, such as the PyScope.GIL class in the PyScope proposal.

Following are my other proposals for the embedding API design,

Merge the PythonEngine class and Py class. PythonEngine called the methods in Py such as Throw, Py called the PythonEngine in PythonEngine, I don‘t know the principle of where to put a method, Py or PythonEngine.
The KeywordArguments class can be moved out from Py class, the Py.kw method can be put into the KeywordArguments class as a static convert method, and other convert methods can also be putted into the KeywordArguments as overloading methods. For example, I used a convert method to transform C# Dictionary into KeywordArguments.
It seems there is no way to flatten a list and pass it to a python function. A new TupleArguments class similar to KeywordArguments can do this.
The GILState class can be moved out from Py class, and should be extended as shown above.
A class wrapping threading may be needed. (explore later)

For the docs of embedding CPython, @vmuriart said the easier approach is writing a few more example based on the unittests, supplementary approach is providing a doc according to topics. Topics come from the list of issues users encontered. For example, following is what I have currently,

how to embed CPython into .NET as a independ module. I developed a python module to extract and pack the moduled into a zip file.
Pass .NET object as attribute of CPython object. For example,
how to redirect sys.stdout and sys.stdin
Call python function
how to call a python function of signature "def func(arg1, *arg2, **kwargs)"
Transform CPython object (custom and build-in) into .NET Object. For example,
how to convert a CPython OrderedDict object into C# Dictionary
Callback
how to pass CPython object to the callback function defined in .NET (I encontered a bug here)

here is a library for embedding python with pythonnet and using numpy, matplotlib:

https://github.com/Lodomir/PythonInterface

@denfromufa
Can you take a look why in PythonInterface
the pyplot.cs

Line 96 and 97 fail with the latest Python.Runtime.dll (compile from git clone)

        public void plot(double[] x, double[] y)
        {
            Python.PushLocal(Local);
            Python.Set("x", x);  //line96
            Python.Set("y", y);  //line97
            Python.RunString(
                "fig = plt.figure()\n" +
                "ax = fig.add_subplot(111)\n" +
                "ax.plot(x, y)\n" +
                "del x\n" +
                "del y\n"
                );
            Local = Python.PopLocal();
            UpdatePlot();
        }

using example

                double[] x = { 1.0, 2.0, 3.0, 4.0 };
                double[] y = { 1.0, 2.0, 1.0, 2.0 };
                this.pyplot.plot(x, y);

        public void Set(string name, dynamic content)
        {
            using (Py.GIL())
            {
                System.Type ValueType = content.GetType();
                // Arrays are always converted to python lists
                if (ValueType.IsArray)
                {
                    if (!NumpyLoaded)
                    {
                        throw new PythonException("numpy must be loaded for array conversion.");
                    }

                    // BlockCopy possibly multidimensional array of arbitrary type to onedimensional byte array
                    System.Type ElementType = ValueType.GetElementType();
                    int nbytes = content.Length * Marshal.SizeOf(ElementType);
                    byte[] data = new byte[nbytes];
                    System.Buffer.BlockCopy(content, 0, data, 0, nbytes);

                    // <=== HERE IS THE EFFOR===> Create an python tuple with the dimensions of the input array 
                    PyObject[] lengths = new PyObject[content.Rank];
                    for (int i = 0; i < content.Rank; i++)
                        lengths[i] = new PyInt(content.GetLength(i));
                    PyTuple shape = new PyTuple(lengths);

                    // Create an empty numpy array in correct shape and datatype
                    dynamic dtype;
                    if (ElementType == typeof(int))
                        dtype = np.int32;
                    else if (ElementType == typeof(double))
                        dtype = np.float64;
                    else
                        throw new System.Exception("Datatype not supported!");
                    dynamic pydata = np.empty(shape, dtype);

                    // Copy the data to that array
                    System.IntPtr ptr = (System.IntPtr)PyInt.AsInt(pydata.__array_interface__["data"][0]).ToInt32();
                    Marshal.Copy(data, 0, ptr, nbytes);

                    // Push the variable to local dictionary
                    UpdateLocals(name, pydata);
                }
                else
                {
                    if (ValueType == typeof(int))
                        UpdateLocals(name, new PyInt(content));
                    else if (ValueType == typeof(string))
                        UpdateLocals(name, new PyString(content));
                    else if (ValueType == typeof(double))
                        UpdateLocals(name, new PyFloat(content));
                    else if (ValueType == typeof(bool))
                    {
                        UpdateLocals(name, new PyInt(System.Convert.ToInt32(content)));
                        RunStringTry(name + " = (" + name + " == 1)");
                    }
                    else
                        throw new PythonException("Datatype not supported!");
                }
            }
        }

previously, I thought was due to regression of Numpy. With the latest compile python.Runtime, the regression issue is gone, yet, there is a problem of converting double[] array to python.net

Appreciate

PyPlot for matplotlib
#442

@JimSEOW please don't hijack this issue with a separate feature request, mapping containers between languages is not easy. Especially arrays, since Python does not have built-in arrays. There is array module in Python, but everybody uses numpy arrays.

…

On Wed, Apr 5, 2017, 6:14 PM JimSw2016 ***@***.***> wrote: PyPlot for matplotlib #442 <#442> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#410 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHgZ5Yo_h3-0QUZL6D5ewYWGUbbwsCEjks5rtCBhgaJpZM4MMs2N> .

@denfromufa sorry, I am not aware.

@JimSEOW

@JimSEOW I tried the SimplePythonGraph project. I guess you ran the program in x64 mode. If so, the bug you encountered was caused by the limitation of method PyInt.AsInt().

A solution is,

replace all the expressions PyInt.AsInt(...).ToInt32() and PyInt.AsInt(...).ToInt64() with (...).AsManagedObject(typeof(long)). They appear in both the Get() and Set() methods of PythonConnection class.

P.S.
Since AsManagedObject method is widely used and not very convenient, I am trying to add a new Generic method As<T> to the PyObject class, after that, one can use
(...).As<long>()
instead of
(...).AsManagedObject(typeof(long))

The limitation of the PyInt.AsInt method is,

This method called the CPython API PyNumber_Int, which converts a python number to C int (maybe C long?). When the target platform of the program was x64, the address of a variable may out of the range of integer. PyNumber_Long is also not working here, because on windows the length of C long is also 4 bytes, no matter the target platform is x86 or x64.

@yagweb

 public void Set(string name, dynamic content)  //Only 1 modification
//System.IntPtr ptr = (System.IntPtr)PyInt.AsInt(pydata.__array_interface__["data"][0]).ToInt32();
   System.IntPtr ptr = (System.IntPtr)pydata.__array_interface__["data"][0].AsManagedObject(typeof(long));

public dynamic Get(string name_in)  //only 2 modifications
 //System.IntPtr ptr = (System.IntPtr)PyInt.AsInt(Local.GetItem("__" + name + "__ptr")).ToInt32();
System.IntPtr ptr = (System.IntPtr)Local.GetItem("__" + name + "__ptr").AsManagedObject(typeof(long)); 

//int nbytes = PyInt.AsInt(Local.GetItem("__" + name + "__nbytes")).ToInt32();
 long nbytes = (long)Local.GetItem("__" + name + "__nbytes").AsManagedObject(typeof(long));

I get error of {"Specified cast is not valid."}
public dynamic Get(string name_in)

RunString("__" + name + "__dtype = str(" + name + ".dtype)"); <=========

@yagweb Do you have a running SimplePythonGraph project to share?

Relevant issue

@yagweb

@yagweb will your scope class address the GET, SET issues above?

The reference documentation is auto-generated now.

Navigation Menu

Repository navigation

embedding docs - expose all available API #410

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

embedding docs - expose all available API #410

Description

Activity

vmuriart commented on Feb 27, 2017

vmuriart commented on Feb 27, 2017

yagweb commented on Feb 28, 2017

den-run-ai commented on Mar 1, 2017

JimSEOW commented on Apr 5, 2017

JimSEOW commented on Apr 5, 2017

den-run-ai commented on Apr 5, 2017

JimSEOW commented on Apr 5, 2017

yagweb commented on Apr 6, 2017

JimSEOW commented on Apr 9, 2017

JimSEOW commented on Apr 27, 2017

filmor commented on Nov 22, 2022

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions