pythonjs

Rusthon and WebWorkers

2015-07-22T18:11:00.001-07:00

Rusthon is the fork of PythonJS that I currently maintain and use on all my personal and work projects. It has an improved JavaScript transpiler, with a powerful syntax that mixes Python and Golang. Recently I added special syntax for timed processing and webworkers, see these posts: [1], [2]

WebWorkers

The WebWorker standard is many years old, but often web developers do not have time to break apart their code in order to use it. Code inside a webworker has no access to the DOM, and passing objects back and forth requires packing them into a message protocol. You also need to worry about loading your shared JavaScript with importScripts, or have some logic to create a URL Blob at runtime, and splitting off the worker code into its own file. In other words, refactoring your JavaScript to leverage webworkers, can be a big pain in the ass.

Rusthon has simple syntax inspired by Golang for using webworkers, see the wiki here. The transpiler and runtime take care of all the details of passing messages, reconstructing objects, splitting apart your code, and generating a URL blob at runtime. Example source code.

transpiler input

This example passes instances of SharedClass back and forth from webworker to the main thread. Python3 style function annotations are used to type the input arguments and return type, this allows the transpiler to inject code to restore the objects __proto__ class|prototype property, this is required when you intend to call methods on objects that cross from webworker to main, or main to webworker.

#backend:javascript
from runtime import *

def show(txt):
 document.getElementById('CONTAINER').appendChild(
  document.createTextNode(txt + '\n')
 )

class SharedClass:
 def __init__(self, x,y,z):
  self.x = x
  self.y = y
  self.z = z

 def foobar(self):
  return self.x + self.y + self.z


with webworker:
 def somefunction() -> SharedClass:
  s = SharedClass(10,20,30)
  return s

 class MyWorker:
  def send(self, obj:SharedClass ) -> SharedClass:
   print obj
   print obj.foobar()
   obj.x = 10
   return obj

  def somemethod(self) -> SharedClass:
   s = SharedClass(100,200,300)
   return s



def main():
 global WORKER
 show('spawn worker...')
 WORKER = spawn( MyWorker() )

 show('creating SharedClass')
 a = SharedClass(1,2,3)
 print(a)
 show(a.foobar())

 show('sending data to worker')
 WORKER <- a

 show('getting data from worker')
 b = <- WORKER
 show(b.foobar())

 c = <- somefunction()
 show(c.foobar())

 d = <- WORKER.somemethod()
 show(d.foobar())

javascript output - webworker

note below: somefunction.returns = "SharedClass"; is the result of this input from above def somefunction() -> SharedClass: that sets the return type of the function. The WebWorker runtime manager checks the returns property of functions it calls, and sets the __proto__ of the result in the main thread.

var somefunction = function()
{
 var s;
 s =  new SharedClass(10, 20, 30);
 return s;
}
somefunction.returns = "SharedClass";

var MyWorker = function(){}

MyWorker.prototype.send = function(obj)
{
 var obj;
 obj.__proto__ = SharedClass.prototype;
 console.log(obj);
 console.log(obj.foobar());
 obj.x = 10;
 return obj;
}
MyWorker.prototype.send.returns = "SharedClass";

MyWorker.prototype.somemethod = function()
{
 var s;
 s =  new SharedClass(100, 200, 300);
 return s;
}
MyWorker.prototype.somemethod.returns = "SharedClass";

javascript output - main

var show = function(txt)
{
 document.getElementById("CONTAINER").appendChild(document.createTextNode((txt + "\n")));
}

var SharedClass = function(x, y, z)
{
 this.__init__(x, y, z);
}

SharedClass.prototype.__init__ = function(x, y, z)
{
 this.x = x;
 this.y = y;
 this.z = z;
}

SharedClass.prototype.foobar = function()
{
 return ((this.x + this.y) + this.z);
}

var main = function()
{
 var a,c,b,d;
 show("spawn worker...");
 WORKER = __workerpool__.spawn({new:"MyWorker", args:[]});
 show("creating SharedClass");
 a =  new SharedClass(1, 2, 3);
 console.log(a);
 show(a.foobar());
 show("sending data to worker");
 __workerpool__.send({message:a,id:WORKER})
 show("getting data from worker");
  __workerpool__.recv( WORKER, function (b) {
  show(b.foobar());
   __workerpool__.call( "somefunction", [],  function (c) {
   show(c.foobar());
    __workerpool__.callmeth( WORKER, "somemethod", [],  function (d) {
    show(d.foobar());
   });
  });
 });
}

Rapydscript Kicks Ass

2015-05-12T12:51:00.002-07:00

Rapydscript by Alexander Tsepkov is a Python to JavaScript translator that really kicks ass. It is stable, the transpiler can be run fully client side, produces very fast JavaScript, and is well maintained by Alex. Rapydscript syntax is very similar to PythonJS, and both can be used together in the same markdown file compiled by Rusthon.

Rapydscript has the right mix of Pythonic-style and features to make it inter-operate well with JavaScript. Alex made all the right choices on what parts of Python to throw away, and adapting the syntax to the requirements imposed by generating fast JavaScript. Check out this example for how to use Rapydscript in Rusthon.

Python to Go

2014-09-07T23:25:00.000-07:00

Golang is a fully typed language with simple and clean syntax. The Go compiler will throw an error if you made a typo or called a method on the wrong object. This means you need to worry less about runtime errors, and writting unit tests. Go programs are compiled to native executables, or can be compiled to JavaScript using GopherJS.

PythonJS has a new backend in development that translates to Go. See the documentation here. The Go backend supports translation of Python classes and list comprehensions to Go. The PythonJS syntax has also been extended to support Go features like: go-routines and channels.

Go/Python Meetup

The Barcelona Python users group is hosting a free meetup on Oct 13th, see the details here. I will be going and giving a talk about using PythonJS to translate to Go, and using GopherJS to translate it back to JavaScript.

new pythonjs syntax

2014-07-22T21:09:00.000-07:00

PythonJS is being extended with optional static typing and new syntax. The translator pre-processes input source code and transforms it into an intermediate form that can be parsed by Python's ast module. This simplifies adding new syntax to PythonJS, see this commit that implements Exception Expressions.

a = d[ 'somekey' ] except KeyError: 'mydefault'

inline anonymous functions

I am now researching new syntax that can be added to PythonJS. After looking at RapydScript, I see that inline anonymous functions are a must have, and recently implemented them in PythonJS. Inline functions can be used in function calls as a named keyword argument, or in a dict literal. This solves the problem that Lambda functions can only be a single statement, which is often not that useful.


a.func(
 callback1=def (x,y,z):
  x += y
  return x - z,
 callback2= def (x, y):
  return x * y
)

b = {
 'cb1' : def (x):
  return x,
 'cb2' : def (y):
  return y
}

switch


switch a==b:
  case True:
    pass
  case False:
    pass
  default:
    pass

Go channels and select

def send_data( A:chan int, B:chan int, X:int, Y:int):
 while True:
  print('sending data..')
  A <- X
  B <- Y

def select_loop(A:chan int, B:chan int, W:chan int) -> int:
 print('starting select loop')
 y = 0
 while True:
  print('select loop:',y)
  select:
   case x = <- A:
    y += x
    W <- y
   case x = <- B:
    y += x
    W <- y
 print('end select loop', y)
 return y

def main():
 a = go.channel(int)
 b = go.channel(int)
 w = go.channel(int)

 go(
  select_loop(a,b, w)
 )


 go(
  send_data(a,b, 5, 10)
 )

 z = 0
 while z < 100:
  z = <- w
  print('main loop', z)

 print('end test')

WebGL CSS3D Hybrid part2

2014-07-17T01:34:00.000-07:00

Despite WebGL having been around since 2011, you probably do not run into many websites with 3D. And when you do, its likely a fullscreen game or demo, not a typical website. Traditional web technologies do not mix well with WebGL. Libraries like Voodoo.js try to bridge this gap by allowing you to insert 3D objects inside a normal webpage. Voodoo.js graphics are not anti-aliased when rendering with WebGL, the contrast with the rest of the smooth text and graphics on a webpage then becomes very harsh and downright ugly.

Threepy is an experimental library built on THREE.js that allows some HTML and CSS to be mixed together with WebGL rendering and post-processing FX. The library wraps many standard HTML element types with special hacks like: callbacks, catching events, and rendering to proxy clones. Not every combination of HTML and CSS is compatible with this method. The video below tests the basic UI elements: checkboxes, text input, select drop down lists, number input, color selector, tab menus, slider controls, images and videos. Works with GoogleChrome and NodeWebkit.

WebGL CSS3D Hybrid

2014-07-07T22:09:00.001-07:00

Jerome Etienne has an interesting post where he combines CSS3D and WebGL, see his post here. However, his method is not compatible with Three.js post-processing shaders that is required to blur and perform other 2D effects on the 3D rendering.

The videos below show integration of WebGL and CSS3D with post-processing in THREE.js. The main hack do to this requires double rendering the DOM in two CSS3D layers. The first layer is rendered under the WebGL layer, and is a clone of the top CSS layer. The top CSS layer is rendered transparent over the WebGL layer and catches keyboard and mouse events. The WebGL layer is rendered with alpha blending. See the source code here.

dnd, iframes, video, collada

Update july10th, testing new hacks to integrate: iframes, html5 video, and collada 3D files by drag and drop. Starting new module dddom.py with simple API for creating 3D widgets mixing WebGL and CSS3D.

GLSL Classes

2014-06-24T18:24:00.000-07:00

GLSL is not object oriented, there are no classes or method calls. GLSL4.0 has subroutine variables that can provide an object oriented solution, but this is not compatible with GLSL in WebGL.

PythonJS GPU Class

In the example below an array of MyObject is created and uploaded to the GPU, where it is iterated over. The method call s.mymethod(1.1, 2.2) translates to MyObject_mymethod(s, 1.1, 2.2) in the GLSL shader.

@gpu.object
class MyObject:
 @gpu.method
 float def subroutine(self, x,y):
  float x
  float y
  return x + y * self.attr2

 @gpu.method
 float def mymethod(self, x,y):
  float x
  float y
  if self.index == 0:
   return -20.5
  elif self.index == 0:
   return 0.6
  else:
   return self.subroutine(x,y) * self.attr1

 def __init__(self, a, b, i):
  self.attr1 = a
  self.attr2 = b
  self.index = int16(i)


class myclass:
 def run(self, w):
  self.array = [MyObject(1.1,1.2,x) for x in range(w)]

  @returns( array=64 )
  @gpu.main
  def gpufunc():
   struct* A = self.array
   float b = 0.0

   for s in iter(A):
    b += s.mymethod(1.1, 2.2)

   return b

  return gpufunc()

GLSL Array of Arrays

2014-06-22T21:01:00.001-07:00

WebGL shader code written in GLSL is based on the OpenGL-ES standard, see GLSL 1.2 spec pdf. GLSL 1.2 has no support for multidimensional arrays (array of arrays), and loops require a constant expression for iteration. These limitations make it very hard to write generic shader programs.

PythonJS shader translation provides a workaround, and supports one-level-deep array of arrays and iteration over them. The input array data and sizes can change at runtime because the shader is fully recompiled each call to its wrapper function. Attributes from the current scope in JavaScript can also be inlined into the shader. Read the syntax documentation here.

array of array example

class myclass:
 def __init__(self, s):
  self.s = s
 def my_method(self):
  return self.s

 def run(self, w, h):
  self.array = [ [x*y*0.5 for y in range(h)] for x in range(w) ]

  @returns( array=64 )
  @gpu.main
  def gpufunc():
   float* A = self.array
   float b = self.my_method()

   for subarray in A:
    for j in range( len(self.array[0]) ):
     b += subarray[j]
   return b

  return gpufunc()

GLSL output

The inner function gpufunc becomes main below. Inside gpufunc above, the assignment to A as a float pointer float* A = self.array triggers the wrapper code to unroll A into A_𝑛 and inline the values, '𝑛' is the length of the array.

 void main() {
 float A_0[4];
 A_0[0]=0.0;A_0[1]=0.0;A_0[2]=0.0;A_0[3]=0.0;float A_1[4];A_1[0]=0.0;A_1[1]=0.5;A_1[2]=1.0;A_1[3]=1.5;float A_2[4];A_2[0]=0.0;A_2[1]=1.0;A_2[2]=2.0;A_2[3]=3.0;float A_3[4];A_3[0]=0.0;A_3[1]=1.5;A_3[2]=3.0;A_3[3]=4.5;float A_4[4];A_4[0]=0.0;A_4[1]=2.0;A_4[2]=4.0;A_4[3]=6.0;float A_5[4];A_5[0]=0.0;A_5[1]=2.5;A_5[2]=5.0;A_5[3]=7.5;float A_6[4];A_6[0]=0.0;A_6[1]=3.0;A_6[2]=6.0;A_6[3]=9.0;float A_7[4];A_7[0]=0.0;A_7[1]=3.5;A_7[2]=7.0;A_7[3]=10.5;
 ...

the wrapper inlines the runtime value of float b = self.my_method(). Iteration over the list for subarray in A: is translated into a for loop that copies the data from A_𝑛 into the iterator target subarray.

    float b;
 b = 0.1;
 for (int _iter=0; _iter < 4; _iter++) {
  float subarray[4];
  if (_iter==0) { for (int _J=0; _J<4; _J++) {subarray[_J] = A_0[_J];} }
  if (_iter==1) { for (int _J=0; _J<4; _J++) {subarray[_J] = A_1[_J];} }
  if (_iter==2) { for (int _J=0; _J<4; _J++) {subarray[_J] = A_2[_J];} }
  if (_iter==3) { for (int _J=0; _J<4; _J++) {subarray[_J] = A_3[_J];} }
  for (int j=0; j < 4; j++) {
   b += subarray[j];
  }
 }
    out_float = b;
 }

array of structs example

To iterate over an array of structs, wrap the struct* with iter in a for loop. The struct below contains a float number 'num' and array of floats 'arr'. The nested loop iterates over the indices of the structs array 'arr'.

class myclass:
 def new_struct(self, g):
  return {
   'num' : g,
   'arr' : [0.1 for s in range(6)]
  }

 def run(self, w):
  self.array = [ self.new_struct( x ) for x in range(w) ]
  @returns( array=64 )
  @gpu.main
  def gpufunc():
   struct* A = self.array
   float b = 0.0
   for s in iter(A):
    b += s.num
    for i in range(len(s.arr)):
     b += s.arr[i]
   return b
  return gpufunc()

GLSL output

The assignment struct* A = self.array triggers the wrapper code to generate a struct typedef that is inserted into the shader header. self.array is inlined at runtime as 𝙎𝙩𝙧𝙪𝙘𝙩𝙉𝙖𝙢𝙚 A_𝑛 Before the struct is constructed the array attribute arr is assigned to the variable _arrA_𝑛. The for loop switches the iterator target s based on the loop index 𝑛.

void main( ) {
    float b;
 b=0.0;

 float _arrA_0[6];_arrA_0[0]=0.1;_arrA_0[1]=0.1;_arrA_0[2]=0.1;_arrA_0[3]=0.1;_arrA_0[4]=0.1;_arrA_0[5]=0.1;
 𝙎𝙩𝙧𝙪𝙘𝙩𝙉𝙖𝙢𝙚 A_0 = 𝙎𝙩𝙧𝙪𝙘𝙩𝙉𝙖𝙢𝙚(0.0,_arrA_0);
 ...
 𝙎𝙩𝙧𝙪𝙘𝙩𝙉𝙖𝙢𝙚 A_6 = 𝙎𝙩𝙧𝙪𝙘𝙩𝙉𝙖𝙢𝙚(6.0,_arrA_6);
  float _arrA_7[6];_arrA_7[0]=0.1;_arrA_7[1]=0.1;_arrA_7[2]=0.1;_arrA_7[3]=0.1;_arrA_7[4]=0.1;_arrA_7[5]=0.1;
 𝙎𝙩𝙧𝙪𝙘𝙩𝙉𝙖𝙢𝙚 A_7 = 𝙎𝙩𝙧𝙪𝙘𝙩𝙉𝙖𝙢𝙚(7.0,_arrA_7);

 for (int _iter=0; _iter < 8; _iter++) {
  𝙎𝙩𝙧𝙪𝙘𝙩𝙉𝙖𝙢𝙚 s;
  if (_iter==0) { s=A_0;}
  if (_iter==1) { s=A_1;}
  if (_iter==2) { s=A_2;}
  if (_iter==3) { s=A_3;}
  if (_iter==4) { s=A_4;}
  if (_iter==5) { s=A_5;}
  if (_iter==6) { s=A_6;}
  if (_iter==7) { s=A_7;}
  b += s.num;
  for (int i=0; i < 6; i++) {
      b += s.arr[i];
     }
    }
    out_float = b;
  }

PythonJS GPU Mandelbrot

2014-06-18T01:02:00.000-07:00

PythonJS now supports translation of a limited subset of Python syntax into GPU code. This is done with a new GLSL backend using the WebCLGL library by Roberto Gonzalez. The above benchmark calculates the Mandelbrot set for a size of 512x512, CPython3 with NumPy takes over eight seconds to complete, PythonJS specialized version takes just 0.2 seconds.

source code: mandelbrot.py

40X faster is probably not the best that WebCLGL can do, this benchmark was performed with a low-end GPU. To run the benchmarks yourself, install Python2 and Python3, and extract the latest PyPy to your home directory, and install NumPy for each. Install the NodeWebkit NPM package. Then run these commands, and it will save the file /tmp/mandelbrot.py.eps:

cd
git clone https://github.com/3DRoberto/webclgl.git
git clone https://github.com/PythonJS/PythonJS.git
cd PythonJS/regtests
./run ./bench/mandelbrot.py

PythonJS SIMD Vectors part2

2014-06-13T21:30:00.002-07:00

The Dart backend has a new class float32vec that encapsulates a list of Float32x4 SIMD vectors. The class acts like a normal vector allowing you to do element wise operations and look-ups, while under-the-hood it will index and operate on the appropriate sub-vector. This allows you to write SIMD accelerated code and not have to manually break things apart into chunks of four. Looping over the sub-vectors and encapsulation adds some overhead and slows down performance for arrays with more than 32 elements. The micro benchmark above was performed with 32 elements and shows better performance than CPython with NumPy.

benchmark source code

float32vec

class float32vec:
 def __init__(self, items):
  self[...] = new( List() )
  self.length = items.length

  i = 0; s = 0
  while i < items.length:
   x = items[i]
   y = items[i+1]
   z = items[i+2]
   w = items[i+3]
   vec = new( Float32x4(x,y,z,w) )
   self[...].add( vec )
   i += 4


 def __getitem__(self, index):
  if index < 0: index = self.length + index

  float32x4 vec = self[...][ index // 4 ]
  lane = index % 4
  if lane == 0: return vec.x
  elif lane == 1: return vec.y
  elif lane == 2: return vec.z
  elif lane == 3: return vec.w

 def __setitem__(self, index, value):
  if index < 0: index = self.length + index

  vec = self[...][ index // 4 ]
  lane = index % 4
  if lane == 0: vec = vec.withX(value)
  elif lane == 1: vec = vec.withY(value)
  elif lane == 2: vec = vec.withZ(value)
  elif lane == 3: vec = vec.withW(value)

  self[...][ index // 4 ] = vec

 def __add__(self, other):
  arr = new( List() )
  for i, vec1 in enumerate( self[...] ):
   vec2 = other[...][ i ]
   arr.add( vec1+vec2 )

  v = inline("new float32vec([])")
  v.length = self.length
  v[...] = arr
  return v

 def __mul__(self, other):
  arr = new( List() )
  for i, vec1 in enumerate( self[...] ):
   vec2 = other[...][ i ]
   arr.add( vec1*vec2 )

  v = inline("new float32vec([])")
  v.length = self.length
  v[...] = arr
  return v

PythonJS SIMD Vectors

2014-06-13T04:59:00.000-07:00

PythonJS using direct SIMD via the Dart backend and running in the Dart VM is about 6X faster than CPython with Numpy in the following micro benchmark testing float32x4 multiplication. SIMD stands for single instruction multiple data, and it allows you to instruct the CPU to perform the same math operation on a vector of data to increase performance. Read more about SIMD on my old research blog, here.

I was expecting NumPy would have specialized the case of an array with four float32 elements to use SIMD. Searching around for why this is the case in NumPy, I could not find any clear answers why: [1], [2]. More confused and curious, I jumped into the PyPy IRC chat room, and Matti Picus gave me the answer: NumPy has no direct support for SIMD, instead it relies on helper libraries like: MKL, BLAS, and lapack.

The DartVM includes SIMD and float32x4 and int32x4 primities as part of the core language, you simply import dart:typed_data. Google Chrome and FireFox are also in the process of supporting SIMD.

SIMD multiply micro benchmark

The PythonJS translator has been updated to pass type information to the Dart backend, and translate numpy.array into a Float32x4 vector if it has been typed as float32x4. See my previous blog post about optional static typing, here.

def main():
 start = time()
 float32x4 a = numpy.array( 
  [1.0001, 1.0002, 1.0003, 1.0004], 
  dtype=numpy.float32 )
 float32x4 b = numpy.array( 
  [1.00009, 1.00008, 1.00007, 1.00006], 
  dtype=numpy.float32 )
 float32x4 c = numpy.array( 
  [1.00005, 1.00004, 1.00003, 1.00002], 
  dtype=numpy.float32 )

 arr = []
 for i in range(20000):
  c *= a*b
  arr.append( a*b*c )

 print(time()-start)

PythonJS faster than CPython part2

2014-06-11T20:43:00.000-07:00

My earlier post PythonJS now faster than CPython struck a nerve with the Python community, and some heated debate on Hacker News. The fact is that CPython is embarrassingly slow compared to V8, and the adoption rate of Python3 is very slow compared to JavaScript's explosive growth.

As pointed out by a few people in Hacker News thread, my first benchmarks were performed with PyPy1.9, so I have updated the test runner to use the latest PyPy 2.3.1 and PyPy1.9. I have also removed the micro-benchmarks, and now only include benchmarks from the Unladen Swallow test suite.

PythonJS (with the fast-backend) is faster than CPython in four of the following five benchmarks. Some may argue that this is not a fair comparison because the fast-backend is only a subset of the Python language standard. Keep in mind that: 1. it is large and useful subset, and 2. PythonJS is designed to allow you to mix both modes with in the same script by blocking out code as fully compliant or fast using special with blocks or calling pythonjs.configure.

nbody

pystone

richards

fannkuch

float

benchmark source code

Python has some big problems not just speed, like running on mobile devices. Translation to JavaScript, and other targets like: C++ and Rust, make it possible to side step the Python interpreter and that makes deployment much simpler.

64bit integer long type

2014-06-07T00:23:00.000-07:00

PythonJS now has a new static type long that can be used to mark a variable as a 64bit integer, see this commit. JavaScript has no native support for native 64bit integers, so the translator will use the Long.js API to construct a Long object and call the appropriate methods for math and comparison logic.

python input

def main():
 long x = 65536
 long y = x * x
 long z = 4294967296
 TestError( y==z )

 long a = z + z
 long b = 8589934592
 TestError( a==b )

 TestError( y < b )
 TestError( b > y )

 TestError( y <= b )
 TestError( b >= y )

javascript output

main = function() {
  if (__NODEJS__==true) var long = require('long');
  
  x = long.fromString("65536");
  y = x.multiply(x);
  z = long.fromString("4294967296");
  TestError( y.equals(z) );
  a = z.add(z);
  b = long.fromString("8589934592");
  TestError( a.equals(b) );
  TestError( y.lessThan(b) );
  TestError( b.greaterThan(y) );
  TestError( y.lessThanOrEqual(b) );
  TestError( b.greaterThanOrEqual(y) );
}

optional static typing

2014-06-06T09:31:00.000-07:00

People have been asking for optional types for Python for at least 10 years, and only recently have a couple of good solutions appeared, one is MyPy by Jukka Lehtosalo, but the github project hasn't been updated in several months, and no recent updates on the dev blog either. Python could greatly benefit from optional typing, it makes your code more clear and readable, can provide better compile or time checks, and better performance.

typed_int.py

The above benchmark shows PythonJS in normal mode with statically typed variables is 20 times faster compared to below without statically typed variables.

untyped_int.py

Most of the performance gain comes from typing the arr variable as list, this allows the translator to bypass a runtime method lookup and instead directly call arr.push.

new syntax

def f(arr, x):
  list arr
  int x
  int y = somefunction()
  arr.append( x+y )

PythonJS allows you to type each variable in a function body as a plain statement int x or in an assignment expression int z = somefunction(). This is different from the approach of MyPy where things can only be typed in the function's annotations. It is implemented as a simple text pre-processor, for the details see here, and source code here. This is still a very new feature and can only optimize a few cases. In the future it can be leveraged to further increase performance, and provide translation-time error checking.

automatic synchronous to async transform

2014-06-05T09:05:00.001-07:00

Asynchronous programming in JavaScript can quickly become a mess of callbacks, sometimes it is called `callback hell`. This hell gets worse when you need to move some logic to WebWorkers and pass data back and forth. Your once simple synchronous function must be rewritten into many callbacks that shuffle data around using postMessage and onmessage, and trigger the next callback.

PythonJS allows you to code in a synchronous style, and when your code it translated to JavaScript, it will also be transformed into async callbacks. You can call the sleep function to stop a function for a set amount of time while other things take place, the function will resume after the timeout.

These commits: [1], [2], [3], allow the webworker to directly call functions in the main thread, passing it data as normal function arguments. Under the hood, PythonJS will call postMessage with the function name and arguments. In the main thread this triggers the requested function with the result sent back to the worker. The function in the worker halts until it gets this response from the main thread. What ends up being alot of async code, can be expressed in just a few lines of sync code.

python input

import threading
from time import sleep

shared = []

def blocking_func(x,y):
 shared.append( x )
 shared.append( y )
 return x+y

def async_func( a ):
 shared.append( a )

def main():
 w = threading.start_webworker( worker, [] )
 sleep(1.0)
 assert len(shared)==3
 assert shared[0]==10
 assert shared[1]==20
 assert shared[2]==30
 print('main exit')

## marks this block of code as within the webworker
with webworker:

 def worker():
  ## blocks because the result is assigned to `v`
  v = blocking_func( 10, 20 )
  ## non-blocking because result is not used
  async_func( v )
  self.terminate()

javascript output - main

shared = [];

blocking_func = function(x, y) {
  shared.append(x);
  shared.append(y);
  var __left16, __right17;
  __left16 = x;
  __right17 = y;
  return ((( typeof(__left16) ) == "number") ? (__left16 + __right17) : __add_op(__left16, __right17));
}

async_func = function(a) {
  shared.append(a);
}

main = function() {
  var w;
  w = __start_new_thread(worker, []);
  __run__ = true;
  var __callback0 = function() {
    __run__ = true;
    var __callback1 = function() {
      console.log("main exit");
    }

    if (__run__) {
      setTimeout(__callback1, 1000.0);
    } else {
      if (__continue__) {
        setTimeout(__callback2, 1000.0);
      }
    }
  }

  if (__run__) {
    setTimeout(__callback0, 200.0);
  } else {
    if (__continue__) {
      setTimeout(__callback1, 200.0);
    }
  }
}
worker = "/tmp/worker.js";

javascript output - webworker


onmessage = function(e) {
  if (( e.data.type ) == "execute") {
    worker.apply(self, e.data.args);
    if (! (threading._blocking_callback)) {
      self.postMessage({ "type":"terminate" });
    }
  } else {
    if (( e.data.type ) == "append") {
      __wargs__[e.data.argindex].push(e.data.value);
    } else {
      if (( e.data.type ) == "__setitem__") {
        __wargs__[e.data.argindex][e.data.key] = e.data.value;
      } else {
        if (( e.data.type ) == "return_to_blocking_callback") {
          threading._blocking_callback(e.data.result);
        }
      }
    }
  }
}
self.onmessage = onmessage;

worker = function() {
  var v;
  v = self.postMessage({
    "type":"call",
    "function":"blocking_func",
    "args":[10, 20] 
  });
  var __blocking = function(v) {
    self.postMessage({
       "type":"call",
        "function":"async_func",
        "args":[v] 
    });
    self.postMessage({ "type":"terminate" });
    threading._blocking_callback = null;
  }

  threading._blocking_callback = __blocking;
}

PythonJS IDE

2014-06-04T20:41:00.001-07:00

Above Pypubjs running in NodeWebkit, with project preview in the Android emulator and native HTML window. Pypubjs is simple example that shows how to write a desktop applications using PythonJS and NodeWebkit. Pypubjs is itself written almost entirely in Python and HTML. It integrates with the python-js NodeJS package to dynamically compile itself, and code you write in the editor.

https://github.com/PythonJS/pypubjs

Escaping the GIL with WebWorkers

2014-05-27T23:37:00.000-07:00

The Global Interpreter Lock (GIL) prevents threads in Python from using all cores for CPU bound tasks, and using more threads can even make your program slower. These limitations of the GIL can be bypassed with the faking threading module in PythonJS. The fake threading module provides a function threading.start_webworker(f,args) that will run the function f with args in a new WebWorker. If you pass a list or dict as arguments to the worker function they will become shared between the parent and worker process (each append operation on a list will call self.postMessage to keep the parent and child in sync).

The following benchmark measures how many prime numbers can be found per-second. These tests were done with a dual core CPU, NodeJS 0.10.22, and workerjs. The dual core version of the script is 2X faster than the single core version when translated by PythonJS.

prime numbers per-second (single core)

https://github.com/PythonJS/PythonJS/blob/master/regtests/bench/webworker_single.py

prime numbers per-second (dual core)

https://github.com/PythonJS/PythonJS/blob/master/regtests/bench/webworker.py

sleep(T)

If you looked at the source code for the benchmarks above you might have been surprised to see the sleep function imported from the time module. I recently added experimental support for sleep to PythonJS in these commits: [1], [2]. This allows you to write code in a simpler synchronous style, and behind the scenes the code will be broken down into blocks and async callbacks generated.

python input

from time import sleep

def main():
 sleep(0.01)
 a = []
 sleep(0.1)
 a.append(1)
 sleep(0.1)
 a.append(2)

 TestError( len(a)==2 )

javascript output

main = function() {
  var a;
  __run__ = true;
    var __callback0 = function() {
    a = [];
    __run__ = true;
        var __callback1 = function() {
      a.append(1);
      __run__ = true;
            var __callback2 = function() {
        a.append(2);
        TestError("sleep.py",10,len(a)==2,"len(a)==2");
      }

      if (__run__) {
        setTimeout(__callback2, 100.0);
      } else {
        if (__continue__) {
          setTimeout(__callback3, 100.0);
        }
      }
    }

    if (__run__) {
      setTimeout(__callback1, 100.0);
    } else {
      if (__continue__) {
        setTimeout(__callback2, 100.0);
      }
    }
  }

  if (__run__) {
    setTimeout(__callback0, 10.0);
  } else {
    if (__continue__) {
      setTimeout(__callback1, 10.0);
    }
  }
}

PythonJS now faster than CPython

2014-05-18T18:48:00.000-07:00

pystone benchmark

PythonJS using the dart backend is 6x faster than CPython.

pystone.py

nbody benchmark

PythonJS using the fast javascript backend is faster than CPython and even 2x faster than PyPy.

nbody.py

richard's benchmark

PythonJS using the dart backend is 7x faster than CPython.

richards.py

float benchmark

PythonJS using the dart backend is 3x faster than CPython.

float.py

Micro-Benchmarks

recursive fibonacci micro-benchmark

PythonJS using the dart backend is 27x faster than CPython and 7x faster than PyPy.

recursive_fib.py

simple add loop micro-benchmark

PythonJS using any backend, is about 30x faster than CPython.

add.py

The benchmarks were run with node.js 0.10.22, pypy 1.9, and dart-sdk 1.0. To run them yourself: get the latest PythonJS source from github, and download and extract the dart-sdk to ~/dart-sdk-1.0. The benchmark eps graphs are written to "/tmp"

cd PythonJS/regtests
./run.py