Improve GIL in 2.7

@loewis

BPO	8299
Nosy	@loewis, @rhettinger, @terryjreedy, @pitrou, @kristjanvalur, @bitdancer, @florentx, @Bluehorn
Files	gil.patch gil2.patch ccbench.patch: improved ccbench with -y option and 'balance' fair.py: Fair CPU use test fair.py: A modified version of David's "fair.py" thread_pthread.h: Modified cond var GIL with fairness. evalsrv.rar

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2010-08-06.16:50:13.023>
created_at = <Date 2010-04-03.10:27:23.069>
labels = ['interpreter-core', 'performance']
title = 'Improve GIL in 2.7'
updated_at = <Date 2010-08-06.16:50:13.022>
user = 'https://github.com/kristjanvalur'

bugs.python.org fields:

activity = <Date 2010-08-06.16:50:13.022>
actor = 'terry.reedy'
assignee = 'none'
closed = True
closed_date = <Date 2010-08-06.16:50:13.023>
closer = 'terry.reedy'
components = ['Interpreter Core']
creation = <Date 2010-04-03.10:27:23.069>
creator = 'kristjan.jonsson'
dependencies = []
files = ['16744', '16770', '16867', '16946', '16951', '16958', '17034']
hgrepos = []
issue_num = 8299
keywords = ['patch']
message_count = 64.0
messages = ['102234', '102235', '102248', '102250', '102252', '102254', '102255', '102256', '102290', '102291', '102416', '102500', '102502', '102506', '102507', '102508', '102758', '102759', '102823', '102827', '102828', '102830', '102831', '102838', '102840', '102845', '103087', '103088', '103093', '103098', '103215', '103219', '103224', '103225', '103228', '103230', '103232', '103233', '103234', '103235', '103236', '103238', '103239', '103258', '103310', '103311', '103355', '103384', '103385', '103396', '103397', '103399', '103500', '103525', '103533', '103551', '103782', '103785', '103922', '103923', '113028', '113039', '113087', '113111']
nosy_count = 11.0
nosy_names = ['loewis', 'rhettinger', 'terry.reedy', 'beazley', 'pitrou', 'kristjan.jonsson', 'techtonik', 'r.david.murray', 'flox', 'dabeaz', 'torsten']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = None
status = 'closed'
superseder = None
type = 'performance'
url = 'https://bugs.python.org/issue8299'
versions = ['Python 2.7']

This patch does several things:

Creates a separate lock type PyThread_type_gil and locking functions for that. This allows tweaking of the GIL without affecting regular lock behaviour.
Creates a uniform implementation of the GIL on windows/pthreads using macros, with emulated condition variables on windows (Lifted Antoine's code from py3k, adding own improvements to the slightly problematic windows implementation). This makes the GIL behave the same on windows and pthreads platforms, if we so choose, and allows cross-platform development.
provide three GIL implementations:
a) legacy gil, which is the same as the one used on pthreads
b) a roundrobin gil, which fixes the multicore problem on pthreads and exhibits the same behaviour as the legacy GIL on windows did (no jumping the gil queue)
c) a priority based gil, with n given priority levels, and optionally, the ability to request immediate GIL drop by the ceval.c loop.

See thread_gil.h for details of the three modes.

In my experiments using David Beazley's scripts from http://www.dabeaz.com/blog/dablog.html, implementation "b" fixed the performance problems encountered on multicore machines. This is, I believe, the original impetus for Antoine Pitrou's work on the new GIL.
Implementation "c" improved data transfer still, by allowing faster wakeup of completed IO.

Please note that I was not able to test this patch on a pthreads machine, I can only hope that it compiles :)

I think this is too late for 2.7. 2.7b1 will be released RSN, and we should not implement such a change after the first beta release.

Without looking at this patch, I think it would wise to proceed with caution on incorporating any kind of GIL patch into 2.X. If there is anything to be taken away from my own working studying the GIL, it's that the problem is far more tricky than it looks and that any kind of fix has the potential to adversely affect something that you might not expect.

That said, I would only suggest that any kind of "new gil" incorporated in 2.X be disabled by default and enabled with special configuration options for those who want to try it out.

Martin: Well, this patch was originally conceived more as a demonstration of the GIL problem and an alternative fix proposal.
However, it is possible to configure it so that there is no change from existing functionality, simply by not including thread_gil.h in thread_pthread.h and thread_nt.h. The only change would then be the presence of the new PyThread_type_gil and associating locking functions which delegate directly to the old PyThread_type_lock functions.

Antoine: Please take a look, the change is really simple, particularly the ROUNDROBIN_GIL variant which fixes the originally observed problem.
the GIL is still a lock, implemented using a mutex and a semaphore. It is modified to work exactly as the lock always has done on windows (which is why the original problem isn't present on that platform).

The simplicity of the change stems from the fact that the gil is still just a mutex-type object, which is aqcuired and released just as it has always been. The change is in the internal rules of the mutex, making sure that threads queue up properly and (optionally) that they are released in a priority based order.

I'm not sure where you're getting your information, but the original GIL problem *DEFINITELY* exists on multicore Windows machines. I've had numerous participants try it in training classes and workshops they've all observed severely degraded performance for CPU-bound threads on Windows (XP, Vista, and Windows 7).

Just ran the CPU-bound GIL test on my wife's dual core Windows Vista machine. The code runs twice as slow using two threads as it does using no threads (original observed behavior in my GIL talk).

Kristjan, I agree with Martin, it's probably too late to make such
changes for 2.7.
Additionally, your "round-robin" scheme only seems round-robin when
there are two threads competing. Otherwise, you could have three threads
A, B and C, and the GIL bouncing between A and B.

I would advocate opening a separate issue to improve the Windows
condition variable code under 3.x.

Silly question, I know, but why isn't the GIL just implemented as a lock of the host operating system? After all, we want mutual exclusion, I don't see why condition variables are required for this.

I have to admin that I did not look at the source, so the reason might be documented there.

It's not a simple mutex because if you did that, you would have performance problems much worse than those described in bpo-7946. http://bugs.python.org/issue7946

Sorry, what I meant with the "original problem" was the phenomenon observed by Antoine (IIRC) that the same CPU thread tends to hog the gil, even when releaseing it in ceval.c.
What I have been looking at up to now is chiefly IO performance using David's iotest.py, and improving the poor performance of IO. IO will not suffer as badly on windows because the IO thread will get its fair slice of execution time. Promted by you, I added this bit of code to the iotest.py:
spins = 0
laststat = 0
def spin():
global spins, laststat
task,args = task_pidigits()
while True:
r= task(*args)
spins += 1
t = time.clock()
if t-laststat > 1:
print spins/(t-laststat)
spins = 0
laststat = t

You are right, however that cpu throughput of multiple cpu bound thread suffers. And in fact, on windows, it appears to suffer the least using the LEGACY_GIL implementation. This is, I conjecture, because there are far fewer context switches (because relinqushing the GIL fails). My conjecture is that context switches between threads on two cores are so expensive as to dramatically affect performance. Normal multithreaded programs don't suffer from this because the threads are kept busy. But in our case, we are stopping one thread on one core, and starting another on a separate core, and this causes latency.

Now, I've improved my patch somewhat. First off, I fixed some minor errors in the PRIORITY_GIL implementation. But more importantly, I added something called FIFOCOND. It is a condition variable that guarantees the FIFO property. This was prompted by my observation that even Windows' Semaphore doesn't do that, rather the windows scheduler may allow the currently executing thread to jump ahead in the semaphore queue. The FIFOCOND condition variable fixes that using explicit scheduling, and is intended as a diagnostic tool.
(Antoine, your comment from 13:04 about "roundrobin" inasfar as that we don't know anything about the condition variable behaviour. I was assuming FIFO behaviour for the sake of argument, and I thought I´ put it in to the comments that we assume a general 'fairness' there. Put in the FIFOCOND and you will have that fairness guaranteed.)

At any rate, I believe my patch provides a useful platform for further experimentation.

Factoring out the gil as a separate type of lock (which it must be)
allowing for different implementation of the GIL
shoring up the Condition variable implementation on Windows
Providing a FIFOCOND_T type to enforce a particular scheduling order, and demonstrating how we can be explicit about thread scheduling.

I have already demonstrated that using the PRIORITY_GIL method fixes the problem with IO threads in the presence of CPU bound threads. Your iotest.py script is perfect for this, using 2 worker threads. On windows, the problem with IO wasn't so grave as I have explained (windows by default works as the ROUNDROBIN_GIL implementation, not the LEGACY_GIL mode used on pthreads). The PRIORITY_GIL solution is particularly effective with multicore on Windows, but it also improves IO throughput if cpu affinity of the server is fixed to one CPU, i.e. on singlecore.

I have no fix for CPU bound threads, and I honestly don't think such a fix exists, except by causing switches to happen far less frequently, e.g. by raising the checkinterval, and so mitigating the problem (which is what the new gil in py3k does with its timeout implementation) But the IO fix for pthreads

To summarise then:

The GIL has two problems on multicore machines
a) performance of CPU threads goes down
b) performance of IO in the presence of CPU threads is abysmal, but not on Windows
We can fix problem b) on pthreads with the ROUNDROBIN_GIL implementation.
We can improve IO performance in the presence of CPU threads on pthreads and Windows using the PRIORITY_GIL implementation, even to become faster than on a single core.
We cannot do anything about decreased performance of co-operatively switching CPU threads on multicore except switching less frequently. But this is quite feasible now with the PRIORITY_GIL implementation because it can request an immediate gil drop when IO is ready. So raising the checkinterval will not affect IO performance in a negative way.

Please have a look at the latest patch with IO thread performance in mind. It is currently configured to enable the PRIORITY_GIL implementation without the FIFOCOND on windows and pthreads.

I just did some profiling. I´m using visual studio team edition which has some fancy built in profiling. I decided to compare the performance of the iotest.py script with two cpu threads, running for 10 seconds with processor affinity enabled and disabled. I added this code to the script:
if affinity:
import ctypes
i = ctypes.c_int()
i.value = 1
ctypes.windll.kernel32.SetProcessAffinityMask(-1, 1)

Regular instruction counter sampling showed no differences. There were no indications of excessive time being used in the GIL or any strangeness with the locking primitives. So, I decided to sample on cpu performance counters. Following up on my conjecture from yesterday, that this was due to inefficiencies in switching between cpus, I settled on sampling the instruction fetch stall cycles from the instruction fetch unit. I sample every 1000000 stalls. I get interesting results.

With affinity:
Functions Causing Most Work
Name Samples %
_PyObject_Call 403 99,02
_PyEval_EvalFrameEx 402 98,77
_PyEval_EvalCodeEx 402 98,77
_PyEval_CallObjectWithKeywords 400 98,28
call_function 395 97,05

affinity off:
Functions Causing Most Work
Name Samples %
_PyEval_EvalFrameEx 1.937 99,28
_PyEval_EvalCodeEx 1.937 99,28
_PyEval_CallObjectWithKeywords 1.936 99,23
_PyObject_Call 1.936 99,23
_threadstartex 1.934 99,13

When we run on both cores, we get four times as many L1 instruction cache hits! So, what appears to be happening is that each time that a switch occurs the L1 instruction cache for each core must be repopulated with the python evaluation loop, it having been evacuated on that core during the hiatus.

Note that for this effect to kick in we need a large piece of code excercising the cache, such as the evaluation loop. Earlier today, I wrote a simple (python free) C program to do similar testing, using a GIL, and found no performance degradation due to multi core, but that program only had a very simple "work" function.

So, this confirms my hypothesis: The downgrading of the performance of python cpu bound threads on multicore machines stems from the shuttling about of the python evaluation loop between the instruction caches of the individual cores.

How best to combat this? I'll do some experiments on Windows. Perhaps we can identify cpu-bound threads and group them on a single core.

[...]

_PyObject_Call 403 99,02
[...]
affinity off:
Functions Causing Most Work
Name Samples %
[...]
_PyObject_Call 1.936 99,23
[...]
_threadstartex 1.934 99,13

When we run on both cores, we get four times as many L1 instruction cache hits!

You mean we get 4x the number of cache /misses/, right?

This analysis is gratuitous if you can't evaluate/measure/calculate the
actual cost (in proportion of total elapsed or CPU time) of the
instruction cache misses. Perhaps it is actually negligible and the
slowdown is caused by something else.

How best to combat this? I'll do some experiments on Windows.
Perhaps we can identify cpu-bound threads and group them on a single
core.

IMHO, the OS should handle this. I don't think ad-hoc platform-specific
CPU affinity tweaks belong in the Python core.

Martin, I´ve explained it in my other dissue, bpo-8411, with a step by step example.

Hmm. Can't find it there. What message or file should I be looking at?

Here are the results of running the fair.py test on a Mac OS-X system using a "fair" GIL implementation (modified condition variable):

[ Fair GIL, Dual-Core, OS-X ]
Sequential execution
slow: 5.490943 (0 left)
fast: 0.369257 (0 left)
Threaded execution
slow: 6.122093 (0 left)
fast: 6.179179 (0 left)
Treaded, balanced execution:
fast C: 3.345452 (0 left)
fast B: 3.389235 (0 left)
fast A: 3.426407 (0 left)
Treaded, balanced execution, with quickstop:
fast C: 2.557972 (0 left)
fast B: 2.558551 (35087 left)
fast A: 2.558914 (13142 left)

Here is the same test with the original GIL.

[Unfair GIL, original implementation]

Sequential execution
slow: 5.444754 (0 left)
fast: 0.361340 (0 left)
Threaded execution
slow: 5.542008 (0 left)
fast: 5.225690 (0 left)
Treaded, balanced execution:
fast C: 1.381929 (0 left)
fast B: 1.499969 (0 left)
fast A: 1.549571 (0 left)
Treaded, balanced execution, with quickstop:
fast A: 1.284043 (0 left)
fast B: 1.295507 (32490 left)
fast C: 1.294981 (274777 left)

Please observe that the performance of threads under the "fair" GIL are significantly worse than with the "unfair" GIL.

Having studied this in more depth, I have to say that I would much rather have fast-running unfair threads than slow-running fair threads. Although I agree that there are other benefits to fairness, they just aren't enough to compensate for the huge performance hit.

Sorry Martin, I meant bpo-8410.
I have so many of these going on :)

It is interesting to see, David, the difference in the behaviour of the semaphore based and condition variable based lock on linux. It is clear that the semaphore and the condition varable have different queuing characteristics. I wouldn't be surprised if the semaphore were "fair" on average, but that it somehow colludes with the scheduler allow the current thread to cut in early based on, perhaps, timeslices or something. The condition variable seems to be much more FIFO.

It is odd to me that you don't seem to be able to shake the performance hit even with a checkinterval of 10000. In the end, the performance hit is because of successful switches during the GIL yield in ceval.c. You ought to be able to raise the checkinterval up to the same level as the 'effective' one with the unfair condition variable approach and have the same sort of behaviour (inclusive IO latency). I'll do some modifcations locally to gather stats on successful yields vs unsuccessful ones.

As for this being academic, yes it is. As I stated early on, I never intended this to be checked in as is. My aim was to take a step back from the radical GIL rewrite in 3.x. Provide a simple platform for GIL experiments in the know and trusted 2.x series. See if we could achieve the desired features without the added unknowns of the new GIL.

And in fact, I also didn't want to expend this amount of ammunition on the "ROUNDROBIN_GIL" implementation. It was included merely as an example of one thing that was wrong with the old one and why it was so unpredictable. What I really wanted to look at was to combine the benefit of a large checkinterval (100000, say) with a priority based mechanism giving us a low IO latency. I've gotten so sidetracked with this whole fair/unfair business that I have neglected to provide useful performance benchmark of the PRIORITY_GIL implementation.

The fact that I had to expend so much effort explaining exactly how the gil on mac (LEGACY_GIL) was not fair, also shows how tricky working with synchronization primitives and threads can be, even using such relatively modern primitives as mutexes and condition variables. There are many pitfalls for the unwary, and in my experience, only a lot of exposure will bend your brain into taking all those pesky race conditions into acccount. This is why I think it is important to _understand_ the problem completely before trying to fix it, and why fixing it stepwise, with a full understanding of each step along the way, is necessary for robust behaviour.

As for missing the boat on 2.7, well, there is always 2.8 isn't there :)

David, trying to get some more realistic IO benchmarks I did some more tests. The idea is to have a threaded socket server, serving requests that take different amounts of time to process, and see how io response measures up for two classes of requests being serviced simultaneously.

Please see the evalsrv.rar for the client and server scripts. The client uses multiprocessing to distance itself from the GIL issue. The results on my dual core windows box are as follows (LEGACY_GIL is the mac, unfair GIL, ROUNDROBIN_GIL is the same with the fairness fix. "with affinity" means that the server process is restricted to running on one core.

label time avg time std.dev

serial
((30, 500), (2.7145072907310848, 0.09047581466359553, 0.0041867466462554535))
((300, 10), (0.46542703053041656, 0.0015481250643121787, 0.0002282114778449236))

3.36s (3.18s) (total time, sum of individual classes)
simultaneous
((30, 500), (2.8820070707310563, 0.09605833283280416, 0.004430198440914231))
((300, 10), (3.2082978423235358, 0.010690429928943495, 0.014415958519681225))
3.21s (6.09s)

(for each test, you get the indvidual timing for each request class, and then a sum of total time and sum of individual times.)
Please don't read too much into small differences, this is a roughly one-off test here and likely contains noise.
A few things become apparent:

with LEGACY_GIL, affinity appears not to matter. The 300 fast requests take longer to complete than the 30 slow requests if done in parallel, even though their serial execution time is roughly 1/5th.
With ROUNDROBIN_GIL, serial performance appears not to be affected, but simultaneous performance is much better: end-to-end time is the same, but the sum of individual classes is lower. That means that the client had to wait less for their IO results.
With ROUNDROBIN_GIL, if we put affinity on, we get the same kind of performance as with the LEGACY_GIL.

The most important points here are the two last ones, I think. The fact that the sum of the individual request waits goes down is significant, and it is by no small amount that it drops. But equally perplexing is the fact that forcing the server to one cpu, removes the "fairness" again. It would appear that the behaviour of the synchronization object (an windows Semaphore in this case) changes depending on the number of cores, just as you had previously mentioned. This is, however, a windows only effect, I think. I must try to find out what is going on.

Sorry, all the benchmarks were missing from my last comment. Here they are:
total time avg time/request stddev
LEGACY_GIL
serial
((30, 500), (2.6908777512225717, 0.08968708313486773, 0.0015164644061278888))
((300, 10), (0.4470272758129874, 0.0014862882945551276, 0.00024585959927491124))
3.30s (3.14s) [end-to-end time, ( sum of total times)
simultaneous
((30, 500), (2.876280574765787, 0.09586895587753937, 0.0011848842024515473))
((300, 10), (3.2645357161315194, 0.010877886016239504, 0.027372366395914234))
3.27s (6.14s)

LEGACY_GIL, affinity on
serial
((30, 500), (2.8003825905247735, 0.09333925587376796, 0.013360667457052446))
((300, 10), (0.45636945477707364, 0.001517769716542185, 0.00020067298808990378))
3.43s (3.26s)
simultaneous
((30, 500), (2.8963266281049687, 0.09653735553913509, 0.0096323348697044))
((300, 10), (3.212417639672081, 0.010703065845891957, 0.014433573531885001))
3.21s (6.11s)

ROUNDROBIN_GIL
serial
((30, 500), (2.703059536896449, 0.09009326837163198, 0.0020036630273102198))
((300, 10), (0.4539455433581643, 0.0015096094615377107, 0.00041274624690973134))
3.32s (3.16s)
simultaneous
((30, 500), (3.2613636649350686, 0.10870530565570016, 0.02613516235517462))
((300, 10), (1.4737764855589188, 0.004908413639163637, 0.002818028070766402))
3.26s (4.74s)

ROUNDROBIN_GIL, affinity on
serial
((30, 500), (2.7145072907310848, 0.09047581466359553, 0.0041867466462554535))
((300, 10), (0.46542703053041656, 0.0015481250643121787, 0.0002282114778449236))

3.36s (3.18s)
simultaneous
((30, 500), (2.8820070707310563, 0.09605833283280416, 0.004430198440914231))
((300, 10), (3.2082978423235358, 0.010690429928943495, 0.014415958519681225))
3.21s (6.09s)

ROUNDROBIN_GIL, USE_FIFO_COND
serial
((30, 500), (2.7050386292112547, 0.09016071176643957, 0.0017078174777967025))
((300, 10), (0.4478294727402505, 0.001488698051474884, 0.00012887034661806298))
3.31s (3.15s)
simultaneous
((30, 500), (3.262343957123042, 0.10873750248518549, 0.02640198429431565))
((300, 10), (1.5242242379967286, 0.005076519036171731, 0.0033917786033315026))
3.26s (4.79s)

For better or worse, this did not make it into 2.7.
Is it the sort of thing that could still go into a bug-fix release, without alpha/beta testing?
Or should this be closes as out-of-date?

That question should probably raised on python-dev, not the bug tracker.

Although I did finally manage to explain the point of this patch (after a long, long discussion), I think the issue is still too controversial. We did, for example, see some strange behaviour in my last comment (Date: 2010-04-21 23:22) regarding affinity fixing of the process!

What I hope comes out of this is that I think I have put my point across that with multithreading, a lock is not a lock. While a mutex may be indeed a mutex, its behaviour towards the threads that want to claim it can be different and can affect program behaviour and performance.

This also goes for "emulated" or "constructed" entities, built out of something more primitive such as condition variables.

Since 2.x is now frozen, and everyone seems happy (I think) with the more complicated 3.x method (although, being more complex, probably has more surprises in store), we should probably just let this fade away.

OK. I would probably be better to expend energy on the 3.x new GIL, should issues arise.

I realise that this comment is VERY late. But for the record I would like to point out that the entire discussion about O/S schedulers being good at scheduling threads may well be off-target. The O/S scheduler is attempting to schedule multiple threads in all processes (not just Python's) across multiple cores - but it knows NOTHING about the GIL, and so has to consider running all threads just so they can compete for the GIL which is both inefficient AND unpredictable as to which thread will manage to get the GIL.

In this specific case, it seems to me that it would be more sensible for Python to track which threads are requesting the GIL and to decide which thread should get the GIL next and make that specific thread runnable and give it the GIL. In faster-cpython/ideas#328 this is referred to as a GIL scheduler, but because the GIL is attempting to schedule one thread on one core (as determined by the O/S) a GIL scheduler would be far, far simpler than an O/S scheduler.

In my opinion (based on sound Computer Science principles established back when computers had a single core and schedulers were accordingly much simpler), a simple scheduler would try to give priority to threads that need short bursts of CPU, and scheduler threads that need lots of CPU once all I/O threads are waiting. In this case all that would be needed would be two FIFO queues, one containing now runnable threads that previously had voluntarily given up the GIL (e.g. to undertake I/O - what I call CPU-light threads) and the other containing now runnable threads that reached the CPU limit and had the GIL taken from them (CPU-heavy threads), with the GIL scheduler taking the first item in the CPU-light queue first, and only taking items from the CPU-heavy queue if the CPU-light queue is empty.

See also:
#52194
faster-cpython/ideas#328

This is the wrong place to post such a comment as, except possibly for R. Hettinger, whose only comment was to post elsewhere, none of the discussants above are still active in core python development. Only A. Pitrou is still commenting on what others do. To put it another way, none except possibly one of the people who would be involved in such a core change will read this.

Before posting again, read the GIL-related discussions earlier this month.
https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-python_11.html
https://pyfound.blogspot.com/2022/05/the-2022-python-language-summit-per.html
and possibly others from the language summit.

@terryjreedy

@terryjreedy I did say that it was A) very late, and B) for the record.

But thanks for the two links which are definitely relevant.

Navigation Menu

Repository navigation

Improve GIL in 2.7 #52546

41 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Improve GIL in 2.7 #52546

Description

Activity

kristjanvalur commented on Apr 3, 2010

loewis commented on Apr 3, 2010

dabeaz commented on Apr 3, 2010

kristjanvalur commented on Apr 3, 2010

kristjanvalur commented on Apr 3, 2010

dabeaz commented on Apr 3, 2010

dabeaz commented on Apr 3, 2010

pitrou commented on Apr 3, 2010

Bluehorn commented on Apr 3, 2010

dabeaz commented on Apr 3, 2010

kristjanvalur commented on Apr 5, 2010

kristjanvalur commented on Apr 6, 2010

pitrou commented on Apr 6, 2010

41 remaining items

loewis commented on Apr 18, 2010

dabeaz commented on Apr 19, 2010

kristjanvalur commented on Apr 20, 2010

kristjanvalur commented on Apr 20, 2010

kristjanvalur commented on Apr 21, 2010

kristjanvalur commented on Apr 21, 2010

terryjreedy commented on Aug 5, 2010

rhettinger commented on Aug 5, 2010

kristjanvalur commented on Aug 6, 2010

terryjreedy commented on Aug 6, 2010

Sophist-UK commented on May 23, 2022

terryjreedy commented on May 23, 2022

Sophist-UK commented on May 24, 2022

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions