Docker, Singularity, et. al #1606

Since you're here: is there a reason you opted to use meson instead of uv? I know the latter is the new hotness and so maybe invites worries on the newness front, but the former is also quite new and not as nimble.

We're trying meson (still very much a work in progress and I'm not 100% sure it will stick as I'm banging my head against the wall on trying to get it to handle static files / data properly) because that's what scipy and numpy are moving to. We were previously using numpy.distutils (which is deprecated) and thought that following the same path as numpy might make sense. There is a fair bit of c code and cython in PYME in that needs to be built along with the python modules. We were previously using the numpy runtime dispatch feature (to enable support of different SMID instruction sets in one binary) within PYMECompress - currently disabled while we try and get meson working - and figured that using the same tooling and numpy would give us the best chance of getting that running again.

I haven't looked into uv in any great detail, but if I can't find a good solution for the static files problem might have a look.

Really pleased to see this conversation happening. In case it is at all helpful, here's a link to @zacsimile 's go at docker #1219

Why wasn't that PR ever accepted? It doesn't appear to affect any of the mainline work and only adds value?
@zacsimile or @barentine, do you remember?

I don't recall closing it, but as far as I remember it was never quite fully working (looking at the comment history it looks like, e.g. pymecompress didn't get bundled) and it was developed at a time when all the distributed analysis code was changing rapidly meaning that the mainline had moved far enough away that it would no longer run.

For sure, it seems like acquisition is part of the front-end: interacting with the investigator and the hardware and such. Is the way that such hardware is presented to Python via manufacturer-provided packages? Or does the hardware present itself in some way that's standard to Windows?

It's mainly the camera drivers that are hard. You get a driver (kernel code + user space DLL to talk to the driver), usually as an executable installer from the manufacturer. These are often behind a login or some other form of authentication, and often not re-distributable. They need admin privileges to install (but not to use). We talk to the DLL using ctypes. No standardisation of APIs etc across manufacturers (ie Hamamatsu has a different API to Andor etc). Often a single driver interface for different cameras from the same manufacturer, but not always the case.

I'd suggest using a message queue fronted by an API that allows workers to call in, find/check out a qualified task, and check in results (assuming they got it done in time/didn't die/etc.). Workers shouldn't need to be called if there's a single well-known endpoint for handing out tasks within the cluster.
Celery might be a solution here?

The task distribution is a bit special (needs to be low latency, high task rate (thousands of tasks per second), and data-locality aware, all while keeping within a fairly tight bandwidth limit). This meant that there were not any off the shelf solutions which ticked all the boxes - at least when it was written. But you are right in that the workers themselves are purely clients. There is, however, a global master queue server as well as an overseeing server and one or more data servers on each node that provide a virtualised distributed filesystem. The node-level servers use mDNS to discover the global server, the workers use mDNS to discover the local server and all the data servers. It they need data which resides on a different node they will access it by connecting directly to the data server running on that node. The global server advertises a list of tasks, and the node-level servers "bid" on those tasks, with a cost which depends on whether they hold the data locally or not. The workers poll the node server for tasks. If it all sounds a bit convoluted, it is, but it allows real time analysis of GByte/s data streams on a cluster which only has 1Gbit internal networking and around 100MB/s disk bandwidth on each node (ie HDDs rather than SSDs).

I know it was a horrid option when you started all of this (in the aughts, iirc?), but Apache Spark has matured greatly since then and seems to account for many of the needs you're trying to address vis-a-vis getting processes to run where there data is, if possible, and accounting for the ins & outs of those processes

In an ideal world we'd be able to support a range of different schedulers, but I strongly suspect our bespoke one would still have an edge for the use case(s) it was developed for (at the expense of not being very general). We take a bunch of shortcuts (e.g. no random-access to files, but rather atomic chunk access which allows us to forego / simplify locking) which you simply can't do in a general purpose solution. For that reason I would want to keep it available. That said it's super contrived on e.g. a larger shared storage / shared memory kind of scenario, and it would be great if we could switch it out on a case-by-case basis for alternative schedulers and storage backends. This would likely add more rather than less complexity, however.

The one you've written is unique and capable, and does a great job of meeting the constraints it was built to meet. I'd love to see it made available for any project that needs to invert the usual relationship between locality and workload: filesystems like Ceph and Gluster get data to where it's needed -- but only after a request has been made and the requesting process is presumably blocked until the data shows up. If I understand correctly, your system spreads data out among nodes that can host workloads for that data, and then tasks are meted out based on locality and suitability?

Unfortunately test coverage is pretty rubbish, and it's not safe to assume that all the important functionality is covered. I suspect it would take the best part of 3-4 person-weeks to get the test coverage into a reasonable state. In general the UI stuff has the worst coverage, and the distributed analysis coverage is better. The tests are generally good enough to capture major breakages, but can let some more subtle ones through.

There are also a couple of tests (typically the fitting algorithm ones) that fail stochastically as they use randomly generated synthetic data and test whether the result lies within a confidence interval of the expected value (ie we would expect the test to fail ~5% of the time). We should probably change those to use a fixed random seed.

Started discussion #1608

I think the main appeal of containerisation is that it there are so many parts (and hence dependencies). Splitting the project up has it's own benefits, risks and challenges and is a much bigger project than containerisation (and is really orthorgonal - you'd still likely want to include all components in a container, along with the chunks that we have already split out).

Lol okay, so I've got an HPC that researchers have been writing code for, but it's not great code and doesn't make very good use of constructs in that environment. My team loves looking at tiny things using SMLM, not code. Part of my job is to make it easy for them to get the science done fast without having to fiddle around with code so much. Part of my work is also applying the DRY principle to work done by researchers: collectively, Don't Repeat Yourself. I see my team reinventing infrastructural things every few years as people leave the lab and new projects start. PyME does a great job of presenting a standard research flow and giving some pre-built implementations of well-known tools, and that structure helps alleviate some of that tedium and repetition I've seen.

Therefore, I'd like to use PyME as a basis for developing a facility for getting data from scopes to HPC storage, coordinating work, and archiving results. In order of importance, that breaks down to:

1. Doing Work: Analysis routines that are portable and scalable, both horizontally and vertically. PyME's analysis modules cover a lot of needs here and seem to be well-organized.
2. Coordinating Work: I likely have a different set of needs from many current PyME users here, but making the coordination layer a pluggable interface that can be met by whatever provider meets the user's/lab's needs seems useful? I suspect that something like Spark, which can reason about infrastructure, storage, and heterogenous environments, will fit the bill for me and may do for others.
3. Coordinating Storage: again, my needs are likely different, but making storage a pluggable concept rather than something that the core of PyME needs to worry about seems like it'd reduce complexity of that core. The work that was done to integrate with OMERO comes to mind as an example.
4. Getting data: that's acquisition, and I think this is really where PyME shines (no pun intended but lol). Something that helps a researcher quickly get to the point of having usable data that's good enough to start refining questions seems like a powerful tool, and PyME's got the bones to be that in a world that's increasingly automating the process of refinement.

Apologies if this is a long answer to a short question, but y'all have clearly put a lot of consideration into PyME, so I figure it'd be meet for me to do the same.

Hi @gynnantonix , @zacsimile and I have been traveling (at a conference together) so am sorry for radio silence here, but it's great to hear where you're coming from and see your interest in using and improving PYME! In my postdoc lab I've seen PYME be pretty useful to folks who need most of pre-canned acquisition to just work, and end up benefiting from some performance they wouldn't have coded in themselves, while still having the ability to add in that one-off weird thing they need to add to it for their setup. Alls to say I can appreciate some of the reasons for your interest. Welcome, and if you ever want to chat about parts of it I'm happy to help on-board (though strategic/refactoring things are @David-Baddeley 's domain [thankfully ;) ])!

Hi @barentine! :) So yeah, I'm tasked with getting the heavy lifting for segmentation and whatnot for Rothenberg Lab @ NYU onto the HPC so that they can run faster and not have to keep buying increasingly expensive GPUs that sit idle too much. They're running what appears to be an older version of the Ries Lab's SMLM package, and that's tied to Windows, so it's not possible to run it on HPC. Also proprietary.
I have no doubt I'll have questions for you and @David-Baddeley soon, as I'm currently packaging the analysis code so it can run separately from the frontend, and I'm seeing stuff that's frontend-specific (like, having to do with calibration of optical chain, e.g.).