Avoiding Thread Contention When Using Multiprocessing with Finesse and Cython

When running Monte Carlo simulations or other computational workloads, it's common to use Python's ProcessPoolExecutor to parallelize multiple independent tasks. This approach works well—until it interacts with low-level libraries that themselves use multi-threading under the hood.

The Problem: Thread Over-Subscription

In a recent project, I ran into a significant performance issue while executing a large number of Monte Carlo trials using a process pool with 30 worker processes on Megatron (with 48 cores). Each trial ran a function that internally used cython.parallel.prange for fast, element-wise operations, which is what Finesse uses under the hood for many internal numerical calculations. Cython, via OpenMP, was configured to use all available CPU threads per process by default.

This resulted in severe thread over-subscription. With 30 parallel processes and each process attempting to use all 48 threads, the system was launching over 1,400 threads concurrently. The CPU quickly became saturated, and the tasks stalled. In some cases, the system became unresponsive, and the jobs had to be canceled repeatedly.

This happens because when the function calls into these libraries from within a Python multiprocessing context, each subprocess will attempt to use the full number of threads available to the machine.

The Solution: Limit Threads per Process

The solution is simple: explicitly limit the number of threads each subprocess is allowed to use. This can be done by setting the environment variable at the top of your script, before importing any thread-hungry libraries like Finesse.

      
import os
os.environ["OMP_NUM_THREADS"] = "1"

    

By setting OMP_NUM_THREADS to "1", we ensure that each multiprocessing worker uses only one thread internally, preventing them from overloading the system and allowing the tasks to run more efficiently.

Action items for facilities

• Ceiling lights replacement
• Fume hood cleaned and refurbished
• Sink countertop cleaning
• Floor cleaned and waxed
•  

Today we started re-configuring the vacuum chamber components.

Particle count
• 10:45 am : Starting the particle count in the clean room.
• 11:13 am : Finished the particle count in the clean room.
• Zone 3 :
  • 0.3 u: 2535
  • 0.5 u: 1413
  • 1.0 u: 789
• Zone 4 :
  • 0.3 u: 457
  • 0.5 u: 290
  • 1.0 u: 207
  Starting the reconfiguration.
  • 11:21 am: Started removing up-to-air valve
  • 11:30 am: Finished removing up-to-air valve.
  • 11:33 am: Started removing calibrated Ar Leak and the elbow.
  • 11:35 am: Finished removing calibrated Ar Leak and started assembling the up-to-air valve.
  • 11:47 am: Finished assembling up-to-air valve.
  • 11:48 am: Started removing magnetron gauge.
  • 11:55 am: Finished removing magnetron gauge.
  • 11:57 am: Started installing the elbow and calibrated Ar Leak and started removing the RGA probe
  • 12:13 pm: Finished installing the elbow and calibrated Ar Leak.
  • 12:20 pm: Break for lunch.
  • 01:25 pm: Come back from lunch break.
  • 01:30 pm: Started removing RGA line.
  • 01:39 pm: Finished removing RGA line.
  • 01:39 pm: Started removing gate valve [RGA line] and finished removing gate valve.
  • 02:00 pm: Started installing gate valve in the new position.
  • 02:08 pm: Fished installing gate valve [RGA line].
  • 02:09 pm: Checked the screws in the elbow to gate valve. [RGA line].
  • 02:28 pm: Finished checking the screws in the elbow to gate valve [RGA line].
  • 02:29 pm: Started installing RGA line.
  • 02:39 pm: Finished installing RGA line.
  • 02:50 pm: Started installing magnetron gauge.(In this part we assembled gauge with the used gasket)
  • 03:02 pm: Finished installing magnetron gauge.
  • 03:04 pm: Started installing RGA probe [RGA line].
  • 03:13 pm: Finished installing RGA probe [RGA line].
  • 03:15 pm: Connected the cables to the magnetron gauge.
  • 03:20 pm: Started the testing in the vacuum chamber.
  • 04:30 pm: Started installing the RGA 200 and connected the cables (power cable and DB9 cable).
  • 04:52 pm: Finished installing the RGA 200 and connected the cables (power cable and DB9 cable).
  • 04:53 pm: Started testing the RGA connections. Logrus is able to recognize and connect to RGA unit. We are leaving the turbo pump on for a few hours before checking back for pressure readouts.
  • 05:00 pm : Starting the particle count in the clean room.
    1. Zone 3 :
       • 0.3 u: 2244
       • 0.5 u: 997
       • 1.0 u: 207
    2. Zone 4 :
       • 0.3 u: 2993
       • 0.5 u: 1579
       • 1.0 u: 498
  • 05:30 pm : Finished the particle count in the clean room.
  • 05:34 pm: Test pressure readout from the gauges.
    1. Channel 1 : 1.06E-5 mbar
    2. Channel 2 :8.89E-6 mbar
    3. Channel 3 :5.0E-4 mbar
  • 05:59 pm: Test pressure readout from the gauges.
    1. Channel 1: 9.09E-6 mbar
    2. Channel 2: 7.75E-6 mbar
    3. Channel 3: 5.0E-4 mbar

[Jon, Cao]

1. Re-routing of cables

We re-routed the connections between the turbo pump and its fan to the controller. Instead of going through the side of the server rack, they are now routed along the the cable tray and came down from the top of the server rack.

2. Planning for vacuum assembly re-configuration

• Move the entire RGA arm to the mirrored CF port, where the Up-to-Air valve is at
• Move the Up-to-Air valve to the calibrated leak port
• Move the calibrated Ar leak the main chamber full-range gauge port
• Move the full-range gauge to the RGA line port

3. First test pump-down

1. With all valves closed, we started scroll pump, pump line quickly got down to 6.08 mbar from atmospheric 1000 mbar (measured by Pirani gauge, channel 3 on controller )
2. We open Lesker angled valve and let the RGA arm pumped down, Pirani gauge read 6.3 mbar while the full-range guage on RGA line reads 4.9 mbar ( channel 1 on controller )
3. We open the pump line gate to expose the pump to the main volume, all gaugues readout immediate rise back up 1000 mbar. After 3 minutes, we started to see channel 3 slowly dropped down. A minute later channel 1 and 2 (main body) also dropped down. The slow pressure dropping speed and 6.3 mbar measured earlier got us suspected that there is some large leaks
4. We proceed to tighten all the ports as the vacuum is pumped down. In particular, we found that large feedthrough port still required a lot of tightening up
5. As we tighten up all the ports, after 40 minutes, the gauges are now
   • Channel 1 : RGA line full-range gauge: 2.55E-1 mbar
   • Channel 2 : Main chamber full-range gauge: 2.60E-1 mbar
   • Channel 3 : Pump line Pirani gauge: 2.94E-1 mbar
   Compare this to the scroll pump manual , Table 1, page 3, the ultimate pressure of the scroll pump is 2.5E-1 Torr (3.3E-1 mbar), we thus managed to achieve scroll pump ultimate pressure
6. Turn on turbo pump : Change turbo pump controller from REMOTE to FRONT PANEL mode by pressing both "COUNTERS" and "MEASURE" buttons at the same time, select "MODE=FRONT"
7. Shorting interlock pin: since we do not have an interlock signal for the controller, use the provided DB-9 connector that has pin 3 and 8 shorted and connect this to the P1 IN connection at the rear of the controller (see attachment 1 )
8. Press "START" on the controller to start the turbo pump
9. The pressure readout from the gauges quickly dropped down. After 3 minutes, the Pirani range is maxed out at 0.5E-3 mbar. After 20 minutes, we recorded the following values:
   • Channel 1 : RGA line full-range gauge: 1.50E-5 mbar
   • Channel 2 : Main chamber full-range gauge: 1.89E-5 mbar
   • Channel 3 : Pump line Pirani gauge: 5.0E-4 mbar
   This is Medium vacuum , we want to further reduce this by 2 orders of magnitude. However, we can run RGA test + helium leak test at this pressure
10. Turn off turbo pump, wait for 10 minutes, turn off scroll pump, open Up-to-Air valve, all pressure gauges indicated pressure returned back to atmospheric pressure

3. To-do actions

• Run RGA test to get information about contamination status of vacuum
• Implement suggested changes in section 2
• Check and modify suspected poor connection: Pirani gauge on pump line. A gap can be seen between connection. There's no good way to tighten it with the screw. Maybe use threaded pin + hex bolt?
• Controller communications

Summary : We resolved problems with heaters tripping power and were able to proceed with chamber baking

1. Circuit connection adjustment

After yesterday elog 137, today we resolved most of these issues. After Jon contacted Facilities, the LP3B 6 circuit was reset and the cleanroom filter & light panels resumed to work as normal.

Regarding the connection of the heaters. we made the following adjustments:

• High-temperature controller powering lid + upper volume heater: connect to LP3B 8 circuit (clean room sides, 2 outlets)
• High-temperature controller powering bottom + lower volume heater : connect to LP3B 4 circuit (workstation side, 2 outlets)

2. Replacement of vacuum nipple insulation

We had also received new insulation pieces from Worbo today to replace the existing insulation with the new ones (see images). These cover the 2 4" tubes (for 6" flanges ) and the 4 1.5" tubes (for 2.75" flanges). The new insulations fit perfectly on these tubes. I also placed all insulation taken off from the last elog back onto the chamber ( these are insulations for the pumps and RGA lines).

3. Baking

• Starting set point: 40 deg C
• Step increase: 10 deg C up to 80 deg C, 5 deg C from 80 deg to 120 deg C
• At each step, the temperature readouts show approx. 2.1 deg C overshooting, wait to settle back to approx 1.5 deg C overshoot before increase the set point again

The temperature of the chamber reached as stable 120 deg C without any power issues at 3:45 pm. I waited another 15 minutes to verify its stability and the official baking duration started at 4:00 pm Jun 29 2023. Since we are baking at 120 deg C instead of the standard 150 deg C for Aluminium, the duration for the bake will be over three days until Monday morning, upon which we will slowly ramp down the the temperature.

[Jon, Shane, Luis]

My repair of the internal ribbon connecting the ADC to the adapter board resolved the timing signal problem. After this repair, we were able to start the front-end IOP model and checked out the RTS diagnostic screens (pictured below). All indicator lights were green except for the DK flag (indicating the DAC outputs are not enabled) and the DAQ flag (indicating that the system is recording data to disk). Those were both as expected, because the DAQD data acquisition service was not set up yet and the DAC outputs are not enabled until at least one user model (which outputs signals to the DAC) is started. I created and installed a simple user model (C1MSC) and confirmed that the DK flag clears once this model starts.

I later attempted to set up the DAQD service, which is needed to save data, but am yet to successfully debug it. I have received some guidance from one of LIGO's CDS experts and will try it at my next opportunity for lab work.

Cross-linking instructions: How to run a Jupyter notebook in your custom Conda environment

The new FLIR A70 infrared camera has arrived. Tyler and I unpacked it in the lab yesterday. In less than an hour, we succeeded in powering it on and connecting it to the lab network. We have assigned it the static IP address 192.168.1.6.

Online Configuration Portal

The FLIR camera can be configured, as well as stream live data, through a web browser interface. It can be accessed from any workstation on the lab network by navigating in the browser to http://192.168.1.6. The login credentials are stored here (log in with your LIGO.ORG credentials).

Next Steps

The next step is to install FLIR's Python API for controlling and reading out the camera on chimay. The API comes with demo codes which we can use to test the basic connectivity and which will serve as a reference for developing our own Python interface over the summer.

Summary

I have installed the requisite software on chimay for interfacing the FLIR A70 camera in Python. There are two packages required from FLIR:

• Spinnaker SDK, which provides the low-level camera drivers and a C/C++ API.
• PySpin, a wrapper of the Spinnaker library which provides the Python API.

These installations did not work out-of-the-box for Debian 11 (only Ubuntu is officially supported). I had to make several modifications which are documented below for future reference.

This setup has not yet been tested with the camera connected to chimay.

Documentation and Demo Codes

The PySpin package comes with a number of Python demo codes and a complete API reference. These can be found on chimay at the following locations.

• Example codes: /opt/spinnaker/python/Examples/Python3/
• Python API reference manual: /opt/spinnaker/python/docs/PySpinDoc.pdf

Installing Spinnaker SDK

Below were the steps required to install Spinnaker on chimay (Debian 11).

1. Download the Spinnaker binaries (AMD64 architecture) and copy the tarball to, e.g., /home/controls on chimay.
   
   
2. Unpack the tarball contents and enter the new directory:
   $ tar -xf spinnaker-2.6.0.160-Ubuntu20.04-amd64-pkg.tar.gz
$ cd spinnaker-2.6.0.160-amd64
   
   
3. Next, install the dependencies (on chimay, these were all already installed):
   $ sudo apt-get install libavcodec58 libavformat58 \
libswscale5 libswresample3 libavutil56 libusb-1.0-0 \
libpcre2-16-0 libdouble-conversion3 libxcb-xinput0 \
libxcb-xinerama0
 
4. There is one additional dependency, qt5-default, which is obsolete in Debian and no longer available via the package manager (that is, its functionality was absorbed into other Qt packages). I was able to find a workaround based on these instructions.
   
   
   1. Install all the dependencies of qt5-default:
      $ sudo apt-get install qtbase5-dev qtchooser qt5-qmake qtbase5-dev-tools
 
   2. Manually remove the qt5-default dependency from the Spinnaker package.
      
      Unpack the spinview-qt_2.6.0.160_amd64.deb package:
      $ mkdir tmp
$ cd tmp
$ ar -x ../spinview-qt_2.6.0.160_amd64.deb
$ tar xf control.tar.xz

Open the file control in a text editor and delete the qt5-default dependency from the Depends list.
      
      Then repackage the contents:
      $ tar cfJ control.tar.xz control
$ ar rcs ../spinview-qt_2.6.0.160_amd64.deb debian-binary control.tar.xz data.tar.xz
$ cd ..
$ rm -rf tmp

This overwrites the original package with a version no longer containing the qt5-default dependency.
      
      
5. Now proceed with running the install script:
   $ sudo sh install_spinnaker.sh

This will install the Spinnaker library at /opt/spinnaker. Spinnaker also provides a standalone GUI application, SpinView, which can be executed from the terminal (from any directory) via the command spinview.

Installing PySpin

The main challenge with installing PySpin was that it is currently only supported for Python <=3.8. The system installation on Debian 11 is Python 3.9 and 3.8 is not available within the package manager. Following these instructions, I manually installed a second version of Python (3.8) on chimay, in a way that should not interfere with the system installation.

The Python 3.8 executable is in the system path and can be run only via the command python3.8. It is not symlinked to python or to python3. Those remain linked to the preexisting Python 3.9.

After installing Python 3.8, I proceeded with the installation as follows:

1. Download the PySpin package (x86_64 architecture) and copy the tarball to, e.g., /home/controls on chimay.
   
   
2. Unpack the tarball contents and into a new directory:
   $ mkdir python
$ mv spinnaker_python-2.6.0.160-Ubuntu20.04-cp38-cp38-linux_x86_64.tar.gz python
$ tar xf spinnaker_python-2.6.0.160-Ubuntu20.04-cp38-cp38-linux_x86_64.tar.gz


3. Move the new directory into the Spinnaker installation directory:
   $ sudo mv python /opt/spinnaker
$ cd /opt/spinnaker/python
 
4. Install the dependencies:
   $ sudo python3.8 -m pip install --upgrade numpy matplotlib
   
   
5. Finally, install PySpin itself:
   $ sudo python3.8 -m pip install spinnaker_python-2.6.0.160-cp38-cp38-linux_x86_64.whl

If this succeeded, you should now be able to enter import the package PySpin as

$ python3.8
>>> import PySpin

without error.

Today I tested the Spinnaker/PySpin software installations (detailed in ELOG #4) with the FLIR camera connected to chimay. It works!

Example codes

I was able to run several of the PySpin example codes. In particular, there is one which connects to the camera and streams live data to a pop-up Matplotlib window that looks very useful. It is called AcquireAndDisplay.py.

When running these, it is important to keep in mind that PySpin requires Python 3.8, which is not the default system version on chimay. So to run AcquireAndDisplay.py, for example, you must explicitly call the correct version of Python:

$ python3.8 AcquireAndDisplay.py

The standard python and python3 aliases are still linked to the system version (3.9), so calling these will result in a PySpin import error.

Git repository

I have set up a git repo for our FLIR camera control code. I have populated it with an Examples directory which contains the PySpin Reference Manual as well as all the example codes (see the README). There is a local copy of this repo on chimay at /home/controls/FLIR.

Other FLIR streaming software

In addition to the PySpin demos, there are several fully developed applications provided by FLIR. While we do not plan to use these long term, they may be very useful for debugging and cross-validation of our Python interface during development:

• Browser interface: From any web browser on the local lab network, navigate to http://192.168.1.6 and log in (credentials here). This interface supports live data streaming as well as full control of the camera settings.
  
  
• SpinView: A standalone application provided as part of the Spinnaker SDK. It supports streaming live camera data as well as saving images and videos. It can be launched from the terminal on chimay via the command: $ spinview
  
  
• Research Studio: This is FLIR's proprietary software, for which we have a one-year license. It can be launched from the terminal on chimay via the command: $ FLIRResearchStudio

Permanent cabling

Since everything appears to be working, I ran a permanent Cat 6 cable from the lab switch to the camera's power+I/O adapter. The adapter is plugged into a UPS-protected power strip overhanging the optical table, as pictured below. To prevent the adapter from unplugging itself under its own weight, I attached a zip tie around the adapter to hold it securely in place.

I set up the new Windows 10 laptop (pictured below), which arrived yesterday. This laptop is intended to be used for running lightweight Windows-only programs, such as the Thorlabs beam profiler software or the SRS RGA client. However, none of that software is installed yet.

Configuration details

As usual, the computer is configured with one shared account (username: controls) and the standard password. Note that it is connected to the campus wifi (UCR-SECURE).

If a connection to the lab's local network is required, then the laptop must be connected by an Ethernet cable to the switch in the top of the server rack.

Jon, Cao

Today we raised the height of the shelf overhanging the cleanroom laser table by 8 inches. This was done to create more vertical clearance between the top-loading vacuum chamber and the bottom of the shelf. The added clearance should make both removing the chamber lid and inserting large parts easier.

The procedure required unmounting the shelf and removing all eight vertical support posts (1" x 1" x 18.5" pieces of 80/20 unistrut). The support posts were taken to the machine shop and cut, retapped, and cleaned (coarsely, with IPA wipes) prior to reinstallation. We took care to minimize the contamination introduced into the cleanroom, but some amount of particulate from disturbing the shelf was unavoidable.

This work is completed, and the cleanroom is now ready for final cleaning (HEPA vac, mopping, and wiping down of all surfaces including the softwalls).

The Linux workstation (ws2) that used to sit on the blue workbench (now inside the cleanroom) has been mounted on a mobile cart, as pictured below. This is intended to be a clean cart that will be housed inside the cleanroom.

The cart is currently dirty and will need to be throughly wiped down (along with the computer monitor and peripherals) prior to being moved into the cleanroom. Once the cleaned cart has been moved inside, it should never be brought back outside the cleanroom and should never be touched with ungloved hands.

I also upgraded the OS to Debian 11.6 and upgraded the CDS workstation tools.

Jon, Cao, Peter

This morning Facilities delivered and installed two new cabinets with sliding glass doors.

The smaller of the two (36" W x 13" D x 84" H) has been installed in the Clean & Bake area adjacent the flow bench. The larger one (48" x 16" D x 84" H) has been installed in the back of the room next to the electronics bench. Both cabinets have been securely anchored to the wall in two places each for earthquake safety.

We also installed the sliding glass doors and leveled them. However, we have not installed any of the shelves yet because the cabinets are quite dirty from the installation. Everything needs to be wiped down with IPA wipes, and it will be easier to do that before the shelves are in place.

Jon, Cao

In effort to try to reduce the noise level inside the cleanroom, we have dialed all four HEPA fan-filter units (FFUs) down from HIGH to MEDIUM speed. These dials can only be accessed from inside the cleanroom, by bringing in the large ladder and opening adjacent ceiling tiles.

We tested three configurations, in each case with all the FFUs on either HIGH (initial state), MEDIUM, or LOW. We measured the ambient noise in each configuration.

Going from HIGH to MEDIUM yields the largest improvement, reducing the ambient sound intensity by 6 dB (i.e., by a factor of 4, corresponding to a ~35% reduction in perceived volume).

An additional 3 dB of noise reduction can be achieved by further reducing the fan speeds to LOW. However, even after allowing some extended settling time (few hours), we found the particle counts to be fluctuating right at the threshold zone for ISO Class 5. Thus we dialed the fan speeds back up to MEDIUM with the expectation that this will be sufficient for Class 5 performance.

The cleanroom now needs to be recertified with a fresh round of five-zone particle count measurements.

The two parts needed to complete the vacuum assembly (ELOG 70) have arrived.

• (10) 5/16"-24 x 1 3/4" threaded rods - for attaching the turbo pump reducing nipple to the CF 4.5" gate valve;
• (1) 45 degree CF 2.75" elbow for attaching the calibrated Ar/He leak to the chamber.

I left them laying on top of the ultrasonic washer. They both need to cleaned and baked following the standard procedure for stainless steel, as the threaded rods are visibly dirty.

This afternoon I made up a green 10 AWG grounding cable and connected it to the vacuum system.

One end is tightly connected to the bottom flange of the vacuum chamber (photo 1). It is run along and up the table framing to the top of the cleanroom, where it exits into the overhead cable tray in the same location as the other power cables. It drops down from the top of the server rack all the way to the bottom, where the other end is connected to the lab's electrical ground in the rear of the 240 V UPS (photo 2).

The connections were confirmed to be secure, but continuity testing with an ohmmeter remains to be done to confirm that the chamber and tabletop are indeed grounded.

Today I brought in a fresh supply of zip ties (we now have 1500 in the tool chest) and used them to permanentize the cable routing for the gauges, pumps, and RGA.

I also brought and installed a 3-foot 15A extension cable for powering the scroll pump. Installing the cable required shutting down the pumps, which I did and then reverted via the following procedure:

1. Close the 4.5" gate valve, 2.75" gate valve, and the bypass line angle valve.
2. Shut down the turbo pump.
3. Shut down the scroll pump.
4. Unplug the scroll pump and install the extension cable.
5. Power on the scroll pump.
6. Power on the turbo pump.
7. Open all three valves.

Incidentally, before I started, I noticed that the pressure in the main volume had reached 7E-7 torr, which is lower than the pressures seen last week. The system quickly returned to this pressure after I restarted the pumps.

For future reference, the calibrated Argon source has a leak rate of 7.55E-8 atm cc/s, or equivalently 5.74E-8 torr L/s. This can be used to calibrate RGA scans to units of physical leakage (outgassing) rate.

The electrical overload problem encountered in ELOG 130 has been resolved. The two heater controllers, which draw up to 14.1 A each, overloaded the UPS and tripped one of its circuit breakers, shutting off power to both power strips mounted above the optical table.

I reset the circuit breaker and rerouted the two heater power cables instead to two separate 20 A outlets in the overhead cable tray outside the cleanroom (both on the LP3B 6 circuit). The two high-limit temperature controllers are now permanently positioned, as shown in the photos. For now, the PID controllers have been left sitting at the table level. I am ordering extension cords that will enable us to move those up to the overhead shelf, as well. I ran the heaters in their new configuration for several minutes without issue. Thus we should be able to now proceed with baking the chamber.

For future clarity, I added labels to power strips around the lab indicating which ones are powered by the UPS. To avoid overloading the UPS, only sensitive electronics or devices that could be damaged by a sudden loss of power should be connected to these.

In response to reports that ants have been observed in the lab, I placed five ant bait traps around the room today. Each is sitting on the floor on top of a piece of aluminum foil, in areas unlikely to be inadvertently stepped on. They contain liquid which will spill out if picked up, so please take care not to disturb them.

Update: After seeing the strong response to the first set, I redistributed them to the hottest spots and added one more trap.