One ongoing work is to make all lab machines automatically backed-up on Scribe on a daily basis. The updates should be boatable and stored for some time (potentially a few weeks) on Scribe. Making whole disk images has already been tried for some of the machines with no problems. (e.g., Cymac and WorkStations)

The same thing can not be done with Chimay though, as it currently has one huge RAID-controlled volume that stores all the information (OS, home directories, and NDS-downloaded data). Creating daily full disk images of such a system is not practical. 

Here is the plan we came up with to overcome this issue:

1. Create one full disk image of Chimay and store it on Scribe (it was already done)
2. Move the nds-downloaded raw data temporarily to Scribe and remove it from Chimay
3. Make another full disk image of Chimay
4. Burn this image to a single disk and boot chimay with it

5. Restore the rest of Chimay disks and move the NDS data back

    

On the weekend (Sat and Sun 9/7-8) I tried to execute these steps. There wasn't enough free space left on Scribe to move all the NDS data to it, so I stored part of this data temporarily on WS4. Then I also checked storage-consuming directories and, in one case, removed some non-important stored files. As there was no free space left on Scribe to execute step no. 3, I initiated storing the image on WS3. Unfortunately, a couple of different trials of the image-creation process failed as there was not enough free space on WS3 to accommodate Chiamy image as well. I was not able to reduce the image size such that it can be stored on WS3. 

These are the options left for us to get this work done. 

1. Distribute the NDS files between different machines to make enough free space on Scribe for the image and then follow the previous plan
2. Shrink the Chimay drive size, create the image and then follow the previous plan
3. Temporarily transfer some services to Megatrone (Network gateway, Wiki, elog) and recreate chimay and its services from scratch

All Sorensen power supplies are turned off. The Cymac system is down (All Chassis are down)

If you need more information or if you need to turn them back on please contact Xuesi Ma.

Date and Time: Around 2:20 PM on May 21, 2025

Location and Temperature:

• Back of the room, around the working station: 85.5 °F
• Front of the room, around the doorway: 82.3 °F

Date and Time: Around 3:50 PM on June 2, 2025

Location and Temperature:

• Back of the room 1119, around the working station: 86 °F
• Front of the room 1119, around the doorway: 84.1 °F
• Back of the room 1129, around the working station: 78.6 °F
• Front of the room 1129, around the doorway: 78.3 °F
• In the hallway in front of room 1119: 75.2 °F

Date and Time: Around 4:50 PM on July 1, 2025

Location and Temperature:

• Back of the room 1119, around the working station: 85.7 °F
• Front of the room 1119, around the doorway: 83.5 °F
• Back of the room 1129, around the working station: 79.1 °F
• In the hallway in front of room 1119: 76.5 °F

Action items for facilities

• Ceiling lights replacement
• Fume hood cleaned and refurbished
• Sink countertop cleaning
• Floor cleaned and waxed
•  

Cleaned off the tapes on the optical table with acetone.

The working document of tasks regarding moving the clean room is attached. Needs LIGO account to access and edit.

  https://docs.google.com/document/d/14HhMYBcHmM62Hc1YJhinMsG5uPx6xoW0WEBmDcbNMMg/edit?usp=sharing

[Audrey, Cece, Jon, Ma, Mary, Michael, Pooyan, Tyler, Shane]

We have completed the preliminary reorganization of our Physics 1119 lab space. This includes:

• Moving all cabinets to their final positions
• Clearing out all equipment from the cleanroom in prep for HVAC work, and storing them in either the HEPA cabinet or double-bagging in regular storage.
• Moving all non-essential equipment into other lab spaces.
• Wrapping vacuum system and VLC for protection
• Removing all ceiling tiles and fans from the ceiling of the cleanroom
• Moving the cleanroom frame towards the front of the room.

Prepping the new clean-and-bake station in 1125 will begin within the next week, along with the construction of the stainless steel table planned to be put into the cleanroom.

[Cece, Ma, Mary, Pooyan, Tyler, Paul]

We have continued the reorganization of our Physics 1125 and 1129 lab spaces. This includes:

• Moving two workbenches from Room 1129 into Room 1125
• The workbench with the oven is still in Room 1129
• Assembling the stainless steel table
• Moving the cleanroom frame toward the whiteboard side to make room for HVAC work

The sink in Room 1125 was painted, and we stood Frosti up.

[Pooyan, Ma, Mary, Tyler]

While re-installing the clean-room HEPA fan filters, we realized that one of the blowers is broken. The issue seems to be that the plastic damper bracket that is keeping the blower in place has failed. The fix is to open up the fan unit, and replace the broken vibration damper. We installed the defected unit on top of the clean room entrance, for easy future access.

• concrete Caulk to fill the drill holes in 1119
  • Example [Pooyan: I have a caulk gun and silicon caulk at home, if needed.]
• Water hose for the washer in 1125
  • 1/2" diameter
  • almost 8 Ft 
• Exhaust for the oven in 1125
  • Question: not sure which way we want the exhaust to go. Measurements depend on the path. 
  • Oven exhaust to the ceiling vertical distance: 47"
  • Oven exhaust to the ceiling exhaust distance: 6 ft
• Rubber vibration dampers for the cleanroom fans (see the first 4 pictures)
  • 25 mm diameter, 13 mm height, ~5-6 mm thread, 6 mm deep
  • Example (This is the male version. We need one with female on both sides)
• Cabling conduit from the cable tray to the optical table frame (through the cleanroom ceiling)
  • Aluminum example
  • Galvanized Steel example 
• Wire cable tray for 1125
  • ~10 Ft, exactly similar to the one already installed in 1129

Port 80 is kept closed by default. This might be causing the certbot auto-renewal cronjob to fail. Therefore we must renew the certificate manually.

Step 1: Open port 80. (This is needed as the certificate renewal process runs some tests which requires client communication over port 80)

Step 2: Run the following command

sudo certbot certonly --force-renew -d richardsonlab.ucr.edu

Step 3: Confirm the certificate was renewed by running the following command

sudo certbot certificates

Cross-linking instructions: How to run a Jupyter notebook in your custom Conda environment

1. Turn off the RGA filament and disconnect it.

2. Record the pressure and then close the angle valve and gate valve on the RGA line to isolate it from the system.

3. Turn off the backing and turbo pump and let the turbo pump spin to a stop by slowly leaking the chamber. The slower the safer.

4.Slowly open the vents on the chamber and the turbo pump. Making sure not to raise the pressure too fast.

5. Once done venting the chamber, open the lid by removing the bolts and prying the lid open with a long flat head screw driver.

6. Removed previous parts and placed them into clean bags. Place new parts inside of the chamber and group them together close to the center.

7. Place the lid back onto the chamber and tighten the bolts in a circular pattern until tightened all the way with the wrench.

8. Make sure all vents are closed and turn the backing pump back on.

9. The turbo pump can be turned back on when the pressure reaches e-1 torr.

10. Wait until the chamber reaches a pressure of e-6 torr before opening the angle and gate valve on the RGA line. After the RGA line is open and the pressure is still below e-6 torr, the RGA filament can be turned on and a leak test of the lid should be performed.

11. After the leak test shows no leaks, proceed with starting the low temp bake.

Avoiding Thread Contention When Using Multiprocessing with Finesse and Cython

When running Monte Carlo simulations or other computational workloads, it's common to use Python's ProcessPoolExecutor to parallelize multiple independent tasks. This approach works well—until it interacts with low-level libraries that themselves use multi-threading under the hood.

The Problem: Thread Over-Subscription

In a recent project, I ran into a significant performance issue while executing a large number of Monte Carlo trials using a process pool with 30 worker processes on Megatron (with 48 cores). Each trial ran a function that internally used cython.parallel.prange for fast, element-wise operations, which is what Finesse uses under the hood for many internal numerical calculations. Cython, via OpenMP, was configured to use all available CPU threads per process by default.

This resulted in severe thread over-subscription. With 30 parallel processes and each process attempting to use all 48 threads, the system was launching over 1,400 threads concurrently. The CPU quickly became saturated, and the tasks stalled. In some cases, the system became unresponsive, and the jobs had to be canceled repeatedly.

This happens because when the function calls into these libraries from within a Python multiprocessing context, each subprocess will attempt to use the full number of threads available to the machine.

The Solution: Limit Threads per Process

The solution is simple: explicitly limit the number of threads each subprocess is allowed to use. This can be done by setting the environment variable at the top of your script, before importing any thread-hungry libraries like Finesse.

      
import os
os.environ["OMP_NUM_THREADS"] = "1"

    

By setting OMP_NUM_THREADS to "1", we ensure that each multiprocessing worker uses only one thread internally, preventing them from overloading the system and allowing the tasks to run more efficiently.

Below are a basic diagram of what the RTD measurement circuit logically looks like and an example schematic of the actual wiring. The schematic wiring will be placed internally into a chassis, connected to the RTDs via DB25 cable.

The custom front and rear panels for the RTD readout chassis arrived last Friday. I installed them in the chassis frame to check their fit. They fit very well, so all that now remains is to complete the internal wiring and test the connections.

The chassis panel designs are archived to LIGO-D2300452 and LIGO-D2300453.

Below is the current state of the RTD readout chassis wiring. Initial continuity tests seem good, will run through one more time to confirm.

I performed another continuity test on the RTD chassis wiring, and everything seems to be set up correctly. The chassis should be ready for installation.

Below is the proposed schematic for FROSTI optical testing, chosen so enough space is allotted for prototype assembly.

Steps to be taken include:

1. Reconstruct FLIR staging apparatus
2. Move test mass stand-in to cleanroom
3. Mark FLIR camera position on cleanroom optical table at correct distance
4. Run ethernet cable into cleanroom
5. Move FLIR aside to allow for more assembly space
6. Upon assembly completion, reposition FLIR onto optical table again

Tentative plan is to begin setup early next week.

Group Meeting slides for Non-deterministic Heater Response.