[Jon, Tyler, Aiden, Shane, Pooyan, Michael, Cynthia, Luke]

On Wednesday, we completed long list of work towards making the new lab (1129) fully operational and enabling the next phase of FROSTI testing. 

Cabinet Installation

Three new VWR cabinets with sliding glass doors were installed in 1129. Each unit measures 48" (W) x 22" (D) x 84" (H) and sits along the back wall (see attachment 1). The 350-lb. cabinets were laid in place by Facilities on Monday and permanentized on Wednesday. Work included:

• Earthquake anchoring to the masonry wall
• Sliding glass doors leveled
• Shelving installed
• Wiping down of interior and exterior surfaces with IPA wipes

Server Rack Installation

A new Tripp Lite 42U open-frame rack was laid in place in 1129 and anchored to the floor (see attachment 1). This rack will house all of our general-purpose and simulation computers, which will be relocated from the 1119 rack at a later time.

Lab Clean-Up

Following installation of the new cabinets and rack, we proceeded to organize and clean both labs. Work included:

• Moved parts and equipment into permanent storage in 1129 cabinets
• Wiped down surfaces in 1119 and 1129 with polypropylene IPA wipes
• HEPA-vacuumed floors of 1119 and 1129
• Mopped floor in 1119 with Liquinox solution
• Installed new sticky mats in 1119 and 1129
• Regular cleanroom cleaning and particle counts (see 302)
• Positioned new stainless steel gowning bench outside the cleanroom (see attachment 2)

At this point, the only piece of lab equipment still to be delivered is a HEPA garment cabinet for reusing our (semi-disposable) bunny suits. It is schedule to arrive in mid-February and will sit outside the cleanroom in 1119, in the former location of the HEPA flow bench.

On Thursday we installed the power conditioning/distribution equipment and networking equipment in the new 1129 rack. The hardware is identical to the setup in the 1119 rack and includes:

• Tripp Lite SU5KRT3UTF - 208V, 5kVA on-line UPS with 120V transformer
• CyberPower PDU20M2F12R - metered power distribution unit, (14) NEMA 5-20R
• Ubiquiti USW-Pro-48 - 48 port 10Gbps network switch

The UPS is connected to a 208V NEMA 6-30R outlet in the overhead cable tray, which is on the building's "standby" (backup power) circuit. An 8-ft L6-30 extension cord has been ordered to permanently run the power cable through the cable tray.

The network switch will be connected to a Cat6 cable that was recently run by ITS from the 1119 rack, allowing the lab's LAN to be extended into 1129. This Ethernet link remains to be tested.

[Jon, Pooyan, Tyler]

A few computer machine changes have been made. 

• Logrus moved from 1119 to 1129. It is up and running with the same IP address as before. 
• A new Windows machine (host name: spica, IP:192.168.1.14) is installed in the 1119 server rack. It is connected to the RGA scanner with the serial port and is specifically used for that purpose. 
  • Update: The machine was off on 6/25, although it was left on 6/24. We think that it might have been because of Windows' default setting to suspend/hibernate the machine after idleness. To resolve this, I used  "powercfg /change" command to set all the following parameters equal to zero. The machine is still running on 6/26. 

    monitor-timeout-ac
    monitor-timeout-dc
    disk-timeout-ac
    disk-timeout-dc
    standby-timeout-ac
    standby-timeout-dc
    hibernate-timeout-ac
    hibernate-timeout-dc

• A new Debian machine (hostname: megatron, IP:192.168.1.16) is installed in the 1129 server rack. This machine is intended to be used for FEA/simulation work. A new 2TB WD Green SSD is used as its main disk drive. 

   At the moment, “controls” is the only user, and there are no apps/libraries installed on the machine.

  • Update: Jon installed LIGO cds-workstation tools and MiniConda on 6/26. 
  • Update: Pooyan and Liu set the following conda environments:
    • Env named “finesse” with Python 3.12.3 and Finesse version 3.0a24 installed. Finesse was installed via the source code. The subdirectory “/home/controls/packages” is used to store the package sourcecodes.
    • Env named “fenicsx” with the same version of Python and Finesse as the previous env, with the latest version of FEniCSx (0.8) and the test-mass-thermal-state installed.  

Set-up of the first CDS workstation for 1129, ws3, is complete and the machine is ready for use. The login credentials are the same as the other lab machines.

All that now remains is to install a permanent cable tray for running the new Ethernet cables between the electronics rack and bench (they are currently dangling from the suspended lights).

Today I installed the second desktop workstation in 1129. The new machine is an Intel NUC13ANHi5, with a 12-Core Intel i5-1340P CPU, 32GB DDR4 RAM, and a 1TB SSD.

I loaded it with a fresh installation of Debian 12 and installed the LIGO CDS workstation (control room) tools. It is assigned the hostname ws4 and and the static IP address 192.168.1.19 on the local lab network. Like the other CDS workstations, there is just one user account accessible with the usual credentials.

The machine is fully set up and ready for use.

One ongoing work is to make all lab machines automatically backed-up on Scribe on a daily basis. The updates should be boatable and stored for some time (potentially a few weeks) on Scribe. Making whole disk images has already been tried for some of the machines with no problems. (e.g., Cymac and WorkStations)

The same thing can not be done with Chimay though, as it currently has one huge RAID-controlled volume that stores all the information (OS, home directories, and NDS-downloaded data). Creating daily full disk images of such a system is not practical. 

Here is the plan we came up with to overcome this issue:

1. Create one full disk image of Chimay and store it on Scribe (it was already done)
2. Move the nds-downloaded raw data temporarily to Scribe and remove it from Chimay
3. Make another full disk image of Chimay
4. Burn this image to a single disk and boot chimay with it

5. Restore the rest of Chimay disks and move the NDS data back

    

On the weekend (Sat and Sun 9/7-8) I tried to execute these steps. There wasn't enough free space left on Scribe to move all the NDS data to it, so I stored part of this data temporarily on WS4. Then I also checked storage-consuming directories and, in one case, removed some non-important stored files. As there was no free space left on Scribe to execute step no. 3, I initiated storing the image on WS3. Unfortunately, a couple of different trials of the image-creation process failed as there was not enough free space on WS3 to accommodate Chiamy image as well. I was not able to reduce the image size such that it can be stored on WS3. 

These are the options left for us to get this work done. 

1. Distribute the NDS files between different machines to make enough free space on Scribe for the image and then follow the previous plan
2. Shrink the Chimay drive size, create the image and then follow the previous plan
3. Temporarily transfer some services to Megatrone (Network gateway, Wiki, elog) and recreate chimay and its services from scratch

All Sorensen power supplies are turned off. The Cymac system is down (All Chassis are down)

If you need more information or if you need to turn them back on please contact Xuesi Ma.

Date and Time: Around 2:20 PM on May 21, 2025

Location and Temperature:

• Back of the room, around the working station: 85.5 °F
• Front of the room, around the doorway: 82.3 °F

Date and Time: Around 3:50 PM on June 2, 2025

Location and Temperature:

• Back of the room 1119, around the working station: 86 °F
• Front of the room 1119, around the doorway: 84.1 °F
• Back of the room 1129, around the working station: 78.6 °F
• Front of the room 1129, around the doorway: 78.3 °F
• In the hallway in front of room 1119: 75.2 °F

Date and Time: Around 4:50 PM on July 1, 2025

Location and Temperature:

• Back of the room 1119, around the working station: 85.7 °F
• Front of the room 1119, around the doorway: 83.5 °F
• Back of the room 1129, around the working station: 79.1 °F
• In the hallway in front of room 1119: 76.5 °F

Action items for facilities

• Ceiling lights replacement
• Fume hood cleaned and refurbished
• Sink countertop cleaning
• Floor cleaned and waxed
•  

Cleaned off the tapes on the optical table with acetone.

The working document of tasks regarding moving the clean room is attached. Needs LIGO account to access and edit.

  https://docs.google.com/document/d/14HhMYBcHmM62Hc1YJhinMsG5uPx6xoW0WEBmDcbNMMg/edit?usp=sharing

[Audrey, Cece, Jon, Ma, Mary, Michael, Pooyan, Tyler, Shane]

We have completed the preliminary reorganization of our Physics 1119 lab space. This includes:

• Moving all cabinets to their final positions
• Clearing out all equipment from the cleanroom in prep for HVAC work, and storing them in either the HEPA cabinet or double-bagging in regular storage.
• Moving all non-essential equipment into other lab spaces.
• Wrapping vacuum system and VLC for protection
• Removing all ceiling tiles and fans from the ceiling of the cleanroom
• Moving the cleanroom frame towards the front of the room.

Prepping the new clean-and-bake station in 1125 will begin within the next week, along with the construction of the stainless steel table planned to be put into the cleanroom.

[Cece, Ma, Mary, Pooyan, Tyler, Paul]

We have continued the reorganization of our Physics 1125 and 1129 lab spaces. This includes:

• Moving two workbenches from Room 1129 into Room 1125
• The workbench with the oven is still in Room 1129
• Assembling the stainless steel table
• Moving the cleanroom frame toward the whiteboard side to make room for HVAC work

The sink in Room 1125 was painted, and we stood Frosti up.

[Pooyan, Ma, Mary, Tyler]

While re-installing the clean-room HEPA fan filters, we realized that one of the blowers is broken. The issue seems to be that the plastic damper bracket that is keeping the blower in place has failed. The fix is to open up the fan unit, and replace the broken vibration damper. We installed the defected unit on top of the clean room entrance, for easy future access.

• concrete Caulk to fill the drill holes in 1119
  • Example [Pooyan: I have a caulk gun and silicon caulk at home, if needed.]
• Water hose for the washer in 1125
  • 1/2" diameter
  • almost 8 Ft 
• Exhaust for the oven in 1125
  • Question: not sure which way we want the exhaust to go. Measurements depend on the path. 
  • Oven exhaust to the ceiling vertical distance: 47"
  • Oven exhaust to the ceiling exhaust distance: 6 ft
• Rubber vibration dampers for the cleanroom fans (see the first 4 pictures)
  • 25 mm diameter, 13 mm height, ~5-6 mm thread, 6 mm deep
  • Example (This is the male version. We need one with female on both sides)
• Cabling conduit from the cable tray to the optical table frame (through the cleanroom ceiling)
  • Aluminum example
  • Galvanized Steel example 
• Wire cable tray for 1125
  • ~10 Ft, exactly similar to the one already installed in 1129

[Ma, Mary, Tyler, Pooyan]

HEPA fan air flow measurements are done with the fans sitting on their side in the lab, and the anemometer being 24in away from the filter face of the fan. Per company's documents, we need to replace the filter if the air flow velocity drops under 38-45 fpm for the medium settings.)
 

All numbers are in fpm unit. The measurement for each unit is done twice in front of the fan positions.  

Port 80 is kept closed by default. This might be causing the certbot auto-renewal cronjob to fail. Therefore we must renew the certificate manually.

Step 1: Open port 80. (This is needed as the certificate renewal process runs some tests which requires client communication over port 80)

Step 2: Run the following command

sudo certbot certonly --force-renew -d richardsonlab.ucr.edu

Step 3: Confirm the certificate was renewed by running the following command

sudo certbot certificates

Cross-linking instructions: How to run a Jupyter notebook in your custom Conda environment

1. Turn off the RGA filament and disconnect it.

2. Record the pressure and then close the angle valve and gate valve on the RGA line to isolate it from the system.

3. Turn off the backing and turbo pump and let the turbo pump spin to a stop by slowly leaking the chamber. The slower the safer.

4.Slowly open the vents on the chamber and the turbo pump. Making sure not to raise the pressure too fast.

5. Once done venting the chamber, open the lid by removing the bolts and prying the lid open with a long flat head screw driver.

6. Removed previous parts and placed them into clean bags. Place new parts inside of the chamber and group them together close to the center.

7. Place the lid back onto the chamber and tighten the bolts in a circular pattern until tightened all the way with the wrench.

8. Make sure all vents are closed and turn the backing pump back on.

9. The turbo pump can be turned back on when the pressure reaches e-1 torr.

10. Wait until the chamber reaches a pressure of e-6 torr before opening the angle and gate valve on the RGA line. After the RGA line is open and the pressure is still below e-6 torr, the RGA filament can be turned on and a leak test of the lid should be performed.

11. After the leak test shows no leaks, proceed with starting the low temp bake.

Avoiding Thread Contention When Using Multiprocessing with Finesse and Cython

When running Monte Carlo simulations or other computational workloads, it's common to use Python's ProcessPoolExecutor to parallelize multiple independent tasks. This approach works well—until it interacts with low-level libraries that themselves use multi-threading under the hood.

The Problem: Thread Over-Subscription

In a recent project, I ran into a significant performance issue while executing a large number of Monte Carlo trials using a process pool with 30 worker processes on Megatron (with 48 cores). Each trial ran a function that internally used cython.parallel.prange for fast, element-wise operations, which is what Finesse uses under the hood for many internal numerical calculations. Cython, via OpenMP, was configured to use all available CPU threads per process by default.

This resulted in severe thread over-subscription. With 30 parallel processes and each process attempting to use all 48 threads, the system was launching over 1,400 threads concurrently. The CPU quickly became saturated, and the tasks stalled. In some cases, the system became unresponsive, and the jobs had to be canceled repeatedly.

This happens because when the function calls into these libraries from within a Python multiprocessing context, each subprocess will attempt to use the full number of threads available to the machine.

The Solution: Limit Threads per Process

The solution is simple: explicitly limit the number of threads each subprocess is allowed to use. This can be done by setting the environment variable at the top of your script, before importing any thread-hungry libraries like Finesse.

      
import os
os.environ["OMP_NUM_THREADS"] = "1"

    

By setting OMP_NUM_THREADS to "1", we ensure that each multiprocessing worker uses only one thread internally, preventing them from overloading the system and allowing the tasks to run more efficiently.

Below are a basic diagram of what the RTD measurement circuit logically looks like and an example schematic of the actual wiring. The schematic wiring will be placed internally into a chassis, connected to the RTDs via DB25 cable.

The custom front and rear panels for the RTD readout chassis arrived last Friday. I installed them in the chassis frame to check their fit. They fit very well, so all that now remains is to complete the internal wiring and test the connections.

The chassis panel designs are archived to LIGO-D2300452 and LIGO-D2300453.

Below is the current state of the RTD readout chassis wiring. Initial continuity tests seem good, will run through one more time to confirm.

I performed another continuity test on the RTD chassis wiring, and everything seems to be set up correctly. The chassis should be ready for installation.

Below is the proposed schematic for FROSTI optical testing, chosen so enough space is allotted for prototype assembly.

Steps to be taken include:

1. Reconstruct FLIR staging apparatus
2. Move test mass stand-in to cleanroom
3. Mark FLIR camera position on cleanroom optical table at correct distance
4. Run ethernet cable into cleanroom
5. Move FLIR aside to allow for more assembly space
6. Upon assembly completion, reposition FLIR onto optical table again

Tentative plan is to begin setup early next week.

Group Meeting slides for Non-deterministic Heater Response.