Cleaned off the tapes on the optical table with acetone.

The working document of tasks regarding moving the clean room is attached. Needs LIGO account to access and edit.

  https://docs.google.com/document/d/14HhMYBcHmM62Hc1YJhinMsG5uPx6xoW0WEBmDcbNMMg/edit?usp=sharing

[Audrey, Cece, Jon, Ma, Mary, Michael, Pooyan, Tyler, Shane]

We have completed the preliminary reorganization of our Physics 1119 lab space. This includes:

• Moving all cabinets to their final positions
• Clearing out all equipment from the cleanroom in prep for HVAC work, and storing them in either the HEPA cabinet or double-bagging in regular storage.
• Moving all non-essential equipment into other lab spaces.
• Wrapping vacuum system and VLC for protection
• Removing all ceiling tiles and fans from the ceiling of the cleanroom
• Moving the cleanroom frame towards the front of the room.

Prepping the new clean-and-bake station in 1125 will begin within the next week, along with the construction of the stainless steel table planned to be put into the cleanroom.

[Cece, Ma, Mary, Pooyan, Tyler, Paul]

We have continued the reorganization of our Physics 1125 and 1129 lab spaces. This includes:

• Moving two workbenches from Room 1129 into Room 1125
• The workbench with the oven is still in Room 1129
• Assembling the stainless steel table
• Moving the cleanroom frame toward the whiteboard side to make room for HVAC work

The sink in Room 1125 was painted, and we stood Frosti up.

[Pooyan, Ma, Mary, Tyler]

While re-installing the clean-room HEPA fan filters, we realized that one of the blowers is broken. The issue seems to be that the plastic damper bracket that is keeping the blower in place has failed. The fix is to open up the fan unit, and replace the broken vibration damper. We installed the defected unit on top of the clean room entrance, for easy future access.

• concrete Caulk to fill the drill holes in 1119
  • Example [Pooyan: I have a caulk gun and silicon caulk at home, if needed.]
• Water hose for the washer in 1125
  • 1/2" diameter
  • almost 8 Ft 
• Exhaust for the oven in 1125
  • Question: not sure which way we want the exhaust to go. Measurements depend on the path. 
  • Oven exhaust to the ceiling vertical distance: 47"
  • Oven exhaust to the ceiling exhaust distance: 6 ft
• Rubber vibration dampers for the cleanroom fans (see the first 4 pictures)
  • 25 mm diameter, 13 mm height, ~5-6 mm thread, 6 mm deep
  • Example (This is the male version. We need one with female on both sides)
• Cabling conduit from the cable tray to the optical table frame (through the cleanroom ceiling)
  • Aluminum example
  • Galvanized Steel example 
• Wire cable tray for 1125
  • ~10 Ft, exactly similar to the one already installed in 1129

[Ma, Mary, Tyler, Pooyan]

HEPA fan air flow measurements are done with the fans sitting on their side in the lab, and the anemometer being 24in away from the filter face of the fan. Per company's documents, we need to replace the filter if the air flow velocity drops under 38-45 fpm for the medium settings.)
 

All numbers are in fpm unit. The measurement for each unit is done twice in front of the fan positions.  

[Ma, Cece, Pooyan, Tyler]

We installed the repaired HEPA fans on top of the cleanroom, and cleaned the fans themselves. The table of all the Serial Numbers for the fans, filters, and where they are installed is added for the record. 

The electrical wiring diagram of the cleanroom is on page (1) of the attached file.

• Blue lines are protected by the lab UPS, and power the power strips attached to the frame of the optical tables. The same power strips feed the frame lights and the vacuum system. 
• Red lines are connected to the red outlet on the celling cable tray, presumabaly connected to the building's backup system. The feed everything on the ceiling of the cleanroom by first going through two switches attached to the frame of the clean room. Each outlet feeds two fans and one ceiling lamp via power strips sitting on top of the cleanroom ceiling. (see pictures on page (2))

We will need two sets of conduits feeding cables into the cleanroom, possibly secured on the legs of the cleanroom frame and going through the ceiling tile.

• The eastern conduit will have all of the connection/control/CyMAC cables, and two power cables. (See page (3))
• The western conduit will have two power cables, and the table's air feeder. 
• I suggest that each conduit be either a two inch wide, or two one inch wide coduits. Anything less than one inch wide will not be able to contain the current batch of chables. 

Page 4 is a possible cartoon of where we want to install the conduits. We can have electric boxes either on the both sides of the conduit, or just on one side. 

Note that a couple of the optical posts still need to be measured for their diameters. Images are located in a Google Folder and linked on the spreadsheet so to view them you have to go to the Google Sheet.

[Shane, Joshua, Cece, Ma, Mary, Abhi, Jacob]

The clean room (1119) was re-cleaned on Thursday (7/2). All surfaces were wiped down, the floor was vacuumed and mopped, and new sticky mats were installed. Additionally, a particle count measurement was performed on Monday (7/6) after allowing sufficient time for dust to settle. As this is the first particle count conducted after the clean room modification, the updated clean room zone configuration is attached for reference.

Particle Count Measurements:

• Zone 1:
  • 0.3 µm: 11556
  • 0.5 µm: 8563
  • 1.0 µm: 5237
• Zone 2:
  • 0.3 µm: 1039
  • 0.5 µm: 623
  • 1.0 µm: 374
• Zone 3:
  • 0.3 µm: 748
  • 0.5 µm: 290
  • 1.0 µm: 207
• Zone 4:
  • 0.3 µm: 1829
  • 0.5 µm: 748
  • 1.0 µm: 374
• Zone 5:
  • 0.3 µm: 1454
  • 0.5 µm: 665
  • 1.0 µm: 415

[Shane, Joshua, Cece, Ma, Mary, Abhi, Jacob]

The clean room (1119) was re-cleaned on Tuesday (7/7). All surfaces were wiped down, and the wire rack was rewiped. A particle count measurement was performed on zone 1.

Particle Count Measurements:

• Zone 1:
  • 0.3 µm: 5861
  • 0.5 µm: 2036
  • 1.0 µm: 249

[Joshua, Cece, Ma, Mary, Abhi, Jacob, Dhatri]

The clean room (1119) was re-cleaned on Tuesday (7/14). A particle count measurement was performed on zones 1, 3, and 5. We noticed the burning smell again in 1129. As a precaution, we closed the two doors separating the three rooms.

Particle Count Measurements:

• Zone 1:
  • 0.3 µm: 4873
  • 0.5 µm: 1041
  • 1.0 µm: 124
• Zone 3:
  • 0.3 µm: 708
  • 0.5 µm: 83
  • 1.0 µm: 83
• Zone 5:
  • 0.3 µm: 4165
  • 0.5 µm: 1916
  • 1.0 µm: 583

[Joshua, Mary, Ma]

On Wednesday (7/15), a particle count measurement was performed in the clean room in all five zones.

Particle Count Measurements:

• Zone 1:
  • 0.3 µm: 958
  • 0.5 µm: 333
  • 1.0 µm: 240
• Zone 2:
  • 0.3 µm: 541
  • 0.5 µm: 458
  • 1.0 µm: 416
• Zone 3:
  • 0.3 µm: 624
  • 0.5 µm: 166
  • 1.0 µm: 83
• Zone 4:
  • 0.3 µm: 958
  • 0.5 µm: 458
  • 1.0 µm: 124
• Zone 5:
  • 0.3 µm: 2499
  • 0.5 µm: 1082
  • 1.0 µm: 499

Backup plan for the CyMAC machine:

Cymac

• Full image backup
  • Every Saturday at 3:00 am
  • One image for the whole machine. can be copied to a new disk and booted.
  • Only one version is kept
  • uses abb-cli on cymac. It’s the Synology agent script
  • Controlled by the Synology Active Backup for Business, Available in Scribe’s apps.
  • Image files are stored at `/volume1/ActiveBackupforBusiness/ActiveBackupData/Server-cymac-controls-Default` on Scribe
• Things that will be periodically zipped and backed-up to Scribe
  • controls home directory
  • `/etc/advligorts/`
  • `/opt/`
  • Done weekly on Sundays at 4:00 am
  • Script at /opt/backup/weekly_zip.sh
  • cronjob at /etc/cron.d/cron.cymac
• Things that will be periodically rsynced in the archive mode
  • `/frames`
  • Every night at 1 am
  • script at `/opt/backup`
  • cronjob at `/etc/cron.d/cron.cymac`
• Biweekly purge of the raw frame files from the source (cymac)
  • on the 1st and the 15th of the month at 2:00 am
  • moves the raw frame files older than two weeks to Scribe, and removes them from the source

Schedule for Scribe:

• Empty recycle bin every night at 1:00 am
  • Files that have been deleted more that 7 days ago will be permanently deleted
  • This is set in Synology control panel
• Remove raw frames older than 2 months (60 days)
  • Script at /opt/backup/raw_purge.sh
    • Removes files from /volume1/Backups/cymac/full/
  • Runs every night at midnight to remove any raw frames older than 60 days

Port 80 is kept closed by default. This might be causing the certbot auto-renewal cronjob to fail. Therefore we must renew the certificate manually.

Step 1: Open port 80. (This is needed as the certificate renewal process runs some tests which requires client communication over port 80)

Step 2: Run the following command

sudo certbot certonly --force-renew -d richardsonlab.ucr.edu

Step 3: Confirm the certificate was renewed by running the following command

sudo certbot certificates

Cross-linking instructions: How to run a Jupyter notebook in your custom Conda environment

1. Turn off the RGA filament and disconnect it.

2. Record the pressure and then close the angle valve and gate valve on the RGA line to isolate it from the system.

3. Turn off the backing and turbo pump and let the turbo pump spin to a stop by slowly leaking the chamber. The slower the safer.

4.Slowly open the vents on the chamber and the turbo pump. Making sure not to raise the pressure too fast.

5. Once done venting the chamber, open the lid by removing the bolts and prying the lid open with a long flat head screw driver.

6. Removed previous parts and placed them into clean bags. Place new parts inside of the chamber and group them together close to the center.

7. Place the lid back onto the chamber and tighten the bolts in a circular pattern until tightened all the way with the wrench.

8. Make sure all vents are closed and turn the backing pump back on.

9. The turbo pump can be turned back on when the pressure reaches e-1 torr.

10. Wait until the chamber reaches a pressure of e-6 torr before opening the angle and gate valve on the RGA line. After the RGA line is open and the pressure is still below e-6 torr, the RGA filament can be turned on and a leak test of the lid should be performed.

11. After the leak test shows no leaks, proceed with starting the low temp bake.

Avoiding Thread Contention When Using Multiprocessing with Finesse and Cython

When running Monte Carlo simulations or other computational workloads, it's common to use Python's ProcessPoolExecutor to parallelize multiple independent tasks. This approach works well—until it interacts with low-level libraries that themselves use multi-threading under the hood.

The Problem: Thread Over-Subscription

In a recent project, I ran into a significant performance issue while executing a large number of Monte Carlo trials using a process pool with 30 worker processes on Megatron (with 48 cores). Each trial ran a function that internally used cython.parallel.prange for fast, element-wise operations, which is what Finesse uses under the hood for many internal numerical calculations. Cython, via OpenMP, was configured to use all available CPU threads per process by default.

This resulted in severe thread over-subscription. With 30 parallel processes and each process attempting to use all 48 threads, the system was launching over 1,400 threads concurrently. The CPU quickly became saturated, and the tasks stalled. In some cases, the system became unresponsive, and the jobs had to be canceled repeatedly.

This happens because when the function calls into these libraries from within a Python multiprocessing context, each subprocess will attempt to use the full number of threads available to the machine.

The Solution: Limit Threads per Process

The solution is simple: explicitly limit the number of threads each subprocess is allowed to use. This can be done by setting the environment variable at the top of your script, before importing any thread-hungry libraries like Finesse.

      
import os
os.environ["OMP_NUM_THREADS"] = "1"

    

By setting OMP_NUM_THREADS to "1", we ensure that each multiprocessing worker uses only one thread internally, preventing them from overloading the system and allowing the tasks to run more efficiently.

Below are a basic diagram of what the RTD measurement circuit logically looks like and an example schematic of the actual wiring. The schematic wiring will be placed internally into a chassis, connected to the RTDs via DB25 cable.

The custom front and rear panels for the RTD readout chassis arrived last Friday. I installed them in the chassis frame to check their fit. They fit very well, so all that now remains is to complete the internal wiring and test the connections.

The chassis panel designs are archived to LIGO-D2300452 and LIGO-D2300453.

Below is the current state of the RTD readout chassis wiring. Initial continuity tests seem good, will run through one more time to confirm.

I performed another continuity test on the RTD chassis wiring, and everything seems to be set up correctly. The chassis should be ready for installation.

Below is the proposed schematic for FROSTI optical testing, chosen so enough space is allotted for prototype assembly.

Steps to be taken include:

1. Reconstruct FLIR staging apparatus
2. Move test mass stand-in to cleanroom
3. Mark FLIR camera position on cleanroom optical table at correct distance
4. Run ethernet cable into cleanroom
5. Move FLIR aside to allow for more assembly space
6. Upon assembly completion, reposition FLIR onto optical table again

Tentative plan is to begin setup early next week.

Group Meeting slides for Non-deterministic Heater Response.