All Sorensen power supplies are turned off. The Cymac system is down (All Chassis are down)

If you need more information or if you need to turn them back on please contact Xuesi Ma.

Slides

I have re-run the optimization plots with the changes mentioned last week. 

I have added the heat maps of the HOM scattering from HR deformation and the HOM scattering due to the OPD difference as well as the curvature actuation of the surface. 

[Luke, Mohak, Luis]

We were also able to remove the residue that has built up on the soft walls by donning a skinny bunny suit and with two wipes rubbing both sides of the soft walls at the same time. The wipes would get gummy very quickly and need to be replaced frequently.

Particle Count Measurements:

• Pre-cleaning (1:20 pm):
  • Zone 3:
    • 0.3 µm: 3577
    • 0.5 µm: 1369
    • 1.0 µm: 485
  • Zone 4:
    • 0.3 µm: 1062
    • 0.5 µm: 442
    • 1.0 µm: 177

• Started Cleaning (3:00 pm)
• Finished Cleaning (4:00 pm)

• Post-cleaning (4:22 pm):
  • Zone 3:
    • 0.3 µm: 3577
    • 0.5 µm: 618
    • 1.0 µm: 132
  • Zone 4:
    • 0.3 µm: 2031
    • 0.5 µm: 397
    • 1.0 µm: 132

[Ma, Luis]

After taking the RGA scans of the new system, here are the results of the raw and normalized data.We can see that in the post baked system the He peak is way lower, and the peaks around 20AMU are also significantly lower.

Slides

I have changed some properties of the thermal model and am re-running the thermal sweeping

I have also been messing around with making a logo

The vacuum chamber has stopped baking. 

Current state as of 12:30, 5/2/2025:

The gate valve is open, and the filament of the RGA is on.

The temperatures were steady at:

PID right: barrel upper: 108°C,  RGA volume: 120°C

PID left: barrel lower: 120°C, Lid: 91°C

[Luke, Ma, Tyler]

The vacuum chamber is currently baking. 

Current state as of 5:00

The gate valve is open, and the filament of the RGA is on. Argon leak is opened

The temperatures are as follows:

PID right barrel upper: 99°C,  RGA volume: 120°C

PID left barrel lower: 123°C, Lid: 92°C

[Ma, Luke, Luis]

April 29, 2025

3:30 PM
We performed another leak test on the vacuum system, this time focusing on the welds around the main cylinder. The leaks from these junctions were in the mid 10⁻¹¹ range, but we noticed some strange behavior after completing the test. The partial pressure of He kept rising at a slow, constant rate for a while—even after stopping the test for about 15–20 minutes and restarting the measurement. We're not sure what’s causing this and wanted to check in before moving forward with the bake.

4:30 PM
We performed an RGA of the system (see below) and the findings are that there was a higher contraction of He than usual. We have attached another scan from 2/24/25 for reference.

5:00 PM
We turned off the electron multiplier and closed the RGA Program.

[Ma, Luke, Luis]

April 24, 2025

3:00 PM
Ma and Luis attempted to connect to the RGA to initialize the leak test but encountered some difficulties.

3:20 PM
After restarting both the computer and the RGA, we realized that the DB9 cable was not connected to Spica. After connecting it, we had no further issues communicating with the RGA.

3:30 PM
Luke and Luis performed the leak test. There were no significant leaks in the vacuum system. The highest leak rates we found are listed in the table below; the rest of the entries were limited by noise, as we could not detect any change in the program after spraying the ports with He.

I have created two more plots. One on the transmitted power for a specific configuration. This was done by taking the product of the efficiency and the power released by surface area of the source plane for a configuration assuming it is at 300C. The second is a plot of how close the reflector is to the test mass I have included a contour line at 2mm the nominal proximity constraint. Both of these plots are generated for a 2cm wide target plane.

Somethings of note: Because the proximity requires no modeling and can be determined purely mathematically it is easy to re-generate this plot of other target widths. If you increase the target width the proximity contour moves upward and downwards if reduced.

Test 7: Reconnect ribbon cables

• Setup: Reconnect all the ribbon cables, and turned on the AI Chassis while isolated rack
• Observation: No spike observed.

Test 8: Reconnect to rack

• Setup: Reconnect the AI Chassis to the rack.
• Observation: No spike observed.

Test 9: Restore to original position.

• Setup: CLose up the AI Chassis, and put it back to its original location.
• Observation: No spike observed.

I have redone the width and location optimization with a 40kg test mass. This was to see where the nominal ring heater would fall in this optimization map see attached.

There are two concerns I have with this modeling:

1. I am using normalized power only imparting 1W, which may effect the single pass scattering used for optimization.
2. I am only looking at the surface deformation and I believe that looking at OPD may also be something that is important to consider. 

Note z_RH is measured from the HR surface and w_RH is the half width of the target plane

Avoiding Thread Contention When Using Multiprocessing with Finesse and Cython

When running Monte Carlo simulations or other computational workloads, it's common to use Python's ProcessPoolExecutor to parallelize multiple independent tasks. This approach works well—until it interacts with low-level libraries that themselves use multi-threading under the hood.

The Problem: Thread Over-Subscription

In a recent project, I ran into a significant performance issue while executing a large number of Monte Carlo trials using a process pool with 30 worker processes on Megatron (with 48 cores). Each trial ran a function that internally used cython.parallel.prange for fast, element-wise operations, which is what Finesse uses under the hood for many internal numerical calculations. Cython, via OpenMP, was configured to use all available CPU threads per process by default.

This resulted in severe thread over-subscription. With 30 parallel processes and each process attempting to use all 48 threads, the system was launching over 1,400 threads concurrently. The CPU quickly became saturated, and the tasks stalled. In some cases, the system became unresponsive, and the jobs had to be canceled repeatedly.

This happens because when the function calls into these libraries from within a Python multiprocessing context, each subprocess will attempt to use the full number of threads available to the machine.

The Solution: Limit Threads per Process

The solution is simple: explicitly limit the number of threads each subprocess is allowed to use. This can be done by setting the environment variable at the top of your script, before importing any thread-hungry libraries like Finesse.

      
import os
os.environ["OMP_NUM_THREADS"] = "1"

    

By setting OMP_NUM_THREADS to "1", we ensure that each multiprocessing worker uses only one thread internally, preventing them from overloading the system and allowing the tasks to run more efficiently.

Test 4: AI Chassis ground isolation

• Setup: AI Chassis isolated from the ground with readout directly connected to adapter board
• Observation: No spike observed.

Test 5: Ribbon cable check

• Setup: AI Chassis remains isolated from the ground. The signal is measured after the ribbon cable. (Attachment 3)
• Observation: No spike observed.

Test 6: Single ground connection check

• Setup: AI chassis powered on but remains isolated from the frame. The signal is measured from the output of the AI Chassis. (Attachment 1 ,2)
• Observation: No spike observed.

Test 1: Adapter Board Connected to Filter Board

• Setup: AI Chassis is powered on. Signal is measured directly from the adapter board. All other ports on the adapter board are connected to the filter board.
• Observation: Two distinct spikes observed at the beginning of the measurement.

Test 2: Adapter Board Disconnected from Filter Board

• Setup: AI Chassis remains powered on. Signal is again measured directly from the adapter board. This time, the remaining ports on the adapter board are not connected to the filter board.
• Observation: Multiple spikes observed, distributed evenly across the entire measurement.

Test 3: Isolation Test with Spare Adapter Board (ongoing)

• Setup: Suspecting the original adapter board may be faulty, a spare adapter board is used for comparison. Signal is measured directly from this spare board.

Update, 04/08/25, Tue 17:30

Took the old adapter board out of the AI chassis and and used it to connect DAC to AA chassis. If no spikes are seen, it means that the glitches are not originating from this board.

Also checked al the previous spikes using the raw data (65536 Hz sample rate). The duration of the glitches are ~1 second, despite the previous guess that they are happening in the Milli-second scale.

The result from AI Chassis bypass test showed that the AI Chassis may be the problem. There are no spikes from DAC's direct output.

All chassis except timing chassis are turned off. Power supplies for 24V, 18V, and -18V are turned off

The AI chassis has been taken out of the rack for further inspection

https://dcc.ligo.org/LIGO-E2300117

https://dcc.ligo.org/cgi-bin/private/DocDB/ShowDocument?.submit=Identifier&docid=D2300124&version=

Upon final review of the FROSTI analysis included in the (hopefully) soon-to-be submitted instrument paper, I've made some adjustments to the reflectivity analysis that estimates the amount of power delivered to the test mass by the FROSTI. Initially, as detailed in elog 447, the delivered power was approximated to be 12.6 W (later adjusted to 12.9 W), with roughly 18% of this power being reflected by the test mass. After some final adjustments, this value now is closer to 12.0 W, with 10.2 W absorbed (15% power reflection). Below is a table showing the updated values for the FROSTI prototype tests:

Continue investigation in the spikes

Pulsed ADC with a function generator and find no spikes. Rules out ADC for causing the spikes

Loopback test that bypasses the FROSTI Chassis shows spikes (spikes happened on all channels at the same time)

Next step is to bypass the AI Chassis to find the source of the spikes

[Shane, Jon]

[Shane, Jon]

Update on the serial interfacing with the vacuum system: both turbo pumps are now able to successfully communicate through the code. The older pump, for unknown reasons, required a newer version of the communication syntax (despite the newer, small pump being able to communicate fine with the old communication format). We have also confirmed that the newer pump works with this new format as well. Next step is interfacing the pressure gauge.

Spikes Appear Again, need to address it systematically.

Power on and off before reaching steady state ✔

At 12V, the rise and fall time of heater elements are different from 24V.

Initial guess is due to temperature in the chamber. However, it does not seem to be the case, 24V with 30c have the same hall time as 24V at lower temperature.

Attached are some graphs for rise and fall time

After a weekend of powering on, the main chamber pressure stabilized in the UHV region.

Temperature in the chamber seems also not to change.

1424564097.243212 2025/02/26 00:14:39 UTC Time start

12V 2A right after start all 8 elements

1424564478.079428 2025/02/26 00:21:00 UTC Time stop

12V 1.8A right before stop all 8 elements

1424564832.211584 2025/02/26 00:26:54 UTC Time start

12V 1.9A right after start all 8 elements

1424565208.359935 2025/02/26 00:33:10 UTC Time stop

12V 1.8A right before stop all 8 elements

1424565573.565066 2025/02/26 00:39:15 UTC Time start

12V 1.8A right after start all 8 elements

1424565931.242394 2025/02/26 00:45:13 UTC Time stop

12V 1.7A right before stop all 8 elements

1424566292.67104 2025/02/26 00:51:14 UTC Time start

12V 1.8A right after start all 8 elements

1424566648.619952 2025/02/26 00:57:10 UTC Time stop

12V 1.7A right before stop all 8 elements

1424566996.312246 2025/02/26 01:02:58 UTC Time start

12V 1.8A right after start all 8 elements

1424567381.748943 2025/02/26 01:09:23 UTC Time stop

12V 1.7A right before stop all 8 elements

1424567756.528736 2025/02/26 01:15:38 UTC Time start

12V 1.7A right after start all 8 elements

0.2A right before start

spikes!?

1424643001.864687 2025/02/26 22:09:43 UTC

change c_0(VEXC0 & VCXC0) to 2V (why is it on 5V ?)

2025/02/26 22:18:51 UTC

Main chamber pressure:5.92e-9

RGA chamber pressure:1.98e-9

1424650528.240947 2025/02/27 00:15:10 UTC Time start (increase voltage)

24V 2.1A right after start all 8 elements

disconnect and reconnect (exc 1-4)(out 9-12) & (exc 5-8)(out 13-16)

2025/02/27 19:34:26 UTC

Main chamber pressure:1.34e-8

RGA chamber pressure:4.33e-9

Main chamber temp: 29

RGA chamber temp:29

2025/02/28 18:02:29 UTC

Main chamber pressure:1.05e-8

RGA chamber pressure:3.53e-9

Main chamber temp: 29

RGA chamber temp:30

1424801097.374902 2025/02/28 18:04:39 UTC Time stop

24V 1.8A right before stop all 8 elements

0.1A right after stop

1424827460.71372 2025/03/01 01:24:02 UTC Time start

24V 2.9A right after start all 8 elements

0.1A right before start

2025/03/01 01:25:34 UTC

Main chamber pressure:5.98e-9

RGA chamber pressure:2.06e-9

Main chamber temp:27

RGA chamber temp:27

2025/03/03 20:28:03 UTC

Main chamber pressure:7.8e-9

RGA chamber pressure:2.64e-9

Main chamber temp:27

RGA chamber temp:27