I have created two more plots. One on the transmitted power for a specific configuration. This was done by taking the product of the efficiency and the power released by surface area of the source plane for a configuration assuming it is at 300C. The second is a plot of how close the reflector is to the test mass I have included a contour line at 2mm the nominal proximity constraint. Both of these plots are generated for a 2cm wide target plane.

Somethings of note: Because the proximity requires no modeling and can be determined purely mathematically it is easy to re-generate this plot of other target widths. If you increase the target width the proximity contour moves upward and downwards if reduced.

Test 7: Reconnect ribbon cables

• Setup: Reconnect all the ribbon cables, and turned on the AI Chassis while isolated rack
• Observation: No spike observed.

Test 8: Reconnect to rack

• Setup: Reconnect the AI Chassis to the rack.
• Observation: No spike observed.

Test 9: Restore to original position.

• Setup: CLose up the AI Chassis, and put it back to its original location.
• Observation: No spike observed.

I have redone the width and location optimization with a 40kg test mass. This was to see where the nominal ring heater would fall in this optimization map see attached.

There are two concerns I have with this modeling:

1. I am using normalized power only imparting 1W, which may effect the single pass scattering used for optimization.
2. I am only looking at the surface deformation and I believe that looking at OPD may also be something that is important to consider. 

Note z_RH is measured from the HR surface and w_RH is the half width of the target plane

Avoiding Thread Contention When Using Multiprocessing with Finesse and Cython

When running Monte Carlo simulations or other computational workloads, it's common to use Python's ProcessPoolExecutor to parallelize multiple independent tasks. This approach works well—until it interacts with low-level libraries that themselves use multi-threading under the hood.

The Problem: Thread Over-Subscription

In a recent project, I ran into a significant performance issue while executing a large number of Monte Carlo trials using a process pool with 30 worker processes on Megatron (with 48 cores). Each trial ran a function that internally used cython.parallel.prange for fast, element-wise operations, which is what Finesse uses under the hood for many internal numerical calculations. Cython, via OpenMP, was configured to use all available CPU threads per process by default.

This resulted in severe thread over-subscription. With 30 parallel processes and each process attempting to use all 48 threads, the system was launching over 1,400 threads concurrently. The CPU quickly became saturated, and the tasks stalled. In some cases, the system became unresponsive, and the jobs had to be canceled repeatedly.

This happens because when the function calls into these libraries from within a Python multiprocessing context, each subprocess will attempt to use the full number of threads available to the machine.

The Solution: Limit Threads per Process

The solution is simple: explicitly limit the number of threads each subprocess is allowed to use. This can be done by setting the environment variable at the top of your script, before importing any thread-hungry libraries like Finesse.

      
import os
os.environ["OMP_NUM_THREADS"] = "1"

    

By setting OMP_NUM_THREADS to "1", we ensure that each multiprocessing worker uses only one thread internally, preventing them from overloading the system and allowing the tasks to run more efficiently.

Test 4: AI Chassis ground isolation

• Setup: AI Chassis isolated from the ground with readout directly connected to adapter board
• Observation: No spike observed.

Test 5: Ribbon cable check

• Setup: AI Chassis remains isolated from the ground. The signal is measured after the ribbon cable. (Attachment 3)
• Observation: No spike observed.

Test 6: Single ground connection check

• Setup: AI chassis powered on but remains isolated from the frame. The signal is measured from the output of the AI Chassis. (Attachment 1 ,2)
• Observation: No spike observed.

Test 1: Adapter Board Connected to Filter Board

• Setup: AI Chassis is powered on. Signal is measured directly from the adapter board. All other ports on the adapter board are connected to the filter board.
• Observation: Two distinct spikes observed at the beginning of the measurement.

Test 2: Adapter Board Disconnected from Filter Board

• Setup: AI Chassis remains powered on. Signal is again measured directly from the adapter board. This time, the remaining ports on the adapter board are not connected to the filter board.
• Observation: Multiple spikes observed, distributed evenly across the entire measurement.

Test 3: Isolation Test with Spare Adapter Board (ongoing)

• Setup: Suspecting the original adapter board may be faulty, a spare adapter board is used for comparison. Signal is measured directly from this spare board.

Update, 04/08/25, Tue 17:30

Took the old adapter board out of the AI chassis and and used it to connect DAC to AA chassis. If no spikes are seen, it means that the glitches are not originating from this board.

Also checked al the previous spikes using the raw data (65536 Hz sample rate). The duration of the glitches are ~1 second, despite the previous guess that they are happening in the Milli-second scale.

The result from AI Chassis bypass test showed that the AI Chassis may be the problem. There are no spikes from DAC's direct output.

All chassis except timing chassis are turned off. Power supplies for 24V, 18V, and -18V are turned off

The AI chassis has been taken out of the rack for further inspection

https://dcc.ligo.org/LIGO-E2300117

https://dcc.ligo.org/cgi-bin/private/DocDB/ShowDocument?.submit=Identifier&docid=D2300124&version=

Upon final review of the FROSTI analysis included in the (hopefully) soon-to-be submitted instrument paper, I've made some adjustments to the reflectivity analysis that estimates the amount of power delivered to the test mass by the FROSTI. Initially, as detailed in elog 447, the delivered power was approximated to be 12.6 W (later adjusted to 12.9 W), with roughly 18% of this power being reflected by the test mass. After some final adjustments, this value now is closer to 12.0 W, with 10.2 W absorbed (15% power reflection). Below is a table showing the updated values for the FROSTI prototype tests:

Continue investigation in the spikes

Pulsed ADC with a function generator and find no spikes. Rules out ADC for causing the spikes

Loopback test that bypasses the FROSTI Chassis shows spikes (spikes happened on all channels at the same time)

Next step is to bypass the AI Chassis to find the source of the spikes

[Shane, Jon]

[Shane, Jon]

Update on the serial interfacing with the vacuum system: both turbo pumps are now able to successfully communicate through the code. The older pump, for unknown reasons, required a newer version of the communication syntax (despite the newer, small pump being able to communicate fine with the old communication format). We have also confirmed that the newer pump works with this new format as well. Next step is interfacing the pressure gauge.

Spikes Appear Again, need to address it systematically.

Power on and off before reaching steady state ✔

At 12V, the rise and fall time of heater elements are different from 24V.

Initial guess is due to temperature in the chamber. However, it does not seem to be the case, 24V with 30c have the same hall time as 24V at lower temperature.

Attached are some graphs for rise and fall time

After a weekend of powering on, the main chamber pressure stabilized in the UHV region.

Temperature in the chamber seems also not to change.

1424564097.243212 2025/02/26 00:14:39 UTC Time start

12V 2A right after start all 8 elements

1424564478.079428 2025/02/26 00:21:00 UTC Time stop

12V 1.8A right before stop all 8 elements

1424564832.211584 2025/02/26 00:26:54 UTC Time start

12V 1.9A right after start all 8 elements

1424565208.359935 2025/02/26 00:33:10 UTC Time stop

12V 1.8A right before stop all 8 elements

1424565573.565066 2025/02/26 00:39:15 UTC Time start

12V 1.8A right after start all 8 elements

1424565931.242394 2025/02/26 00:45:13 UTC Time stop

12V 1.7A right before stop all 8 elements

1424566292.67104 2025/02/26 00:51:14 UTC Time start

12V 1.8A right after start all 8 elements

1424566648.619952 2025/02/26 00:57:10 UTC Time stop

12V 1.7A right before stop all 8 elements

1424566996.312246 2025/02/26 01:02:58 UTC Time start

12V 1.8A right after start all 8 elements

1424567381.748943 2025/02/26 01:09:23 UTC Time stop

12V 1.7A right before stop all 8 elements

1424567756.528736 2025/02/26 01:15:38 UTC Time start

12V 1.7A right after start all 8 elements

0.2A right before start

spikes!?

1424643001.864687 2025/02/26 22:09:43 UTC

change c_0(VEXC0 & VCXC0) to 2V (why is it on 5V ?)

2025/02/26 22:18:51 UTC

Main chamber pressure:5.92e-9

RGA chamber pressure:1.98e-9

1424650528.240947 2025/02/27 00:15:10 UTC Time start (increase voltage)

24V 2.1A right after start all 8 elements

disconnect and reconnect (exc 1-4)(out 9-12) & (exc 5-8)(out 13-16)

2025/02/27 19:34:26 UTC

Main chamber pressure:1.34e-8

RGA chamber pressure:4.33e-9

Main chamber temp: 29

RGA chamber temp:29

2025/02/28 18:02:29 UTC

Main chamber pressure:1.05e-8

RGA chamber pressure:3.53e-9

Main chamber temp: 29

RGA chamber temp:30

1424801097.374902 2025/02/28 18:04:39 UTC Time stop

24V 1.8A right before stop all 8 elements

0.1A right after stop

1424827460.71372 2025/03/01 01:24:02 UTC Time start

24V 2.9A right after start all 8 elements

0.1A right before start

2025/03/01 01:25:34 UTC

Main chamber pressure:5.98e-9

RGA chamber pressure:2.06e-9

Main chamber temp:27

RGA chamber temp:27

2025/03/03 20:28:03 UTC

Main chamber pressure:7.8e-9

RGA chamber pressure:2.64e-9

Main chamber temp:27

RGA chamber temp:27

1424039912.625576 2025/02/19 22:38:14 UTC Time start

24V 2.8A right after start all 8 elements

1424044804.902443 2025/02/19 23:59:46 UTC Time stop

24V 1.8A right before stop all 8 elements

0.1A right before start and right after stop

note: turned on briefly to check current right before stop

2/20 RGA Scan

spikes??

1424129207.857777 2025/02/20 23:26:29 UTC Time start

24V 2.9A right after start all 8 elements

1424140789.856096 2025/02/21 02:39:31 UTC Time stop

24V 1.7A right before stop all 8 elements

0.1A right before start and right after stop

2025/02/21 02:43:19 UTC

Main chamber pressure: 1.54e-8

RGA chamber pressure:5.06e-9

spikes are due to loose connection between connectors.

1424210410.173863 2025/02/21 21:59:52 UTC Time start

24V 2.9A right after start all 8 elements

1424218357.96404 2025/02/22 00:12:19 UTC Time stop

24V 1.7A right before stop all 8 elements

each elements: 0.3A (0.2A increment)(all)

0.1A right before start and right before stop

2/24 RGA Scan

1424467918.129082 2025/02/24 21:31:40 UTC Time start

12V 2A right after start all 8 elements

1424478936.635821 2025/02/25 00:35:18 UTC Time stop

12V 1.6A right before stop all 8 elements

each elements: 0.4A (0.2A increment)(all)

0.2A right before start

rise time: A(1-exp(-t/tau))+B

fall time: Aexp(-t/tau) +B

[Ma, Luis]

February 24, 2025 - 1:00PM

Ma performed an RGA scan with the vacuum system containing the heater elements. The results are attached below.

These are some plots:

The first shows the convergence of the 02 mode reducing the size of the mesh. The second shows the the numerical error of the zernike.

The first is found by sweeping a parameter that changes the size of the mesh. The HR surface was set to half a cm for all values.

The second by taking the inner product of a zernike mode with its self and calculating its deviation from pi for varying fineness of the mesh.

[Luke, Mary, Luis]

Particle Count Measurements:

• Pre-cleaning (8:40 am):
  • Zone 3:
    • 0.3 µm: 441
    • 0.5 µm: 44
    • 1.0 µm: 0
  • Zone 4:
    • 0.3 µm: 353
    • 0.5 µm: 176
    • 1.0 µm: 44

• Started Cleaning (3:16 pm)
• Finished Cleaning (4:10 pm)

• Post-cleaning (4:15 pm):
  • Zone 3:
    • 0.3 µm: 574
    • 0.5 µm: 265
    • 1.0 µm: 132
  • Zone 4:
    • 0.3 µm: 177
    • 0.5 µm: 44
    • 1.0 µm: 0

[Ma, Luis]

February 20, 2025 - 3:30 PM

Ma performed an RGA scan with the vacuum system containing the heater elements. The results are attached below.

As we can see, we are not yet under cleanliness standards. The passing threshold is 4 × 10⁻¹⁰, we are very close to it.

Debugging update for vac system serial communications: we were able to successfully connect the smaller turbo pump to the Varian (manufacturer) software today by plugging the controller db9 directly into spica, and verified that the pump was able to send and receive information. Still no success on communication with the larger pump via python serial code, though. Issue potentially lies in the pinout mapping of the connectors, as the pinouts we'd been using work successfully for the small pump which only requires 3 connections (RXD, TXD, GND), but the large pump may require more. Tried to test this on the big pump by keeping the pinout setup for the small pump, and incrementally adding the remaining 6 connections on a one to one scale (Pin 1 matched to Pin 1, etc) between the controller DB9 and the field DB9 connected to the IOLAN ethernet cable. After each new connection was added, I retried running the serial connection code. None of the additional connections yielded any results, and the large pump remained unable to connect through the code. Next step is further looking into documentation on the pinouts to see if the large pump requires a different connection setup than the small pump.

Below is a preview of the final RIN figure that will be included in the FROSTI instrument paper. A quick summary of what is shown below:

The original RIN CSD measurement is shown on the top panel in red. Frequency bins that exhibit external electronics noise (ex. ADC, photodetector noise, etc.) are identified and shaded in gray. These noisy bins are then excluded from the dataset before beginning the next step in the analysis process: rebinning. Here, the resolution of the spectrum can be changed by averaging frequency bins together within a specified interval, with the goal of pushing the measurement curve closer to the A+ requirement shown in the figure. For demonstration below, the spectrum goes from a resolution of ~0.93 Hz to 14.90 Hz.

https://docs.google.com/presentation/d/1WiV2VqS0BzXNCK6VYYQ-Ty8xlHnzXatQaFbMf1-0rsY/edit?usp=sharing

[Ma, Pooyan, Tyler]

On Friday, we connected the vacuum chamber with the Cymac.

Over the last couple weeks I have been working on finding the optimal position of the ringheater's thermal profile. 

Today I would like to give an update of where I am at and my next steps.

Using the python COMSOL interface I have been able to run and save deformation data sweeping through a great deal of potential combinations of widths and positions.

I then calculated their zernike coefficients and using a very simple "quality" function (02 - (40)*10 - (4-4)*10), where 02 stands for the zernike mode of quadratic deformation, I was able to generate a heat map of comparing the "quality" verses the width and position used. 

Seen below are four plots. The first is simply the 02 coefficient of the decomposition followed by the negative of 40 and 4-4 then the "quality."

This 'quality' function was an arbitrary choice on my part. I believe that my next step would either be defining a more useful function or using finesse to model the actual effects of this surface deformation.

[Luis, Luke, Ma]

February 12, 2025 - 4:00 PM

We performed an RGA scan with the vacuum system containing the heater elements. The results are attached below.

As we can see, we are not yet under cleanliness standards. The passing threshold is 4 × 10⁻¹⁰, and we are currently measuring nearly double this value. We will continue performing scans to monitor any changes in these readings.

[Luis]

Feb/11/25 9:30am

Pressure gages were connected.