CYMAC Remote Control Cable Assembly

Cable Functions

The custom cable assembly serves two primary functions:

Note: All connections are verified with a multimeter.

Note: This cable allows remote control of the Sorensen power supply with an external voltage source (Cymac DAC).

Note: For more details, check section 2.11.1 from the attached Sorensen power supply manual.

• Hosted Lukes model on Chimay. It's available at https://richardsonlab.ucr.edu/real-time-frosti
• The sourcecode is at Github. It is clones at /var/www/html/real-time-frosti, and this is the block added to nginx config file at /etc/nginx/sites-available/default :
  
  location /real-time-frosti {

root /var/www/html;
index index.html index.htm index.php;
}
• Also added this line to the crontab, so the code will be checked for updated from the sourcecode every five minutes. 
  
  */5 * * * * cd /var/www/html/real-time-frosti/ && git pull 
  
  
• TODO: Move the sourcecode from Github to git.ligo.org, and make the repository public. 

• Fast channels recorded to .gwf frame files
  • Had to add a DAQ Channels block to the c1msc model for those channel to be recorded in the .gwf frame files. The block is in the CDS_PARTS, and I just had to copy it to our model. Here is the example text for the channels:
    
    

    #DAQ Channels

    ONE_DAQ_CHANNEL 2048
ANOTHER_DAQ_CHANNEL 1024
SCIENCE_FRAME_CHAN* 1024
UINT32_CHAN uint32 2048
DAQ_CHANNEL_AT_DEFAULT_RATE

    I added two channels to the list (V0 and V1). Remember that the channel name should not be the full name, but just the part's name. In this case, I added these two channel names:
    
    V0_OUT
V1_OUT
    
    with the default sample rate. 
• The fast channels are recorded with an extra _DQ in their names. So, in ndscope and diaggui these two channels should be addressed as
  • C1:MSC-V0_OUT_DQ
  • C1:MSC-V1_OUT_DQ

• Update 6/13: enabled recording of all the ADC voltage channels so that Tyler can use all the data for the noise floor calculation. attached is an example of diagggui with almost 30 hrs of data.

• TODO: Enable daqd_wiper

By measuring the resistance of the ladder, I obtained a graph of resistance as a function of voltage.
All the heater elements behave consistently, showing only a small standard deviation.
However, heater element 5 shows a significantly larger standard deviation.
Upon examining the initial ladder graph, I noticed that the resistance of heater element 5 increases following a power outage that resets the system.
At this point, I am unsure why this behavior occurs.

• The MEDM models issue was fixed. The issue was that building the c1msc model alone was not enough and we had to re-build the c1iop as well, although no changes has been made to the c1iop. Added a note in the wiki on this for future reference.
• For writing the fast channels to the frame files, our CDS system should have a GDS broadcaster. I followed this wiki page to set one, but it breaks the current daqd system. Asked about it earlier today on the CDS Mattermost channel and waiting for a response. 
• Maybe I should continue the distributed scheme we have been trying for the nds2-server and let another machine on the network do this. 
• Question: daqd has a feature to zero-out the bad data (NaNs), and it is currently on for our system. I don't think that it's a good idea to have this. Should I turn it off? 

Date and Time: Around 3:50 PM on June 2, 2025

Location and Temperature:

• Back of the room 1119, around the working station: 86 °F
• Front of the room 1119, around the doorway: 84.1 °F
• Back of the room 1129, around the working station: 78.6 °F
• Front of the room 1129, around the doorway: 78.3 °F
• In the hallway in front of room 1119: 75.2 °F

We currently have the parts for a DB9 connector but need parts for a DB25 connector, where did we get the parts for the DB9 connector? We may be able to use the same company for the DB25 connector.

Also need a cable for the connectors. Find a DB9 to DB9 cable in 1129, not sure if I can disassemble it and use just the cable.

The pressure in the vacuum chamber rises to 2.12e-8 torr, which needs baking.

There are three major issues with the lab's CDS system that should be addressed. They are ordered from the highest priority to the lowest.

• CyMAC has stopped serving the fast channels (65k Hz)
  The channels are still there, and the data from the slow (16 Hz) channels is served correctly, but the fast data is not served. As a result, diagggui can not retrieve any information at the moment.
  Probably the issue is the same as the version conflict between daqd and nds2-server. After installing nds2-server on CyMAC and realizing that the conflict can not be resolved, I reverted the changes so that the CyMAC could be compatible with daqd, but there is a chance that there are still version conflicts that block the fast channels.
  I suspect that we have broken the MSC model when adding the new button. This could explain both the fasr-channels fault and the unexpected crashes.
  
  UPDATE 6/5
  Fixes this by rebuilding the models.
  Note for the future: The c1iop and c1msc models should be rebuilt together, even if no change has been made to c1iop.
• The fast channels were not saved to the frames
  Reviewing the frame files showed that the daqd was only saving the 16 Hz data in the .gwf frame files. I had missed this because I only checked the existence and the integrity of the raw frame files, but never retrieved the high-frequency data itself to check it.
• NDS2-server is installed, but does not distribute the data
  I managed to install the nds2-server and its satellite programs on the Chimay, and NFS mount the frame files on it. The server successfully chaches the channel names, frame files, and the data, but fails to list the channels for distribution.

Note: CyMAC machine was strangely disconnected from the network, and required manual reboot. I couldn't find any abnormal logs that could explain what happened.

Slides

I have added some slides to my meeting updates. The updates include: my choice in position and width, some figures I have added to my write up, and what I think my immediate steps are to wrap up.

I have also made a new FROSTI model that conducts its raytracing in real time. Something that was severely lacking in the previous model.

Using the configuration in section 2.11.1 in the Sorensen installation and operation manual (attached below), I'm able to remote control the Sorensen power supply.

I use DAC VEXC8 as the external voltage source and a breakout board to match the configuration of connector J3.

I have recorded 7 cycles (attached below), and am now working on analyzing the data.

slideshow

Slides

Date and Time: Around 2:20 PM on May 21, 2025

Location and Temperature:

• Back of the room, around the working station: 85.5 °F
• Front of the room, around the doorway: 82.3 °F

All Sorensen power supplies are turned off. The Cymac system is down (All Chassis are down)

If you need more information or if you need to turn them back on please contact Xuesi Ma.

Slides

I have re-run the optimization plots with the changes mentioned last week. 

I have added the heat maps of the HOM scattering from HR deformation and the HOM scattering due to the OPD difference as well as the curvature actuation of the surface. 

[Luke, Mohak, Luis]

We were also able to remove the residue that has built up on the soft walls by donning a skinny bunny suit and with two wipes rubbing both sides of the soft walls at the same time. The wipes would get gummy very quickly and need to be replaced frequently.

Particle Count Measurements:

• Pre-cleaning (1:20 pm):
  • Zone 3:
    • 0.3 µm: 3577
    • 0.5 µm: 1369
    • 1.0 µm: 485
  • Zone 4:
    • 0.3 µm: 1062
    • 0.5 µm: 442
    • 1.0 µm: 177

• Started Cleaning (3:00 pm)
• Finished Cleaning (4:00 pm)

• Post-cleaning (4:22 pm):
  • Zone 3:
    • 0.3 µm: 3577
    • 0.5 µm: 618
    • 1.0 µm: 132
  • Zone 4:
    • 0.3 µm: 2031
    • 0.5 µm: 397
    • 1.0 µm: 132

[Ma, Luis]

After taking the RGA scans of the new system, here are the results of the raw and normalized data.We can see that in the post baked system the He peak is way lower, and the peaks around 20AMU are also significantly lower.

Slides

I have changed some properties of the thermal model and am re-running the thermal sweeping

I have also been messing around with making a logo

The vacuum chamber has stopped baking. 

Current state as of 12:30, 5/2/2025:

The gate valve is open, and the filament of the RGA is on.

The temperatures were steady at:

PID right: barrel upper: 108°C,  RGA volume: 120°C

PID left: barrel lower: 120°C, Lid: 91°C

[Luke, Ma, Tyler]

The vacuum chamber is currently baking. 

Current state as of 5:00

The gate valve is open, and the filament of the RGA is on. Argon leak is opened

The temperatures are as follows:

PID right barrel upper: 99°C,  RGA volume: 120°C

PID left barrel lower: 123°C, Lid: 92°C

[Ma, Luke, Luis]

April 29, 2025

3:30 PM
We performed another leak test on the vacuum system, this time focusing on the welds around the main cylinder. The leaks from these junctions were in the mid 10⁻¹¹ range, but we noticed some strange behavior after completing the test. The partial pressure of He kept rising at a slow, constant rate for a while—even after stopping the test for about 15–20 minutes and restarting the measurement. We're not sure what’s causing this and wanted to check in before moving forward with the bake.

4:30 PM
We performed an RGA of the system (see below) and the findings are that there was a higher contraction of He than usual. We have attached another scan from 2/24/25 for reference.

5:00 PM
We turned off the electron multiplier and closed the RGA Program.

[Ma, Luke, Luis]

April 24, 2025

3:00 PM
Ma and Luis attempted to connect to the RGA to initialize the leak test but encountered some difficulties.

3:20 PM
After restarting both the computer and the RGA, we realized that the DB9 cable was not connected to Spica. After connecting it, we had no further issues communicating with the RGA.

3:30 PM
Luke and Luis performed the leak test. There were no significant leaks in the vacuum system. The highest leak rates we found are listed in the table below; the rest of the entries were limited by noise, as we could not detect any change in the program after spraying the ports with He.

I have created two more plots. One on the transmitted power for a specific configuration. This was done by taking the product of the efficiency and the power released by surface area of the source plane for a configuration assuming it is at 300C. The second is a plot of how close the reflector is to the test mass I have included a contour line at 2mm the nominal proximity constraint. Both of these plots are generated for a 2cm wide target plane.

Somethings of note: Because the proximity requires no modeling and can be determined purely mathematically it is easy to re-generate this plot of other target widths. If you increase the target width the proximity contour moves upward and downwards if reduced.

Test 7: Reconnect ribbon cables

• Setup: Reconnect all the ribbon cables, and turned on the AI Chassis while isolated rack
• Observation: No spike observed.

Test 8: Reconnect to rack

• Setup: Reconnect the AI Chassis to the rack.
• Observation: No spike observed.

Test 9: Restore to original position.

• Setup: CLose up the AI Chassis, and put it back to its original location.
• Observation: No spike observed.

I have redone the width and location optimization with a 40kg test mass. This was to see where the nominal ring heater would fall in this optimization map see attached.

There are two concerns I have with this modeling:

1. I am using normalized power only imparting 1W, which may effect the single pass scattering used for optimization.
2. I am only looking at the surface deformation and I believe that looking at OPD may also be something that is important to consider. 

Note z_RH is measured from the HR surface and w_RH is the half width of the target plane

Avoiding Thread Contention When Using Multiprocessing with Finesse and Cython

When running Monte Carlo simulations or other computational workloads, it's common to use Python's ProcessPoolExecutor to parallelize multiple independent tasks. This approach works well—until it interacts with low-level libraries that themselves use multi-threading under the hood.

The Problem: Thread Over-Subscription

In a recent project, I ran into a significant performance issue while executing a large number of Monte Carlo trials using a process pool with 30 worker processes on Megatron (with 48 cores). Each trial ran a function that internally used cython.parallel.prange for fast, element-wise operations, which is what Finesse uses under the hood for many internal numerical calculations. Cython, via OpenMP, was configured to use all available CPU threads per process by default.

This resulted in severe thread over-subscription. With 30 parallel processes and each process attempting to use all 48 threads, the system was launching over 1,400 threads concurrently. The CPU quickly became saturated, and the tasks stalled. In some cases, the system became unresponsive, and the jobs had to be canceled repeatedly.

This happens because when the function calls into these libraries from within a Python multiprocessing context, each subprocess will attempt to use the full number of threads available to the machine.

The Solution: Limit Threads per Process

The solution is simple: explicitly limit the number of threads each subprocess is allowed to use. This can be done by setting the environment variable at the top of your script, before importing any thread-hungry libraries like Finesse.

      
import os
os.environ["OMP_NUM_THREADS"] = "1"

    

By setting OMP_NUM_THREADS to "1", we ensure that each multiprocessing worker uses only one thread internally, preventing them from overloading the system and allowing the tasks to run more efficiently.

Test 4: AI Chassis ground isolation

• Setup: AI Chassis isolated from the ground with readout directly connected to adapter board
• Observation: No spike observed.

Test 5: Ribbon cable check

• Setup: AI Chassis remains isolated from the ground. The signal is measured after the ribbon cable. (Attachment 3)
• Observation: No spike observed.

Test 6: Single ground connection check

• Setup: AI chassis powered on but remains isolated from the frame. The signal is measured from the output of the AI Chassis. (Attachment 1 ,2)
• Observation: No spike observed.

Test 1: Adapter Board Connected to Filter Board

• Setup: AI Chassis is powered on. Signal is measured directly from the adapter board. All other ports on the adapter board are connected to the filter board.
• Observation: Two distinct spikes observed at the beginning of the measurement.

Test 2: Adapter Board Disconnected from Filter Board

• Setup: AI Chassis remains powered on. Signal is again measured directly from the adapter board. This time, the remaining ports on the adapter board are not connected to the filter board.
• Observation: Multiple spikes observed, distributed evenly across the entire measurement.

Test 3: Isolation Test with Spare Adapter Board (ongoing)

• Setup: Suspecting the original adapter board may be faulty, a spare adapter board is used for comparison. Signal is measured directly from this spare board.

Update, 04/08/25, Tue 17:30

Took the old adapter board out of the AI chassis and and used it to connect DAC to AA chassis. If no spikes are seen, it means that the glitches are not originating from this board.

Also checked al the previous spikes using the raw data (65536 Hz sample rate). The duration of the glitches are ~1 second, despite the previous guess that they are happening in the Milli-second scale.