cleaning cleanroom and particle count

NOTE: particle counter was found dead, with the charging dock unplugged. For future reference, if you need to unplug the dock, please either plug it back in when you're done, or make a note in an elog so someone else can come down and charge it.

• 1:07 pm: ran zero count test on particle counter
• 1:26 pm: started particle count
  • zone 3:
    • 0.3 u:1787
    • 0.5 u:457
    • 1.0 u:41
• zone 4: We were only able to charge the particle counter a small amount before starting the cleaning, so it died again halfway through this measurement, and we didn't get results for zone 4. In interest of time, we just put it on the charger and started the cleaning.
• 1:49 pm: began surface check and wipedown, including softwalls
• 2:16 pm: started vacuuming the floor
• 2:24 pm: finished vacuuming the floor
• 2:26 pm: started mopping the floor
• 2:32 pm: finished mopping the floor
• 2:32 pm: started cleaning the buckets
• 2:35 pm: started mopping with IPA wipes
• 2:43 pm: finished mopping with IPA wipes
• 2:43 pm: changed sticky floor mats
• 2:45 pm: started particle count
  • zone 3:
    • 0.3 u: 1288
    • 0.5 u: 124
    • 1.0 u: 0
• zone 4:
  • 0.3 u: 415
  • 0.5 u: 290
  • 1.0 u: 0

cleaning cleanroom and particle count

• 1:45 pm: ran zero count test on particle counter
• 1:50 pm: started particle count
  • zone 3:
    • 0.3 u: 2660
    • 0.5 u: 332
    • 1.0 u: 41
• zone 4:
  • 0.3 u: 8106
  • 0.5 u: 2494
  • 1.0 u: 831
• 2:08 pm: began surface check and wipedown, including softwalls
• 2:22 pm: started vacuuming the floor
• 2:39 pm: finished vacuuming the floor
• 2:43 pm: started mopping the floor
• 2:53 pm: finished mopping the floor
• 2:54 pm: started cleaning the buckets
• 2:59 pm: started mopping with IPA wipes
• 3:06 pm: finished mopping with IPA wipes
• 3:10 pm: changed sticky floor mats
• 3:06 pm: started particle count
  • zone 3:
    • 0.3 u: 1496
    • 0.5 u: 83
    • 1.0 u: 0
• zone 4:
  • 0.3 u: 789
  • 0.5 u: 83
  • 1.0 u: 41

cleaning cleanroom and particle count

• 10:45 am: ran zero count test on particle counter
• 11:02 am: started particle count
  • zone 3:
    • 0.3 u: 3284
    • 0.5 u: 1247
    • 1.0 u:332
• zone 4:
  • 0.3 u: 1829
  • 0.5 u: 581
  • 1.0 u: 207
• 11:15 am: began hepavac of rest of lab
• 11:19 am: began surface check and wipedown, including softwalls
• 11:32 am: finished hepavac of rest of lab
• 11:35 am: started vacuuming the cleanroom floor
• 11:45 am: finished vacuuming the floor
• 11:47 am: started mopping the floor
• 11:55 am: finished mopping the floor
• 11:56 am: started cleaning the buckets
• 11:57 am: started mopping with IPA wipes
• 12:02 pm: finished mopping with IPA wipes
• 12:03 pm: changed sticky floor mats
• 12:04 pm: started particle count
  • zone 3:
    • 0.3 u: 3117
    • 0.5 u: 374
    • 1.0 u: 124
• zone 4:
  • 0.3 u: 4531
  • 0.5 u: 540
  • 1.0 u: 0

Finished ADC-DAC loopback testing today (see attached xlsx file or access directly here). All looks well with the first 8 channels, with the gain hovering just under 2. Also edited c1msc model in simulink to add channels 7-15 (the last 8 channels), and changed the rate back to 64K. See image 1 for the updated c1msc model. The last 8 channels are also looking good and show no problems, with slightly more scattering for the gain, but all values very close to 2.

We also installed the new eight-channel Perle IOLAN SDS8 terminal server today. Image attached.

NOTE: When we were installing it, we noticed an energized wire dangling from the 24V power supply. Has since been fixed and put back into place.

cleaning cleanroom and particle count

• 12:25 pm: started particle count
  • zone 3:
    • 0.3 u: 4614
    • 0.5 u: 872
    • 1.0 u:83
• zone 4:
  • 0.3 u: 2411
  • 0.5 u: 415
  • 1.0 u: 83
• 12:48 pm: began surface check and wipedown, including softwalls
• 1:20 pm: started vacuuming the floor
• 1:30 pm: finished vacuuming the floor
• 1:34 pm: started mopping the floor
• 1:40 pm: finished mopping the floor
• 1:40 pm: started cleaning the buckets
• 1:42 pm: started mopping with IPA wipes
• 1:50 pm: finished mopping with IPA wipes
• 1:51 pm: changed sticky floor mats
• 3:44 pm: started particle count
  • zone 3:
    • 0.3 u: 3207
    • 0.5 u: 624
    • 1.0 u: 83
• zone 4:
  • 0.3 u: 916
  • 0.5 u: 83
  • 1.0 u: 0

[Ma, Luis, Shane]

Working theory for the timing chassis issues had been that the 1A breaker was tripping and causing the failure of the Valon 5015 and 3010 to output the timing signal correctly. We just tried bypassing the breaker, running 6 V on the benchtop power supply (set the current limit to 1.5A), with the 5010 generating the sine wave to pass to the 3010. All worked correctly, and there were no issues. Square wave outputted by the 3010 was exactly as desired (image attached) at the correct frequency, and this confirms the issue was the breaker, not the valon 5015. Ready to go ahead with ordering a new replacement breaker.

We replaced the 1A breaker in the timing chassis today with a 4A one, and tested that all is working well. The chassis successfully outputted the correct signal (image attached). The real time models have also been restarted and the CyMAC diagnostics screen is showing all green flags. Timing chassis has been closed up and reinstalled in the server rack.

Debugging update for vac system serial communications: we were able to successfully connect the smaller turbo pump to the Varian (manufacturer) software today by plugging the controller db9 directly into spica, and verified that the pump was able to send and receive information. Still no success on communication with the larger pump via python serial code, though. Issue potentially lies in the pinout mapping of the connectors, as the pinouts we'd been using work successfully for the small pump which only requires 3 connections (RXD, TXD, GND), but the large pump may require more. Tried to test this on the big pump by keeping the pinout setup for the small pump, and incrementally adding the remaining 6 connections on a one to one scale (Pin 1 matched to Pin 1, etc) between the controller DB9 and the field DB9 connected to the IOLAN ethernet cable. After each new connection was added, I retried running the serial connection code. None of the additional connections yielded any results, and the large pump remained unable to connect through the code. Next step is further looking into documentation on the pinouts to see if the large pump requires a different connection setup than the small pump.

[Shane, Jon]

Update on the serial interfacing with the vacuum system: both turbo pumps are now able to successfully communicate through the code. The older pump, for unknown reasons, required a newer version of the communication syntax (despite the newer, small pump being able to communicate fine with the old communication format). We have also confirmed that the newer pump works with this new format as well. Next step is interfacing the pressure gauge.

[Shane, Jon]

Port 80 is kept closed by default. This might be causing the certbot auto-renewal cronjob to fail. Therefore we must renew the certificate manually.

Step 1: Open port 80. (This is needed as the certificate renewal process runs some tests which requires client communication over port 80)

Step 2: Run the following command

sudo certbot certonly --force-renew -d richardsonlab.ucr.edu

Step 3: Confirm the certificate was renewed by running the following command

sudo certbot certificates

[Jon, Pooyan]

Moved Chimay from the server rack in Physics 1119 to a new rack in Physics 1129. It is connected to the switch in that rack and has the same ip address as before.

All services are up and running.

It appears that JupyterHub creates some processes whenever a user connects to an instance of it, but in some cases does not stop those processes after the user is not using that instance. This results in having lots of running idle processes, each using a small bit of the resources. Those processes are killed now as a result of rebooting. It might be a good idea to manually restart JupyterHub (or the whole machine) every few months to avoid this.

Attached is a brief recap PDF file. A video file showing separate HOMs plots for the cavity scan with ETM08 surface map is also attached.

The codes are available at https://git.ligo.org/uc_riverside/hom-rh/-/tree/main/SIS

[Jon, Pooyan, Tyler]

A few computer machine changes have been made. 

• Logrus moved from 1119 to 1129. It is up and running with the same IP address as before. 
• A new Windows machine (host name: spica, IP:192.168.1.14) is installed in the 1119 server rack. It is connected to the RGA scanner with the serial port and is specifically used for that purpose. 
  • Update: The machine was off on 6/25, although it was left on 6/24. We think that it might have been because of Windows' default setting to suspend/hibernate the machine after idleness. To resolve this, I used  "powercfg /change" command to set all the following parameters equal to zero. The machine is still running on 6/26. 

    monitor-timeout-ac
    monitor-timeout-dc
    disk-timeout-ac
    disk-timeout-dc
    standby-timeout-ac
    standby-timeout-dc
    hibernate-timeout-ac
    hibernate-timeout-dc

• A new Debian machine (hostname: megatron, IP:192.168.1.16) is installed in the 1129 server rack. This machine is intended to be used for FEA/simulation work. A new 2TB WD Green SSD is used as its main disk drive. 

   At the moment, “controls” is the only user, and there are no apps/libraries installed on the machine.

  • Update: Jon installed LIGO cds-workstation tools and MiniConda on 6/26. 
  • Update: Pooyan and Liu set the following conda environments:
    • Env named “finesse” with Python 3.12.3 and Finesse version 3.0a24 installed. Finesse was installed via the source code. The subdirectory “/home/controls/packages” is used to store the package sourcecodes.
    • Env named “fenicsx” with the same version of Python and Finesse as the previous env, with the latest version of FEniCSx (0.8) and the test-mass-thermal-state installed.  

Created a Google Slides presentation to summarize all the mirror surface map information that we use for simulating interferometers. 

A+ expected maps are based on correspondence with G. Billingsley. The estimate for the A+ ITMs will be to take the “as polished” data and add coating non-uniformity to it. (T2000398) Neither of these are scaled for the precise thickness of the Ti:Ge coatings.

Google Slides link: https://docs.google.com/presentation/d/1ge-ciAiEdNyyTvSShYdZz2JpACFRY2W3JDpxHRqMnOQ/edit?usp=sharing

Installed a Synology NAS server (Synology RackStation RS1221) in lab room 1129, with host name “scribe” and ip “192.168.1.17”. It is mounted on the rack and each of its 8 storage bays has a 2TB SSD disk. It will be used to set up automated backups of all the lab machines (e.g., chimay, logrus, megatron).

One shared storage is set on it with SHR-2 as its RAID type. It can tolerate the failure of two disks and has 10.4TB of total capacity. 

We can use both rsync and dd to create backups of the system. A suggested backup schedule could be daily rsync backups and bi-weekly disk snapshots using dd. 

One ongoing work is to make all lab machines automatically backed-up on Scribe on a daily basis. The updates should be boatable and stored for some time (potentially a few weeks) on Scribe. Making whole disk images has already been tried for some of the machines with no problems. (e.g., Cymac and WorkStations)

The same thing can not be done with Chimay though, as it currently has one huge RAID-controlled volume that stores all the information (OS, home directories, and NDS-downloaded data). Creating daily full disk images of such a system is not practical. 

Here is the plan we came up with to overcome this issue:

1. Create one full disk image of Chimay and store it on Scribe (it was already done)
2. Move the nds-downloaded raw data temporarily to Scribe and remove it from Chimay
3. Make another full disk image of Chimay
4. Burn this image to a single disk and boot chimay with it

5. Restore the rest of Chimay disks and move the NDS data back

    

On the weekend (Sat and Sun 9/7-8) I tried to execute these steps. There wasn't enough free space left on Scribe to move all the NDS data to it, so I stored part of this data temporarily on WS4. Then I also checked storage-consuming directories and, in one case, removed some non-important stored files. As there was no free space left on Scribe to execute step no. 3, I initiated storing the image on WS3. Unfortunately, a couple of different trials of the image-creation process failed as there was not enough free space on WS3 to accommodate Chiamy image as well. I was not able to reduce the image size such that it can be stored on WS3. 

These are the options left for us to get this work done. 

1. Distribute the NDS files between different machines to make enough free space on Scribe for the image and then follow the previous plan
2. Shrink the Chimay drive size, create the image and then follow the previous plan
3. Temporarily transfer some services to Megatrone (Network gateway, Wiki, elog) and recreate chimay and its services from scratch

There are three major issues with the lab's CDS system that should be addressed. They are ordered from the highest priority to the lowest.

• CyMAC has stopped serving the fast channels (65k Hz)
  The channels are still there, and the data from the slow (16 Hz) channels is served correctly, but the fast data is not served. As a result, diagggui can not retrieve any information at the moment.
  Probably the issue is the same as the version conflict between daqd and nds2-server. After installing nds2-server on CyMAC and realizing that the conflict can not be resolved, I reverted the changes so that the CyMAC could be compatible with daqd, but there is a chance that there are still version conflicts that block the fast channels.
  I suspect that we have broken the MSC model when adding the new button. This could explain both the fasr-channels fault and the unexpected crashes.
  
  UPDATE 6/5
  Fixes this by rebuilding the models.
  Note for the future: The c1iop and c1msc models should be rebuilt together, even if no change has been made to c1iop.
• The fast channels were not saved to the frames
  Reviewing the frame files showed that the daqd was only saving the 16 Hz data in the .gwf frame files. I had missed this because I only checked the existence and the integrity of the raw frame files, but never retrieved the high-frequency data itself to check it.
• NDS2-server is installed, but does not distribute the data
  I managed to install the nds2-server and its satellite programs on the Chimay, and NFS mount the frame files on it. The server successfully chaches the channel names, frame files, and the data, but fails to list the channels for distribution.

Note: CyMAC machine was strangely disconnected from the network, and required manual reboot. I couldn't find any abnormal logs that could explain what happened.

• The MEDM models issue was fixed. The issue was that building the c1msc model alone was not enough and we had to re-build the c1iop as well, although no changes has been made to the c1iop. Added a note in the wiki on this for future reference.
• For writing the fast channels to the frame files, our CDS system should have a GDS broadcaster. I followed this wiki page to set one, but it breaks the current daqd system. Asked about it earlier today on the CDS Mattermost channel and waiting for a response. 
• Maybe I should continue the distributed scheme we have been trying for the nds2-server and let another machine on the network do this. 
• Question: daqd has a feature to zero-out the bad data (NaNs), and it is currently on for our system. I don't think that it's a good idea to have this. Should I turn it off? 

• Fast channels recorded to .gwf frame files
  • Had to add a DAQ Channels block to the c1msc model for those channel to be recorded in the .gwf frame files. The block is in the CDS_PARTS, and I just had to copy it to our model. Here is the example text for the channels:
    
    

    #DAQ Channels

    ONE_DAQ_CHANNEL 2048
ANOTHER_DAQ_CHANNEL 1024
SCIENCE_FRAME_CHAN* 1024
UINT32_CHAN uint32 2048
DAQ_CHANNEL_AT_DEFAULT_RATE

    I added two channels to the list (V0 and V1). Remember that the channel name should not be the full name, but just the part's name. In this case, I added these two channel names:
    
    V0_OUT
V1_OUT
    
    with the default sample rate. 
• The fast channels are recorded with an extra _DQ in their names. So, in ndscope and diaggui these two channels should be addressed as
  • C1:MSC-V0_OUT_DQ
  • C1:MSC-V1_OUT_DQ

• Update 6/13: enabled recording of all the ADC voltage channels so that Tyler can use all the data for the noise floor calculation. attached is an example of diagggui with almost 30 hrs of data.

• TODO: Enable daqd_wiper

• Hosted Lukes model on Chimay. It's available at https://richardsonlab.ucr.edu/real-time-frosti
• The sourcecode is at Github. It is clones at /var/www/html/real-time-frosti, and this is the block added to nginx config file at /etc/nginx/sites-available/default :
  
  location /real-time-frosti {

root /var/www/html;
index index.html index.htm index.php;
}
• Also added this line to the crontab, so the code will be checked for updated from the sourcecode every five minutes. 
  
  */5 * * * * cd /var/www/html/real-time-frosti/ && git pull 
  
  
• TODO: Move the sourcecode from Github to git.ligo.org, and make the repository public. 

We have tested the heater to find emissivity, mounted the heater system to the optical table, and have taken irradiance maps of the heater projected onto the screen.

The heater's emissivity was determined by using a thermocouple in conjunction with the FLIR's temperature calibration. To attach the thermocouple to the heater initially, I used Kapton tape and ran both the wires of the heater and the thermocouple through the heater bridge. This allowed for the heater to rest on an optical post and be observed without anyone directly holding it, but there were some measurement issues. The thermocouple had a very wide range of temperatures it was reading, which may have been due to intermittent contact or a short between the two legs of the thermocouple. To solve this and make the temperature measurements more stable, we pried apart the two ends of the thermocouple (to ensure there was no short) and put tape on either side, leaving the end connection bare. This was then taped to the heater, and the thermocouple was much more stable. We also used a K-type thermocouple that has an adhesive tape on it already, which assisted with the intermittent contact as well. With the thermocouple measuring the temperature of the heater, we could point the FLIR directly at the heater and calibrate the emissivity until the FLIR and the thermocouple agreed. Cassidy's emissivity calculator was also used, as I could input a temperature and observe what the emissivity of an area was based on that temperature. We found the emissivity of the heater to be 0.57.

As a note, when observing the heater with the FLIR, it appeared that there was a hot spot in the center, where the Kapton tape sat. Because the Kapton has a different emissivity than the 304 stainless steel of the heater, the FLIR will read it as having a different temperature than it actually does. When using the FLIR in the future, be sure to ascertain whether there is a temperature difference somewhere or if there may be different emissivities.

Additionally, the first heater that I used was taken to a very high temperature and oxidized. The emissivity of this oxidized heater is not known, but could be good information for knowing how oxidation affects these heaters specifically.

To mount the heater system in front of the screen, I used 1/2'' optical posts and the mount I designed using COMSOL's CAD program. The heater was originally 2.5 inches away from the screen, and has since been moved back by an additional two inches so that we could observe the heater side of the screen with the FLIR. We wanted to see what temperature the heater side of the screen was when irradiated by the heater, and how that compared to the camera side of the screen. When the heater ran at 1.12 W of input power, the heater side of the screen had a max temperature of around 29.7 C, and the camera side of the screen read at about 29.5 C. This means that there is very little thermal loss between the two sides of the screen, and any insulation that the screen's adhesive may have is largely negligible. Additionally, the camera was placed at an angle and undetermined distance for these tests, confirming that the temperature measurements compensate well/don’t depend on changes in angle or distance between the camera and the screen. However, there was spots on the back of the screen that the camera was measuring as hot spots where there shouldn’t have been any. I have included an example below. It would be useful to run a test where the camera is directly on the back of the screen without the heater to characterize the screen and see if the hot spots are physically present on the screen or if this is an artifice of the camera because of something like angle of viewing.

Taking irradiance maps of the screen was straightforward. After checking that the emissivity of the screen is 0.99 by viewing it at room temperature, we monitored the max temperature while slowing increasing the wattage the heater was running at. There is not a large change until the heater is at around 95 C, at which point the screen began to rise in temperature from 27 C to 28 C. We took measurements of this while the heater was 2.5 and 4.5 inches away from the screen. The irradiance map has a very symmetrical and circular shape, but does not have the ring pattern that we expected. There may be a few reasons for this: there could be some conduction between the two sides of the screen that is causing the pattern to spread further, the heater setup may not be as ideal as it was modeled to be, or there could be a different, unknown issue.

TO DO:

- It would be useful to run a test of the camera in multiple different positions to ensure our conclusion that the camera’s measurements don’t depend on angle or distance (or that these factors are well accounted for in the current temperature calculations) is correct.

- Measure the back of the screen straight on to identify bright spots and possible reasons as to their appearance.

- Recalibrate camera to ensure it is still correct after testing in multiple positions.

- Take another irradiance map of the screen at a higher input power, as well as moving the heater close/further away to try and replicate the COMSOL irradiance maps. It would be useful to also redo the COMSOL modeling at lower powers and variable distances.

Pictures included of full table setup, the heater mount, the heater with Kapton tape attaching the thermocouple as well as FLIR's measured irradiance map.