This are my notes from Winter 2022 before the Richardson Lab ELOG was setup. TODO: Need to write a proper "HowTo" ELOG for jupyterhub + web server setup based on this.

JupyterHub on Chimay

[Early Jan 2022]

Cannot install using TLJH since it's only available for Ubuntu. So instead, installing step-by-step using guide available here - https://github.com/jupyterhub/jupyterhub-the-hard-way/blob/master/docs/installation-guide-hard.md

• Nginx webserver is used for HTTPS reverse-proxy for the jupyterhub. I installed nginx via apt-get, config at - /etc/nginx/sites-available/default

• I used cerbot to generate SSL certificates.

To start the webserver sudo systemctl start nginx.service

Goal: Setup JupyterHub as a systemd service and make it available over HTTPS.

To start the services needed for the hub sudo systemctl start jupyterhub.service

Note: Reload the daemon if you edited a service before you restart it, otherwise it will complain.

sudo systemctl daemon-reload

Logs

[Jan 26, 2020]

Ended up using the JupyterHub guide available here - https://github.com/jupyterhub/jupyterhub-the-hard-way/blob/master/docs/installation-guide-hard.md and SudoSpawner guide available here - https://jupyterhub.readthedocs.io/en/stable/reference/config-sudo.html

I was struggling with this quite a bit but eventually I figured out that I was running into trouble because I was trying to run jupyterhub command as a password-less sudo user as I thought that's what sudospawner needed. That seems not to be the case.

1. The systemd service jupyter.service has to be run as User=root and the ExecStart=/opt/jupyterhub/bin/jupyterhub -f /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py and ensure that /usr/sbin is in PATH. (This necessary as we need to spawn single-user jupyter servers for our users on-demand, so the User that runs the ExecStart needs to have permissions to run single-user server init commands.)
2. In jupyterhub_config.py, the spawner is configed as sudospawner - c.JupyterHub.spawner_class=sudospawner.SudoSpawner. I installed sudospawner in our jupytherhub environment via pip.
3. In jupyterhub_config.py, the authenticator used is PAM which allows linux users on chimay to login with there username and password and opens their shell in their home directories.

[Jan 27, 2020]

[10:30]

Attempted MATLAB integration following this guide - https://github.com/mathworks/jupyter-matlab-proxy

It annoyingly broke nodejs which broke jupyterhub as the proxy failed without it. I have now restored nodejs and the jupyterhub is back up.

[12:00]

User controls now has admin access on the hub. To add more admins - edit /opt/jupyterhub/etc/jupyterhub/jupyterhub_config.py by adding the username in c.Authenticator.admin_users

[15:15]

Argh! User's cannot access user-created conda environments in notebooks besides the default one i.e. the one for which kernelspec is available in /usr/local/share/jupyter/kernels. Non-sudo users also can't access this path to put a kernelspec in it! Sudo users who can do this end up having their custom environments visible to everyone. Argh! !FIX NEEDED!

[15:54]

After reading this (https://github.com/Anaconda-Platform/nb_conda_kernels/issues/185) I am putting

"CondaKernelSpecManager": { "kernelspec_path": "--user" }

in /opt/miniconda3/etc/jupyter/jupyter_config.json

WebServer down, fix yet to be tested.

[16:04]

Fix does not seem to be working.

[Tue Feb 1 17:02:46 2022]

ISSUE: User unable to use their custom conda environments in notebooks

STATUS: Fixed, refer to elog 'ligo-ds' no. 1

All future logs will be on the ELOG.

Prerequisites - Make sure you have your Mathworks login details handy. UCR provides a single-user MATLAB license to students.

Steps:

1. Login to JupyterHub and select Open MATLAB from the Launcher tab. Click on Open MATLAB.
2. Enter your Mathworks login detail in the Online License Manager window that pops up.
3. Status Information window will popup and MATLAB startup process will begin. This can take a few minutes the first time. You'll eventually land on a standard MATLAB GUI.
4. From here on, you can continue developing in the MATLAB GUI or you can navigate back to your JupyterHub launcher to create a notebook and run MATLAB code in a notebook. Just choose MATLAB kernel to execute the code.
5. You can close the MATLAB session and sign out by using the navigation button that pops up on the MATLAB GUI.

Fix:
Used the fix on this blog [https://danct12.github.io/Fix-X11-Forwarding-sudo/] to add the current user's SSH session authentication to the root's .Xauth file.

$ echo $DISPLAY
$ xauth list $DISPLAY
$ sudo xauth add chimay/unix:12 MIT-MAGIC-COOKIE-1 c9acf72143c05334f58f**********220

after this, the user can run matlab as super user, "sudo matlab", and use the GUI. This should be done by a user that has sudo access.
This process seems to install the toolboxes for all users, so there shouldn't be any need of doing this by others.

Cannot access SegDB on Chimay. See error below.

from gwpy.segments import DataQualityFlag
from gwpy.time import to_gps, from_gps
ifo, run = 'L1', 'O4'

if run == 'O3':
O3a_start, O3a_end = '2019-04-01T08:00:00', '2019-10-01T08:00:00'
O3b_start, O3b_end = '2019-11-01T08:00:00', '2020-03-27T10:00:00'
start_gps, end_gps = to_gps(O3a_start), to_gps(O3b_end)
if run == 'O4':
O4_start, O4_end = '2023-05-24T15:00:00', '2024-01-16T16:00:00'
start_gps, end_gps = to_gps(O4_start), to_gps(O4_end)

print(start_gps, end_gps)
segs = DataQualityFlag.query(ifo+':DMT-ANALYSIS_READY:1',start_gps, end_gps, url="https://segments.ligo.org")

HTTPError: 401 Client Error: Unauthorized for url: https://segments.ligo.org/dq/L1/DMT-ANALYSIS_READY/1?e=1389456018&include=metadata,known,active&s=1368975618

Issue: Currently, the only kernelspec present in /usr/local/share/jupyter/kernels is the miniconda environment called 'python' (/opt/miniconda3/envs/python/).

Goal: We want users to be able to create their own conda environs and use them in their jupyter notebooks. Currently they can only use their custom environs in terminal.

ToDo: Figure out a way to place user1's custom environment 'mytestenv' kernelspec in /home/user1/.local/share/jupyter/kernels/mytestenv

[UPDATE] Fixed this issue. Users should follow the following steps to have their custom conda environs available in their jupyter notebooks.

1. Say you have a conda env named mytestenv. Activate this env.

2. Install ipykernel in the conda env you want to add to Jupyter:

$ conda install ipykernel

3. Install the kernel

$ python -m ipykernel install --user --name mytestenv --display-name "My Test Env"

You can always change the --name (how this kernel appears in your jupyter directory) and --display-name (how it appears in the JupyterLab's 'Select Kernel' window) but it is recommended to match it with your conda environment name to avoid future confusion.

4. Check if you installed the kernel successfully:

$ jupyter kernelspec list

[UPDATE: JR] After completing step 4, you must restart your server for the newly installed environment to be visible in your Jupyter notebooks.

5. Click: File > Hub Control Panel > Stop My Server > Start My Server

6. Reopen your notebook. Then click: Kernel > Change Kernel... and select the name of your custom conda environment from the drop-down list.

Followed the instructions in this installation guide - https://github.com/mathworks/jupyter-matlab-proxy - to install the jupyter-matlab-proxy in the jupyterhub python environment using command

sudo /opt/jupyterhub/bin/python3 -m pip install jupyter-matlab-proxy

But upon first execution I have to provide Mathworks accounts details.

The license on Chimay belongs to Prof. Jon. So, he'll have to insert those.

Right now MATLAB kernels for jupyter notebook as well as console show up (as shown in screenshot below) but code will not execute.

If I click on Open MATLAB tab, it will ask for Mathworks account login details.

The installation guide mentioned above says,
"When you execute MATLAB code in a notebook for the first time, enter your MATLAB license information in the dialog box that appears. See Licensing for details. The MATLAB session can take a few minutes to start."

Follow-up with Prof. Jon needed.

I was incorrect about Prof. Jon needing to activate the MATLAB license for JupyterHub for all users to use the Jupyter MATLAB Proxy. Turns out the license on Chimay cannot be used by multiple users. But no worries, a user can click on Open MATLAB tab and then on the Online License Manager window login to your Mathworks account to use your UCR-provided license. If you instead click on Existing License tab and try to use the currently active license, your startup will eventually fail. See attached screenshots.

Log:

Two days ago I noticed a high CPU usage on Chimay. The proccess "kswapd0" run by user "testuser" was taking all of the ram. [htop screenshot attached]
I followed the answer here [https://askubuntu.com/a/530661] to make kswapd0 use swap only if there is no usable RAM left on the system.
also restarted the swap manually and changed the swappiness by sysctl, but the process was still there. Killing the process didn't work as it was back running in a few seconds.
I then came across this question [https://askubuntu.com/questions/1224927/cpu-100-with-kswapd0-process-although-no-swap-is-needed], showing that there is a coin miner malware called "Multios.Coinminer.Miner", hiding itself to look like the kswapd0 process.
Contamination by the malware is confirmed by looking at the directory that the code resides in the testuser directory. No other user seems to be infected. Logs show that this user was created three days ago and is only used for JupyterHub testing.

from /var/log/auth.log:
Feb 26 13:25:49 localhost sudo: rutuja : TTY=pts/167 ; PWD=/home/rutuja ; USER=root ; COMMAND=/usr/sbin/useradd --create-home testuser
...
Feb 26 18:22:17 localhost sshd[845020]: Failed password for testuser from 14.18.92.211 port 54280 ssh2
...
Feb 26 23:36:03 localhost sshd[934183]: Accepted password for testuser from 43.128.69.133 port 52690 ssh2

Apparently the user was compromised a few hours after its creation, and started its job a little bit later. Looking at the failed login logs show that there is a constant attack on Chimay of Brute-force ssh logins with dictionary usernames and passwords.

I killed al the processes by the testuser, deleted the malicious files and its home directory, and deleted the user itself. Thee seems to be no residues of it. No cronjob related to the malware is seen. CPU usage is back to normal.

Suggestions:
It seems that no other user accounts were compromised. But because there is these brute-force login attempts on Chimay, I have two suggestions:‌
Changing the passwords and having stronger ones, replacing ssh-keys (The user was running rsync, but probably they didn't care about the files that we have)
Disabling ssh login by password and asking everyone to use ssh-keys