Summary: There was a Cryptocurrency mining malware running on Chimay for ~2 days and using all of its CPU. Only one user was compromised, most probably by Brute-force ssh login. The malware, it's files, and the user was removed.
Log:
Two days ago I noticed a high CPU usage on Chimay. The proccess "kswapd0" run by user "testuser" was taking all of the ram. [htop screenshot attached]
I followed the answer here [https://askubuntu.com/a/530661] to make kswapd0 use swap only if there is no usable RAM left on the system.
also restarted the swap manually and changed the swappiness by sysctl, but the process was still there. Killing the process didn't work as it was back running in a few seconds.
I then came across this question [https://askubuntu.com/questions/1224927/cpu-100-with-kswapd0-process-although-no-swap-is-needed], showing that there is a coin miner malware called "Multios.Coinminer.Miner", hiding itself to look like the kswapd0 process.
Contamination by the malware is confirmed by looking at the directory that the code resides in the testuser directory. No other user seems to be infected. Logs show that this user was created three days ago and is only used for JupyterHub testing.
from /var/log/auth.log:
Feb 26 13:25:49 localhost sudo: rutuja : TTY=pts/167 ; PWD=/home/rutuja ; USER=root ; COMMAND=/usr/sbin/useradd --create-home testuser
...
Feb 26 18:22:17 localhost sshd[845020]: Failed password for testuser from 14.18.92.211 port 54280 ssh2
...
Feb 26 23:36:03 localhost sshd[934183]: Accepted password for testuser from 43.128.69.133 port 52690 ssh2
Apparently the user was compromised a few hours after its creation, and started its job a little bit later. Looking at the failed login logs show that there is a constant attack on Chimay of Brute-force ssh logins with dictionary usernames and passwords.
I killed al the processes by the testuser, deleted the malicious files and its home directory, and deleted the user itself. Thee seems to be no residues of it. No cronjob related to the malware is seen. CPU usage is back to normal.
Suggestions:
It seems that no other user accounts were compromised. But because there is these brute-force login attempts on Chimay, I have two suggestions:
Changing the passwords and having stronger ones, replacing ssh-keys (The user was running rsync, but probably they didn't care about the files that we have)
Disabling ssh login by password and asking everyone to use ssh-keys
|