/mnt/smart is filling up QUICK! Clean up!

[pablo-icm]

Please, check whether you can remove things there. This is serious!

[pablo-icm]

98% people!

[xabilopalf]

I tried guys, however, I can't even do a simple "ls" without getting stucked.

[franlat]

🚨 Urgent – /mnt/smart is completely full

Hi everyone 👋,

Our shared workspace /mnt/smart (Lustre) has reached 100 % capacity (again).

---

Why a full Lustre file system is a big deal:

What you might notice	Why it happens
Jobs stop, hang, or crash.	Lustre needs “scratch room” to record every change. With zero room, it can’t finish even tiny writes.
Even simple reads become slow.	The system keeps trying (and failing) to free space.
Possible downtime for repairs.	If Lustre can’t update its own “index”, we might have to take it offline eventually.

Think of Lustre like a giant shared whiteboard: when every square is filled, nobody can add anything until space is cleared.

---

How Lustre differs from our HDisk (/mnt/cold02):

	Lustre (/mnt/smart)	HDisk (/mnt/cold02)
Purpose	Fast, temporary workspace for running jobs.	Slower, long-term storage for completed results or files.
When full	Cluster-wide freeze of reads and writes.	Shared slowdown, but jobs can keep running.
Ideal Free-space margin	Keep ≥ 15 % free (max. 85% - we are 100%!!).	Keep ≥ 10 % free (max. 90%).

Both are shared, but Lustre is much more sensitive because it must constantly update its internal table of contents while jobs stream data.

---

What to do right now ✅

1. Check your space

   lfs quota -u $USER /mnt/smart   # usage on Lustre
   df -h /mnt/cold02               # quick look at HDisk

2. Delete or move anything you don’t need

3. Old outputs, temp files, logs, duplicates.

4. Move large, rarely-used files to /mnt/cold02 or your own personal storage disks: rsync -aP /path/to/my_results /path/to/destination (⚠️ remember to always have an external back-up of your data!)

5. Compress bulky folders: tar -czf my_results.tar.gz my_results/

What is being done ⌛

From the e-mail received by everyone in the smart-users list yesterday, and roughly translated to English by me:

Hello,

As shown on the Migration tab of the storage dashboard (https://docs.google.com/spreadsheets/u/0/d/e/2PACX-1vSrkhbptpqiLsZXOrGP6vqhSDKvuwYT77YIu01q-aCAJhpXTyatiTT2AiIsfcYjH_LZg3t4Hd1spyUS/pubhtml?pli=1#), we are currently making a full backup of the /users directory, which was outside the intended scratch area.

Transfer speed is quite slow because COLD02 is not yet connected to the high-speed private network (we will address this soon). For now, we must copy the data over the public network, with the usual bandwidth limits. The average rate is approximately 2.5 hours per terabyte, so we estimate the backup will be completed before Friday, July 17th.

If everything looks good, on Monday, July 21st, we will delete /mnt/smart/users, freeing roughly 100 TB on Lustre. That space will then be redistributed to groups according to their quotas.

[aleixop]

Thanks for the info, Fran. How reliable is the connection to cold02, though? Is it more stable now? Did something change?

[franlat]

The connection is very slow, but reliable. Although, you should always make sure the things are copied correctly (I had some files copied incorrectly in the past).

However, Servei de Computació Científica (aka Juanlu & Co) are making a big dump of data into cold02 at the moment. They are copying all the /mnt/smart/users into it, so those directories can be removed from smart-lustre and free some space. If your data is safe in /mnt/smart/scratch, I would wait until that copy is finished, so we don't clock the connection even more.

The message I wrote yesterday was before Juanlu's last e-mails, so right now, I would just focus on removing / compressing things in /mnt/smart/scratch.