r/Ubuntu • u/el_beef_chalupa • 22h ago
26.04 changes /tmp to tmpfs
Long time Ubuntu user here but I don't follow what happens to other Linux distros outside of Ubuntu since Ubuntu is relevant to me, so this was news to me.
Ubuntu adopted /tmp as tmpfs meaning files in /tmp use system RAM instead of disk space. On the surface, I completely get it...
However, our company had a history of using /tmp more like /var/cache. And under 24.04 and before this has been fine. You reboot the server, the cache gets cleared, no harm no foul. Maybe not exactly following conventions but nothing was necessarily wrong with this.
When testing the upgrade to 26.04, I kept having weird OOM killed processes, scripts were dying in strange spots. Especially around gziped files in /tmp. Well a 10MB tar.gz can blow up to 300+MB pretty easily for a product catalog. Now that 300MB is using system RAM instead of disk space... You see the problem.
Anyway, PSA for anyone else out there that used /tmp like this. I can't be the only one.
11
u/mgedmin 20h ago
Back in 2000 I was preparing an ISO image with random entertainment media for a 5-month study abroad, where I expected not to have Internet access. So I mkdir /tmp/stuff, and start a couple of recursive wget's of various webcomics (User Friendly and such) then go to sleep, because it's 2000 and Internet is slow (I had a luxurious 256 Kbit/s downlink!).
In the morning I discover that
- wget --mirror copies the Last-Modified response header into file mtimes
- Debian runs tmpwatch from cron by default
- tmpwatch deletes all files older than a week, according to their mtime
All my apes webcomics, gone! And my flight is in a few hours.
(Somehow I made it. I re-ran the right wget commands, using a staging directory outside /tmp this time, and then cdrecord'ed the files into a CD-R, and then made my flight in time.)
Lessons Were Learned about appropriate use of /tmp.
3
u/el_beef_chalupa 20h ago
I suppose the difference between your outlined use case here and mine is really don't care that the files in /tmp or any subdirectory get deleted after a certain time or on reboot. The files are theoretically temporary and it's fine. The surprising thing now is they eat system RAM.
15
u/ThePerplexedSynopsis 22h ago
old habit from the sysadmin days, /tmp was always the junk drawer. 300mb in ram is nothing until you've got six jobs unpacking at once and suddenly your box is gasping for air. we had a similar mess with a jenkins pipeline that used /tmp for scratch space, took us way too long to figure out why the nodes kept dying after an upgrade. moved it all to a proper directory under /var and the problem vanished
5
u/el_beef_chalupa 22h ago
Your team is in the same exact spot my team is in. Yes, 300MB is not a problem until 15 jobs all write their own files here and there and before you know it you have 4GB of RAM eaten up when you don't expect it.
I'm moving everything to /var/cache right now too. Just have to test and break other devs old habbit of checking /tmp for their files!
3
u/CharacterUse 19h ago
Changing /tmp back to being an actual filesystem is less work and less likely to miss a script.
1
u/el_beef_chalupa 19h ago
I understand that. I already fixed our system actually. Almost all of the configuration was in one spot so I just updated the variable to point to a new spot. I'm monitoring whether files still get put in the old spot, /tmp, on which case I'll have something more to fix. Our production systems are using 24.04 so there is no immediate risk here. I just need to fix it all before the migration to 26.04.
7
u/john2095 21h ago
I'd guess you are the first canary gasping in a very large mine. I am a mere desktop/workstation user and I certainly use /tmp as a dumping ground for boundless crap and work stuff. I have no expectation that it will survive a reboot, but I certainly never expect it to cause OOM difficulties.
Which committee voted on this decision?
5
u/el_beef_chalupa 21h ago
When I googled it earlier, I read that Fedora and other distros adopted this a while ago, the debian line is just now following.
Like I understand the change theoretically, tmpfs auto cleans itself on reboot, it's super fast... But also /tmp seems like a just fine place to place compressed files and unzip them which is obviously going to cause problems on some systems.
5
u/lathiat 21h ago
For what it's worth you could either disable tmpfs or alternatively add more swap space to deal with the increased space requirements.
It's most likely to be an issue in small-ish VM environments I would suspect.
3
u/el_beef_chalupa 21h ago
Yes, I thought about that, but I also realize we probably shouldn't have been using /tmp in this manner anyhow and should just fix the application.
I am curious how many Linux programs in the world use /tmp to do certain tasks that were expected to write files to disk space temporarily and now it's going to eat a ton of RAM. Maybe all the bugs out there with that have already been worked out, idk.
6
5
u/SAJewers 16h ago
For reference, Debian apparently made this change in Trixie, while Fedora seems to have had this since Fedora 18
https://fedoraproject.org/wiki/Features/tmp-on-tmpfs
https://linuxiac.com/debian-13-moves-to-tmpfs-for-tmp-directory/
3
u/linmanfu 19h ago
Thank you for pointing this out. In fairness, it is mentioned in the Release Notes, though in the systemd section (not where I'd look for a filesystem change), and without really explaining the most important consequence.
3
u/sgorf 14h ago
Here's the Ubuntu discussion related to this change: https://lists.ubuntu.com/archives/ubuntu-devel/2025-May/043401.html
tl;dr: it was for alignment with upstream systemd and Debian, and there wasn't a strong reason to diverge from the rest of the ecosystem.
2
u/tes_kitty 19h ago
I prefer /tmp in tmpfs. Self cleaning /tmp, nothing better.
But nothing keeps you from limiting the size of that filesystem. I have 'size=8g,nr_inodes=100k' as arguments for /tmp in my /etc/fstab.
That 'size' argument is the size limit, not a static assignment of 8 GB. If you have less in /tmp, it'll use only what is needed.
1
u/schnufi666 10h ago
What happens when the size of tmp is larger than the available ram? E.g. if you have 8 GB Ram, in fstab you write "size=20G"?
1
u/tes_kitty 6h ago
Never tried that since it would be pointless, it would have to use swap and end up on disk again. I usually limit /tmp to half the amount of real RAM I have. But that's for a desktop, on a server you might choose a different number or keep /tmp on disk.
2
u/Aggressive_Ad_5454 18h ago edited 13h ago
I’ve been working in the UNIX world since 1978. I’ve always understood that small files go in /tmp and larger ones in /var/tmp. But it seems that bit of lore hasn’t survived.
1
u/el_beef_chalupa 18h ago
That's the thing about Linux... There are no hard fast rules for anything, only convention. The convention of putting "small" files in /tmp is not a rule. There was no reason why you couldn't put any file in /tmp you wanted as long as you understood it can and will be deleted.
Is a 10MB zip file "small"? If unzipping it is much larger than that, then what? The only enforcement is to understand the files will be deleted from there. If I am fine with it, then there is no problem. This change has now went back to the convention of "keep small files in tmp but understand not only it will be deleted but it will also live in system ram until cleared"
1
u/alec_me 14h ago edited 14h ago
There are no hard fast rules for anything, only convention.
No, that's absurd. Of course there are standards: STANDARDS (7). And there are also best practices. Both those things go beyond mere convention.
Like RTFM is a best practice. RTFM about /tmp would provide the following information:
FILE-HIERARCHY (7)
/tmp/
The place for small temporary files. This directory is usually mounted as a "tmpfs"
instance, and should hence not be used for larger files. (Use /var/tmp/ for larger
files.) This directory is usually flushed at boot-up. Also, files that are not
accessed within a certain time may be automatically deleted.If applications find the environment variable $TMPDIR set, they should use the
directory specified in it instead of /tmp/ (see environ(7) and IEEE Std 1003.1[4] for
details).Since /tmp/ is accessible to other users of the system, it is essential that files and
subdirectories under this directory are only created with mkstemp(3), mkdtemp(3), and
similar calls. For more details, see Using /tmp/ and /var/tmp/ Safely[5].1
u/el_beef_chalupa 14h ago
Thanks, you really didn't add anything to the conversation. "/tmp is usually tmpfs" which Ubuntu has never been as far as I remember for the nearly 20 years I've used it.
Additionally standards are constantly broken. Usually programs should put config files in /etc/ but that's not a rule, merely a convention, all kinds of programs break that. User config files should belong in .local, .config .local/config what is the rule? There are no rules! Hahahaha
1
u/alec_me 13h ago
https://ubuntu.com/blog/data-driven-analysis-tmp-on-tmpfs
And it was enabled 24.10
2
u/el_beef_chalupa 13h ago
Thanks I read that. For production environments I (and probably nearly every other company using Ubuntu for production systems) only use the lts versions. I don't necessarily follow the changes to every intermediate release because it's not real important what happens in between. The goal of the post is to help other people like me prepare their systems before an update. I'm sure the pattern we followed of putting discardable files in /tmp now living in system memory will be a surprise to a few others too.
1
u/alec_me 7h ago
You work in a production environment? That's a laugh. With an attitude like, "There are no hard fast rules for anything..." and "There are no rules! Hahahaha"? I feel a little sorry for your H-1B replacement who's going to be tasked to have to clean up the shit-show anarchial mess of your "production environment" created by admins like you.
1
2
u/Uncle-Osteus 14h ago
back in the day /tmp was often tmpfs by default on many distros, and we encouraged each other to check the mount table on an unfamiliar system to make sure that reality aligns with your expectations, which was the style at the time
1
u/john2095 11h ago
Is the (new) /tmp a RAM disk?
A RAM-disk would use a fixed amount of RAM? Or least have a size ceiling?
So OOM can't be caused directly via the /tmp getting too big? The "disk" would report being full and writes report failing?
1
u/el_beef_chalupa 9h ago
Right, well mostly. So tmpfs sets a size ceiling and will consume up to x% of system ram as files are places in /tmp. OOM can be triggered when you have 4GB of RAM now eaten by temporary files and other processes that use 2GB of ram and a series of other smaller 100+MB processes on an 8GB server. So the other processes start hitting OOM. Not only that but writing to the tmpfs starts failing the writes once it hits x% which causes random processes who don't handle failed disk writes well to crash.
30
u/Ok-386 21h ago
Instead of changing scripts you could simply run:
systemctl mask tmp.mount
And restore the old behavior.