My server (fedora) stops all podman containers after 2-3 hours since 3 days. I can start all containers again, and the same happens after a while. I do not know where to look for the problem.
In top, I found a oom message. I assume that the system runs out of memory and stops all services. How can I find the problem? I can’t find anything in the container logs.
I can see that systemctl status is always starting. It doesn’t become “running”. But I do not know how to proceed.
You must log in or register to comment.
The issue with diagnosing memory issues is that it usually results in no memory available to handle the logging of such a problem when it happens.
I’ve found that the easieat approach is to set up a file as additional swap space, and swapon, then see if the problem disappears, either partially or fully.
I’ve got way too much RAM for swap being useful at all. Good idea though.
There is no such thing as too much RAM…
If something you’re running has a memory leak then it doesn’t matter how much RAM you have.
You can try adding memory limits to your containers to see if that limits the splash damage. That’s to say you would hopefully see only one container (the bad one) dying.
Then you didn’t understand how the system uses swap.
They could mean that they have swap but it’s not being used.
When I had the issue with mariadb demon been killed, I think either in dmesg or syslog there was an entry reading "Out of memory: Kill process… " or similar.
I’ll have a look, thx
If you saw an OOM anything, it’s getting OOMkill’d by the kernel trying to keep the machine up. Check syslogs and dmesg, and it should say what was killed, and there’s your problem container. You probably have a memory leak, so just check your container stats every so often and see what is growing out of control with memory usage.
Enable swap regardless. Would also help to know what you’re running.
Are you running them from your user session? If so, when you log out it will stop your processes, unless you have enabled ‘linger’ mode.
I would start all containers except one. If everything works that one is the cause of the problem. Keep trying with a different container every time.
