Thanks, I appreciate your answers. I also found tmpwatch utility for removing stale files: https://linux.die.net/man/8/tmpwatch

Regards, Vitaliy

пт, 2 июля 2021 г., 1:08 Jeremy Stanley <fungi@yuggoth.org>:
On 2021-07-01 15:00:02 -0700 (-0700), Clark Boylan wrote:
[...]
> Before OpenDev uploaded logs to swift we had our CI system copy
> logs using scp/rsync to a log server with 12TB of disk. On that
> server we ran cron jobs to run find and expire any files older
> than our desired expiration date. I think this,
> https://opendev.org/opendev/puppet-openstackci/src/branch/master/files/log_archive_maintenance.sh,
> is the script we were using at the time. It is fairly complicated
> and probably worth understanding what it is doing before you try
> running it since it can be destructive. It is also possible you
> can simplify it further. Note that it also tries to compress
> recent files in addition to deleting old files.
>
> The major downside to this approach is that you have to stay on
> top of your disk usage and keep track of whether or not the
> cleanups are producing enough headroom for new logs otherwise your
> test jobs will all start failing and then you have to fix things
> really quickly.
[...]

Also if you've got too many files and too little CPU or I/O
bandwidth on the system hosting it, you can get in a situation where
it takes longer to iterate over all the files than the expired files
accumulate, at which point your filesystem utilization grows without
bound until you run out of space.

One solution I was considering back then was rotating logical
volumes in a Linux LVM2 volume group, so have a new volume for each
day and then just drop each old volume as it exceeds your expiration
and create and mount a new empty volume formatted with a fresh
filesystem to replace it. This however requires that you shard your
file path by date, which may take some effort with the existing log
publication roles. It also probably means a bit of additional wasted
block storage unless you use something like thin-provisioned
volumes.
--
Jeremy Stanley
_______________________________________________
Zuul-discuss mailing list
Zuul-discuss@lists.zuul-ci.org
http://lists.zuul-ci.org/cgi-bin/mailman/listinfo/zuul-discuss