Preventing Full Disk Issues

Running out of disk space in a ClickHouse environment can cause query failures, part merge errors, and even full service downtime. ClickHouse is highly dependent on disk for storing columnar data, part files, metadata, temporary sort buffers, and backups. On platforms like Elestio, infrastructure is managed, but users are still responsible for monitoring storage, managing data retention, and optimizing resource usage. This guide explains how to monitor and clean up disk usage, configure safe retention policies, and implement long-term strategies to prevent full disk scenarios in ClickHouse when running under Docker Compose

Monitoring Disk Usage

Inspect the host system storage

Run this on the host machine to check which mount point is filling up:

df -h

This shows usage across all mounted volumes. Look for the mount used by your ClickHouse volume—usually mapped to something like /var/lib/docker/volumes/clickhouse_data/_data.

Check disk usage from inside the container

Enter the ClickHouse container shell:

docker-compose exec clickhouse bash

Inside, check total ClickHouse disk usage:

du -sh /var/lib/clickhouse

To inspect usage of specific folders like data/, tmp/, or store/:

ls -lh /var/lib/clickhouse

Configuring Alerts and Cleaning Up Storage

Inspect Docker’s storage usage

On the host, check space used by containers, images, volumes:

docker system df

Identify and remove unused Docker volumes

List all Docker volumes:

docker volume ls

Remove unused volumes (only if you’re sure they’re not needed):

docker volume rm <volume-name>
Warning: Never delete your active ClickHouse data volume unless you’ve backed it up.

Drop data manually using SQL

To free space by removing outdated partitions or tables:

ALTER TABLE logs DROP PARTITION '2024-01';
TRUNCATE TABLE temp_events;

Clean up local backups

If you’re storing backups under /var/lib/clickhouse/backup, list and delete old ones:

ls -lh /var/lib/clickhouse/backup
rm -rf /var/lib/clickhouse/backup/backup-<timestamp>

Ensure important backups are offloaded before removing.

Managing Temporary Files

Monitor temporary file usage

Check the temp directory inside the container:

du -sh /var/lib/clickhouse/tmp

Old files may remain if queries or merges crashed. Clean up when the system is idle.

Redirect temporary paths to persistent storage

Modify the tmp_path in config.xml to use a volume-backed directory:

<tmp_path>/var/lib/clickhouse/tmp/</tmp_path>

Restart the container after applying changes.

Best Practices for Disk Space Management

ALTER TABLE logs MODIFY TTL created_at + INTERVAL 90 DAY;
ALTER TABLE logs DROP PARTITION '2023-12';
CREATE TABLE logs (...) ENGINE = MergeTree() SETTINGS compression = 'ZSTD';
<background_pool_size>8</background_pool_size>
<max_bytes_before_external_sort>500000000</max_bytes_before_external_sort>
logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
SELECT table, sum(bytes_on_disk) AS size FROM system.parts GROUP BY table ORDER BY size DESC;
volumes:
  - /mnt/backups:/backups

Revision #1
Created 11 June 2025 10:04:24 by kaiwalya
Updated 11 June 2025 11:12:29 by kaiwalya