# Preventing Full Disk Issues

Running out of disk space in a ClickHouse environment can cause query failures, part merge errors, and even full service downtime. ClickHouse is highly dependent on disk for storing columnar data, part files, metadata, temporary sort buffers, and backups. On platforms like Elestio, infrastructure is managed, but users are still responsible for monitoring storage, managing data retention, and optimizing resource usage. This guide explains how to monitor and clean up disk usage, configure safe retention policies, and implement long-term strategies to prevent full disk scenarios in ClickHouse when running under Docker Compose

### **Monitoring Disk Usage**

#### **Inspect the host system storage**

Run this on the host machine to check which mount point is filling up:

```
df -h
```

This shows usage across all mounted volumes. Look for the mount used by your ClickHouse volume—usually mapped to something like <span class="s1">/var/lib/docker/volumes/clickhouse\_data/\_data</span>.

#### **Check disk usage from inside the container**

Enter the ClickHouse container shell:

```
docker-compose exec clickhouse bash
```

Inside, check total ClickHouse disk usage:

```
du -sh /var/lib/clickhouse
```

To inspect usage of specific folders like <span class="s1">data/</span>, <span class="s1">tmp/</span>, or <span class="s1">store/</span>:

```
ls -lh /var/lib/clickhouse
```

## **Configuring Alerts and Cleaning Up Storage**

#### **Inspect Docker’s storage usage**

On the host, check space used by containers, images, volumes:

```
docker system df
```

#### **Identify and remove unused Docker volumes**

List all Docker volumes:

```
docker volume ls
```

Remove unused volumes (only if you’re sure they’re not needed):

```
docker volume rm <volume-name>
```

> <span class="s1">**Warning:**</span> Never delete your active ClickHouse data volume unless you’ve backed it up.

#### **Drop data manually using SQL**

To free space by removing outdated partitions or tables:

```
ALTER TABLE logs DROP PARTITION '2024-01';
TRUNCATE TABLE temp_events;
```

#### **Clean up local backups**

If you’re storing backups under <span class="s2">/var/lib/clickhouse/backup</span>, list and delete old ones:

```
ls -lh /var/lib/clickhouse/backup
rm -rf /var/lib/clickhouse/backup/backup-<timestamp>
```

Ensure important backups are offloaded before removing.

### **Managing Temporary Files**

#### **Monitor temporary file usage**

Check the temp directory inside the container:

```
du -sh /var/lib/clickhouse/tmp
```

Old files may remain if queries or merges crashed. Clean up when the system is idle.

#### **Redirect temporary paths to persistent storage**

Modify the <span class="s2">tmp\_path</span> in <span class="s2">config.xml</span> to use a volume-backed directory:

```
<tmp_path>/var/lib/clickhouse/tmp/</tmp_path>
```

Restart the container after applying changes.

### **Best Practices for Disk Space Management**

- <span class="s1">**Avoid storing binary blobs:**</span> Do not store large files like PDFs or images in ClickHouse. Use external object storage and only store references.
- <span class="s1">**Use TTL to expire old data:**</span> Automatically delete old data based on timestamps:

```
ALTER TABLE logs MODIFY TTL created_at + INTERVAL 90 DAY;
```

- <span class="s1">**Drop old partitions regularly:**</span> If partitioned by month/day, remove outdated partitions:

```
ALTER TABLE logs DROP PARTITION '2023-12';
```

- <span class="s1">**Enable efficient compression:**</span> Use ZSTD for better compression and lower disk usage:

```
CREATE TABLE logs (...) ENGINE = MergeTree() SETTINGS compression = 'ZSTD';
```

- <span class="s1">**Split large inserts into smaller batches:**</span> Avoid memory and disk spikes during large ingest operations.
- <span class="s1">**Optimize background merge load:**</span> Tune merge concurrency and thresholds using:

```
<background_pool_size>8</background_pool_size>
```

- <span class="s1">**Limit disk spill during queries:**</span> Prevent massive temp usage during large operations:

```
<max_bytes_before_external_sort>500000000</max_bytes_before_external_sort>
```

- <span class="s1">**Rotate Docker logs:**</span> Prevent logs from filling up your disk using log rotation:

```yaml
logging:
  driver: "json-file"
  options:
    max-size: "10m"
    max-file: "3"
```

- <span class="s1">**Monitor disk usage from ClickHouse itself:**</span> Track table-level disk usage using system tables:

```
SELECT table, sum(bytes_on_disk) AS size FROM system.parts GROUP BY table ORDER BY size DESC;
```

- <span class="s1">**Offload backups to remote storage:**</span> Backups inside containers should be copied off-host. Use Elestio’s backup tool or mount a backup volume:

```
volumes:
  - /mnt/backups:/backups
```