BackAR 
My disk hit 100% full in production. Postgres crashed. But my monitor lost zero customer check results, retries caught every failed write.
Then I found the cause. ClickHouse was quietly writing gigabytes of logs about itself, mostly system.text_log and trace_log. Logs I never asked for.
The fix: truncate the log tables (12 GB back), then a config file that caps them for good. I also added a disk alert at 80%, so next time I get days of warning, not minutes.
Full postmortem and the copy-paste config:
Uptimepage
ClickHouse system tables ate my disk (and the fix)
My disk filled and I started dropping writes. Real data: 20 MB; ClickHouse had logged 12 GB about itself and burned idle CPU doing it. The config that fixes both.
Sign in to reply.