Troubleshooting a self-hosted install
This page covers failure modes with a documented cause in the repo — real incidents that shaped a memory cap or a config default, not speculation. If a container is unhealthy and it isn't listed here, start with docker compose logs <service>.
deckgauge-worker Restarting (137) — killed at its memory cap
deckgauge-clickhouse Restarting (1) — broken parts after an unclean shutdown
Two of the crash-loop patterns this page explains.
Container crash-loops
Every service in docker-compose.yml carries a hard memory limit; several were raised in response to a specific incident, which means going back under them still reproduces the same failure.
| Symptom | Cause | Fix |
|---|---|---|
The worker restarts every 20–30 minutes; its RestartCount keeps climbing | It hit its memory cap mid-sync. A large Azure DevOps source (thousands of work items, dual-written to ClickHouse and promoted batch by batch) can peak well past the container's limit | The cap is already 768M for this reason; if you've raised the number of large synced projects further, give the container more memory headroom in docker-compose.yml |
| Each worker kill restarts the sync from the top of its project list, and the same tail projects never finish syncing | The ADO sync processor runs projects sequentially — a kill mid-list starves whatever comes after the one that was running | Same fix as above: keep the container comfortably under its cap so a sync pass completes uninterrupted |
| The API container OOMs with "JS heap out of memory" for orgs with tens of thousands of work items | The timesheet engine runs two full passes per report (a monthly CapEx view and a year-window epic breakdown), and the annual pass allocates roughly 12x the monthly one | The container's Node heap is already pinned below its cgroup cap (--max-old-space-size) so it GCs instead of getting killed outright; very large orgs may still need the memory limit raised further |
| Redis stops accepting writes | BullMQ queue state outgrew an earlier, smaller memory cap; the background save (BGSAVE) fork was OOM-killed, and stop-writes-on-bgsave-error then blocked all writes | Redis is capped well below its container's memory limit specifically to leave room for the BGSAVE fork's copy-on-write pages — don't raise maxmemory close to the container cap |
| ClickHouse won't start after an unclean shutdown (e.g. the host or VM crashed) | A table has more broken/truncated parts than max_suspicious_broken_parts allows, so ClickHouse refuses to boot rather than risk touching them | The shipped config already raises this cap; a table hit by this moves its broken parts to detached/ and starts anyway — data for that table (e.g. ado_transitions) re-fills on the next full sync |
ClickHouse returns Code 241 MEMORY_LIMIT_EXCEEDED during a large sync (thousands of rows in one insert) | The server-side memory cap is a fixed byte value, tuned for the container's 3G limit | The shipped cap already accounts for large inserts (raised from 800MiB to 2.5GiB after exactly this); if you tune clickhouse/config/memory.xml further, keep the cap safely under the Docker container limit |
ClickHouse refuses to start with Code 36 BAD_ARGUMENTS mentioning background_pool_size | The pool was shrunk below the minimum a mutation needs to execute | Don't lower background_pool_size below the default — the note explaining why lives in clickhouse/config/memory.xml (not merge_tree.xml, which only overrides max_suspicious_broken_parts); the memory it would save is negligible next to that file's cache-size caps |
Migrations
| Symptom | Cause | Fix |
|---|---|---|
pnpm --filter @deckgauge/db migrate:dev fails with a shadow-database error (P3006) | migrate:dev isn't supported in this repo | Use migrate:deploy; for a new migration, hand-write the SQL and apply it the same way |
migrate:deploy fails to connect | It ran before Postgres was up and healthy | Run docker compose up -d first, confirm postgres shows healthy in docker compose ps, then retry |
A source failing its connection test
Testing a connection (Jira, GitHub, GitLab, or Azure DevOps) calls the provider with the stored credential before saving it. A failure returns a structured error and a hint rather than a bare status code — the UI surfaces both as a badge and a reconnect prompt instead of a silent failure.
| Symptom | Cause | Fix |
|---|---|---|
| Connection test fails immediately on save | The credential is wrong, expired, or scoped too narrowly for the check the test performs | Read the returned hint — it names the specific problem — then re-enter the credential and test again |
| A previously working source starts showing a health warning | Its token expired or was revoked upstream | Reconnect from the board's Sources tab; see Connection health |
Related
- Services & architecture — every container's role and memory limit.
- Backup & restore — recovering data after a container failure.
- Connection health — reading the health badge and reconnecting a source.
Last updated