# BARAAA-70: Ofelia Container Restart Loop - RESOLVED ✅

**Date**: 2026-05-02  
**Server**: 192.168.9.23 (LAN)  
**Status**: ✅ DONE

## Problem

agenthub-ofelia-1 container was in continuous restart loop with error:
```
unable to start a empty scheduler
```

Ofelia scheduler was unable to find any scheduled jobs and crashed immediately.

## Root Cause Chain

1. **Backup container crash** → Permission denied writing to `/backups/` directory
2. **Backup container exits** → Has `restart: 'no'` policy, container stops after running
3. **Ofelia finds no jobs** → Was looking for labels on backup container, but container not running
4. **Ofelia crashes** → Cannot start with empty scheduler (no jobs found)
5. **Restart loop** → Docker restarts Ofelia, cycle repeats

## Fixes Applied

### 1. Fixed Backup Permissions
```bash
sudo mkdir -p /opt/agenthub/backups
sudo chmod 777 /opt/agenthub/backups
```

### 2. Relocated Ofelia Labels

**Problem**: Labels were on `backup` service which has `restart: 'no'` and exits after running.

**Solution**: Moved labels to `postgres` service which runs continuously.

**Modified**: `/opt/agenthub/compose.lan.yml`

```yaml
postgres:
  image: postgres:16-alpine
  environment:
    POSTGRES_DB: agenthub
    POSTGRES_USER: agenthub
    POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
  volumes:
    - pgdata:/var/lib/postgresql/data
  labels:
    ofelia.enabled: 'true'
    ofelia.job-exec.backup-daily.schedule: '0 0 3 * * *'
    ofelia.job-exec.backup-daily.container: 'agenthub-backup-1'
    ofelia.job-exec.backup-daily.command: '/usr/local/bin/backup.sh'
  restart: unless-stopped
```

### 3. Fixed YAML Syntax Issues

Multiple YAML syntax errors were introduced during manual editing:
- Incorrect indentation causing `services.restart must be a mapping`
- Empty `labels:` line in backup section
- Redis command in flow style instead of block style

All fixed via SSH access using programmatic file editing.

### 4. Restarted Services

```bash
docker compose -f compose.lan.yml up -d postgres
docker compose -f compose.lan.yml restart ofelia
```

## Verification Results

### Container Status
```bash
docker compose -f compose.lan.yml ps ofelia
```
**Result**: Container shows "Up" status (not "Restarting") ✅

### Restart Count
```bash
docker inspect agenthub-ofelia-1 --format '{{.State.Status}} - Restarts: {{.RestartCount}}'
```
**Result**: `running - Restarts: 0` ✅

### Ofelia Logs
```bash
docker logs agenthub-ofelia-1 --tail 20
```
**Result**: 
```
New job registered 'backup-daily' - '/usr/local/bin/backup.sh' - '0 0 3 * * *'
Starting scheduler with 1 jobs
```
✅ Job successfully registered and scheduler started

### Uptime Stability
Container maintained stable "Up" state for 27+ seconds after restart with zero restarts.

## Acceptance Criteria Met

- [x] Ofelia container in "Up" state (not "Restarting")
- [x] Scheduler starts successfully with registered job
- [x] Zero restart count after fix applied
- [x] Backup job registered with correct schedule (3am UTC daily)

## Next Verification

Monitor backup-daily job execution at **03:00 UTC on 2026-05-03** to confirm scheduled task runs successfully.

Expected: `/opt/agenthub/backups/` should contain new dump file after 3am execution.

## Files Modified

- `/opt/agenthub/compose.lan.yml` - Added Ofelia labels to postgres service, fixed YAML syntax
- `/opt/agenthub/backups/` - Created directory with correct permissions (777)

## Technical Notes

**Ofelia Job Discovery**: Ofelia scans **running** containers for labels. Jobs on containers with `restart: 'no'` that exit immediately are not discoverable.

**Solution Pattern**: For job-exec mode, place Ofelia labels on a service that runs continuously (like postgres, redis) rather than on the ephemeral service being executed.

**Alternative**: Use job-run mode instead of job-exec if you need to schedule one-shot containers, but this wasn't necessary here since backup.sh already existed in the backup service.