Case study: one agent install, start to finish
Case study: one agent install, start to finish
The brief
A small marketing agency wanted an assistant to carry its admin. Read the shared inbox, keep the contact records, remember what it learns, run on a schedule without being asked. They had one condition: nothing leaves their hardware.
What we found
They already ran a three node Proxmox cluster on site with about thirty containers on it, and it had room for four more. The local LVM pool showed 349 GB free. The shared NVMe pool held 4.9 TB at 44 percent used. A Debian 13 template was already downloaded.
The cluster ran Proxmox VE 9.2.21 on kernel 6.17. Debian 13 containers cannot be created on Proxmox 8 at all, so the same install on a cluster one release back would have failed on its first command.
A root password had been shared in a plain text file on a laptop. A public hostname pointed at one of their machines with nothing standing in front of it. Nothing on that network answered from outside.
The shape of the build
Four services, one container each.
| Service | What it holds | vCPU | RAM | Disk |
|---|---|---|---|---|
| Agent | Chat, routines, logs | 2 | 4 GB | 20 GB |
| Board | Work queue and handovers | 4 | 8 GB | 20 GB |
| CRM | Contacts and companies | 4 | 10 GB | 20 GB |
| Memory | What the agent has learned | 1 | 4 GB | 10 GB |
| Total | 11 | 26 GB | 70 GB |
There is no graphics card in any of the four. The agent calls an API model instead of running one on the box, so the only running cost is that call, and it is metered.
How we reached the hardware
Their network is private. The work ran from a throwaway container of ours that joined their tailnet for the job, and we never needed root on their nodes.
Every change to the agent went in through her own dashboard API, so nothing on their hosts was hand-edited. That container was destroyed when the job finished.
What went wrong
Nine faults, all nine in the install runbook. Four of them:
- The board lost its own task list. The command the agent used to read her assigned work called a route that returned 404. The route that worked sat under the company and assignee identifiers. Her task list came back with three open items.
- A stale record was holding her name. An agent row left over from an earlier onboarding still owned the name, and the API refused every attempt to rename it. We archived the row and changed the name in the database, then checked that no other row carried it.
- The mailbox refused to authenticate. Google no longer accepts an account password from a mail client and demands an app password generated inside the account. What we had been handed was an account password.
- Configuration arrived and nothing read it. The mail platform only reads its settings at gateway start, so the change looked successful and did nothing. Its log shows the restart in seconds.
What we proved before we left
- A real message arrived and was answered twice from the agent's own address. The gateway log carries the send line.
- All four services answered from inside the customer's network: dashboard, board, CRM and memory.
- One routine fired with nobody watching. The board held the run row, marked completed, and the agent wrote eight durable facts into memory. The graph went from three entries to eleven.
- Ten skills were in place, counted by listing the directory on the box, after the first push silently missed one.
- Five routines were live, with real triggers, in the customer's timezone.
- A dated archive is written nightly. Fourteen days of archives are kept and the oldest rolls off. Nothing the customer put on the box is deleted by it.
Still open at handover
- The mailbox was assigned and not connected, waiting on access to the customer's own Google account.
- No key to the customer database had been issued, so the agent could not file a contact. She reported that herself in her end of day note.
- Two stale records sat on the board. Removing them is the customer's call.
- The addresses were handed out by DHCP, so a reboot could move them. Pinning them needs a person at the router.
- The CRM configuration still pointed at the temporary doorway we were about to remove, so links it sent would not resolve until that line changed.
The exit
- Every script of ours came off their nodes and containers, archived before removal: 536 files in seven archives, kept on our side.
- The jump container was stopped, its services disabled, the container destroyed. The address it published answers nothing.
- Credentials were shredded on both sides, including staged copies that had held a dashboard password inside an email body.
- Their install was probed healthy immediately before teardown and left running.
Their cluster, their logins, and nothing of ours left on it.
What we would change next time
- Ask for the right credential in week one. The mailbox cost a full pass because an account password arrived where an app password was needed.
- Probe from the far side. Every address that matters was tested from inside their network. Two faults showed up nowhere else.
- Settle public exposure before the build starts. A machine that can run terminal commands has no business on a public hostname by itself.
- Log open items the day they appear. At handover that page is the one the customer actually reads.
The same install without the commands is in the walkthrough. For one on your own hardware, email [email protected].