The job log said FATAL six weeks ago. The disk alert arrived on Tuesday.

The alert that finally arrives is about a disk. Storage on the archive server has passed ninety percent and somebody in infrastructure wants to know whether they should extend the volume. A week later, if nobody has, SAP stops being able to archive anything at all and the conversation changes considerably.

Neither of those is the actual event. The actual event happened six weeks earlier, on a Tuesday night, when the server was restarted for patching while a job was in the middle of clearing out a buffer. The job never finished. Archive Center noticed, wrote three lines beginning with the word FATAL into a job log, locked the buffer for safety, and then went on doing everything else perfectly well.

That is the shape of this problem. It is not a silent failure. It is a loud failure in a place nobody reads, and by the time it reaches somebody's screen it has been rewritten as a capacity question. The fix is three commands. The prevention is one line in a restart procedure.

Why a buffer has to be emptied at all

A document does not go straight from SAP to its final storage. It lands first in a buffer, which is local disk on the archive server, and then works through a short lifecycle: written onward to the real storage device, verified there, and only then removed from the buffer.

That last step is the point. The buffer is finite local disk and the final storage is the S3 bucket, the Azure container, the NetApp volume or the filesystem device that the storage contract is actually about. Purging is what keeps the staging area from becoming permanent residence, and it runs as a scheduled job called Purge_Buffer.

Two consequences follow, and the second one is the one people miss.

A buffer that is not purged fills up, and a full buffer means the archive cannot accept new documents. That is the disk alert, six weeks late.

But a document still sitting in the buffer is also a document that has not yet reached final storage, which means it is not covered by whatever protects final storage. No replication, no immutability, no retention lock, no cloud durability. Whatever the storage design promises, it does not apply to content that has not arrived yet.

A document that has not reached final storage is a document with one copy, whatever the storage contract says.

— a stuck buffer, found during a disaster recovery test rather than a disk alert

What "stuck" actually means

The job log is explicit, and it is worth reading closely because it contains more than it appears to:

FATAL: The following command did not terminate properly or is still running:
FATAL: --     dsHdskRm -i 100 -t 4 -x 0 -p <PoolName>
FATAL: It was started at 27.08.18 00:06:08
FATAL: After careful consideration, call dsHdskRm -clear or retry this command with -f option

Three things are in there. The pool, so you know which buffer. The exact command, so you know what was running. And the start time, which is the useful one, because it dates the incident precisely. A job that started at six minutes past midnight and is still described as running at four in the afternoon is not running.

The lock itself is deliberate rather than a defect. A purge that was interrupted partway through leaves the archive unable to say with certainty which documents were removed and which were not, and rather than guess, it stops. It also refuses to let dsHdskVerify run for that buffer again until a person resolves it.

That refusal is why the problem grows instead of sitting still. Verification is the step that confirms documents are safely on final storage, and purging is what happens after it. Lock one and you have stopped the whole chain for that buffer, so the backlog builds from the moment of the interruption rather than from the moment somebody notices.

Meanwhile the archive keeps serving documents normally. Retrievals work, users see nothing unusual, and the accounting file records an entirely healthy day. Nothing about the running system suggests anything is wrong, because from the perspective of anyone using it, nothing is.

Why it happens

Almost always because the server or the services were restarted while the job was running. Patch night, a hypervisor moving the machine, a cluster failover, a backup that quiesces the guest, somebody bouncing services after a configuration change at half past five.

None of those are mistakes. They are ordinary operations meeting a job that happened to be mid-purge, and the prevention is correspondingly ordinary: check that no jobs are running before a planned restart, and put that check in the runbook rather than in somebody's memory. It costs one command and it is the entire difference between a routine reboot and six weeks of accumulating backlog.

Clearing it, in order

The step people skip is the first one, and nothing else works without it. The command line tools need the Archive Center environment loaded before they will run at all. On Windows that is:

%ECM_ARCHIVE_SERVER%\bin\profile.bat

On Linux the equivalent profile has to be sourced first. If the commands below come back as not found, this is why, and it is not a sign that anything is broken.

Then check before you clear. This is the only judgement call in the whole procedure:

dsHdskRm -check

Read the start times in the output. A job that started hours ago is stuck and is no longer doing anything. A job that started a few minutes ago is genuinely working, and clearing it would interrupt a running purge, which is precisely the thing that created the problem in the first place. If anything is legitimately in progress, wait for it to finish before going further.

Then clear. This releases all currently running buffer purge jobs:

dsHdskRm -clear

Then start it again from the Administration Client, manually, rather than waiting for the next scheduled run. Watch the first one through to completion.

The error message also offers -f to force a retry of the original command. The documented route is -check, then -clear, then a clean restart from the Administration Client, and on a buffer that is the safe path.

The rest of the family

dsHdskRm is one of a set, and they follow the same pattern across the device types. The same three verbs, different tool per storage kind:

Job type	Size outstanding	Check pending	Clear pending
Single file, filesystem	dsHdsk -size	dsHdsk -check	dsHdsk -clear
ISO	dsCd -size	dsCd -check	dsCd -clear
Single file VI	dsGs -size	dsGs -check	dsGs -clear
Buffer purge		dsHdskRm -check	dsHdskRm -clear
Buffer verify		dsHdskVerify -check	dsHdskVerify -clear

Note that dsHdskVerify has its own -check and -clear, which matters because the verify step can be the one that is stuck rather than the purge. If dsHdskRm -check comes back clean and the buffer is still growing, that is where to look next.

The column that deserves more attention than it gets is the first one. -size reports how much work is outstanding, which is a number rather than a symptom, and it is available long before any disk gets uncomfortable. Run it on each device type once, write the answer down, and you have a baseline. Run it weekly and a backlog that is quietly building announces itself weeks ahead of the alert, in a form somebody can act on calmly.

---

There is nothing subtle happening here. A job was interrupted, the archive did the safe thing and stopped, and it wrote down exactly what it had done and when. Everything that followed was a consequence of nobody reading it.

Which makes the useful work small and boring, in the way that useful work usually is. Add a job check to the procedure that shuts down or restarts the archive server, so the cause stops recurring. Run -size against each of your device types this week and note what normal looks like, so a backlog is visible as a trend rather than as a disk alert. Neither takes an afternoon, and between them they turn a six week problem into a five minute one.