Nobody reports a broken agent. They report an email that never arrived.

Nobody has ever raised a ticket saying that the Event Processor is not loaded on any instance in the cluster. What arrives instead is a message from somebody in finance who never received the notification about a document she was supposed to approve, three weeks ago, and who is only mentioning it now because it has happened again.

Between those two sentences sits the part of Content Server that nobody sees. A set of background workers, each identified by a number, quietly doing the things everybody assumes the software does by itself: sending notifications, expiring passwords, emptying the recycle bin, moving content between storage locations, running disposition searches, importing whatever arrived overnight. They are called agents, and when one of them stops, nothing breaks. Work simply stops happening, silently, until a person notices its absence.

That silence is the point of this article. Almost every mystery of this kind comes down to the same thing: an agent was switched on in one place and not loaded in another, so it runs nowhere at all, and nothing anywhere reports it.

The part of Content Server nobody sees

An agent is a scheduled job that runs inside the Content Server process rather than in the browser. Each has a numeric identifier, an AgentID, and the system carries a short description of every one of them in the AgentSchedule table, which is the closest thing to an inventory that exists.

In a clustered environment, agents are not meant to run everywhere. The normal arrangement is that exactly one instance runs them and every other instance does not. That instance is usually a back-end machine, one carrying the search processes rather than serving users, because agents want processor time and memory and there is no reason to take either away from somebody's screen.

The rule holds in the other direction too, and more sharply. Running the same agents, notifications or relagent on several instances at once is not a redundancy strategy. It is a way to spike processor use on the database and degrade the whole cluster, because several machines end up competing over the same queues.

Two settings, and both have to agree

Whether an agent runs is decided in two places in opentext.ini, and this is the single most useful thing to understand about the whole subject.

Setting	Section	What it decides
`load`	`[loader]`	Which threads this machine starts at all
`AgentID=1` or `0`	`[scheduleactivity]`	Which individual agents are switched on

They are not alternatives. They are both required, and this is where the silence comes from. An agent set to 1 in [scheduleactivity] will not run unless the thread that carries it appears in [loader]. The Event Processor, 9000, is the classic example: set it to 1 all you like, and if notify is missing from the load line it never runs, and nobody is told.

An instance designated to run agents carries a long loader line:

[loader]
load=sockserv;agents;notify;wfagent;wrscheduleagent;wrcollectionagent;commagent
load_relagent=relagent
load_distributedagent=distributedagent
load_daagent=daagent
load_verify=verifyAgent

Every other instance in the cluster carries a short one, and the difference between the two files is the entire configuration:

[loader]
load=sockserv;notify

Underneath, the individual switches, one line per agent:

[scheduleactivity]
1000=1
4000=1
4100=0
4101=0
4102=0
8900=1
8999=1
9000=1
9001=1
3502=1
8000=1
3601=1

Before touching any of this: take a copy of opentext.ini and stop the Content Server service on the instance you are editing. The file is read at startup, and editing it under a running service is how people lose an afternoon.

Three switches that bite quietly

EnableAgents should stay TRUE on every instance, including the ones running no agents at all. It reads like a master switch and it is not one. The eLink module depends on it everywhere. The same applies to EnableNotification if notifications are in use, because that setting is what puts the Notification option in each user's personal menu. Turn it off on a front-end server and you have not saved any work, you have removed a menu item for everyone using that server.

javaserver is the opposite trap. Older documentation shows it in the load line, and on newer versions its presence can stop Content Server from starting altogether. The safe rule is to leave it exactly as the installer left it. If a default installation did not put it there, do not add it back because an older article said so.

And 4100, 4101 and 4102, the three password agents, are set to 0 by default. That is deliberate, not an oversight, but it does mean password expiry does nothing at all until somebody enables them on the Content Server Administration page. Setting them to 1 in the INI file alone is not enough.

The agents, grouped by what people notice

The full list is worth having somewhere findable, so here it is, arranged by the kind of complaint each group produces rather than by number.

Notifications, events and email. This group is a chain, and that matters more than any individual entry.

ID	Agent	What it does
`1000`	Late event producer	Runs at midnight, raising events for late tasks and workflows and for closing polls
`1290`	Tempo Notification Digest	Builds the digest notifications
`4000`	ExpireMessages	Clears expired messages out of `NotifyMessages`
`4001`	ExpireHandlerEvents	Clears expired handler events out of `LLEventQueue`
`5000`	eLink inbound	Handles incoming email
`5001`	eLink outbound	Handles outgoing email
`8999`	Clear Notify Events	Moves rows from `notifyevents` into `LLEventQueue`
`9000`	Generic Event Processor	Processes events by running the node processors, `5001` and `9001`
`9001`	Node Event Processor	Transfers `LLEventQueue` entries into `NotifyMessages`
`9002`	Reserved node event processors	Held in reserve
`9999`	Internal User Agent	Hands the finished messages to the SMTP server

People, passwords and identity.

ID	Agent	What it does
`2000`–`2500`	DirectorySynchronize	Directory Services synchronisation
`2501`–`2504`	eSign	Electronic signature processing
`4100`	ExpirePwds	Disables accounts whose passwords have expired
`4101`	MonitorPwdExpiration	Warns users before their password expires
`4102`	FailedLoginNotification	Tells the administrator when failed logins spike
`8100`	FailedLoginProcessor	Produces failed login events for auditing
`9120`	QuotaManagementAgent	Emails users who pass their quota or threshold

Content arriving and moving.

ID	Agent	What it does
`1964`	SpcJobAgent	Content Move, also known as spchange
`2990`–`2999`	Object Importer	The importer itself, one agent per slot
`3000`	Object Importer Consumer	Consumes what the importer produces
`3001`	Importer Process	Prospector
`3501`	Rendition Agent	Works through the rendition queue
`3504`	ProviderBlobDeleteRetry	Retries blob deletions that failed earlier
`3601`	Pstage ItemAdded	Handles items added to the WebDAV personal staging folder
`20296`	Brava Publish	Publishes for the Brava viewer

Records management and compliance.

ID	Agent	What it does
`9051`	RCServerGuard	Superseded, no longer used
`9052`	RCQueueInsert	Clears old rows from `RCQueue` and adds the ones now due
`9101`	DispositionSearchEvents	Checks for scheduled disposition searches
`9102`	Search Enabled Holds	Holds processing for records management
`9103`	Security Clearance Defined Rules	Applies clearance rules
`20055`	CLM resubmission	Resubmission for Contract Lifecycle Management
`31040`	Internal Controls snapshot	Refreshes the Internal Controls Management report snapshot

Housekeeping and the rest.

ID	Agent	What it does
`3502`	SOVSystemManagement	Records the condition of system objects and reports it to the alert
`3503`	OnTime	Scheduling support
`5858`	Recommender gather agent	Collects the data behind recommendations
`6969`	Search Stats	Purges search statistics automatically
`8000`	UndeletePurgeAgent	Purges undeleted documents
`8010`	Recycle bin purge	Empties the recycle bin
`8900`	FiveMinuteAgent	The five minute cycle that carries other work
`20200`	WebReports Collection	Runs WebReports collections
`20201`	WebReports Scheduling	Runs scheduled WebReports

Four questions this settles

Why did nobody get the notification?

Because a notification is not one agent, it is four in sequence. An event is captured, then 8999 moves it into the event queue, then 9000 processes it and 9001 turns it into a message, and only then does 9999 hand that message to the mail server. Every one of those steps has to be enabled, and the thread that carries it has to be loaded on the same machine.

Break any single link and the outcome is identical from the outside: no email, no error, no entry anywhere saying a message was due. The work waits in a queue table nobody looks at. This is why the failure is usually discovered by a person rather than by monitoring, and why it has typically been running for weeks by then.

Nothing here reports an agent that never ran. The absence of work looks exactly like the absence of work to do.
— three weeks of missing approvals, found by accident

Why is the overnight import still running at nine in the morning?

Because by default one instance is doing all of it. The Object Importer occupies a range of agent IDs, 2990 through 2999, and that range exists precisely so the load can be split across machines.

Enable agents in [loader] on a second instance, then give each machine a different importer from the range: 2990=1 on the first and 2991=0, and the mirror image on the second. The two now work in parallel on the same import, and the window shrinks. It is one of the few Content Server performance problems with a genuinely simple answer, and it costs nothing but a restart.

Why did the recycle bin never empty, and why is the database growing?

Because purging is an agent too. 8010 empties the recycle bin, 8000 purges undeleted documents and 6969 clears down search statistics. None of them announce themselves, and a cluster that was rebuilt or migrated without carrying the [scheduleactivity] section forward will look completely healthy while quietly retaining everything anyone has ever deleted.

This is the group worth auditing first after any upgrade or platform move, because the symptom is a storage bill rather than an error, and it takes months to become visible.

Why did password expiry never happen?

Because 4100, 4101 and 4102 ship disabled and are enabled from the Administration page rather than the INI file. A security policy that says passwords expire after ninety days is, until somebody does that, a document rather than a behaviour.

When one agent server is not enough

The single agent server is the normal arrangement, not a rule. A specific agent can be given a machine of its own, which is worth doing when it is heavy enough to interfere with everything else.

The simplest form is one instance that loads nothing but the agent you want to isolate:

[loader]
load_verify=verifyAgent

Another instance carries the general load, and every remaining instance carries only sockserv. The more interesting form is moving one agent onto a different thread, which is how Content Move gets separated out. Two settings do the work, on two different machines. On the general agent server, 1964 is excluded so it does not run there:

[agents]
ExcludeActivityIDs=3000,3501,5000,8999,9000,9001,9999,12568,20201,20200,1964

On the specialised instance, the notify thread is told to run that one agent and nothing else:

[notify]
SleepIntervalSec=300
ActivityIDs=1964

Which means that in a mature cluster the answer to does this agent run here comes from four places at once: whether it is enabled in AgentSchedule in the database, whether it is set to 1 in [scheduleactivity], whether it appears in ExcludeActivityIDs under [agents], and whether it is listed in ActivityIDs under [notify]. Any one of them can veto. None of them will tell you they did.

There is no report that lists the agents which did not run last night. Archive Center at least writes a line for every request it answers, which is a separate story and a more forthcoming one. Here, silence is the default, and it looks precisely like a system with nothing to do.

So the useful exercise is not a project. Take the [loader] and [scheduleactivity] sections from every instance in your cluster, put them side by side, and check that each agent you depend on is switched on somewhere and loaded on that same machine. Most of the time the list is fine. The value is in the one or two lines where it is not, and in knowing that before somebody in finance mentions an email she never received.