Hi,
Our environment is cluster of 3 nodes and each node has 1 job server. Luckily there is maximum concurrent jobs set to 5 on each job server, this morning suddenly all jobs started on job server on node1 went to permanent running mode and all subsequent job requests were run on other 2 nodes job servers.
I couldn't able to find what happened in the server application log! I don't see any errors leading to this problem. Everything is normal and jobs were run successful on the same job server, But I see lot of errors on the same server after jobs went to permanent running state for "crproc" error as seen below
A failure occurred while the server was processing report 'REPORT NAME' (id=159605) for user 5109497 (RCIRAS0567)
and few crcahe Timeout. (RCIRAS0244) errors.
Attached, snapshot of application log. Highlighted jobs are the jobs went to permanent running mode.
Can you please help us, where could I able to find possible cause?
Thanks!