Memory Leaks in DM Service (DOCSFusion.exe)

It was the OpenText Technical Alert message sent out November 4, 2013 that drew my attention to memory leaks in DM Server:
In OpenText Document Management, eDOCS Edition (eDOCS DM) 5.3.1, with .NET Framework 4.0 installed on the client and server machines, the DM Server may stop responding. The tracking number for this issue is DM-31701.

The alert suggested installing .NET 4.5 as soon as possible to fix memory leaks.

I searched for known memory leaks in .NET 4.0, but couldn't find reports on any major issues. I then asked OpenText which specifically known issue in .NET 4.0 negatively impacts DM, but their answer was: "We tested with .NET 4.5 and couldn't detect memory leaks. You should install .NET 4.5 as recommended." Uh...

So I installed .NET Framework 4.5.1 on DM servers and started monitoring the memory of the DOCSFusion.exe process. It was soon noticed that memory consumption of DOCSFusion.exe is steadily increasing over time. Here is what I see now on a server:

This particular instance of DOCSFusion.exe was running for 20 days now and it climbed up to using 550 Mb of memory.

The issue was brought to OpenText Support Team's attention. After months of exchanging emails with OpenText Support and sending a ton of server and memory monitoring logs (OpenText has a special tool for this - WatchMemory.exe), I gave up on them admitting and fixing the issue.

This issue may be not critical in your environment, still you may want to keep your eye on it while monitoring your DM servers. The method I use for DOCSFusion.exe remote monitoring is checking the value of Process.WorkingSet64. Below is some C# code.

using System.Diagnostics;
...
public long MaxMemLimit = 800 * 1024 * 1024; // 800Mb
...
var procs = Process.GetProcessesByName(ProcessName, MachineName);
if(procs == null || procs.Length == 0)
    return new TestResult(false, null, subj,
        string.Format("Process {0} was not found on {1}.", ProcessName, MachineName), null);

var mem = procs.Max(p => p.WorkingSet64);
var isTestPassed = mem <= MaxMemLimit;

OpenText eDOCS DM Server Issues Every Administrator Must Be Aware Of

If everything seems to be going well, 

you have obviously overlooked something.

Basic Murphy's Laws

There are quite a few things an eDOCS DM administrator must be aware of and keep them under strict control to avoid the surprise of end-users calling and telling that DM isn't working for them. Here are my favorite.

Memory leaks in the DM service process
DOCSFusion.exe is leaking memory. The memory consumption steadily grows until it becomes critical, at which point you'd better restart the DM service than let your entire server to go down.

Database connection leaks
The DOCSFusion process manages a pool of database connections. Once in a while the pool manager goes mad and stops releasing database connections. Soon after that DM becomes extremely slow to end-users and can even crash.

Indexer sticks and stops progressing
As the subject says, the indexing appears to be working, but in fact it's not: all services are running and not reporting any errors, however, the index database files are not being updated. You have to restart the DM services on the server to give the indexer a kick to keep going.

These issues I observe with the most recent version of OpenText eDOCS DM: v5.3.1 Patch 5. I shall write separate blog posts on each of them to describe the "early-warning" detection methods I use in hope it would help other DM administrators...

DM Server Log Parser has been updated (v.1.0.4)

I've just updated the EDocsLogParser tool. Now it can process Hummingbird DM 5.2.1 server logs in addition to OpenText eDOCS DM 5.3.1 supported from the initial version. Additionally, a few issues were fixed, and I've made some minor cosmetic tweaks.

Binaries: EDocsLogParser v1.0.4 x64/x86 for Windows
Source: https://github.com/dotnetnick/edocs/
User Guide: EDocsLogParser Usage

And let's not forget why we parse DM Server logs:
How to Find Slow SQL Queries in eDOCS DM Server Logs
How to Determine the Right Size for SQL Connection Cache
These are just examples. You can also think out other applications for the data from the logs.

If you process huge logs (greater than 1.5 GB), you should use x64 version of EDocsLogParser.exe. For smaller log files it doesn't matter: both 32- and 64-bit versions should work equally well. The tool consumes around the same amount of memory as the size of the log file. If you have 4 GB of RAM on your system, it should be fine for processing logs up to 2 GB, though the more memory your system has, the better.

Any feedback is welcome. Thank you!

DM Server Log Parser Update

EDocsLogParser v1.0.3, a new version, has been uploaded to Drive and GitHub. I tuned up performance a bit and fixed OutOfMemoryException which occurred on some large (~1 GB) log files. The parser was tested on logs larger than 1.3 GB and it worked fine.

While working on these updates, I noticed that DM Server logs become messy when the load on the server increases: messages from different calls are often mixed on same lines. I did my best to resolve this mess in the log parser, but it couldn't be resolved completely. I added new unit tests with snippets from real logs, so you can see a part of that "having fun".

If I wrote the parser from the beginning now, I would based it on loading a whole log file as a single string and not breaking it by lines as I've implemented it. I give it up: I find the OpenText's product being buggy and a bit badly designed to spend too much time and passion on it. With this said, I'm quite happy with EDocsLogParser as it is: it extracts 99.99% of all SQL events out of a messy 1 GB log and this is more than enough for the tasks I intended it for:
How to Find Slow SQL Queries in eDOCS DM Server Logs
SQL Connection Cache Size for OpenText eDOCS DM Server

SQL Connection Cache Size for OpenText eDOCS DM Server

The SQL Connection Cache Size value is one of the DM Server configuration values. You can set it in DM Server Manager on Libraries / Library Properties page.

How to Find Slow SQL Queries in eDOCS DM Server Logs

As a matter of performance optimization of an Hummingbird / OpenText eDOCS DM environment, I tried to detect performance issues and I started my research by checking the backend, i.e. the database. If I were a DBA, I would probably use monitoring and performance profiling tools on the database side. Unfortunately, I don't have that privileged level of access to the database as a DBA has. What I have is the access to DM servers and the DOCSADM login to the database. This should be enough to find the slowest SQL queries and to try to fine-tune performance.