DM Server Log Parser has been updated (v.1.0.4)

I've just updated the EDocsLogParser tool. Now it can process Hummingbird DM 5.2.1 server logs in addition to OpenText eDOCS DM 5.3.1 supported from the initial version. Additionally, a few issues were fixed, and I've made some minor cosmetic tweaks.

Binaries: EDocsLogParser v1.0.4 x64/x86 for Windows
Source: https://github.com/dotnetnick/edocs/
User Guide: EDocsLogParser Usage

And let's not forget why we parse DM Server logs:
How to Find Slow SQL Queries in eDOCS DM Server Logs
How to Determine the Right Size for SQL Connection Cache
These are just examples. You can also think out other applications for the data from the logs.

If you process huge logs (greater than 1.5 GB), you should use x64 version of EDocsLogParser.exe. For smaller log files it doesn't matter: both 32- and 64-bit versions should work equally well. The tool consumes around the same amount of memory as the size of the log file. If you have 4 GB of RAM on your system, it should be fine for processing logs up to 2 GB, though the more memory your system has, the better.

Any feedback is welcome. Thank you!

DM Server Log Parser Update

EDocsLogParser v1.0.3, a new version, has been uploaded to Drive and GitHub. I tuned up performance a bit and fixed OutOfMemoryException which occurred on some large (~1 GB) log files. The parser was tested on logs larger than 1.3 GB and it worked fine.

While working on these updates, I noticed that DM Server logs become messy when the load on the server increases: messages from different calls are often mixed on same lines. I did my best to resolve this mess in the log parser, but it couldn't be resolved completely. I added new unit tests with snippets from real logs, so you can see a part of that "having fun".

If I wrote the parser from the beginning now, I would based it on loading a whole log file as a single string and not breaking it by lines as I've implemented it. I give it up: I find the OpenText's product being buggy and a bit badly designed to spend too much time and passion on it. With this said, I'm quite happy with EDocsLogParser as it is: it extracts 99.99% of all SQL events out of a messy 1 GB log and this is more than enough for the tasks I intended it for:
How to Find Slow SQL Queries in eDOCS DM Server Logs
SQL Connection Cache Size for OpenText eDOCS DM Server

SQL Connection Cache Size for OpenText eDOCS DM Server

The SQL Connection Cache Size value is one of the DM Server configuration values. You can set it in DM Server Manager on Libraries / Library Properties page.

How to Find Slow SQL Queries in eDOCS DM Server Logs

As a matter of performance optimization of an Hummingbird / OpenText eDOCS DM environment, I tried to detect performance issues and I started my research by checking the backend, i.e. the database. If I were a DBA, I would probably use monitoring and performance profiling tools on the database side. Unfortunately, I don't have that privileged level of access to the database as a DBA has. What I have is the access to DM servers and the DOCSADM login to the database. This should be enough to find the slowest SQL queries and to try to fine-tune performance.

eDOCS DM Server Log Parser - Part 3: Usage

This is the third part of the series about converting Hummingbird DM / OpenText eDOCS DM server logs to XML for further processing, e.g. for statistical analysis or for finding performance bottlenecks in SQL queries.

You can download the compiled tool from my Google drive. Here is the link: EDocsLogParser
The tool requires .NET Framework 4.0 or later, which I believe you already have on your Windows system.

Usage:

1. Launch the EDocsLogParser application.
2. Click the 'Directory...' button and select a folder with one or more log files from your DM server.
3. Press the 'Start' button. 
4. XML files will be created in the same directory where source DM logs are located and which you picked for processing.

Here is a sample result of the DM log file processing. A snippet from DM log:

DOCSSQL: [0ECE9D58] SQLObject at 10D48F58 acquired existing connection from pool #1

********** 13:59:28.871  [0ECE9D58] DOCSSQL: EXECute SQL Statement on Library:MYLIB - MYDB  (Oracle7) **********
Autocommit is ON

STATEMENT:
SELECT DOCSADM.APPS.APPLICATION,DOCSADM.APPS.DESCRIPTION,'AMBIENT_PROPERTY' UNK1,DOCSADM.APPS.SYSTEM_ID,'' FROM DOCSADM.APPS WHERE (SYSTEM_ID != 0) AND ((DOCSADM.APPS.APPLICATION = 'MS WORD')) AND (DOCSADM.APPS.DISABLED NOT IN ('Y','T','1') OR DOCSADM.APPS.DISABLED IS NULL) ORDER BY DOCSADM.APPS.APPLICATION ASC

TIMER:   [0ECE9D58] ODBCHandle::ReadItem(): 15 milliseconds  Fetched first row
DOCSSQL: [0ECE9D58] ODBCHandle::IssueCommand(): Statement returned results.  32 rows per fetch.  5248 bytes allocated for result sets.
TIMER:   [0ECE9D58] ODBCHandle::IssueCommand(): 22 milliseconds 
DOCSSQL: [0ECE9D58] ODBCHandle::ClearResults(): 1 row(s) fetched
DOCSSQL: [0ECE9D58] SQLObject at 10D48F58 released connection back to pool

The resultant XML file:

<?xml version="1.0"?>
<ArrayOfBaseEvent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <BaseEvent xsi:type="SqlEvent" key="0ECE9D58" connection="1" isNew="False">
    <Queries>
      <SqlQueryEvent time="13:59:28.871" readItemDuration="0.015" issueCommandDuration="0.022">
        <Command>SELECT DOCSADM.APPS.APPLICATION,DOCSADM.APPS.DESCRIPTION,'AMBIENT_PROPERTY' UNK1,DOCSADM.APPS.SYSTEM_ID,'' FROM DOCSADM.APPS WHERE (SYSTEM_ID != 0) AND ((DOCSADM.APPS.APPLICATION = 'MS WORD')) AND (DOCSADM.APPS.DISABLED NOT IN ('Y','T','1') OR DOCSADM.APPS.DISABLED IS NULL) ORDER BY DOCSADM.APPS.APPLICATION ASC</Command>
      </SqlQueryEvent>
    </Queries>
  </BaseEvent>
</ArrayOfBaseEvent>

Once you have an XML file, you can query data from it. In this case, XPath and PowerShell may be your best option. If you want to load the data into a data grid and visually transform it, try Microsoft Excel.

Alternatively, you may think about writing a program in .NET for log data processing. Please take the definition of the BaseEvent, SqlEvent, and SqlQueryEvent classes from EDocsLog.dll to load data effectively.

See Also:
eDOCS DM Server Log Parser - Part 1: Understanding the Task
eDOCS DM Server Log Parser - Part 2: Implementation

eDOCS DM Server Log Parser - Part 2: Implementation

This is the second post about parsing Hummingbird DM / OpenText eDOCS DM server logs. Part #1 is here.

In this part I'm going to briefly explain the structure of the log parser just in case anyone may want to recompile, modify or extend it. However, if you simply want to get the DM Log Parser executable and use it, please feel free to jump directly to Part #3 - Usage of this article series.

The EDocsLogParser solution is implemented in C# / Visual Studio 2013 and consists of 3 projects.

EDocsLog is a Class Library and this is the core of the parser. The LogFile class represents a log file as a string array. The LogParser class is responsible for converting the log file into an array of events. It applies a set of predefined rules to the log lines. Rules are based on regular expressions. The rules return a RuleResult object, which tells the parser how to recognize event blocks (chunks) and to convert them into events. The result of the LogParser execution is an array of events. The events are serializable to XML via System.Xml.Serialization.
The parser performs two runs on a DM Server "raw" log file. On the first run, it applies the header and footer rules (see Part 1 for details about event blocks) and forms event stubs or chunks. The second run applies the inner or "body" rules to extract additional data from the recognized event block. The second run only processes the log lines within the previously found chunks. It also benefits from Task Parallel Library (TPL) for faster processing of the array of chunks.



EDocsLogParser is a WPF application. It allows you to pick a directory with DM Server logs and then to convert all of them into XML files. It calls the LogParser.Parse method for every log file and displays the progress.


EDocsLogTests is a set of unit tests for the EDocsLog classes.

Currently, the log parser can extract SQL events from a DM Server log file. It works for any log file, but, apparently, events will be extracted only if the log file was created with Log SQL, Log SQL & Calls, or Log All logging level:

Full source code of EDocsLogParser: https://github.com/dotnetnick/edocs/

eDOCS DM Server Log Parser - Part 1: Understanding the Task

A good thing about eDOCS DM is that DM Server can create log files, which give you idea of what is going on behind the scene.