eDOCS DM Server Log Parser - Part 3: Usage

This is the third part of the series about converting Hummingbird DM / OpenText eDOCS DM server logs to XML for further processing, e.g. for statistical analysis or for finding performance bottlenecks in SQL queries.

You can download the compiled tool from my Google drive. Here is the link: EDocsLogParser
The tool requires .NET Framework 4.0 or later, which I believe you already have on your Windows system.

Usage:

1. Launch the EDocsLogParser application.
2. Click the 'Directory...' button and select a folder with one or more log files from your DM server.
3. Press the 'Start' button. 
4. XML files will be created in the same directory where source DM logs are located and which you picked for processing.

Here is a sample result of the DM log file processing. A snippet from DM log:

DOCSSQL: [0ECE9D58] SQLObject at 10D48F58 acquired existing connection from pool #1

********** 13:59:28.871  [0ECE9D58] DOCSSQL: EXECute SQL Statement on Library:MYLIB - MYDB  (Oracle7) **********
Autocommit is ON

STATEMENT:
SELECT DOCSADM.APPS.APPLICATION,DOCSADM.APPS.DESCRIPTION,'AMBIENT_PROPERTY' UNK1,DOCSADM.APPS.SYSTEM_ID,'' FROM DOCSADM.APPS WHERE (SYSTEM_ID != 0) AND ((DOCSADM.APPS.APPLICATION = 'MS WORD')) AND (DOCSADM.APPS.DISABLED NOT IN ('Y','T','1') OR DOCSADM.APPS.DISABLED IS NULL) ORDER BY DOCSADM.APPS.APPLICATION ASC

TIMER:   [0ECE9D58] ODBCHandle::ReadItem(): 15 milliseconds  Fetched first row
DOCSSQL: [0ECE9D58] ODBCHandle::IssueCommand(): Statement returned results.  32 rows per fetch.  5248 bytes allocated for result sets.
TIMER:   [0ECE9D58] ODBCHandle::IssueCommand(): 22 milliseconds 
DOCSSQL: [0ECE9D58] ODBCHandle::ClearResults(): 1 row(s) fetched
DOCSSQL: [0ECE9D58] SQLObject at 10D48F58 released connection back to pool

The resultant XML file:

<?xml version="1.0"?>
<ArrayOfBaseEvent xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema">
  <BaseEvent xsi:type="SqlEvent" key="0ECE9D58" connection="1" isNew="False">
    <Queries>
      <SqlQueryEvent time="13:59:28.871" readItemDuration="0.015" issueCommandDuration="0.022">
        <Command>SELECT DOCSADM.APPS.APPLICATION,DOCSADM.APPS.DESCRIPTION,'AMBIENT_PROPERTY' UNK1,DOCSADM.APPS.SYSTEM_ID,'' FROM DOCSADM.APPS WHERE (SYSTEM_ID != 0) AND ((DOCSADM.APPS.APPLICATION = 'MS WORD')) AND (DOCSADM.APPS.DISABLED NOT IN ('Y','T','1') OR DOCSADM.APPS.DISABLED IS NULL) ORDER BY DOCSADM.APPS.APPLICATION ASC</Command>
      </SqlQueryEvent>
    </Queries>
  </BaseEvent>
</ArrayOfBaseEvent>

Once you have an XML file, you can query data from it. In this case, XPath and PowerShell may be your best option. If you want to load the data into a data grid and visually transform it, try Microsoft Excel.

Alternatively, you may think about writing a program in .NET for log data processing. Please take the definition of the BaseEvent, SqlEvent, and SqlQueryEvent classes from EDocsLog.dll to load data effectively.

See Also:
eDOCS DM Server Log Parser - Part 1: Understanding the Task
eDOCS DM Server Log Parser - Part 2: Implementation

eDOCS DM Server Log Parser - Part 2: Implementation

This is the second post about parsing Hummingbird DM / OpenText eDOCS DM server logs. Part #1 is here.

In this part I'm going to briefly explain the structure of the log parser just in case anyone may want to recompile, modify or extend it. However, if you simply want to get the DM Log Parser executable and use it, please feel free to jump directly to Part #3 - Usage of this article series.

The EDocsLogParser solution is implemented in C# / Visual Studio 2013 and consists of 3 projects.

EDocsLog is a Class Library and this is the core of the parser. The LogFile class represents a log file as a string array. The LogParser class is responsible for converting the log file into an array of events. It applies a set of predefined rules to the log lines. Rules are based on regular expressions. The rules return a RuleResult object, which tells the parser how to recognize event blocks (chunks) and to convert them into events. The result of the LogParser execution is an array of events. The events are serializable to XML via System.Xml.Serialization.
The parser performs two runs on a DM Server "raw" log file. On the first run, it applies the header and footer rules (see Part 1 for details about event blocks) and forms event stubs or chunks. The second run applies the inner or "body" rules to extract additional data from the recognized event block. The second run only processes the log lines within the previously found chunks. It also benefits from Task Parallel Library (TPL) for faster processing of the array of chunks.



EDocsLogParser is a WPF application. It allows you to pick a directory with DM Server logs and then to convert all of them into XML files. It calls the LogParser.Parse method for every log file and displays the progress.


EDocsLogTests is a set of unit tests for the EDocsLog classes.

Currently, the log parser can extract SQL events from a DM Server log file. It works for any log file, but, apparently, events will be extracted only if the log file was created with Log SQL, Log SQL & Calls, or Log All logging level:

Full source code of EDocsLogParser: https://github.com/dotnetnick/edocs/

eDOCS DM Server Log Parser - Part 1: Understanding the Task

A good thing about eDOCS DM is that DM Server can create log files, which give you idea of what is going on behind the scene.

Run Workflow Application Synchronously to Check the Completion State

If you come across this article, you are probably writing a Unit Test for your workflow, in which you need to know when workflow processing is completed. You also may want to examine the CompletionState value from the Complete event, otherwise you would use WorkflowInvoker instead of WorkflowApplication...

The solution turns out to be fairly simple: use ManualResetEvent. Okay, let's get to the code and that will be it:

using System.Activities;
using System.Threading;

[TestMethod]
public void TestExternalCancellationSimple() {
    var wf = new SomeActivity {
        ...
    };

    ActivityInstanceState state = ActivityInstanceState.Executing;
    var mre = new ManualResetEvent(false);

    var app = new WorkflowApplication(wf) {
        Completed = (e) => {
            state = e.CompletionState;
            mre.Set();
        }
    };
    app.Run();
    Thread.Sleep(500);  // wait a bit and then cancel the workflow
    app.Cancel();

    mre.WaitOne();

    Assert.AreEqual(ActivityInstanceState.Canceled, state);
}

UPDATE: Everything had been invented known even before we came in this world... Very similar code to the snippet above can be found in WF's Getting Started How to: Run a Workflow, with the only difference is that in MSDN they use AutoResetEvent, which is, technically speaking more correct, though doesn't make any difference if you call WaitOne only once.

Unit Testing Workflows

Here comes my first Workflow in Unit Tests. I tend to think that "Dot Net" inevitably turns into "Brace Net"... :-)

[TestClass]
public class BackupTests {
    [TestMethod, ExpectedException(typeof(InvalidWorkflowException))]
    public void IndexArgumentMissingTest() {
        Activity backup = new BackupFilesActivity { Copier = new MockCopier() };
        var sequence = new Sequence { Activities = { backup } };
        WorkflowInvoker.Invoke(sequence);
    }

    [TestMethod, ExpectedException(typeof(MyException))]
    public void CopyFailsTest() {
        var index = new IndexInfo() { ID = 1 };
        Activity backup = new BackupFilesActivity { 
            Copier = new MockCopier(new MyException()),
        };
        var inputs = new Dictionary<string, object>() { { "Index", index } };
        WorkflowInvoker.Invoke(backup, inputs);
    }

    [TestMethod]
    public void MockCopyWorksTest() {
        var index = new IndexInfo() { ID = 1 };
        Activity backup = new BackupFilesActivity {
            Copier = new MockCopier(),
            Index = new InArgument<IndexInfo>((ctx) => index)
        };
        var sequence = new Sequence { Activities = { backup } };
        WorkflowInvoker.Invoke(sequence);
    }
}

Pros And Cons of Different Ways to Custom Programming for eDOCS DM

Below I describe 5 different approaches that I know to programming for OpenText eDOCS DM:

DM Extensions API, DM API, Extending DM Webtop, Consuming WCF Service, and Direct Access.

Trustees - How to Get or Set Access Rights to Documents in eDOCS DM

PCDDocObjectClass of DM API has a few methods to work with a document's trustees - the list of users or groups who have access to the document: FetchTrustees, GetTrustees, SetTrustees, UpdateTrustees, etc. eDOCS DM 5.3.1 DM API Reference Guide.pdf includes corresponding code samples.

Uploading a File into eDOCS DM

Once a document profile is created, a file can be uploaded to DM. On the one hand, it's simple - open a stream and upload the content.

How to Programmatically Create Profile in eDOCS DM using .NET

Before you can upload a file into an eDOCS DM library, you need to create a document profile. Profile is metadata of your document. Technically speaking, it's a record in the PROFILE table in the DM database. Of course, we are not going to insert a data record directly into the database. We will be using DM API to let the DOCSFusion application server to execute all necessary business logic (which we do not know much about). However, we are not going to use DM API directly for the reasons explained earlier, but will create a method in our DMApiHelpers library.

Exception-based Error Reporting for DM API Helpers

As mentioned before, DM API calls are old-styled HRESULT functions, which return 0 (S_OK), to indicate the successful execution, or an error code. On the other hand, .NET developers commonly expect handling an exception, if something has gone wrong, not an error code.

Managed Helper Classes (Library) for eDOCS DM API

OpenText (Hummingbird) eDOCS DM API is ActiveX-based. It looks like it's been written decades ago and it's quite "low-levelish". Error reporting is based on return codes and not on exceptions. And it's syntax is VB6-like that is common for ActiveX components.

Custom Programming for eDOCS DM

The most common requests which I hear from the users of eDOCS DM are:

• extract documents from the document library and 
• upload documents from a disk to the library. 

Occasionally a user asks: "We've got documents from one of our offices, which are to be uploaded to DM. Maintain the folder structure, use profile form 'A', set the profile fields like this, and apply security settings like that."

This is a good example where you want to write some code to get the job done in one (well, may be more) clicks. This particular task - uploading the documents - is best to be solved using eDOCS DM API.
Other approaches exist as well, but I decided to use DM API for the following reasons:

• This is my first project of custom coding for eDOCS DM, so, for getting up to speed, I'll need some code samples, documentation and all tools ready.
• I'm going to use managed DM API Interop assemblies and write in C# and in VS2012.
• DM API is already installed as a part of DM Extensions.
• I also have Visual Studio 2012 already installed.
• I have DM Documentation from the eDOCS DM installation disk - eDOCS DM 5.3.1 DM API Reference Guide.pdf
• The last, but not the least, I have the source code of 2 tools written in VB.NET and C#, which I inherited from a guy, who was with the company before I joined and had left soon after :( Thank you, Jory, your code has helped me to start quickly!

Update

I've posted another article on different approaches to custom coding for DM which you may find useful:

Pros And Cons of Different Ways to Custom Programming for eDOCS DM

Opening OpenText eDOCS DM Series

It's been 4 months since I was assigned to support a document management system of a large enterprise. That system is a complex environment built on Hummingbird / OpenText eDOCS DM. My duty was (and still is) to stabilize the environment, make it more reliable. During the past 4 months I built some interesting tools to automate system monitoring, import and export documents, gather data for performance tuning, etc. Unlike Microsoft solutions, information about programming interfaces for OpenText products is difficult to find. I'll post my findings here and hope that someone will find this information helpful.

About Soft Inclinations

I'm starting this blog to write about Software: design, development, troubleshooting, etc., which I'd collectively call my "Inclinations", why not?