Archiving the Import and Export Logs, and viewing them with a stylesheet

Posted on by Carol

A long time ago I wrote up a method that could be used to archive the MIIS import and export logs, while also making them more readable with a stylesheet. I’ve now implemented this on a FIM 2010 server, and it works, so I’m going to write it up again.

The Problem

The FIM Sync Service, just like its predecessors, only stores information about the current state of objects. Run History is almost completely worthless and should be cleared regularly. However there will be times when you will be asked to trace through a series of events – perhaps leading to the CEO’s account being inadvertently disabled. At times like this it is important to be able to show that FIM was just responding appropriately to an imported data change, and not acting maliciously in some skynet-esque awakening.

However, as we know, the import and export run profiles, while allowing a log file to be dumped, then helpfully overwrite it at the next run. We need a way to hang on to that historical data.

The Proposal

If you’re already running your tasks using vbscripts it’s pretty simple to add an extra step which copies the log file off to a datestamped version in an archive location (script below).

At the same time, we can do a little manipulation to the log file to make it more readable. By inserting a couple of lines in the top of the log file it can now be used with an XML Stylesheet, allowing it to be browsed in a nice table format.

Provisos

The log file will only be archived if you run your export and import jobs via your scripts. Anything run directly from the Sync Service GUI may still produce a log file, but it won’t be archived.

Also, the timestamp is a approximate as it represents the time the log was archived, rather than the exact time specific objects were modified in a target directory. But if you archive the log straight after the Export profile runs then it should be close enough for most purposes.

log.xsl

First, you need to create a folder somewhere with the same sub-folders as your MaData folder (in the script example below, I’m using D:\FIM\MALogArchives). Then, into this new folder, create a text file called “log.xsl” and paste in this content.

ArchiveLog.vbs

Now here’s a vbscript that will copy the named log file, while modifying it to work with the stylesheet.

' This script copies the export and import logs to datestamped versions
' and modifies them to work with a stylesheet called ../log.xsl.
'
'   Usage: cscript archivelog.vbs MaName LogFileName
'
'   Eg:    cscript archivelog.vbs HR import.xml
'
' Written by Carol Wapshere

Option Explicit
Const XML_STYLESHEET = "..\log.xsl"
Const MIIS_FOLDER = "C:\Program Files\Microsoft Forefront Identity Manager\2010\Synchronization Service"
Const ARCHIVE_FOLDER = "D:\FIM\MALogArchives"
Const ForReading = 1
Const ForWriting = 2
Const ForAppending = 8
Const Unicode = -1

Dim objFS, MaName, LogName
Set objFS = CreateObject("Scripting.FileSystemObject")

If WScript.Arguments.Count <> 2 Then
  Usage
End If

MaName = WScript.Arguments.Item(0)
LogName = WScript.Arguments.Item(1)

ArchiveLog MaName, LogName

Sub ArchiveLog(MA, LogFile)

  Dim objLogFile, objArchiveFile
  Dim strLogName, strArchiveName, logTime, dateStamp, strLine

  strLogName = MIIS_FOLDER & "\MaData\" & MA & "\" & LogFile
  If objFS.FileExists(strLogName) Then
    logTime = Now()
    dateStamp = DatePart("yyyy", logTime) & TwoChars("m", logTime) &_
                                 TwoChars("d", logTime) & TwoChars("h", logTime) &_
                                 TwoChars("n", logTime) & TwoChars("s", logTime)
    strArchiveName = ARCHIVE_FOLDER & "\" & MA & "\" & Split(LogFile,".")(0) & "_" & dateStamp & ".xml"
    set objLogFile = objFS.OpenTextFile(strLogName, ForReading, false, Unicode)
    set objArchiveFile = objFS.OpenTextFile(strArchiveName, ForWriting, true, Unicode)
    objLogFile.ReadLine()
    objArchiveFile.WriteLine("<?xml version=""1.0"" encoding=""UTF-16""?>")
    objArchiveFile.WriteLine("<?xml-stylesheet type=""text/xsl"" href=""" & XML_STYLESHEET & """?>")
    objArchiveFile.WriteLine("<top>")
    objArchiveFile.WriteLine("<xmlfile-time>")
    objArchiveFile.WriteLine(logTime)
    objArchiveFile.WriteLine("</xmlfile-time>")
    objLogFile.ReadLine() 'skip mmsml
    objLogFile.ReadLine() 'skip directory-entries
    strLine = objLogFile.ReadLine()
    Do Until InStr(strLine, "</directory-entries>") > 0
       objArchiveFile.WriteLine(strLine)
       strLine = objLogFile.ReadLine()
    Loop
    objArchiveFile.WriteLine("</top>")
    objLogFile.Close()
    objArchiveFile.Close()
  End If
End Sub

Function TwoChars(dtvar, time)
  Dim i
  i = DatePart(dtvar, time)
  If i < 10 Then
   TwoChars = "0" & CStr(i)
  Else
   TwoChars = CStr(i)
  End If
End Function

Sub Usage
  Wscript.echo "Usage: cscript archivelog.vbs MaName import|export"
  Wscript.Quit
End Sub

 

Modify the Run scripts

Your last step is to modify your scheduled scripts to archive the import/export log directly after the task has run.

cscript AD_Export.vbs
cscript ArchiveLog.vbs "AD MA" export.xml

 

I've been doing IT for 30 years, and IdM for 15. I live in Australia and build IdM solutions based on Microsoft Identity Manager. I also play the violin, but that doesn't help much with the IdM solutions.

6 Replies to “Archiving the Import and Export Logs, and viewing them with a stylesheet”

  1. Do you happen to know if there is an XSD file that can be used with these logs? For easier reporting to auditors I want to shred them into relational tables. I’ve tried using the XML data source in SSIS and it will create an XSD for you based on the input XML, but of course it fails on subsequent logs.

  2. I haven’t done that, though it sounds like a good idea. I had to modify the logs just to work with the stylesheet so you may have to do something similar. Why does it fail on the second log?

  3. Hi Carol,
    Good stuff. I’m a little fuzzy on the last process when you modify your run scripts. I have all of my run profiles set to run in a single script and I’m not sure how to incorporate that using what you have above. Is AD_Export.vbs a separate script you created just for AD exports? Amazing there is nothing OOB that does this for us, but this is exactly what I need. Hopefully in the near future…

    Thanks,

    Jeff

  4. Yes it is a seperate script, so you’ll need to modify your run script to call this one after the exports and imports where you want archiving.

  5. Thanks for writing this and updating, Carol! I’ve been using your method for about two years, and it looks like the XSL file has been updated since I first “borrowed” it from your blog.

    I just noticed (or probably was just reminded) that the XML log files that ILM creates, when a single-value attribute is updated, have an “add” and “delete” line and they are randomly (grr!) ordered. The older version of the XSL file only shows one. When I get a chance I’m going to test out your new file to see how it looks.

    Have you found any differences in the format of the logs between MIIS, ILM and FIM?

    I’m still hoping we upgrade to FIM sometime in the next 6-12 months and I’d like to keep archiving these logs, though for AD I only archive the exports. The ILM delta import logs have full group membership listings and the resulting files are too large to archive in the long term given the frequency of their creation. The export logs have only the changes I’m interested in.

  6. Hi Chris,

    I pretty sure that’s the same old XSLT – perhaps I just moved it to another page when I tidied my code snippets. I have used that stylesheet on FIM and didn’t get any complaints, but I don’t know how extensivley that customer was using the logs. So in short, I haven’t noticed any differences with FIM but that doesn’t mean there aren’t any. If you come up with an improved stylesheet and are happy to share it let me know so I can link it, or even post it here with your name on it.

Comments are closed.