Friday, December 05, 2008

Action Events in .NET

The Async Pattern used in .NET 1.x and 2.0 code always annoyed me because of the excessive coding noise it introduced. The new support in .NET 3.x in relation to lamdas (convenient delegate or callback syntax) and the generic representation of Actions, Functions and Predicates opens up some possibilities for writing asynchronous oriented code without all that cruft. Check out the IStreamReader class below. It defines a series of events related to the stream reading functionality. Each of these is defined as an event that calls the subscriber back via an Action definition. Each Action type is parameterized based on the needs of the event callback functionality. The beauty of this is that the application code is subsequently much more readable as can be seen by the automated test segment at the end of this blog post.

/// <summary>
/// An object that asynchronously reads bytes from a data stream.
/// </summary>
public interface IStreamReader
{
/// <summary>
/// Occurs when a block of bytes has been read by this reader. The data
/// bytes buffer that were just read and the number of bytes placed in
/// that buffer are supplied to the event callbacks.
/// </summary>
event Action<byte [], int> BytesRead;

/// <summary>
/// Occurs when progress is made by this reader. The event call back
/// provides the number of bytes read to date and the number of bytes
/// expected to be read in total. If the total number of bytes in the
/// stream is unknown then the total count supplied is -1.
/// </summary>
event Action<int, int> ProgressMade;

/// <summary>
/// Occurs when an error stops the read process from proceeding.
/// </summary>
event Action<Exception> ErrorOccurred;

/// <summary>
/// Occurs when the read action is complete. This event provides no
/// additional information so is only practically usable when the
/// <see cref="BytesRead"/> event was subscribed to.
/// </summary>
event Action ReadCompleted;

/// <summary>
/// Occurs when all data bytes have been read from this stream. The
/// entire data block that was read is supplied to the event callback
/// along with the number of bytes that was read. This event should only
/// be used where the files read can practically fit in memory. If this
/// is not the case, then a combination of the <see cref="BytesRead"/>
/// and <see cref="ReadCompleted"/> events can be used instead to process
/// the large file in manageable chunks.
/// </summary>
event Action<byte [], int> EntireFileRead;

/// <summary>
/// Initiates a read of the entire stream.
/// </summary>
void ReadAll();

/// <summary>
/// A convenience method for when this asynchronous oriented class is used
/// in a synchronous manner. This method waits for the read actions to
/// complete or fail before proceeding.
/// </summary>
void WaitForCompletion();
}


In the following test code segment, an error is not expected to occur so rather than handling the error callback we throw the exception object supplied to fail the test. If the entire file is read successfully a lamda is used to set the text that has been read. In typical applications, the WaitForCompletion method would not be used but it useful in test code and single action applications to wait for the read functionality to complete.



[ Test ]
public void Disk_Stream_Reads_Entire_File_Successfully()
{
string text = null;
IStreamReader reader = GetTestReader( "MartinLutherSpeech.txt" );
reader.ErrorOccurred += ex => { throw ex; };
reader.EntireFileRead += ( data, length ) => text = Encoding.ASCII.GetString( data, 0, length );
reader.ReadAll();
reader.WaitForCompletion();

Assert.IsNotNull( text );
Assert.AreEqual( 9190, text.Length );
Assert.IsTrue( text.StartsWith( "I am happy to join" ) );
Assert.IsTrue( text.EndsWith( "we are free at last!\"" ) );
}


The new functional aspects of the .NET 3.x functionality really changes the way we develop with the .NET framework. Practically speaking its going to take some time to get use to the new ways of thinking and applying them to this multi-core world we find ourselves in.

Thursday, December 04, 2008

Torq FxCop Setup

FxCop is a static analysis tool from Microsoft that checks software source code for conformance to the Microsoft .NET framework’s design guidelines. Static analysis is performed on either the source code files that define the software program or on the compiled software files. It is analysis that is performed before the program is run. This contrasts to dynamic analysis tools which act while the program is running. Both have their place in working towards improving software quality.

The name FxCop is derived from “Framework Cop”, so it represents a tool that polices a programmer’s software development so that it meets the .NET framework’s design guidelines. Fortunately the FxCop rule set is configurable so that a software developer can change the static analysis rules to match their internal policies. This is in fact what we are doing with FxCop within our software development processes.

FxCop performs checks on .NET assemblies (dll or exe files) and doesn’t work on the source code directly. There are commercial static analysis tools available including those in Visual Studio Team System and other third parties such as NDepend and CodeIt.Right. As we use Visual Studio Professional and FxCop is appropriate for our needs, we’re going the free static analysis tool route. The goal is to increase code quality and in particular highlight obscure and difficult to remember design rules during the development process.

At the time of writing FxCop isn’t as integrated into Visual Studio as the commercial offerings. Regardless, with some adjustments to our development environment and processes it has been neatly integrated. The approach taken is to add another configuration type to our Visual Studio projects in addition to the defaults of Release and Debug. The additional configuration type has been called Analyse and is intended to be used for any static analysis functionality we choose to add in the future and as such is not limited to FxCop.

The Analyse configuration can be added to a solution in Visual Studio by displaying the Configuration Manager using the “Build-Configuration Manager” menu. Once the Configuration Manager form is displayed, choose the New option in the “Active Solution Configuration” drop down. Once the configuration has been added to the solution, a few more steps are needed as we’re using the Analyse configuration in the same way as the Debug configuration but with additional static analysis actions after every build action.

To make the Analyse configuration operate in a similar fashion to the Debug configuration we do the following for each project that needs static analysis support:

  • Set the “Defined DEBUG constant” checkbox in the Build tab for the project.
  • Set the Debug Info setting to “Full” in the Advanced Build settings form that is also available from the Build tab for the project.
  • Add a call to the AnalyseAssembly batch file to the post build step of the project. A batch file is used to keep the post build step code short and allows us to modify the analysis actions in one location within our development build environment. This then requires the batch file directory to be in the executable path of the development machine.

The AnalyseAssembly batch file is fairly simple, it just runs the command line version of FxCop with the custom dictionary and rule set that we’ve configured.

With the Analysis configuration in place, a developer can easily flip to the Analyse configuration option in Visual Studio and compile to see the static analysis results. In addition, the continuous integration system can include the analysis steps in the automated build process. We’ve just recently started using FxCop and are in the process of updating our Visual Studio projects to use the approach documented here. An involved part of this process is choosing which FxCop rules to use and which ones to disable as some of them are certainly too restrictive for our needs.

Monday, September 29, 2008

Modbus Driver and Rewrite Decisions

There’s an informal definition for Legacy Software that comes up time to time that cuts to the core of the matter: Legacy software is software that actually works. Its been installed, deployed and run against real world situations. As such its a major decision to rewrite any substantial item of legacy software and should typically be resisted. Regardless, there are many situations where doing a rewrite is the best long term decision to make.
Sentient Computing and Torq had to make a rewrite or maintain decision with respect to the mqe driver back in 2004. The mqe driver is an abbreviation for the Modbus Quantum Ethernet driver. It is a MacroView communications driver that implements the Modbus Ethernet TCP protocol. We ended up taking the rewrite route and developed the modbusdrv communications driver. This same technology was then used to develop a Modbus Server implementation for MacroView and some internal Modbus protocol based utilities.
Many of the MacroView installed sites have moved to using the newer modbusdrv communications driver since it was first developed in 2004. Regardless many sites still run the mqe driver which was originally developed in 1996. Recently there was a need to identify the advantages of upgrading to the modbusdrv driver when the mqe had been operating at a site for many years. The question came up because ostensibly there is little functionality difference between the two. From a maintenance perspective, we much prefer the modbusdrv implementation as the source code and internal structure is much cleaner and better organized. So bottom line, the functionality difference is that modbusdrv a better quality communications driver. This translates directly to customer cost savings when any support is needed.
Its interesting that Wikipedia actually has a dedicated page for the word Rewrite with respect to programming. In that article various web pages are referenced with strong opinions on the subject. Its an emotional issue for developers because the cost of a rewrite is significant. There is a preconception that developers will lean towards rewriting software instead of extending it. It is often attributed to a variation of the Not Invented Here syndrome. The Wikipedia article lists the following possible reasons for a software rewrite:
  1. When the source code to be able to extend an existing program is not available.
  2. When the source code is available under an incompatible license.
  3. When the code cannot be adapted to a new target platform.
  4. When the existing code has become too difficult to handle and extend.
  5. When the task of debugging the existing code seems too complicated.
  6. When the programmer finds it difficult to understand the source code.
The last three were the reasons behind effectively rewriting the mqe driver and developing a new from scratch Modbus TCP driver which we called modbusdrv. The approach taken was to implement portions of the mqe functionality as needed on a project by project basis. It was impractical to attempt a total replacement in one go. We added functionality to the modbusdrv implementation over the years such that now they are now on par in terms of functionality. The rewrite was definitely the right decision as the result is that we have a Modbus driver implementation and technology base that is much easier to maintain and extend. Aspects of the modbusdrv implementation and its advantages over the mqe implementation are summarized in the remainder of this blog post.

Leveraging Common Libraries

The modbusdrv implementation leverages a series of common MacroView libraries that Sentient and Torq have developed since taking over the ongoing MacroView maintenance and development. The mqe driver has its own driver specific Modbus protocol and socket communications code. The advantage of using common libraries include:

  • Issues are resolved for multiple applications when they’re resolved in the common libraries
  • Services based on these core libraries can be applied to multiple applications without the need for additional application specific code.
  • Developers don’t have to learn how a particular driver implements a particular aspect of its functionality as there is a common mechanism.
Serialization Library

Newer MacroView drivers and communication utilities leverage the same core Serialization framework. It’s the same serialization library used in drivers such as the Allen Bradley PCCC driver, Simatic 505 driver, Symax ethernet driver, the Modbus server implementation and the MacroView metaserver infrastructure. In all of these implementations, communication messages are modelled as objects. This approach separates out the data structures from the protocol implementations. It makes it easier to add support for additional packet types and Modbus communication layer variations as needed. Note that the diagram below has only the fields of the classes displayed and excludes the class functions to highlight the data structure.

Modbus Protocol Library

A standalone Modbus protocol library was developed and that is used by multiple applications (the driver itself, the Modbus Server application and a general Modbus utility). The protocol library is session type agnostic in its design and currently supports either serial communications or Ethernet TCP scenarios using the same code base.

Debug Logging Library
The debugging and logging options for the modbusdrv driver use a standard MacroView library and leverage the Serialization debugging functionality to provide clear debug logs that show the packet structures in an organized manner. The detailed debug log shows a Modbus request-reply interaction and the associated serialization and deserialization activities. Each structural layer of an object serialization is indented so that the structure of a packet can easily be seen.

Automated Testing

The modbusdrv development included automated unit tests and integration tests. The integration tests can be run against either a physical PLC or a Modbus simulator.

Unit Testing

Within MacroView application programs that we’ve developed, the unit tests are compiled into the Debug versions of executables and can be run with a –test command line option. It is based a unit testing framework built to work with the MacroView C++ classes but inspired by the xUnit family of unit testing frameworks. The following screen cast shows a run of the modbusdrv unit tests. These are all in-memory tests designed to executable quickly and provide developers with rapid feedback of whether any functionality modifications have adversely affected existing functionality.

Integration Testing

The modbusdrv integration tests are script based and exercises the drivers interaction with a Modbus device through a series of get/set actions with confirmation tests run during the test cycle. Both the unit tests and the integration tests are also run within a Purify session to identify and resolve any memory issues.

Saturday, September 20, 2008

General versus Special Case Tension

One of the blogs I subscribe to is by Reginald Braithwaite. He was recently interviewed on InfoQ and part of the conversation related to the development process of generalizing in programming:



……so again I apologize for digressing, but I found this to be an important point in my own programming, and I don't know if that's a general point but quite often we see something and we are very tempted to say "Oh, this is a special case of a more general thing", and then we solve the general thing. However that is an infinite recursion, it's always the special case of something more general. And if you are always climbing up the tree into the more general thing you will eventually wind up with a PhD in computer science and no code.


However on the other hand, if we just scrabble around the Earth, and we never sort of poke our head around to see the more general thing. We are constantly resolving the same problems, we are not even recognizing that two different things we solved are both aspects of the same thing. So what I try to do is I always try to recognize what the general case is and then discipline myself not to solve the general case until it's really an imperative, but not to be ignorant of the general case. So coming back to your question, at the moment I am aware that it is a general case, and I am fighting on an almost daily basis not to make it too general


This is such a fundamental decision that comes up time and time again in software development. I think Reginald is spot on in his approach of not going down the generalization rabbit hole everytime, but never forgetting that there’s an incredible amount of waste associated with always working on just the specialized case. Sentiments such as YAGNI are great guiding principles but can be damaging if they preclude doing any generalization work at all.

Friday, June 06, 2008

dynamic_cast and accessible base classes

We recently upgraded to using Visual Studio 2008 for compiling our MacroView source code on Windows. Previously we had used Visual Studio 2003 so there was a jump over the 2005 compiler version. The same version of IDE is now being used across all of our C, C++ and C# code on Windows. The move went fairly smoothly but one of the porting tasks that I didn’t anticipate was the change in the implementation of dynamic_cast and how it relates to a multiple inheritance hierarchy. A problem occurred where a dynamic_cast failed with a VS2008 compiled executable while it didn’t with the VS2003 compiled version. The inheritance hierarchy of the class in question is shown in the diagram. The diagram is my first use of the C++ class diagram support in VS2008 which is quite nifty. I’m not sure how well it will work with collection relationships in C++, but for quickly creating an inheritance diagram it was very nifty.


OpcMetaBridgeInheritance



I suppose some developers will recoil in horror at the use of multiple-inheritance, but it has actually worked well and well across multiple platforms. The multiple inheritance of IMetaMessageProcessor, IConversationMessageSink and IHandleEventProcessor are all in aid of supporting the interface concept in C++. Each of these interface “classes” have all abstract methods and absolutely no data. The DebugTool class is a convenience class which provides a series of logging functions to any class inheriting from it. This is the aspect of the inheritance hierarchy that feels a bit “dirty”. This approach was taken to minimize the amount of typing needed in a class to call a logging function. This could have also been achieved through composition and some macros to reduce the typing footprint. Regardless, this is the design that was chosen and it has worked well. Purists be damned :^)


Back to the dynamic_cast issue. Testing the VS2008 compiled application showed a dynamic_cast<OpcMetaBridge *>( processor ) where processor is an IMetaMessageProcessor failing. Cracking open the Annotated C++ Reference Manual lead me to the following statement:



The result of a dynamic_cast<T>(v) is of type T. The type T must be a pointer or a reference to a defined class or void *.
If T is a pointer type and v is a pointer of a type for which T is an accessible base class, the result is a pointer to a unique sub object of that class.


So the dynamic_cast was returning a null pointer because MetaCommsServer inherits from IMetaMessageProcessor with protected inheritance. It worked before because VS2003 and all of the other older Unix C/C++ compilers we’re using weren’t matching the standard. VS2008 improves Microsoft’s compliance with the definition of the C++ language.

Thursday, March 13, 2008

DSXP OPC DA 2.0

Man I hate OPC. It just feels like software plumbing gone overboard in the COM based architecture it uses. The term “software plumbing” is used to describe software thats forms the basis of software that people (realy people, not developers) actually see. Software plumbing is important as it needs to be establish for the rest of the structure gets built, though I’m sure the analogy is being taken too far here.

I’m currently working on an OPC application that leverages the Technosoftware C++ API that sits on top of OPC to simplify the OPC development somewhat. It does this, but because of leaky abstractions, there is need to delve into the library code where it interacts with the OPC interfaces directly. It just seems the combination of COM and C++ result in butt ugly hard to do understand code. You hear/read old Microsofties talking about the wonderful COM architecture (COM is love and all that sort of thing) and wonder what the hell were they smoking? It reminds of the STL in that some developers think its a paragon of API design, but to me its difficult to use and difficult to read. Anyway, I’m heading off into bile blog territory here. So back to why I started this blog post…

The Technosoftware OPC C++ Client API treats OPC DA 2.0 and 3.0 servers differently and runs different portions of it’s implementation depending on the OPC version. Its difficult to get access to many OPC servers which require entire systems to practically operate e.g. Yokogawa EXA. Hence there is a need for OPC simulation servers. The one simulation server I’ve found that delivers solely a OPC DA 2.0 interface and sufficiently functional to use in development/testing is the OPC simulator found at http://www.dsxp.com.


Thursday, January 10, 2008

Fedora Core 7 and VMWare

Here’s a quick tip that I kicked myself when I eventually figured out – and then felt subsequently quite thick. After installing Fedora Core 7 within a VMWare host on Windows, the screen resolution couldn’t go above 800x600. I worked with that for a while using an X windows package to interact with FC7 to get around the low resolution. Today I found out that if you change the display monitor type to be an LCD panel of a particular resolution, lo and behold you can then choose a whole bunch of screen resolutions. The performance of the latest VMWare tools and Fedora Core 7 combination is great. Its now as good as a Windows guest running VMware tools.


Saturday, September 22, 2007

Real Men Don't Sleep

There’s a new MSDN show called Striking Pixels that is a “goofy” look at Microsoft’s designer offerings such as Silverlight. There was one excerpt that just cracked me up. Its shown below. The two hosts are playing two sets of characters. The first set of characters are the customers who have a very dodgy set of accents (and dodgy moustaches). The second set of characters are the designers that are providing design services to the customers. I found it so funny because earlier in the week I had a similar experience. A business partner sent over a rough set of functionality requirements for a product extension. It was only on my second pass at reading the documents that I realized the expectation was to have something ready by October 1st which was just 6 working days away at the time. So I totally get the exasperation and stunned silence shown by the designer characters in the following excerpt:


 



 


Now often as developers we can pull out all stops and do some amazing things in a short time. The problem is that this makes a rod for your own back. In other words, if the client or business partner doesn’t understand the intricacies and details of what goes on to achieve the results, their expectations are then set higher than is realistic. Bottom line: we are going to do whatever we can to meet the business needs of the business partner for their deadlines, but… I’m going to need to successfully communicate the compromises and development follow up that will be needed in the following month.


 

Wednesday, September 12, 2007

Inside the Agility Cube Excerpt

I was listening to the Javapolis Inside the Agility Cube session and snorted at the following excerpt:



I know of a project that went into a kind of death march chaos and all kinds of other things. The project that I guess you could have.. probably take twenty to twenty five people to develop. At the last count there was something like seventy people involved. So.. slightly off the objective there. And they had originally instituted a two week cycle. And what happened.. eventually the management decided to shut down the agile process pump and one of the reasons was they said they didn’t like hearing bad news every two weeks.


I have a four year old boy who is just learning the trick of putting his fingers in his ears and saying I can’t hear you, I can’t hear you, I can’t hear you. He hasn’t learned it too well but that is very much what went on at this company. “I don’t want to hear your feedback. Your feedback is not good. I thought that you guys said you would do Agile development and give us this feedback that it was always going to be good news”. No news is good news. This is not what it is about. The whole point of an agile process. The whole point of an iterative and incremental – or in their case decremental – development is you fail fast. If you’re going to get it wrong you may as well find out sooner rather than later. And you have to be prepared for that.


Its tough working with a business partner or customer that expects to hear “good news” all the time. Its just not workable. To guarantee “good news” all the time, the performance bar needs to be set really low – and that’s not in anyone’s best interest.

Saturday, September 01, 2007

MacroView Excel Addin V1.3.0

We’ve released V1.3.0 of the MacroView Excel Add-in software product just this week. Its a package that we’ve developed for Sentient Computing who are one of our business partners. Sentient sell the software as part of their suite of MacroView SCADA products. It allows an end user to get access to both current historical information from their manufacturing, process control or other such industrial plant from within the Excel spreadsheet environment. Being “head-down bum-up” (i.e. really busy) most of the time developing software I don’t talk about our successes or what we can do for business partners enough. Its something thats on the cards to rectify in the coming months.


Back to the Excel addin release. The updated documentation can be found over here. The main feature addition in this release is native support for the Excel 2007 ribbon. Check it out below. I personally find the ribbon approach in the Office 2007 product range to be a huge improvement for the most part. It is frustrating occasionally when a feature you remember from Office V-Previous just “can’t be found” in the ribbon UI. Fortunately the command needs for the MacroView ribbon tab are manageable in number and fit easily within a single ribbon tab:



I particularly like the Value and Table buttons. By default, mouse clicks on these will initiate the most common Value and Table data query respectively (an entity value and a historical trend data table). Its only if the advanced query types are needed does the user need to click on the drop down arrow and select from all the different query types available for that category:



The value queries allow you to associate a MacroView value with a cell and then update the value from the “real world” on demand or in a continuous update mode. You can use these live values in other calculations or as inputs to a graphic visualization such as a pie chart:



The product user can also access MacroView historical data and similarly perform calculations and display charts using that data. Sentient sell the Excel addin as part of the product range. If your business is interested in developing a software product that extends any of the Microsoft Office software then contact me to discuss the possibilities. Similarly if your business would benefit from automating part of your workflow with custom software its worth us talking.