Wednesday, August 22, 2018

Friday, August 03, 2018

Creating scalable RPA processes with Queueing

Often when I am talking with customers and prospects they ask about the concept of queueing transactions and then level loading across multiple agents/servers/bots so that their workloads are instantly scalable up or down as needed by adding additional agents into the mix.

Here's an example where queueing might be useful: Let's say a company needs to interact with a vendor or customer web site, an SAP application or some other UI based ERP or business application that can only process one transaction at a time. Often a single bot won't cut it because it's not fast enough because of application or other response time issues, so you need something that is infinitely scalable. Also a server in many cases is limited to one transaction at a time because it's automating using a user interface. Background processes are another story because a single server/agent/bot can handle many background process threads simultaneously.

By intercepting all transactions as they come in and funneling them to a single queue or inbox, one or more (pick a number) robotic processes can pick up the next transaction as it finishes the previous one. Think of it as being similar to picking up the next phone call in a call center or being next in line at the grocery store. First-in, first out using as many lines available.

So how does this queueing thing work ? Well the idea is that as a transaction comes in and gets added to a database or some other data store, a queue entry is written immediately so the next agent/process/bot can pick up the transaction and get it done. As the next sequential entry gets picked up it gets removed from the queue or marked as in-process or complete.

You might consider using a message queue system such as Microsoft Message Queue (MSMQ), WebSphereMQ, RabbitMQ, ActiveMQ or even a platform such as Apache Kafka. There are probably others that I missed. These queueing systems can be implemented simply and used with any RPA tool set that can call an API or web service. However they do add another layer to the complexity and overhead of your RPA process by adding another moving part for your application or RPA process to monitor: The message queue service.

Another simpler option perhaps might be to write queue entries to a database so that they can be processed easily and also the queue can be visualized via an SQL query as needed for reporting purposes. If you have read articles on table queueing theory on the web, there are lots of things written that recommend not using databases as queue mechanisms. I have researched a lot and found that SQL Server (and probably other databases as well) can work nicely as a queueing system if you're careful. The trick is when reading an entry, how do you make sure you process a particular database queue entry only one time ?

While researching on SQL Server methods for locking I learned about the READPAST and UPDLOCK flags. I won't regurgitate other articles so you can simply google the highlighted keywords for yourself and draw your own conclusions. However the upshot is that when these two flags are used on a query combination that reads and updates/deletes a record all in a single transaction, you should be able insure that each queue table entry is handled correctly and only once. When writing a new entry to a queue table you simply perform a standard record insert, so feeding the queue a bunch of transaction data that needs to get processed is very simple.

Hopefully you'll find this technique to be helpful when you're scaling your RPA, Workflow other process automation workloads. This technique should work with any automation toolset or custom process you create where queueing is required.

Sunday, June 24, 2018

Check out Open Source WinAppDriver UI Recorder

A new open-sourced tool is now available for the Windows Application Driver (WinAppDriver) community: the WinAppDriver UI Recorder tool. This tool will enable users to easily create automated UI tests.
For those of you not familiar with WinAppDriver, it is a UI automation service for Windows 10 that users can use to test their applications. 

Wednesday, June 20, 2018

Setting Up SSH Daemon on IBM i to Use Port Other Than 22

Log on to a 5250 session on your IBM i system. 

Edit the SSH Daemon config file.

On V7R3 the edit command would look as follows:

edtf '/QOpenSys/QIBM/UserData/SC1/OpenSSH/etc/sshd_config'

Once the edit screen comes up it may look as follows: 

# $OpenBSD: sshd_config,v 1.75 2007/03/19 01:01:29 djm Exp $               
# This is the sshd server system-wide configuration file.  See             
# sshd_config(5) for more information.                                     
# This sshd was compiled with PATH=/QOpenSys/usr/bin:/usr/ccs/bin:/QOpenSys
# The strategy used for options in the default sshd_config shipped with    
# OpenSSH is to specify options with their default value where             
# possible, but leave them commented.  Uncommented options change a        
# default value.                                                           
 Port 2022                                                                 

Add a new entry for the new port. Ex: Port 2022 and comment (#) the od entry if not already commented. 

After adding the port entry for port 2022, you can press F3 to save and exit the config file editor.

Stop and restart the SSH daemon with the following commands

Wait 10 seconds

Run the following command to make sure port 2022 is active:


If you see local port 2022 is active then you should be able to log in to the SSH server on port 2022. 

     Remote           Remote     Local                        
Opt  Address          Port       Port       Idle Time  State  
     *                *          as-admi >  100:11:23  Listen 
     *                *          as-admi >  100:17:44  Listen 
     *                *          as-admi >  100:17:44  Listen 
     *                *          as-admi >  100:17:44  Listen 

     *                *          2022       000:06:12  Listen 

Wednesday, June 06, 2018

MobaXterm X Server and SSH Client

If you have a need to connect to remote Linux servers via SSH, VNC, Telnet, RDP and more, check out the MobaXTerm application. 

Saturday, May 12, 2018

Who needs a chatbot when you can have a mailbot ?

While chatbots are the latest new craze, the concept of interacting with a business system has been around for a long time. Back in 2002 I wrote an email monitoring system for a retail customer who was sending reps and managers out into the field with Blackberry devices. When the manager was about to visit a set of stores they would simply send an email request to a selected address such as: along with a subject of: “STINFO store#”.

Once the request was received a background process would be triggered to parse the subject line, build a sales report and then format the data as HTML and send it back as a response. Theoretically a PDF, Excel, CSV or other attachment could be sent back as well in response to a query.

Fast forward sixteen years and we apply the term “bot” to just about any automation process. Since the term “bot” is used so heavily today you might call this request/response scenario an email-bot or mail-bot. I used to call it poor-man’s client/server because a request could be created and processed without opening up a web site or business systems directly to the web or other client application. This also allowed for a nice layer of security. As long as responses are always being returned back to the same authorized requestor email address and that address has been validated, chances are data can’t be compromised. Also since the mail-bot can only respond to a limited set of scenarios, it’s threat vector for being hackable is reduced.

A mail-bot pattern essentially works like this:

An email request is received by a monitored address containing a from address and a request in the subject line or email body.

Once the mailbox monitor process picks up an inbound email message, the message is parsed, information is validated and then a response is compiled if the request is from a valid email address.

A formatted response email message or file attachments are then returned to the requestor.

There’s a little more to it than what I described above, but if you’re thinking about creating a chat-bot, you might want to first do a proof of concept project using the mail-bot pattern to see if your logic is sound and if your customers will get value from such an information request method.

No matter whether you’re using an RPA tool or coding a custom business process the mail-bot pattern can help your team be successful interacting with vendors, employees and customers.

Sunday, April 08, 2018

Free Windows .Net Snippet Keeper Application

Free Snippet Keeper Application. This is my attempt at creating a useful Snipper keeper application. I also used a couple of notable technologies in this app: MetroUIFramework, SQLite for the data store and FastColoredtextBox for the text editing.
Download SnippetKeeper

NancyFx - A Lightweight Web Framework for .Net

I'm always looking for alternate technologies to use for creating easy to install and use app servers, web servers, micro-services and more.

A couple years back I came across the NancyFx framework for this. It works well with .Net and .Net Core to create self hosted web applications and services.

Open Source Spreadsheets for your .Net Applications

If you're building .Net applications chances are you've had the need to convert data into spreadsheet format. Check out the following links for a couple of open source options:
These also work nicely in conjunction with SQL Server, IBM i and other databases to quickly create Excel output.

VietOCR Open Source Front End for Tesseract OCR

Check out this OCR application if you need a basic open source front end to the Tesseract engine. VietOCR is a Java/.NET GUI frontend for Tesseract OCR engine. Supports optical character recognition for Vietnamese and other languages supported by Tesseract.

Ublu Midrange and Mainframe Life Cycle Extension Language

I haven't checked out this project yet, but it looks interesting. From the Github description: Ublu is an interpretive language for remote systems programming of midrange or mainframe hosts from a Java platform such as Linux, Mac, OpenBSD or Windows. It also can run natively on IBM i ®, IBM z/OS USS ® or any other reasonable Java platform.

Windows Software Installation Tools

I have used this product for the past 6 years for building Windows and Web based software installations. Much more affordable than Installshield and works very well.

Advanced Installer

Another open source option is Nullsoft

Nullsoft Scriptable Install System

Free App to Quickly Scan Documents to PDF and other file types

Cool open source scanning application. Scan documents with a single click. Easily scan with your chosen settings, or set up multiple profiles for different devices and configurations. Once you've finished scanning, you can save, email, or print with only a couple clicks. Save to PDF, TIFF, JPEG, PNG, or other file types.
NAPS2 Scan Documents to PDF and TIFF

Free Bootstrap Web Development Themes

Check out these web development themes which can be applied to your .Net and other applications that use the Bootstrap framework.

Free Web Development Books

Check out WebCodeGeeks for FREE web development books and other web development info.

Wednesday, February 14, 2018

Dynamic DotNet Scripting for VB and C# - DotNetScript40

This DotNet Scripting project for VB and C# was Forked from NScript project on CodeProject and Compiled Under .Net 4.0

This scripting framework is a nice way to create dynamic VB or C# scripts that can be run as Console apps and integrated with other applications.

Original NScript CodeProject site link:
Github Site for DotNetScript40 site link:
The project code has been renamed to DotNetScript40 and has been compiled under .Net 4.0, however I believe the software also has a .Net 2.0 dependency for use of the old VSA scripting DLL from .Net 2.0.
The DotNetScript40 solution has three projects:

DotNetScript40.exe - a C# console application for running .CS/.NCS scripts or .VB/.NVB scripts.

DotNetScriptW40.exe - a C# windows application for running .CS/.NCS scripts or .VB/.NVB scripts.

DotNetScriptLib40.DLL - a C# class library for running .CS/.NCS scripts or .VB/.NVB scripts.
DotNetScript40 and DotNetScriptW40 are very much similar to each other; the former can be used to run scripts that can output to console. DotNetScript40 shows error messages in console where as DotNetScriptW40 shows error messages using message boxes. It is DotNScriptW40 which can be associated with file extensions. Since there is lot of code that is shared by the two executables the common code is compiled in DotNetScriptLib40 and both the executables refer to this class library.
Sample usage
DotNetScript40.exe [ScriptFile.ncs/.nvb] [Parm1 Parm2]

Ex: DotNetScript40.exe hello.nvb

Sunday, February 04, 2018

Free Microsoft Training Classes - Microsoft Virtual Academy

If you're looking to learn ASP.Net Core or any other Microsoft technologies take a look at the Microsoft Virtual Academy site. You'll find classes for ASP.Net Core and more.

Monday, January 22, 2018

Using plink.exe and SSH to run a remote CL command on IBM i

If you want to connect to an IBMi system via SSH and plink.exe, run CL system commands and possibly get message responses from the call, the process can get interesting because of the system command and program call syntax causing issues. As it turns out it's not all that hard to format a command line with parameters for running via plink. 

ere’s what you can do if you can’t write a CL command wrapper and want to use plink.exe to run IBMi programs or commands:

Original - Problem - This one passes the numeric parameters as numeric values, but we want them to be character. The command interpreter thinks they are numeric parameters
plink -ssh "user@" -pw pass1 /bin/system 'CALL PGM(LIB/PLINKTESTC) PARM("999   " "222   ")'

Fixed with special characters (,), / and ’ all escaped with a backslash \ and with double quotes around the entire command.
plink -ssh "user@" -pw pass1 /bin/system “CALL PGM\(LIB\/PLINKTESTC\) PARM\(\’999\’  \’222\\)”

An easier option might be to build a CL command wrapper

Sample CL command wrapper which handles all padding and conversion of data to proper typesplink -ssh "user@" -pw pass1 /bin/system 'RJSDEVHS/PLINKTEST PARM1(999) PARM2(222)'

Providing Feedback from plink.exe
You can feedback CL message responses via SNDPGMMSG if you plan to send back any CL responses from plink.exe to be consumed by the caller.

Sample IBMi Code

 PGM        PARM(&PARM1 &PARM2)                         
 DCL        VAR(&PARM1) TYPE(*CHAR) LEN(10)             
 DCL        VAR(&PARM2) TYPE(*CHAR) LEN(10)             
              |> &PARM1)                                
              |> &PARM2)                                

PLINKTEST - CL Command Wrapper
CMD        PROMPT('plink test')                        

Sunday, January 21, 2018

Visual Studio 2017 15.5.4 IDE crashes when typing in C# Source Member

I was working with a C# source member yesterday and my Visual Studio IDE started crashing randomly. I updated from 15.3 to 15.5.4 to see if the problem would go away but it didn't. I saw a few posts when googling that talked about disabling the C# edit and continue option to get rid of this issue. Since I have only ever used edit and continue in VB I thought it was worth a try. 

Sure enough since I disabled edit and continue my C# editor hasn't crashed since.

Here's a MS link for disabling edit and continue

Git for Visual Studio Developers–Get your .gitignore right and save yourself needless merges

When using Visual Studio with Git, you probably want to ignore baggage files that may result in repository merge issues on a regular basis. 

Check out the following link for an explanation on how to add the appropriate files to your .gitignore file list.

The following link takes you to the raw .gitignore example file

Using Alternate Windows FIle Manager

If you're like me you're always looking for tools to make your day more productive. Back in the early days of PCs I used a tool called Norton Commander for file navigation and quick file editing. On Linux and IBMi I now use Midnite Commander. 

Listed below is a list of good FREE alternatives for Windows file management. 

Windows File Explorer Apps

Explorer ++

Free Commander

Norton Commander Look-Alikes

Midnight Commander

Far Manager

Feel free to send me your favorites.

Thursday, January 18, 2018

Capturing IBMi Spool File Print Documents via Windows LPD Service

Some IBM i or Mainframe users may have a need to push spool files from the IBM i/AS400, Mainframe or other systems to a Windows server so the binary or text output can be captured and processed electronically. It's pretty easy to use Windows LPD server to help the process along. 

Steps to Set up Windows Print Services and Print Queue for Capturing Print from Other Systems

Install Windows LPD Server on Windows (By Installing Print Services in Windows 2008, 2012, 2016 or Windows 7, 8 10)

Create a new output directory called: c:\localprint and another subdirectory called c:\localprint\spool

Add a Generic Text Windows Printer and name it MAINFRAME, IBMI or whatever is desired. Make sure it is shared with the same name. MAINFRAME, IBMI, Etc.
During printer creation create a new port for the printer that outputs to c:\localprint\report.txt

After printer creation go in to printer Advanced properties and check/enable the Keep Printed
Documents setting and uncheck/disable the Enable Advanced Printing Features settings.

This will insure the .SHD and .SPL files created by the printing process are kept after printing to disk file c:\localprint\report.txt. 

Note: c:\localprint\report.txt will never be processed. For any custom processing we will process the saved spool files (.SPL) from Windows print creation. We will ignore the .SHD files which appear to hold printer settings. Periodically the c:\localprint and c:\localprint\spool directories should be cleared of files that have been processed by a custom process.

Use regedit to go into Windows Registry and edit the following key: 

Computer\HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\PrintPrinters\MAINFRAME (or whatever your printer name is). Set the SpoolDirectory parm in the registry key to c:\localprint\spool

Re-start Windows Print Spooler service or reboot server.

Sending Over an iSeries Text Based Spool File with Line and Page Breaks via LPR/LPR

Example command to send over spool file from iSeries to Windows server

Create remote outq to send spool files over automatically from the IBM I

Sunday, January 14, 2018

IBM i Access from .Net with JT400

If you have been using Microsoft .Net for Windows development in C# or VB and would like to access IBM i - AS/400 data, programs, IFS files, data areas and more easily from a .Net desktop application, web application or web service, check out this nuget package. 

I converted the jt400.jar from the IBMi Java Toolbox to .Net using an open source project called IKVM. You can do the same or just install the nuget package. 

I've been using this successfully in applications since 2005.

PyDev Python IDE for Eclipse

PyDev is a Python IDE for Eclipse, which may be used in PythonJython and IronPython development.

I discovered this Eclipse based IDE environment for Python development when I was looking for Python IDE and debugging tools. 

Check it out if you plan to do any Python development.

Windows Modern UI for .NET WinForms Applications

If you develop any .Net WinForms applications and you like using open source components, check out the MetroFramework. 

These controls provide rough equivalents to the built-in Windows form controls and they come at no cost.

Using Midnite Commander on IBM i via SSH and Putty

If you used Norton Commander back in the day you probably remember this DOS utility with a side-by-side directory list.

It saved my life back in the early days of PCs.

I discovered Midnite Commander a few years ago which functions as a functional equivalent to Norton Commander in Linux.

There is also an AIX version so now that I have SSH working I thought I would give it a try.

It appears to work just fine. Very cool.

This will be useful addition when doing IBMi command line work via putty.

If you're using SSH and Putty to access the IBMi remotely check it out.

Listed below is a loose list of instructions for getting MC to work on your IBMi.

Create Local IBM i Copy of Perzl RPMs (replaced by OPS product)

Use the above URL and instructions to create a local IBMi copy of Perzl *.deps tree for use with the script.
Thw script will help you download and install AIX binary builds and dependencies to your IBMi PASE environment from the Perzl AIX site.

Installing Midnite Comander from a PASE Command Line

Once the Perzl RPM items have been resolve run the following steps from a PASE or SSH putty terminal.

Note: make sure the updated and files have been installed and has been run prior to trying the next steps.

Start the PASE command line or log in via SSH and Putty 

call qp2term    --enter the PASE command line

cd /QopenSys/download  --Change to location

List to see if Midnite Commander is available
./ aix61 list mc-

Download Midnite Commander via wget
./ aix61 wget mc-

Install Midnite Commander
./ aix61 rpm mc-

Create symbolic link for all users to use MC without path
ln -sf /QOpenSys/opt/freeware/bin/mc /QOpenSys/usr/bin/mc

Now you can simply log in to putty or other SSH terminal session and type:
mc or mc -c to use Midnite Commander. 

Run Midnite Commander

Black and white mode



Color Mode
/QOpenSys/opt/freeware/bin/mc -c


mc -c

Now start enjoying Minite Commander.

You'll find it very easy to navigate the IFS, browse or edit text files right from the IBMi system,

Midnite Commander User Interface

Update for January 2022

Now it's as simple as using yum to install Midnite Commander from a PASE command line if you have the IBM Open Source Package Management installed on your IBM i system.

yum install mc