Posts Tagged ‘FTP’

Using Limilabs’ Ftp.dll with zOS Mainframes

Wednesday, December 17th, 2014

I purchased the Limilabs FTP product for FTP because I needed to send data to and from an IBM mainframe from my VB.NET program running in Windows. In particular I needed to be able to submit jobs, and receive the job output. These notes show how it’s done.

Introduction

FTP to/from IBM computers is pretty much like any other FTP except for two things.
1. IBM Mainframe and Midrange computers mostly use EBCDIC encoding rather than ASCII. With the FTP defaults your data can be garbled and useless when it arrives at the other end.
2. The SITE command is used to submit jobs to the mainframe, and get the results back.

Getting Started

I created a class called “JazzFTP” to wrap the Limilabs’ code. This was going to contain the functions that I wanted for my project, and so I started by defining the common elements that all methods would use. In my situation every FTP would be authenticated, and would be exchanging text data (not binary) with the remote computer.

Here is the initial class definition:

Imports Limilabs.FTP.Client

Public Class JazzFTP
    '   This class wraps the FTP library from Limilabs (/ftp)
    '   All methods
    '   1   Connect and logon using information from MySettings:  Sub LoginFTP
    '   2   Perform their action based on their parameters
    '   3   Close the connection
    Dim ftp As New Ftp()
    Dim response As FtpResponse
    Private Sub LoginFTP()
        ftp.Connect(My.Settings.SubmitIP)
        ftp.Login(My.Settings.Userid, My.Settings.Password)
    End Sub
    '   My functions will be written here
End Class

Basic FTP

Here is my first method, a basic function to upload a text file: –

    Function Upload(DestinationFile As String, Uploadfile As String) As String 
        LoginFTP()
        ftp.TransfersDataType = FtpDataType.Ascii
        response = ftp.Upload(DestinationFile, Uploadfile)
        ftp.Close()
        Return response.message
    End Function

FTP’s default is Binary, which is correct if you are transmitting a .JPG or other binary object, and it probably doesn’t matter if you are transmitting text to/from another Windows computer or a Unix computer. However if you are transmitting to/from an IBM mainframe or midrange computer it probably needs EBCDIC rather than ASCII characters. You must tell it that the file is Ascii text, not binary, otherwise it won’t be converted and it will be gibberish when you examine it on the mainframe.

Although this code above works, it is very fragile: the FTP server has to be up and running, you have to get the connection details exactly right, the source and destination files must exist, and so on. Since I couldn’t guarantee all of these details, I enclosed the code in Try/Catch to deal with any errors. For the time being I’ve simply used MsgBox to display the error message.

    Function Upload(DestinationFile As String, Uploadfile As String) As String
        Try
            LoginFTP()
            ftp.TransfersDataType = FtpDataType.Ascii
            response = ftp.Upload(DestinationFile, Uploadfile)
            ftp.Close()
            Return response.EndLine
        Catch ex As Exception
            MsgBox(ex.Message)
            Return ex.Message
        End Try
    End Function

Download, which is not illustrated, is similar except that you’d use ftp.Download. Again, you specify:

            ftp.TransfersDataType = FtpDataType.Ascii

Submitting Jobs and Receiving Job Output

Submitting a job is essentially an upload with a twist. Instead of uploading the file containing the job to a named file on the mainframe, you upload it to the JES (Job Entry System) input queue. Here is the basic code:

            response = ftp.Site("FILETYPE=JES")
            ftp.TransfersDataType = FtpDataType.Ascii
            response = ftp.Upload("JES", JCLFile)

The first line uses “ftp.Site”. Site means that this is a site-specific command, something that the FTP system at the other end will presumably know about. For an IBM mainframe “FILETYPE=JES” means that the data is going to and from JES.

The third line uploads the file in which we have prepared our job: in this case JCLFile is a file containing something like this: –

//IBMUSERH JOB  ,CLASS=A,MSGCLASS=H,NOTIFY=&SYSUID,COND=(8,LT) 
//*** COPY SOURCE INTO SOURCE LIBRARY
//COPY EXEC PGM=IEBGENER
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT2 DD DSN=IBMUSER.MANAJAZZ.SRCLIB(CRDTA1),DISP=SHR
//SYSUT1 DD *

Like any upload the FTP client syntax requires a destination file name, but because of the preceding FILETYPE=JES this will be ignored. I have written “JES” purely for documentation.

This will submit the job and it will run, appearing in the output like this:

tasks

Of course we can view the job output on the mainframe, but we may want to return it to Windows. We do this by downloading the file by naming the JobID – JOB00594 in this case – and again using the Site command. The essential code is: –

        response = ftp.Site("FILETYPE=JES")
        ftp.TransfersDataType = FtpDataType.Ascii
        ftp.Download(Jobname, LocalPath)

But how do we get the Jobname? JES returns a message with this information when the job is submitted. We need to add code to the Job Submission logic to extract this from the message. In function JobSub

       response = ftp.Upload("JES", JCLFile)

upload the job and (if all goes well) returns a message like
“It is known to JES as JOB00594″
This code extracts JOB00594” and puts it into variable JobName

       Dim TestString As String = "It is known to JES as"
       If Mid(response.Message, 1, Len(TestString)) = TestString Then  'Should be true
           Jobname = Trim(Mid(response.Message, Len(TestString) + 1))
       End If

Now, since it is logical that if we submit a job we’ll want to get it back, I coded this in the JobSub function: –

    Function JobSub(JCLFile As String, ByRef Jobname As String) As FtpResponse
        '   JCLFile is path to a .JCL file (format .txt) containing the job to be submitted
        '   If successful submission, the job name is returned in JobName
        Dim Tstring As String = ""
        Try
            Jobname = "Unknown"
            LoginFTP()
            response = ftp.Site("FILETYPE=JES")
            ftp.TransfersDataType = FtpDataType.Ascii
            response = ftp.Upload("JES", JCLFile)
            Dim TestString As String = "It is known to JES as"
            If Mid(response.Message, 1, Len(TestString)) = TestString Then  'Should be true
                Jobname = Trim(Mid(response.Message, Len(TestString) + 1))
            End If
            JobGet(Jobname, False)
            ftp.Close()
        Catch ex As Exception
            MsgBox(ex.Message, MsgBoxStyle.OkOnly, "Jazz FTP")
            Return response
        End Try
        Return response
    End Function

and I coded JobGet to accept these parameters: –

    Function JobGet(Jobname As String, Optional Login As Boolean = True) As FtpResponse
        '   Get job output, save as Jobname.txt in Jazz Program Library.  
        If Login Then
            LoginFTP()
        End If
        response = ftp.Site("FILETYPE=JES")
        ftp.TransfersDataType = FtpDataType.Ascii
        Dim LocalPath As String = My.Settings.UserCommonPath & "\" & My.Settings.Programs & "\" & Jobname & ".txt"
        Jazzworkbench.ShowBtnResults(Jobname, Jazzworkbench.ResultsStatus.Pending)
        ftp.Download(Jobname, LocalPath)
        ftp.DeleteFile(Jobname)
        Jazzworkbench.ShowBtnResults(Jobname, Jazzworkbench.ResultsStatus.JobReturned)
        ftp.Close()
        Return response
    End Function

Once the job output has been downloaded I didn’t want to leave it cluttering up my Held Job Output Queue, so after the Download

        ftp.DeleteFile(Jobname)

gets rid of it.

This all works for my test jobs (which are very quick) and provided that mainframe FTP server is available.

If you want to know any more about my project to revolutionize mainframe programming, then have a look at www.jazzsoftware.co.nz

Best wishes with your programming,
Robert Barnes.

FTP file and folder permissions

Tuesday, August 14th, 2012

You can access remote file or folder permissions using Limilabs .NET FTP library. First you should learn how to connect and download files from FTP server.

It is important to realize that originally FTP protocol didn’t have the concept of file/folder permissions build-in directly.

LIST command, that allowed listing contents of the remote server, was build around system listing commands: ls on UNIX/LINUX machines, dir on Windows.
ls returns file permissions divided into 3 groups: Owning user, owning group and others e.g.: rwxr–rw-.

When using Ftp.dll Ftp.List and Ftp.GetList methods with UNIX FTP server (or a server that imitates ls output for LIST command) you can access these UNIX permissions using UnixPermissions property on FtpItem object:

// C#

List<FtpItem> items = client.List();
UnixPermissionSet set = items[0].UnixPermissions;

bool ownerCanRead = (set.OwnerUser | UnixPermission.Read) != 0;
bool groupCanRead = (set.OwnerGroup | UnixPermission.Read) != 0;
bool othersCanRead = (set.Others | UnixPermission.Read) != 0;

bool ownerCanWrite = (set.OwnerUser | UnixPermission.Write) != 0;

' VB.NET

Dim items As List(Of FtpItem) = client.List()
Dim [set] As UnixPermissionSet = items(0).UnixPermissions

Dim ownerCanRead As Boolean = ([set].OwnerUser Or UnixPermission.Read) <> 0
Dim groupCanRead As Boolean = ([set].OwnerGroup Or UnixPermission.Read) <> 0
Dim othersCanRead As Boolean = ([set].Others Or UnixPermission.Read) <> 0

Dim ownerCanWrite As Boolean = ([set].OwnerUser Or UnixPermission.Write) <> 0

MLSD and perm fact

As FTP protocol evolved, MLSD command was introduced with RFC 3659.

MLSD standardized the way listings are returned from the server. It introduced facts concept, and in particular perm fact.
Ftp.GetList method uses MLSD command if it is available, that is if MLST extension is supported and advertised by the remote server. It is possible to force this command to be used by calling Ftp.MLSD method directly.

You can access parsed version of perm fact using FtpItem.Permission property:

// C#

List<FtpItem> items = client.GetList();
List<FtpPermission> permissions = items[0].Permissions;

bool canRead = permissions.Contains(FtpPermission.Read);
bool canWrite = permissions.Contains(FtpPermission.Write);
bool canCreateFile = permissions.Contains(FtpPermission.CreateFile);
bool canChangeFolder = permissions.Contains(FtpPermission.ChangeFolder);

' VB.NET

Dim items As List(Of FtpItem) = client.GetList()
Dim permissions As List(Of FtpPermission) = items(0).Permissions

Dim canRead As Boolean = permissions.Contains(FtpPermission.Read)
Dim canWrite As Boolean = permissions.Contains(FtpPermission.Write)
Dim canCreateFile As Boolean = permissions.Contains(FtpPermission.CreateFile)
Dim canChangeFolder As Boolean = permissions.Contains(FtpPermission.ChangeFolder)

As you can see Ftp permissions can contain additional information such as if you can select particular folder or create a file within.

Changing the permissions

There is no standardized way of changing the permissions. You’ll need SITE command (this command is used by the server to provide services specific to his system but not sufficiently universal to be included as commands in protocol) and CHMOD (again UNIX command).

Limilabs .NET FTP library contains useful shortcut for that – SiteChangeMode method:

// C#

client.SiteChangeMode("file.txt", new UnixPermissionSet(UnixPermission.Write));
' VB.NET

client.SiteChangeMode("file.txt", New UnixPermissionSet(UnixPermission.Write))

Note that the same UnixPermissionSet class can is used that is returned by FtpItem.UnixPermissions property

FTP uploading files using patterns

Tuesday, June 26th, 2012

Uploading files using patters is a unique feature of our FTP component. It allows you fast upload of files of certain types.

Here’s the sample that uploads all text files (*.txt) files from C drive, to newly created ‘Uploads’ folder on FTP server. The search includes all child folders recursively – note the true parameter in LocalSearchOptions constructor. Ftp component is going to create all necessary folders on the remote FTP server for you.

Upload using wildcard pattern

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL

    ftp.CreateFolder("Uploads");

    LocalSearchOptions options = new LocalSearchOptions("*.txt", true);
    ftp.UploadFiles("Uploads", @"c:\", options);

    ftp.Close();
}
' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")	' or ConnectSSL
    ftp.CreateFolder("Uploads")

    Dim options As New LocalSearchOptions("*.txt", True)
    ftp.UploadFiles("Uploads", "c:\", options)

    ftp.Close()
End Using

Upload using regex pattern

You can also use LocalSearchOptions.UseRegexMatch method, if you want to use regex patterns:

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL

    ftp.CreateFolder("Uploads");

    LocalSearchOptions options = new LocalSearchOptions();
    options.UseRegexMatch(@"^.*$", @"^.*\.txt$", true);
    ftp.UploadFiles("Uploads", @"c:\", options);

    ftp.Close();
}
' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")	' or ConnectSSL

    ftp.CreateFolder("Uploads")

    Dim options As New LocalSearchOptions()
    options.UseRegexMatch("^.*$", "^.*\.txt$", True)
    ftp.UploadFiles("Uploads", "c:\", options)

    ftp.Close()
End Using

FTP downloading files using patterns

Tuesday, June 26th, 2012

Downloading files using patters is a unique feature of our FTP .NET component. It allows you fast download of files of certain types.

Here’s the sample that download all text files (*.txt) files from remote Uploads folder to Downloads folder located on C drive. Remote search includes all child folders recursively – note the true parameter in RemoteSearchOptions constructor. Ftp component is going to create all necessary folders on the local computer.

Download using wildcard pattern

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL
    
    Directory.CreateDirectory(@"c:\Downloads");

    RemoteSearchOptions options = new RemoteSearchOptions("*.txt", true);
    ftp.DownloadFiles("Uploads", @"c:\Downloads", options);

    ftp.Close();
}
' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")    ' or ConnectSSL

    Directory.CreateDirectory("c:\Downloads")
    
    Dim options As RemoteSearchOptions = _
        New RemoteSearchOptions("*.txt", True)
    ftp.DownloadFiles("Uploads", "c:\Downloads", options)

    ftp.Close()
End Using

Download using regex pattern

You can also use RemoteSearchOptions.UseRegexMatch method, if you want to use regex patterns on remote names:

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL
    
    Directory.CreateDirectory(@"c:\Downloads");

    RemoteSearchOptions options = new RemoteSearchOptions();
    options.UseRegexMatch(@"^.*$", @"^.*\.txt$", true);
    ftp.DownloadFiles("Uploads", @"c:\Downloads", options);

    ftp.Close();
}
' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")	' or ConnectSSL

    Directory.CreateDirectory("c:\Downloads")

    Dim options As RemoteSearchOptions = _
        New RemoteSearchOptions()
    options.UseRegexMatch("^.*$", "^.*\.txt$", True)
    ftp.DownloadFiles("Uploads", "c:\Downloads", options)

    ftp.Close()
End Using

Ftp zlib compression

Tuesday, June 19th, 2012

Ftp.dll FTP component supports zlib compression for all data transfers.

What is zlib

zlib is a compression method which can greatly decrease the size of data, similar to popular compression format – Zip.

How to use zlib

zlib can be used with Ftp.dll FTP/FTPS component thanks to the newly implemented MODE Z command. This command and format have been already adopted by popular FTP servers.

Ftp.dll automatically turns compression on, if it is supported by FTP server. zlib compression is transparent to users. It is activeted using MODE Z command. Once sent, client sends and receives compressed data, so all directory listings and file exchanges between server and client will be compressed.

What is the benefit?

Faster data transfers! Directory listing, which is text, can be highly compressed with zlib thus boosting the server and client network speed and reactivity.

Transfers of html, scripts or large logfiles (which are text) no longer needs to be zipped before being sent via ftp and should generally experience a 3-4 times gain in data transfers.

For example, a 60MB log file can turn into a 5MB data exchange when transferred with MODE Z enabled.

Depending on the file content, you will see different results:

  • Text files : ~15-20% of original size.
  • Html files : ~25-30% of original size.
  • Media, video, sound : ~90-95% of original size.
  • Already compressed documents with Zip, Rar, Ace etc. will see almost no gain at all.