Introduction

FTP to/from IBM computers is pretty much like any other FTP except for two things.
1. IBM Mainframe and Midrange computers mostly use EBCDIC encoding rather than ASCII. With the FTP defaults your data can be garbled and useless when it arrives at the other end.
2. The SITE command is used to submit jobs to the mainframe, and get the results back.

Getting Started

I created a class called “JazzFTP” to wrap the Limilabs’ code. This was going to contain the functions that I wanted for my project, and so I started by defining the common elements that all methods would use. In my situation every FTP would be authenticated, and would be exchanging text data (not binary) with the remote computer.

Here is the initial class definition:

Imports Limilabs.FTP.Client

Public Class JazzFTP
    '   This class wraps the FTP library from Limilabs (/ftp)
    '   All methods
    '   1   Connect and logon using information from MySettings:  Sub LoginFTP
    '   2   Perform their action based on their parameters
    '   3   Close the connection
    Dim ftp As New Ftp()
    Dim response As FtpResponse
    Private Sub LoginFTP()
        ftp.Connect(My.Settings.SubmitIP)
        ftp.Login(My.Settings.Userid, My.Settings.Password)
    End Sub
    '   My functions will be written here
End Class

Basic FTP

Here is my first method, a basic function to upload a text file: –

    Function Upload(DestinationFile As String, Uploadfile As String) As String 
        LoginFTP()
        ftp.TransfersDataType = FtpDataType.Ascii
        response = ftp.Upload(DestinationFile, Uploadfile)
        ftp.Close()
        Return response.message
    End Function

FTP’s default is Binary, which is correct if you are transmitting a .JPG or other binary object, and it probably doesn’t matter if you are transmitting text to/from another Windows computer or a Unix computer. However if you are transmitting to/from an IBM mainframe or midrange computer it probably needs EBCDIC rather than ASCII characters. You must tell it that the file is Ascii text, not binary, otherwise it won’t be converted and it will be gibberish when you examine it on the mainframe.

Although this code above works, it is very fragile: the FTP server has to be up and running, you have to get the connection details exactly right, the source and destination files must exist, and so on. Since I couldn’t guarantee all of these details, I enclosed the code in Try/Catch to deal with any errors. For the time being I’ve simply used MsgBox to display the error message.

    Function Upload(DestinationFile As String, Uploadfile As String) As String
        Try
            LoginFTP()
            ftp.TransfersDataType = FtpDataType.Ascii
            response = ftp.Upload(DestinationFile, Uploadfile)
            ftp.Close()
            Return response.EndLine
        Catch ex As Exception
            MsgBox(ex.Message)
            Return ex.Message
        End Try
    End Function

Download, which is not illustrated, is similar except that you’d use ftp.Download. Again, you specify:

            ftp.TransfersDataType = FtpDataType.Ascii

Submitting Jobs and Receiving Job Output

Submitting a job is essentially an upload with a twist. Instead of uploading the file containing the job to a named file on the mainframe, you upload it to the JES (Job Entry System) input queue. Here is the basic code:

            response = ftp.Site("FILETYPE=JES")
            ftp.TransfersDataType = FtpDataType.Ascii
            response = ftp.Upload("JES", JCLFile)

The first line uses “ftp.Site”. Site means that this is a site-specific command, something that the FTP system at the other end will presumably know about. For an IBM mainframe “FILETYPE=JES” means that the data is going to and from JES.

The third line uploads the file in which we have prepared our job: in this case JCLFile is a file containing something like this: –

//IBMUSERH JOB  ,CLASS=A,MSGCLASS=H,NOTIFY=&SYSUID,COND=(8,LT) 
//*** COPY SOURCE INTO SOURCE LIBRARY
//COPY EXEC PGM=IEBGENER
//SYSPRINT DD SYSOUT=*
//SYSIN DD DUMMY
//SYSUT2 DD DSN=IBMUSER.MANAJAZZ.SRCLIB(CRDTA1),DISP=SHR
//SYSUT1 DD *

Like any upload the FTP client syntax requires a destination file name, but because of the preceding FILETYPE=JES this will be ignored. I have written “JES” purely for documentation.

This will submit the job and it will run, appearing in the output like this:

Of course we can view the job output on the mainframe, but we may want to return it to Windows. We do this by downloading the file by naming the JobID – JOB00594 in this case – and again using the Site command. The essential code is: –

        response = ftp.Site("FILETYPE=JES")
        ftp.TransfersDataType = FtpDataType.Ascii
        ftp.Download(Jobname, LocalPath)

But how do we get the Jobname? JES returns a message with this information when the job is submitted. We need to add code to the Job Submission logic to extract this from the message. In function JobSub

       response = ftp.Upload("JES", JCLFile)

upload the job and (if all goes well) returns a message like
“It is known to JES as JOB00594″
This code extracts JOB00594” and puts it into variable JobName

       Dim TestString As String = "It is known to JES as"
       If Mid(response.Message, 1, Len(TestString)) = TestString Then  'Should be true
           Jobname = Trim(Mid(response.Message, Len(TestString) + 1))
       End If

Now, since it is logical that if we submit a job we’ll want to get it back, I coded this in the JobSub function: –

    Function JobSub(JCLFile As String, ByRef Jobname As String) As FtpResponse
        '   JCLFile is path to a .JCL file (format .txt) containing the job to be submitted
        '   If successful submission, the job name is returned in JobName
        Dim Tstring As String = ""
        Try
            Jobname = "Unknown"
            LoginFTP()
            response = ftp.Site("FILETYPE=JES")
            ftp.TransfersDataType = FtpDataType.Ascii
            response = ftp.Upload("JES", JCLFile)
            Dim TestString As String = "It is known to JES as"
            If Mid(response.Message, 1, Len(TestString)) = TestString Then  'Should be true
                Jobname = Trim(Mid(response.Message, Len(TestString) + 1))
            End If
            JobGet(Jobname, False)
            ftp.Close()
        Catch ex As Exception
            MsgBox(ex.Message, MsgBoxStyle.OkOnly, "Jazz FTP")
            Return response
        End Try
        Return response
    End Function

and I coded JobGet to accept these parameters: –

    Function JobGet(Jobname As String, Optional Login As Boolean = True) As FtpResponse
        '   Get job output, save as Jobname.txt in Jazz Program Library.  
        If Login Then
            LoginFTP()
        End If
        response = ftp.Site("FILETYPE=JES")
        ftp.TransfersDataType = FtpDataType.Ascii
        Dim LocalPath As String = My.Settings.UserCommonPath & "\" & My.Settings.Programs & "\" & Jobname & ".txt"
        Jazzworkbench.ShowBtnResults(Jobname, Jazzworkbench.ResultsStatus.Pending)
        ftp.Download(Jobname, LocalPath)
        ftp.DeleteFile(Jobname)
        Jazzworkbench.ShowBtnResults(Jobname, Jazzworkbench.ResultsStatus.JobReturned)
        ftp.Close()
        Return response
    End Function

Once the job output has been downloaded I didn’t want to leave it cluttering up my Held Job Output Queue, so after the Download

        ftp.DeleteFile(Jobname)

gets rid of it.

This all works for my test jobs (which are very quick) and provided that mainframe FTP server is available.

If you want to know any more about my project to revolutionize mainframe programming, then have a look at www.jazzsoftware.co.nz

Best wishes with your programming,
Robert Barnes.

The post Using Limilabs’ Ftp.dll with zOS Mainframes first appeared on Blog | Limilabs.

It is important to realize that originally FTP protocol didn’t have the concept of file/folder permissions build-in directly.

LIST command, that allowed listing contents of the remote server, was build around system listing commands: ls on UNIX/LINUX machines, dir on Windows.
ls returns file permissions divided into 3 groups: Owning user, owning group and others e.g.: rwxr–rw-.

When using Ftp.dll Ftp.List and Ftp.GetList methods with UNIX FTP server (or a server that imitates ls output for LIST command) you can access these UNIX permissions using UnixPermissions property on FtpItem object:

// C#

List<FtpItem> items = client.List();
UnixPermissionSet set = items[0].UnixPermissions;

bool ownerCanRead = (set.OwnerUser | UnixPermission.Read) != 0;
bool groupCanRead = (set.OwnerGroup | UnixPermission.Read) != 0;
bool othersCanRead = (set.Others | UnixPermission.Read) != 0;

bool ownerCanWrite = (set.OwnerUser | UnixPermission.Write) != 0;

' VB.NET

Dim items As List(Of FtpItem) = client.List()
Dim [set] As UnixPermissionSet = items(0).UnixPermissions

Dim ownerCanRead As Boolean = ([set].OwnerUser Or UnixPermission.Read) <> 0
Dim groupCanRead As Boolean = ([set].OwnerGroup Or UnixPermission.Read) <> 0
Dim othersCanRead As Boolean = ([set].Others Or UnixPermission.Read) <> 0

Dim ownerCanWrite As Boolean = ([set].OwnerUser Or UnixPermission.Write) <> 0

MLSD and perm fact

As FTP protocol evolved, MLSD command was introduced with RFC 3659.

MLSD standardized the way listings are returned from the server. It introduced facts concept, and in particular perm fact.
Ftp.GetList method uses MLSD command if it is available, that is if MLST extension is supported and advertised by the remote server. It is possible to force this command to be used by calling Ftp.MLSD method directly.

You can access parsed version of perm fact using FtpItem.Permission property:

// C#

List<FtpItem> items = client.GetList();
List<FtpPermission> permissions = items[0].Permissions;

bool canRead = permissions.Contains(FtpPermission.Read);
bool canWrite = permissions.Contains(FtpPermission.Write);
bool canCreateFile = permissions.Contains(FtpPermission.CreateFile);
bool canChangeFolder = permissions.Contains(FtpPermission.ChangeFolder);

' VB.NET

Dim items As List(Of FtpItem) = client.GetList()
Dim permissions As List(Of FtpPermission) = items(0).Permissions

Dim canRead As Boolean = permissions.Contains(FtpPermission.Read)
Dim canWrite As Boolean = permissions.Contains(FtpPermission.Write)
Dim canCreateFile As Boolean = permissions.Contains(FtpPermission.CreateFile)
Dim canChangeFolder As Boolean = permissions.Contains(FtpPermission.ChangeFolder)

As you can see Ftp permissions can contain additional information such as if you can select particular folder or create a file within.

Changing the permissions

There is no standardized way of changing the permissions. You’ll need SITE command (this command is used by the server to provide services specific to his system but not sufficiently universal to be included as commands in protocol) and CHMOD (again UNIX command).

Limilabs .NET FTP library contains useful shortcut for that – SiteChangeMode method:

// C#

client.SiteChangeMode("file.txt", new UnixPermissionSet(UnixPermission.Write));

' VB.NET

client.SiteChangeMode("file.txt", New UnixPermissionSet(UnixPermission.Write))

Note that the same UnixPermissionSet class can is used that is returned by FtpItem.UnixPermissions property

The post FTP file and folder permissions first appeared on Blog | Limilabs.

Here’s the sample that uploads all text files (*.txt) files from C drive, to newly created ‘Uploads’ folder on FTP server. The search includes all child folders recursively – note the true parameter in LocalSearchOptions constructor. Ftp component is going to create all necessary folders on the remote FTP server for you.

Upload using wildcard pattern

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL

    ftp.CreateFolder("Uploads");

    LocalSearchOptions options = new LocalSearchOptions("*.txt", true);
    ftp.UploadFiles("Uploads", @"c:\", options);

    ftp.Close();
}

' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")	' or ConnectSSL
    ftp.CreateFolder("Uploads")

    Dim options As New LocalSearchOptions("*.txt", True)
    ftp.UploadFiles("Uploads", "c:\", options)

    ftp.Close()
End Using

Upload using regex pattern

You can also use LocalSearchOptions.UseRegexMatch method, if you want to use regex patterns:

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL

    ftp.CreateFolder("Uploads");

    LocalSearchOptions options = new LocalSearchOptions();
    options.UseRegexMatch(@"^.*$", @"^.*\.txt$", true);
    ftp.UploadFiles("Uploads", @"c:\", options);

    ftp.Close();
}

' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")	' or ConnectSSL

    ftp.CreateFolder("Uploads")

    Dim options As New LocalSearchOptions()
    options.UseRegexMatch("^.*$", "^.*\.txt$", True)
    ftp.UploadFiles("Uploads", "c:\", options)

    ftp.Close()
End Using

The post FTP uploading files using patterns first appeared on Blog | Limilabs.

Here’s the sample that download all text files (*.txt) files from remote Uploads folder to Downloads folder located on C drive. Remote search includes all child folders recursively – note the true parameter in RemoteSearchOptions constructor. Ftp component is going to create all necessary folders on the local computer.

Download using wildcard pattern

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL
    
    Directory.CreateDirectory(@"c:\Downloads");

    RemoteSearchOptions options = new RemoteSearchOptions("*.txt", true);
    ftp.DownloadFiles("Uploads", @"c:\Downloads", options);

    ftp.Close();
}

' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")    ' or ConnectSSL

    Directory.CreateDirectory("c:\Downloads")
    
    Dim options As RemoteSearchOptions = _
        New RemoteSearchOptions("*.txt", True)
    ftp.DownloadFiles("Uploads", "c:\Downloads", options)

    ftp.Close()
End Using

Download using regex pattern

You can also use RemoteSearchOptions.UseRegexMatch method, if you want to use regex patterns on remote names:

// C#

using (Ftp ftp = new Ftp())
{
    ftp.Connect("ftp.example.com");    // or ConnectSSL
    
    Directory.CreateDirectory(@"c:\Downloads");

    RemoteSearchOptions options = new RemoteSearchOptions();
    options.UseRegexMatch(@"^.*$", @"^.*\.txt$", true);
    ftp.DownloadFiles("Uploads", @"c:\Downloads", options);

    ftp.Close();
}

' VB.NET

Using ftp As New Ftp()
    ftp.Connect("ftp.example.com")	' or ConnectSSL

    Directory.CreateDirectory("c:\Downloads")

    Dim options As RemoteSearchOptions = _
        New RemoteSearchOptions()
    options.UseRegexMatch("^.*$", "^.*\.txt$", True)
    ftp.DownloadFiles("Uploads", "c:\Downloads", options)

    ftp.Close()
End Using

The post FTP downloading files using patterns first appeared on Blog | Limilabs.

What is zlib

zlib is a compression method which can greatly decrease the size of data, similar to popular compression format – Zip.

How to use zlib

zlib can be used with Ftp.dll FTP/FTPS component thanks to the newly implemented MODE Z command. This command and format have been already adopted by popular FTP servers.

Ftp.dll automatically turns compression on, if it is supported by FTP server. zlib compression is transparent to users. It is activeted using MODE Z command. Once sent, client sends and receives compressed data, so all directory listings and file exchanges between server and client will be compressed.

What is the benefit?

Faster data transfers! Directory listing, which is text, can be highly compressed with zlib thus boosting the server and client network speed and reactivity.

Transfers of html, scripts or large logfiles (which are text) no longer needs to be zipped before being sent via ftp and should generally experience a 3-4 times gain in data transfers.

For example, a 60MB log file can turn into a 5MB data exchange when transferred with MODE Z enabled.

Depending on the file content, you will see different results:

• Text files : ~15-20% of original size.
• Html files : ~25-30% of original size.
• Media, video, sound : ~90-95% of original size.
• Already compressed documents with Zip, Rar, Ace etc. will see almost no gain at all.

The post Ftp zlib compression first appeared on Blog | Limilabs.

In Active mode client waits for incomming data connections, in Passive mode client establishes data connections.

Passive mode is default. You can switch to Active mode using Mode property:

// C#

using (Ftp client = new Ftp())
{
    client.Mode = FtpMode.Active;

    client.Connect("ftp.example.com");
    client.Login("user", "password");

    // ...

    client.Close();
}

' VB.NET

Using client As New Ftp()
    client.Mode = FtpMode.Active

    client.Connect("ftp.example.com")
    client.Login("user", "password")

    ' ...

    client.Close()
End Using

The post FTP Active vs Passive first appeared on Blog | Limilabs.

// C#

client.Connect("ftp.example.com");

' VB.NET

client.Connect("ftp.example.com")

If you need to specify different port just use overloaded version of Connect method:

// C#

client.Connect("ftp.example.com", 999);
// -or-
client.Connect("ftp.example.com", 999, false);

' VB.NET

client.Connect("ftp.example.com", 999)
' -or-
client.Connect("ftp.example.com", 999, False)

If you are using SSL:

// C#

client.ConnectSSL("ftp.example.com", 999);
// -or-
client.Connect("ftp.example.com", 999, true);

' VB.NET

client.ConnectSSL("ftp.example.com", 999)
' -or-
client.Connect("ftp.example.com", 999, True)

You can also specify the port range used in Active mode.

// C#

client.ActiveModePorts = new Range(1024, 1025);

' VB.NET

client.ActiveModePorts = New Range(1024, 1025)

You can set the IP address announced to the FTP server in Active mode data transfer.
By default, the value of this property is IPAddress.None which means that the address of the network interface is used instead.

// C#

client.ActiveModeAddress = IPAddress.Parse("ftp.example.com");

' VB.NET

client.ActiveModeAddress = IPAddress.Parse("ftp.example.com")

The post Specify different port for FTP first appeared on Blog | Limilabs.

First add appropriate namespaces:

// C# version

using Limilabs.FTP.Client;
using Limilabs.FTP.Client.Hash;

' VB.NET version

Imports Limilabs.FTP.Client
Imports Limilabs.FTP.Client.Hash

Then upload the file, ask the server for hash of the uploaded file, compute the local hash and compare them:

// C# version

using (Ftp client = new Ftp())
{
    client.Connect("ftp.example.org");
    client.Login("user", "password");

    client.Upload("report.txt", @"c:\report.txt");

    byte[] serverHash = client.GetFileHash(
        "report.txt",
        FtpHashType.CRC);

    FileHash hash = new FileHash(FtpHashType.CRC);
    bool hashIsValid = hash.IsValid(@"c:\report.txt", serverHash);

    Console.WriteLine(hashIsValid);

    client.Close();
}

' VB.NET version

Using client As New Ftp()
	client.Connect("ftp.example.org")
	client.Login("user", "password")

	client.Upload("report.txt", "c:\report.txt")

	Dim serverHash As Byte() = client.GetFileHash( _
		"report.txt", _
		FtpHashType.CRC)

	Dim hash As New FileHash(FtpHashType.CRC)
	Dim hashIsValid As Boolean = hash.IsValid("c:\report.txt", serverHash)

	Console.WriteLine(hashIsValid)

	client.Close()
End Using

You can use CRC, MD5 and SHA1 algorithms for hash verification.

You can download Ftp.dll FTP/FTPS client for .NET here.

The post Verify file hash after FTP upload first appeared on Blog | Limilabs.

First add appropriate namespaces:

// C# version

using Limilabs.FTP.Client;
using Limilabs.FTP.Client.Hash;

' VB.NET version

Imports Limilabs.FTP.Client
Imports Limilabs.FTP.Client.Hash

Then download the file, ask server for its hash, compute the local hash and compare them:

// C# version

using (Ftp client = new Ftp())
{
    client.Connect("ftp.example.org");
    client.Login("user", "password");

    client.Download("report.txt", @"c:\report.txt");

    byte[] serverHash = client.GetFileHash(
        "report.txt",
        FtpHashType.CRC);

    FileHash hash = new FileHash(FtpHashType.CRC);
    bool hashIsValid = hash.IsValid(@"c:\report.txt", serverHash);

    Console.WriteLine(hashIsValid);

    client.Close();
}

' VB.NET version

Using client As New Ftp()
	client.Connect("ftp.example.org")
	client.Login("user", "password")

	client.Download("report.txt", "c:\report.txt")

	Dim serverHash As Byte() = client.GetFileHash( _
		"report.txt", _
		FtpHashType.CRC)

	Dim hash As New FileHash(FtpHashType.CRC)
	Dim hashIsValid As Boolean = hash.IsValid("c:\report.txt", serverHash)

	Console.WriteLine(hashIsValid)

	client.Close()
End Using

You can use CRC, MD5 and SHA1 algorithms for hash verification.

You can download Ftp.dll FTP/FTPS client for .NET here.

The post Verify file hash after FTP download first appeared on Blog | Limilabs.

// C#

client.Connect("ftp.example.org");

' VB.NET

client.Connect("ftp.example.org")

Then, before logging-in, start explicit SSL negotiation:

// C#

client.AuthSSL();

' VB.NET

client.AuthSSL()

Now, your connection is secured.

Remember that you can ignore SSL certificate errors using ServerCertificateValidate event:

// C# version

using (Ftp client = new Ftp())
{
    // Use this line to validate self-signed certificates:
    client.ServerCertificateValidate += ValidateCertificate;

    client.Connect("ftp.example.org");
    client.AuthSSL();
    client.Login("username", "password");

    foreach (FtpItem item in client.GetList())
    {
        if (item.IsFolder == true)
            Console.WriteLine("[{0}]", item.Name);
        else
            Console.WriteLine("{0}", item.Name);
    }
    client.Close();
}

private static void ValidateCertificate(
    object sender,
    ServerCertificateValidateEventArgs e)
{
    const SslPolicyErrors ignoredErrors =
        SslPolicyErrors.RemoteCertificateChainErrors |
        SslPolicyErrors.RemoteCertificateNameMismatch;

    if ((e.SslPolicyErrors & ~ignoredErrors) == SslPolicyErrors.None)
    {
        e.IsValid = true;
        return;
    }
    e.IsValid = false;
}

' VB.NET version

Using client As New Ftp()
    ' Use this line to validate self-signed certificates:
    AddHandler client.ServerCertificateValidate, AddressOf ValidateCerificate

    client.Connect("ftp.example.org")
    client.AuthSSL()
    client.Login("username", "password")

    For Each item As FtpItem In client.GetList()
        If item.IsFolder = True Then
            Console.WriteLine("[{0}]", item.Name)
        Else
            Console.WriteLine("{0}", item.Name)
        End If
    Next
    client.Close()
End Using

Private Sub ValidateCerificate( _
    ByVal sender As Object, _
    ByVal e As ServerCertificateValidateEventArgs)

    Const ignoredErrors As SslPolicyErrors = _
        SslPolicyErrors.RemoteCertificateChainErrors Or _
        SslPolicyErrors.RemoteCertificateNameMismatch

    If (e.SslPolicyErrors And Not ignoredErrors) = SslPolicyErrors.None Then
        e.IsValid = True
        Return
    End If
    e.IsValid = False
End Sub

You can download Ftp.dll FTP/FTPS client for .NET here.

The post The handshake failed due to an unexpected packet format first appeared on Blog | Limilabs.