A while back I posted some code for an HTTP handler and module allowing for large files to be upload in ASP.Net. Since then I’ve had a great deal of feedback and there seem to be quite a few people using the module in various forms- even more since the license was changed to LGPL from GPL so that it could be used in commercial projects.

I’ve been working on a new release of the project, which I expect to be available for download soon. This version includes quite a few changes- some have been requested by users, while others just needed doing. Here are some of the updates:

• Support for IIS 7, Cassini, and the inbuild Visual Studio web server.
• A completely rewritten RFC 1867 parser, which in addition to being faster pulls more information from incoming stream- such as the name of the input control the file came from.
• Support for cancelling uploads.
• The ability to specify maximum file sizes.
• A SQL Server upload processor allowing files to be “chunked” directly into SQL databases.
• A completely rewritten user interface. This includes support for multiple uploads and file extension filtering.

Prior to release I’m going to post a few pages about how the new control works and the changes that have been required to develop it. Partly this is to ensure that those who are using the previous version have an easy migration path and also because there are (I think) some interesting bits of code which could be useful to others. I’m going to start today with the first part which deals with one of the most radical changes in the control- support for IIS 7.

I should probably start with a recap of how the upload process works. Essentially when there are input type=”file” controls on a page (whether these are native html elements or ASP.Net FileUpload controls) the HTTP post is in the form of an RFC 1867 stream containing both form data and the uploaded files. If the stream gets too large then ASP.Net runs out of memory and chokes because the uploaded files are retained completely in memory which is inefficient. In ASP.Net 2.0 this situation is improved as files are streamed to disk as they are uploaded through temporary files- but what I wanted was to be able to stream directly to the destination (e.g. a SQL database). In addition to this, uploading large files can also take a while and there isn’t any feedback given to the user which can be frustrating.

The upload module solves these issues by intercepting the incoming stream and pulling out the file sections before they reach the application. The file sections are read in chunks via the ASP.Net worker process and subsequently streamed to processors which deal with them by writing them out to files (or other storage locations such as databases). This means that the entire file never builds up in memory- which is much more efficient. The process also allows for a progress bar to be displayed to the user through the client making calls to an HTTP Handler to get progress updates.

The tricky part is convincing ASP.Net that the newly created input stream (the form data without the files) is in fact the request it’s supposed to see. To understand this process we need to look at the way that the ASP.Net worker process reads incoming requests, particularly the Preloaded Entity Body. At the start of the request, the worker process reads a certain amount of the incoming data into an area known as the Preloaded Entity Body. This allows it to get the header information needed to understand the remainder of the incoming request. The upload module replaces this data with it’s new stream which is essentially the form data with any file uploads stripped out. It then convinces ASP.Net that the entire input stream is contained within the preloaded entity body. Once this is done the maximum upload size is satisfied and available memory isn’t consumed by uploaded files.

In the first version of the module this was accomplished via reflection:

IServiceProvider provider = (IServiceProvider)HttpContext.Current;
worker = (HttpWorkerRequest)provider.GetService(typeof(HttpWorkerRequest));

.......

BindingFlags ba = BindingFlags.Instance | BindingFlags.NonPublic;
Type workerType = worker.GetType();

// Find the actual ISAPIWorkerRequest object
while ((workerType != null) && (workerType.FullName != "System.Web.Hosting.ISAPIWorkerRequest"))
    workerType = workerType.BaseType;

// Set the preloaded content as the contentMinusFiles and
// fool the worker process into thinking that is all the content
// there is by setting the appropriate lengths.
workerType.GetField("_preloadedContent", ba).SetValue(worker, fs.ContentMinusFiles);
workerType.GetField("_preloadedContentRead", ba).SetValue(worker, true);
workerType.GetField("_contentAvailLength", ba).SetValue(worker, fs.ContentMinusFiles.Length);
workerType.GetField("_contentTotalLength", ba).SetValue(worker, fs.ContentMinusFiles.Length);

// Set the content length to the true value to prevent "request exceeded maximum size" errors.
app.Request.GetType().GetField("_contentLength", ba).SetValue(app.Request, fs.ContentMinusFiles.Length);

The above works fine with IIS 6 and IIS 7 in classic mode but fails with IIS 7 in integrated mode and the inbuilt Visual Studio web server. To be fair, the visual studio web server is easily dealt with- the problem is simply that the internal field names for _contentAvailLength etc are different. I thought that the same would be true of IIS 7, so my first attempt at solving the issue was to branch the code depending on the type of worker process something like the following:

if (workerType != null)
{
    if (workerType.FullName == "System.Web.Hosting.ISAPIWorkerRequest")
    {
        // Process IIS 6
    }
    else if (workerType.FullName == "Cassini.Request" || workerType.FullName == "Microsoft.VisualStudio.WebHost.Request")
    {
        // Process VS web server
    }
    else if (workerType.FullName == "System.Web.Hosting.IIS7WorkerRequest")
    {
        // Process II7
    }
}

This worked fine for everything except II7 which has a worker type of System.Web.Hosting.IIS7WorkerRequest. I couldn’t find any corresponding fields to set in this object and eventually found that they are handled internally by WebEngine.dll. The IIS7WorkerRequest uses unmanaged code to access these properties at runtime, so it can’t be fooled by setting variables with reflection.

However, all of the worker process objects extend the base class System.Web.HttpWorkerRequest. An instance of this class is contained in the current HttpContext and is used to determine the size of the request and to get at the loaded form data. This means that we ought to be able to create a new HttpWorkerRequest class which overrides the key methods we need to fool ASP.Net into using our newly created request data. To do this we replace the worker request object in the HTTP context with our new version using reflection- this has the same effect as replacing the variables in the previous version.

internal class UploadWorkerRequest : HttpWorkerRequest
{
    HttpWorkerRequest _request;
    byte[] _buffer;

    /// <summary>
    /// Constructor.
    /// </summary>
    /// <param name="request">The original worker request.</param>
    /// <param name="buffer">The content minus the uploaded files.</param>
    public UploadWorkerRequest(HttpWorkerRequest request, byte[] buffer)
    {
        _buffer = buffer;
        _request = request;
    }

    public override int ReadEntityBody(byte[] buffer, int size)
    {
        // All content is kept in the preloaded entity body so return 0 here
        return 0;
    }

    public override int GetTotalEntityBodyLength()
    {
        return _buffer.Length;
    }

    public override int GetPreloadedEntityBody(byte[] buffer, int offset)
    {
        // Return the buffer
        Buffer.BlockCopy(_buffer, 0, buffer, offset, _buffer.Length);
        return _buffer.Length;
    }

    public override byte[] GetPreloadedEntityBody()
    {
        return _buffer;
    }

    public override int GetPreloadedEntityBodyLength()
    {
        return _buffer.Length;
    }

    public override int ReadEntityBody(byte[] buffer, int offset, int size)
    {
        return 0;
    }

    public override string GetKnownRequestHeader(int index)
    {
        if (index == HttpWorkerRequest.HeaderContentLength)
        {
            return _buffer.Length.ToString();
        }
        else
        {
            return _request.GetKnownRequestHeader(index);
        }
    }

    public override bool IsEntireEntityBodyIsPreloaded()
    {
        return true;
    }

    // All other methods passed through to the original HttpWorkerRequest like the following CloseConnection method

    public override void CloseConnection()
    {
        _request.CloseConnection();
    }

}

This class overrides the core methods which return the preloaded entity content and the total content length. In addition, the GetKnownRequestHeader method is overridden for only the Content-Length header variable. After the new stream has been created and the files removed the _wp variable in the current context is replaced with the new object which has the same effect as the reflection in the earlier version of the module but works with all types of HTTP request- including IIS 7.

BindingFlags ba = BindingFlags.Instance | BindingFlags.NonPublic;

// Replace the worker process with our own version using reflection
UploadWorkerRequest wr = new UploadWorkerRequest(worker, fs.ContentMinusFiles);
app.Context.Request.GetType().GetField("_wr", ba).SetValue(app.Context.Request, wr);

The new process is slightly neater because it minimises the use of reflection to one variable (I’d still prefer none though).

A note about request limits

ASP.Net enforces request limits on applications. These ensure that an individual request never exceeds a pre-set maximum. They exist to prevent denial of service attacks where a bad person can cause your server to go down by frequently sending requests which it can’t handle, thus forcing it to spend most of it’s resources dealing with the problem request to the exclusion of other requests. I think it’s important to mention because request limits aren’t a bad thing- they are an important security feature.

Of course request limits can prevent large files being uploaded, but they’re not the problem. ASP.Net has problems handling large files because it runs out of memory. The upload module solves this by handling memory more efficiently and dealing with files in chunks. The module will still respect the maximum request limits.

In IIS 6 the maximum request limit is set using the maxRequestLength parameter in web.config. In addition to this it is wise to set the executionTimeout parameter to prevent the process from timing out before the file is uploaded. The following example shows how these can be set to 100Mb and 1 hour respectively:

<httpRuntime executionTimeout="3600" maxRequestLength="40960" />

In version 1 of the module the maxRequestLength setting was completely bypassed. In hindsight I think that was probably a bad thing so in version 2 I’ve added in code to ensure that it is respected and that the request is ended if the value is exceeded.

Things are a little different in IIS 7 however. The IIS 7 request filters by default will kick in and limit the maximum content length before the module even gets a chance to do anything. To allow larger uploads we need to set the maximumAllowedContentLength in web.config by entering the statement shown below at a command prompt. This example sets the maximum content length to 100Mb for the web app called “WebApp” on the default web site.


%windir%\system32\inetsrv\appcmd set config "Default Web Site/WebApp" -section:requestFiltering -requestLimits.maxAllowedContentLength:104857600


Note that in IIS 6 the maxRequestLength is in kilobytes while in IIS 7 the maxAllowedContentLength setting is in bytes.