Prerender Application Level Middleware

In the previous post Use Prerender to improve AngularJS SEO, I have explained different solutions at 3 different levels to implement Prerender.

In this post, I will explain how to implement a ASP.NET Core Middleware as a application level middleware to implement prerender. 

Application Level Middleware Architecture

At first, let's review what's the appliaction level middleware solution architecture.

ASP.NET Core Middleware - PrerenderMiddleware

In ASP.NET Core, we can create a Middleware, which has the similar functionality as HttpModule in ASP.NET, but in ASP.NET Core, there is no interface or base class we can use to declare a Middleware.

• Create PrerenderMiddleware class

The default convention is that, we need to:

1. The Middleware class needs to have a constructure which has RequestDelegate parameter as for next delegate.
2. The Middleware class needs to have an async Invoke method with parameter HttpContext

So, the class is as below. I have added PrerenderConfiguration for getting configuration.

#region Ctor
public PrerenderMiddleware(RequestDelegate next, PrerenderConfiguration configuration)
{
    _next = next;
    Configuration = configuration;
}
#endregion

#region Properties
public PrerenderConfiguration Configuration { get; private set; }
#endregion

#region Invoke
public async Task Invoke(HttpContext httpContext)
{
    await Prerender(httpContext);
}
#endregion

• Then, we need to implement Prerender(httpContext) logic

            If you know my implementation for PrerenderHttpModule in ASP.NET, I used HttpWebRequest & HttpWebResponse.

           But for PrerenderMiddleware here, I use HttpClient, as with the HttpWebRequest in ASP.NET Core (at 2/11/2017), there is no way to setup AllowAutoRedirect and other http headers.

private async Task Prerender(HttpContext httpContext)
{
    var request = httpContext.Request;
    var response = httpContext.Response;
    var requestFeature = httpContext.Features.Get<IHttpRequestFeature>();
    
    if (IsValidForPrerenderPage(request, requestFeature))
    {
        // generate URL
        var requestUrl = request.GetDisplayUrl();
        // if traffic is forwarded from https://, we convert http:// to https://.
        if (string.Equals(request.Headers[Constants.HttpHeader_XForwardedProto], Constants.HttpsProtocol, StringComparison.OrdinalIgnoreCase)
         && requestUrl.StartsWith(Constants.HttpProtocol, StringComparison.OrdinalIgnoreCase))
        {
            requestUrl = Constants.HttpsProtocol + requestUrl.Substring(Constants.HttpProtocol.Length);
        }
        var prerenderUrl = $"{Configuration.ServiceUrl.Trim('/')}/{requestUrl}";

        // use HttpClient instead of HttpWebRequest, as HttpClient has AllowAutoRedirect option.
        var httpClientHandler = new HttpClientHandler() { AllowAutoRedirect = true };
        // Proxy Information
        if (!string.IsNullOrEmpty(Configuration.ProxyUrl) && Configuration.ProxyPort > 0)
            httpClientHandler.Proxy = new WebProxy(Configuration.ProxyUrl, Configuration.ProxyPort);

        using (var httpClient = new HttpClient(httpClientHandler))
        {
            httpClient.Timeout = TimeSpan.FromSeconds(60);
            httpClient.DefaultRequestHeaders.CacheControl = new System.Net.Http.Headers.CacheControlHeaderValue() { NoCache = true };
            httpClient.DefaultRequestHeaders.TryAddWithoutValidation(Constants.HttpHeader_ContentType, "text/html");
            httpClient.DefaultRequestHeaders.TryAddWithoutValidation(Constants.HttpHeader_UserAgent, request.Headers[Constants.HttpHeader_UserAgent].ToString());

            if (!string.IsNullOrEmpty(Configuration.Token))
                httpClient.DefaultRequestHeaders.TryAddWithoutValidation(Constants.HttpHeader_XPrerenderToken, Configuration.Token);

            using (var webMessage = await httpClient.GetAsync(prerenderUrl))
            {
                var text = default(string);
                try
                {
                    response.StatusCode = (int)webMessage.StatusCode;
                    foreach (var keyValue in webMessage.Headers)
                    {
                        response.Headers[keyValue.Key] = new StringValues(keyValue.Value.ToArray());
                    }

                    using (var stream = await webMessage.Content.ReadAsStreamAsync())
                    using (var reader = new StreamReader(stream))
                    {
                        webMessage.EnsureSuccessStatusCode();
                        text = reader.ReadToEnd();
                    }
                }
                catch (Exception e)
                {
                    text = e.Message;
                }
                await response.WriteAsync(text);
            }
        }
    }
    else
    {
        await _next.Invoke(httpContext);
    }
}

• At last, let's take  a look at IsValidForPrerenderPage(HttpRequest request, IHttpRequestFeature requestFeature), This method is the same as PrerenderHttpModule class in ASP.NET.

private bool IsValidForPrerenderPage(HttpRequest request, IHttpRequestFeature requestFeature)
{
    var userAgent = request.Headers[Constants.HttpHeader_UserAgent];
    var rawUrl = requestFeature.RawTarget;
    var relativeUrl = request.Path.ToString();
    
    // check if follows google search engine suggestion
    if (request.Query.Keys.Any(a => a.Equals(Constants.EscapedFragment, StringComparison.OrdinalIgnoreCase)))
        return true;

    // check if has user agent
    if (string.IsNullOrEmpty(userAgent))
        return false;

    // check if it's crawler user agent.
    var crawlerUserAgentPattern = Configuration.CrawlerUserAgentPattern ?? Constants.CrawlerUserAgentPattern;
    if (string.IsNullOrEmpty(crawlerUserAgentPattern)
     || !Regex.IsMatch(userAgent, crawlerUserAgentPattern, RegexOptions.IgnorePatternWhitespace))
        return false;

    // check if the extenion matchs default extension
    if (Regex.IsMatch(relativeUrl, DefaultIgnoredExtensions, RegexOptions.IgnorePatternWhitespace))
        return false;

    if (!string.IsNullOrEmpty(Configuration.AdditionalExtensionPattern) && Regex.IsMatch(relativeUrl, Configuration.AdditionalExtensionPattern, RegexOptions.IgnorePatternWhitespace))
        return false;

    if (!string.IsNullOrEmpty(Configuration.BlackListPattern)
      && Regex.IsMatch(rawUrl, Configuration.BlackListPattern, RegexOptions.IgnorePatternWhitespace))
        return false;

    if (!string.IsNullOrEmpty(Configuration.WhiteListPattern)
      && Regex.IsMatch(rawUrl, Configuration.WhiteListPattern, RegexOptions.IgnorePatternWhitespace))
        return true;

    return false;

}

Use PrerenderMiddleware in ASP.NET Core Project

In order to use PrerenderMiddleware in ASP.NET Core project easily, I have created some extension method, so that we can easily setup it in Startup.cs

• AddPrerenderConfig()

            AddPrerenderConfig is used to add PrerenderConfiguration.json to IApplicationBuilder.

        /// <summary>
        /// Add PrerenderConfiguration.json to configuration.
        /// Or you can put the configuration in appsettings.json file either.
        /// </summary>
        /// <param name="builder"></param>
        /// <param name="jsonFileName"></param>
        /// <returns></returns>
        public static IConfigurationBuilder AddPrerenderConfig(this IConfigurationBuilder builder, string jsonFileName = "PrerenderConfiguration.json")
         => builder.AddJsonFile(jsonFileName, false, true);

• ConfigureSection()

             ConfigureSection is used to configure options into servicecollection, so that we can easily get it from servicecollection in the future.

        /// <summary>
        /// Configure Section into Service Collections
        /// </summary>
        /// <typeparam name="TOptions"></typeparam>
        /// <param name="serviceCollection"></param>
        /// <param name="configuration"></param>
        /// <param name="singletonOptions"></param>
        public static void ConfigureSection<TOptions>(this IServiceCollection serviceCollection, IConfiguration configuration, bool singletonOptions = true)
            where TOptions : class, new()
        {
            serviceCollection.Configure<TOptions>(configuration.GetSection(typeof(TOptions).Name));

            if (singletonOptions)
            {
                serviceCollection.AddSingleton<TOptions>(a => a.GetService<IOptions<TOptions>>().Value);
            }
        }

• UsePrerender()

            UsePrerender is used to register PrerenderMiddleware

        #region UsePrerender
        /// <summary>
        /// Use Prerender Middleware to prerender JavaScript logic before turn back.
        /// </summary>
        /// <param name="app"></param>
        /// <param name="configuration">Prerender Configuration, if this parameter is NULL, will get the PrerenderConfiguration from ServiceCollection</param>
        /// <returns></returns>
        public static IApplicationBuilder UsePrerender(this IApplicationBuilder app, PrerenderConfiguration configuration = null)
            => app.UseMiddleware<PrerenderMiddleware>(configuration ?? app.ApplicationServices.GetService<IOptions<PrerenderConfiguration>>().Value);
         // => app.Use(next => new PrerenderMiddleware(next, configuration).Invoke);
         // => app.Use(next => context => new PrerenderMiddleware(next, configuration).Invoke(context));  // either way.        
        #endregion

• With above extension methods, we can easily setup PrerenderMiddleware in Startup.cs
  • Step 1

public Startup(IHostingEnvironment env)
{
    var builder = new ConfigurationBuilder()
        .SetBasePath(env.ContentRootPath)
        .AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
        .AddJsonFile($"appsettings.{env.EnvironmentName}.json", optional: true)
        // Prerender Step 1: Add Prerender configuration Json file.
        .AddPrerenderConfig() 
        .AddEnvironmentVariables();
    Configuration = builder.Build();
}

• Step 2

public void ConfigureServices(IServiceCollection services)
{
    // Add framework services.
    services.AddMvc();

    // Prerender Step 2: Add Options.
    services.AddOptions();
    services.ConfigureSection<PrerenderConfiguration>(Configuration);
}

• Step 3

public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
{
    loggerFactory.AddConsole(Configuration.GetSection("Logging"));
    loggerFactory.AddDebug();

    // Prerender Step 3: UsePrerender, before others.
    app.UsePrerender();

    if (env.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
        app.UseBrowserLink();
    }
    else
    {
        app.UseExceptionHandler("/Home/Error");
    }

.............

PrerenderConfiguration.json

I have added PrerenderConfiguration.json file into ASP.NET Core project, then I can configure for prerender service.

The format of this json file is:

{
  "PrerenderConfiguration": {
    "ServiceUrl": "http://service.prerender.io",
    "Token": null,
    "CrawlerUserAgentPattern": null,
    "WhiteListPattern": null,
    "BlackListPattern": "lib|css|js",
    "AdditionalExtensionPattern": null,
    "ProxyUrl": null,
    "ProxyPort": 80
  }
}

You can go to my github wiki page to get more details about each option: Configuration & Check Priority  

Nuget Package

I have created a nuget package, which is very convenient if you don't want to dive deep into the source code. 

• Install Nuget Package in your project.

           Visual Studio -> Tools -> Nuget Package Manager -> Package Manager Console.

Install-Package DotNetCoreOpen.PrerenderMiddleware

           If you want to take a look more detail about this package, you can go https://www.nuget.org/packages/DotNetCoreOpen.PrerenderMiddleware/

• Use PrerenderMiddleware and configure PrerenderConfiguration.json for prerender service.

            I have fully documented how to do this in my github wiki page, you can go there take a look.

1. Prerender Middleware for ASP.NET Core

2. Configuration & Check Priority            

• Done, try it out.

Github Project

I also have created a github project to host all source code includes sample code for testing: https://github.com/dingyuliang/prerender-dotnet, in this project, it includes ASP.NET HttpModule, ASP.NET Core Middleware, IIS Configuration 3 different solution. 

For ASP.NET Core Middleware, you can go to https://github.com/dingyuliang/prerender-dotnet/tree/master/src/DotNetCorePrerender 

Prerender Related

1. Use Prerender to improve AngularJS SEO
2. Setup Prerender Service for JavaScript SEO
3. Prerender Implementation Best Practice
4. Prerender Application Level Middleware - ASP.NET HttpModule
5. Prerender Application Level Middleware - ASP.NET Core Middleware

------------------------------------------------------------------------------------------------