netcore实践：跨平台动态加载native组件

缘起netcore框架下实现基于zmq的应用。

在.net framework时代，我们进行zmq开发由很多的选择，比较常用的有clrzmq4和NetMQ。 其中clrzmq是基于libzmq的Interop包装，

NetMQ是100%C#的zmq实现（基于AsyncIO组件）。以上两种组件我都有过应用，孰优孰劣各有千秋，本文就不详谈了。

回归正题，netcore下使用zmq首先也是想到引用上述组件，实践后发现clrzmq暂时没有基于netstandard或者netcore的支持，而NetMQ做的比较好，已经基于

netstandard1.3进行了支持。

一番折腾，搭程序，配环境。。。而后 dotnet run ，在windows下正常呈现了zmq的各项功能。

于是继续dotnet publlish，通过Wnscp拷贝到centos7下执行。立即报错，

挺意外的，本以为NetMQ已经基于netstandard进行了支持，也应该对跨平台进行支持。 可事实是AsyncIO的IOControl方法not supported on linux platform/

无奈，网上搜了会，也没找到任何关于netcore在linux下进行zmq的相关内容， 事实上也没有看到AsyncIO或者NetMQ有关于跨平台支持的说明。

既然现有方式行不通那就只好自己动手了，自己操刀通过Interop包装libzmq来实现跨平台的zmq应用吧！

首先看下libzmq的组件目录： 按x86和x64平台分为i386文件夹和amd64文件夹，且都包含windos下的dll组件和linux下的so组件

这挺难办了，要想做好还得考虑平台类型 和 操作系统类型，  还要想想 netcore里有没有相关API提供。 

先是网上搜了圈，也极少有关于netcore进行平台判断相关内容。根据以往在framework下的经验，直接到https://apisof.net搜索相关关键字

：OSPlatform

非常不错，netcore已经提供了，同理搜索了 OSArchitecture 、DllImport 等都发现netcore1.1版本已经实现了相关api （说白了是netstandard已经实现了相关API）

准备就绪，直接开干,首先是平台相关信息判断，并加载目标组件 （具体方式请参照如下代码）

   class ZmqNative
    {
        private const string LibraryName = "libzmq";

        const int RTLD_NOW = 2; // for dlopen's flags
        const int RTLD_GLOBAL = 8;
        [DllImport(@"libdl")]
        static extern IntPtr dlopen(string filename, int flags);
        [DllImport("libdl")]
        protected static extern IntPtr dlsym(IntPtr handle, string symbol);


        [DllImport("kernel32.dll")]
        static extern IntPtr LoadLibrary(string filename);

        private static IntPtr LibPtr = IntPtr.Zero;
        static ZmqNative()
        {
           
            Console.WriteLine("OSArchitecture:{0}",RuntimeInformation.OSArchitecture);
            try {
                var libPath = @"i386";
                if (RuntimeInformation.OSArchitecture == Architecture.X86)
                {
                    libPath = @"i386";
                }
                else if (RuntimeInformation.OSArchitecture == Architecture.X64)
                {
                    libPath = @"amd64";
                }
                else
                {
                    Console.WriteLine("OSArchitecture not suported!");
                }

                if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
                {
                    var libName = $"{AppContext.BaseDirectory}\{libPath}\{LibraryName}.dll";
                    Console.WriteLine("windows:{0}", libName);
                    LibPtr = LoadLibrary(libName);
                   
                }
                else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
                {
                    var libName = $"{AppContext.BaseDirectory}/{libPath}/{LibraryName}.so";
                    Console.WriteLine("linux:{0}", libName);
                    LibPtr = dlopen(libName, RTLD_NOW|RTLD_GLOBAL);

                    
                    if(LibPtr!=IntPtr.Zero)
                    {
                        var ptr1 = dlsym(LibPtr, "zmq_ctx_new");
                        context = Marshal.GetDelegateForFunctionPointer<ZmqContext>(ptr1) ;

                        var ptr2 = dlsym(LibPtr, "zmq_socket");
                        socket = Marshal.GetDelegateForFunctionPointer<ZmqSocket>(ptr2);

                        var ptr3 = dlsym(LibPtr, "zmq_connect");
                        connect = Marshal.GetDelegateForFunctionPointer<ZmqConnect>(ptr3);

                    }
    }
                else
                {
                    Console.WriteLine("OSPlatform not suported!");
                }
                if (LibPtr != IntPtr.Zero)
                    Console.WriteLine("load zmqlib success!");
            }
            catch(Exception ex)
            {
                Console.WriteLine("load zmqlib error:
{0}",ex);
            }
        }

       


        public delegate IntPtr ZmqContext();

        [DllImport(LibraryName, EntryPoint = "zmq_ctx_new", CallingConvention=CallingConvention.Cdecl)]
        public static extern IntPtr zmq_ctx_new();
        public static ZmqContext context = null;


        public delegate IntPtr ZmqSocket(IntPtr context, Int32 type);
        [DllImport(LibraryName, EntryPoint = "zmq_socket", CallingConvention = CallingConvention.Cdecl)]
        public static extern IntPtr zmq_socket(IntPtr context, Int32 type);
        public static ZmqSocket socket = null;


        public delegate Int32 ZmqConnect(IntPtr socket, IntPtr endpoint);
        [DllImport(LibraryName, EntryPoint = "zmq_connect", CallingConvention = CallingConvention.Cdecl)]
        public static extern Int32 zmq_connect(IntPtr socket, IntPtr endpoint);
        public static ZmqConnect connect = null;


        [DllImport(LibraryName, EntryPoint = "zmq_errno", CallingConvention = CallingConvention.Cdecl)]
        public static extern Int32 zmq_errno();

        [DllImport(LibraryName, EntryPoint = "zmq_strerror", CallingConvention = CallingConvention.Cdecl)]
        public static extern IntPtr zmq_strerror(int errnum);
    }

以上为测试代码，请自动忽略代码质量！

简单解释下，如上代码通过平台判断，动态加载组件，采用LoadLibaray的方式。 有心的同学可能会发现几个delegate并且在Linux部分内通过dlsym获取了函数指针，具体原因下面会讲。

以上测试代码，在windows平台下同样正常无误， 而在linux下还是遇到几个小坑~~容我慢慢道来：

1、通过DllImport进行Interop的时候，组件路径必须是确定的，这就引起了如何动态加载不同目录下组件的问题；

    好在windows平台下通过LoadLibaray加载dll到进程空间后，DllImport标记的函数就从进程空间查找，不会重复import组件了。

   而同样的原理在linux下用dlopen却不能实现，还是会提示找不到组件

2、初次部署centos7上时，报找不到libdl.so组件问题，主要原因是系统下没有glibc的原因，该问题可以通过yum安装glibc的方式解决;

  

//先查找系统内是否存在组件
$ sudo find / -name libdl*

//如不存在则安装glibc
# yum install glibc

#安装完毕后进行链接
$ sudo ln -s /usr/lib64/libdl.so.2 /usr/lib64/libdl

3、解决了libdl组件问题后，继续运行还是会发现报找不到libzmq组件的问题，实际就产生了问题1中所描述的，在linux系统下dlopen后，Interop过的函数并不会从进程空间查找。

为了解决上面遇到的问题，我们还有一条办法，就是创建 delegate ， 并且通过LoadLibaray组件后通过GetProcAddress方式获取函数指针了。  具体的解决方案在上述测试代码已经体现了，这里就不过多解释了。

以上，就全部解决了在 netcore框架基础上进行跨平台native组件应用的问题。 真实测试结果如图：

请主动忽略初zmq应用外的其他信息， 本次测试一同测试了通过App入口启动webapi +  websockets + zmq ，api创建为aspnetcore在Web.dll内，websockets在Lib.dll内，zmq在App.dll内。

补充下完整测试代码：

using System;
using System.Collections.Generic;
using System.Text;
using System.Runtime.InteropServices;

namespace App
{
    
    class ZmqNative
    {
        private const string LibraryName = "libzmq";

        const int RTLD_NOW = 2; // for dlopen's flags
        const int RTLD_GLOBAL = 8;
        [DllImport(@"libdl")]
        static extern IntPtr dlopen(string filename, int flags);
        [DllImport("libdl")]
        protected static extern IntPtr dlsym(IntPtr handle, string symbol);


        [DllImport("kernel32.dll")]
        static extern IntPtr LoadLibrary(string filename);

        private static IntPtr LibPtr = IntPtr.Zero;
        static ZmqNative()
        {
           
            Console.WriteLine("OSArchitecture:{0}",RuntimeInformation.OSArchitecture);
            try {
                var libPath = @"i386";
                if (RuntimeInformation.OSArchitecture == Architecture.X86)
                {
                    libPath = @"i386";
                }
                else if (RuntimeInformation.OSArchitecture == Architecture.X64)
                {
                    libPath = @"amd64";
                }
                else
                {
                    Console.WriteLine("OSArchitecture not suported!");
                }

                if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
                {
                    var libName = $"{AppContext.BaseDirectory}\{libPath}\{LibraryName}.dll";
                    Console.WriteLine("windows:{0}", libName);
                    LibPtr = LoadLibrary(libName);
                   
                }
                else if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
                {
                    var libName = $"{AppContext.BaseDirectory}/{libPath}/{LibraryName}.so";
                    Console.WriteLine("linux:{0}", libName);
                    LibPtr = dlopen(libName, RTLD_NOW|RTLD_GLOBAL);

                    
                    if(LibPtr!=IntPtr.Zero)
                    {
                        var ptr1 = dlsym(LibPtr, "zmq_ctx_new");
                        context = Marshal.GetDelegateForFunctionPointer<ZmqContext>(ptr1) ;

                        var ptr2 = dlsym(LibPtr, "zmq_socket");
                        socket = Marshal.GetDelegateForFunctionPointer<ZmqSocket>(ptr2);

                        var ptr3 = dlsym(LibPtr, "zmq_connect");
                        connect = Marshal.GetDelegateForFunctionPointer<ZmqConnect>(ptr3);

                    }
    }
                else
                {
                    Console.WriteLine("OSPlatform not suported!");
                }
                if (LibPtr != IntPtr.Zero)
                    Console.WriteLine("load zmqlib success!");
            }
            catch(Exception ex)
            {
                Console.WriteLine("load zmqlib error:
{0}",ex);
            }
        }

       


        public delegate IntPtr ZmqContext();

        [DllImport(LibraryName, EntryPoint = "zmq_ctx_new", CallingConvention=CallingConvention.Cdecl)]
        public static extern IntPtr zmq_ctx_new();
        public static ZmqContext context = null;


        public delegate IntPtr ZmqSocket(IntPtr context, Int32 type);
        [DllImport(LibraryName, EntryPoint = "zmq_socket", CallingConvention = CallingConvention.Cdecl)]
        public static extern IntPtr zmq_socket(IntPtr context, Int32 type);
        public static ZmqSocket socket = null;


        public delegate Int32 ZmqConnect(IntPtr socket, IntPtr endpoint);
        [DllImport(LibraryName, EntryPoint = "zmq_connect", CallingConvention = CallingConvention.Cdecl)]
        public static extern Int32 zmq_connect(IntPtr socket, IntPtr endpoint);
        public static ZmqConnect connect = null;


        [DllImport(LibraryName, EntryPoint = "zmq_errno", CallingConvention = CallingConvention.Cdecl)]
        public static extern Int32 zmq_errno();

        [DllImport(LibraryName, EntryPoint = "zmq_strerror", CallingConvention = CallingConvention.Cdecl)]
        public static extern IntPtr zmq_strerror(int errnum);
    }

    class ZContext
    {
        private static IntPtr _context = IntPtr.Zero;
        static ZContext()
        {
            if (ZmqNative.context != null)
            {
                _context = ZmqNative.context();
            }
            else
            {
                _context = ZmqNative.zmq_ctx_new();
            }
            
            if (_context == IntPtr.Zero)
            {
                Console.WriteLine("zerror:{0}", ZError.GetLastError());
            }
        }

        public static IntPtr Current
        {
            get
            {
                return _context;
            }
        }
    }


    class ZError
    {
        public static string GetLastError()
        {
            var error = string.Empty;
            var no = ZmqNative.zmq_errno();
            if (no != 0)
            {
                var ptr = ZmqNative.zmq_strerror(no);
                error = Marshal.PtrToStringUTF8(ptr);
            }
            return error;
        }
    }

    enum ZmqSocketType :int {
        SUB = 2
    }

    class ClrZmq
    {
        private IntPtr zmq = IntPtr.Zero;
        public ClrZmq(string url) {
            Url = url;
        }

        public string Url { get; private set; }

        private bool _IsStarted= false;
        public bool IsStarted
        {
            get
            {
                return _IsStarted;
            }
        }

        public void Start() {
            if (IsStarted) return;
            try
            {
                if (ZmqNative.socket != null)
                    zmq = ZmqNative.socket(ZContext.Current, (int)ZmqSocketType.SUB);
                else
                    zmq = ZmqNative.zmq_socket(ZContext.Current, (int)ZmqSocketType.SUB);
                if (zmq == IntPtr.Zero)
                {
                    //error
                    Console.WriteLine("zerror:{0}",ZError.GetLastError());
                    return;
                }
                Console.WriteLine("create mq success!");
                
                var str =  Marshal.StringToHGlobalAnsi(Url);

                var ret = 0;
                if (ZmqNative.connect != null)
                    ret = ZmqNative.connect(zmq, str);
                else
                    ret = ZmqNative.zmq_connect(zmq, str);
                if ( ret!= 0)
                {
                    //error
                    Console.WriteLine("zerror:{0}",ZError.GetLastError());
                    return;
                }
                Marshal.FreeHGlobal(str);

                Console.WriteLine("connect zmq success!");

            }
            catch (Exception ex) {
                Console.WriteLine(ex);
            }
        }
    }
}