[转]Converting a C library to gyp

gyp, short for "Generate Your Projects", is a build system written by Google that was specifically designed to build their Chromium web browser.gyp makes it easy to develop such a large project because the developers can create and test each piece individually by defining self-contained "targets". Then at Release-time, gyp makes it easy to stitch the "targets" together into a single statically linked executable. Ryan Dahl once called it "the module-system of C".

Node.js recently adopted gyp as its build system because V8 was already using it (through Chromium), and because of its explicit Windows support. node-gyp was created to help the community create node.js native addons by writing simple gyp files. This article is targeted towards Node.js native module authors, so hopefully you are somewhat familiar will all of this already.

gyp-ify all the things!

One thing that I want to see more of in the node community from native module authors is the additional work done to convert the dependencies of the native module to use gyp to build, and bundle them with the native module. This allows the library to be statically linked to your native module, rather than dynamically linked. There are pros and cons to both kinds of linking, but node.js has gone with static linking, mostly because of portability and to ease Windows support.

Now converting the build system of some odd C library that you haven't written can be a daunting task, so that's what this article hopes to address. I'll explain my method of converting a C library originally using autotools to build using gyp, and hopefully you will do the same (or improve on my method!) in order to make your native module more portable. Your users will thank you for the easier installation process (i.e. not having toapt-get install some external libraries).

libmp3lame, I choose you!

Our test subject for this article will be libmp3lame, a library that does MP3 encoding and decoding. This is a nice, medium complexity library that will be a good demonstration of my method.

Baseline gyp file

We're gonna start with a baseline gyp file to use. Let's call it libmp3lame.gyp in this case:

{
  'target_defaults': {
    'default_configuration': 'Debug',
    'configurations': {
      'Debug': {
        'defines': [ 'DEBUG', '_DEBUG' ],
        'msvs_settings': {
          'VCCLCompilerTool': {
            'RuntimeLibrary': 1, # static debug
          },
        },
      },
      'Release': {
        'defines': [ 'NDEBUG' ],
        'msvs_settings': {
          'VCCLCompilerTool': {
            'RuntimeLibrary': 0, # static release
          },
        },
      }
    },
    'msvs_settings': {
      'VCLinkerTool': {
        'GenerateDebugInformation': 'true',
      },
    },
  },

  'targets': [
  ]
}

The msvs_settings configuration stuff at the top ensures that we end up with statically built libraries and executables on Windows. This boilerplate isn't strictly necessary, but it matches up with some defaults set in node.js' common.gypi file, and I like to stay consistent. The emptytargets array will end up with the definitions to build libmp3lame, as well as a simple test program.

Adding the platform-specific files

Most C libraries have a ./configure step, and usually that step will create some kind of "config.h" file of some sort, which has appropriate defines for your current platform. So our strategy for converting to gyp is to pre-generate these "config.h" files on all your supported platforms and architectures, and save them for use when building with gyp.

So for example, the first platform + architecture combo I will get working is Mac OS X 64-bit. Therefore, the resulting "config.h" file will be placed atconfig/mac/x64/config.h. In the end, we want to support if possible this baseline of platforms and architectures:

OS	Arch	config.h path
Linux	ARM	`config/linux/arm`
Linux	32-bit	`config/linux/ia32`
Linux	64-bit	`config/linux/x64`
Mac OS X	32-bit	`config/mac/ia32`
Mac OS X	64-bit	`config/mac/x64`
Solaris	32-bit	`config/solaris/ia32`
Solaris	64-bit	`config/solaris/x64`
Windows	32-bit	`config/win/ia32`
Windows	64-bit	`config/win/x64`

Some of the combinations might not be supported by the library you are working with, but (except for linux-arm) this list matches up with the releases of the official node.js binary tarballs, and what node supports, you should try to support too!

Running `./configure`

With C libraries configured using autotools, I've been using a standard set of switches, and adding library-specific ones as necessary to try to get as close of a static build as gyp is going to make. The basic set of flags I use are:

$ ./configure --enable-static --disable-shared --with-pic

These flags are basically saying, build the static version of the library (the .a/.lib files), and don't bother with the shared version (the.so/.dylib/.dll files). Also, build "position independent code".

In some cases, that will be all that you need. But for libmp3lame, I'm also gonna need to specify a few additional flags to disable some unnecessary stuff like the lame program frontend.

$ ./configure --enable-static --disable-shared --with-pic 
              --disable-rpath --disable-frontend --disable-gtktest

After running configure on my 64-bit Mac OS X machine, I'm going to copy the resulting config.h file to config/mac/x64/config.h. The"config.h" file could be called something else for your library, like "expat_config.h" for libexpat for example. In some cases there could be more than 1 ".h" file generated, like libffi had 3 per platform+arch combo. So you kinda just need to take a look around, visually inspect the configureoutput, and see what gets generated.

The first change we're going to make to the gyp file is to make sure that these platform-specific header files are included:

diff --git a/libmp3lame.gyp b/libmp3lame.gyp
index d30085c..cdc5b9b 100644
--- a/libmp3lame.gyp
+++ b/libmp3lame.gyp
@@ -1,4 +1,6 @@
 {
+  'variables': { 'target_arch%': 'ia32' },
+
   'target_defaults': {
     'default_configuration': 'Debug',
     'configurations': {
@@ -24,6 +26,10 @@
         'GenerateDebugInformation': 'true',
       },
     },
+    'include_dirs': [
+      # platform and arch-specific headers
+      'config/<(OS)/<(target_arch)'
+    ],
   },

   'targets': [

So you essentially need to repeat these "configure & copy to config" steps for all of the platforms + arch combinations listed in the table above. I usually do things a little out of order though, and move on to making the library build (the next section) before worrying about the other platforms...

Defining the `targets`

So usually the next step when building these libraries is to run the make command. Sometimes you will need something additonal like overwriting a specific variable, but in this case we'll just invoke it straight up.

$ make

The key to defining proper targets is, again, visually inspecting the make output and seeing exactly which files are compiled, and defines are added, and which flags are used. You can also get part of the picture by looking at the Makefile.am files in the library.

So based off the make output in the case of libmp3lame, there's going to be 3 total "targets" that will be defined:

mpglib - libmp3lame's decoder (an old version of libmpg123 apparently)
liblamevectorroutines - some internal routines used by libmp3lame
libmp3lame the API frontend for libmp3lame

Each of these "targets" are like "modules" in node.js, they're their own independent functional component that make up part of the library.

Ok, so let's define those targets. This is a lot of new gyp code here, so be sure to study it until it makes sense to you:

diff --git a/libmp3lame.gyp b/libmp3lame.gyp
index cdc5b9b..f30319b 100644
--- a/libmp3lame.gyp
+++ b/libmp3lame.gyp
@@ -27,11 +27,88 @@
       },
     },
     'include_dirs': [
+      'include',
+      'mpglib',
+      'libmp3lame',
+      'libmp3lame/vector',
       # platform and arch-specific headers
       'config/<(OS)/<(target_arch)'
     ],
+    'defines': [
+      'PIC',
+      'HAVE_CONFIG_H'
+    ],
   },

   'targets': [
+    # mpglib
+    {
+      'target_name': 'mpgdecoder',
+      'product_prefix': 'lib',
+      'type': 'static_library',
+      'sources': [
+        'mpglib/common.c',
+        'mpglib/dct64_i386.c',
+        'mpglib/decode_i386.c',
+        'mpglib/interface.c',
+        'mpglib/layer1.c',
+        'mpglib/layer2.c',
+        'mpglib/layer3.c',
+        'mpglib/tabinit.c',
+      ],
+    },
+
+    # liblamevectorroutines
+    {
+      'target_name': 'lamevectorroutines',
+      'product_prefix': 'lib',
+      'type': 'static_library',
+      'sources': [
+        'libmp3lame/vector/xmm_quantize_sub.c'
+      ],
+    },
+
+    # libmp3lame
+    {
+      'target_name': 'mp3lame',
+      'product_prefix': 'lib',
+      'type': 'static_library',
+      'sources': [
+        'libmp3lame/VbrTag.c',
+        'libmp3lame/bitstream.c',
+        'libmp3lame/encoder.c',
+        'libmp3lame/fft.c',
+        'libmp3lame/gain_analysis.c',
+        'libmp3lame/id3tag.c',
+        'libmp3lame/lame.c',
+        'libmp3lame/newmdct.c',
+        'libmp3lame/presets.c',
+        'libmp3lame/psymodel.c',
+        'libmp3lame/quantize.c',
+        'libmp3lame/quantize_pvt.c',
+        'libmp3lame/reservoir.c',
+        'libmp3lame/set_get.c',
+        'libmp3lame/tables.c',
+        'libmp3lame/takehiro.c',
+        'libmp3lame/util.c',
+        'libmp3lame/vbrquantize.c',
+        'libmp3lame/version.c',
+        'libmp3lame/mpglib_interface.c',
+      ],
+      'dependencies': [
+        'lamevectorroutines',
+        'mpgdecoder',
+      ],
+      'direct_dependent_settings': {
+        'include_dirs': [
+          'include',
+          'mpglib',
+          'libmp3lame',
+          'libmp3lame/vector',
+          # platform and arch-specific headers
+          'config/<(OS)/<(target_arch)'
+        ],
+      },
+    },
   ]
 }

One important thing to note is how we make the "mp3lame" target depend on "lamevectorroutines" and "mpgdecoder". This allows us to only have to specify "mp3lame" as a dependency in the node module's binding.gyp file. Also note how the direct_dependent_settings are defined within that target as well, for the same reason.

Try it out

At this point, we should be able to build libmp3lame, at least on 64-bit OS X. I like to add a very quick little test program of some sort to the gyp file, to make sure that things get built and statically linked like we expect, and that it works at all.

We have to add an executable target to the gyp file:

diff --git a/libmp3lame.gyp b/libmp3lame.gyp
index f30319b..de2984d 100644
--- a/libmp3lame.gyp
+++ b/libmp3lame.gyp
@@ -110,5 +110,13 @@
         ],
       },
     },
+
+    # test program that prints the version number
+    {
+      'target_name': 'test',
+      'type': 'executable',
+      'dependencies': [ 'mp3lame' ],
+      'sources': [ 'test.c' ]
+    },
   ]
 }
diff --git a/test.c b/test.c
new file mode 100644
index 0000000..1fb3dbf
--- /dev/null
+++ b/test.c
@@ -0,0 +1,7 @@
+#include <stdio.h>
+#include "lame.h"
+
+int main () {
+  printf("get_lame_version(): %s
", get_lame_version());
+  return 0;
+}

This super basic test program simply prints out the result of the get_lame_version() function. Let's download gyp, configure our project, and run this test program:

$ svn co http://gyp.googlecode.com/svn/trunk gyp  # download gyp
$ ./gyp/gyp -f make --depth=. libmp3lame.gyp  # run gyp for the `make` target
$ make V=1  # verbose build
$ ./out/Debug/test  # run the `test` program
get_lame_version(): 3.99.5

Success!

Windows support

So after generating all the config files for the other platforms (mac, linux, solaris), the last one is Windows (win). Windows usually takes some special dedication to get working. At the very least, a Google search asking "how to build library xxxx on Windows" should yield something.

If there are any major differences, you can introduce a conditions block in your gyp file targeting Windows like so:

    'conditions': [
      ['OS=="win"', {
        # windows-specific gyp stuff goes here...
      }],
    ],

The config files

In the case of libmp3lame, the developers have included a hand-crafted configMS.h file, which is the config.h file for Windows, both 32-bit and 64-bit. So all we have to do this time is copy this file to config/win/ia32 and config/win/x64. Piece of cake!

In some cases though you might have to actually run the ./configure script using version of bash compiled for Windows. This was the case forlibffi.

The source files and defines

It's ideal to actually run the build process for your library, ideally using MSVC, doing the method that the library authors suggest. That way you can inspect that build output and be able to determine any differences for the build on Windows.

For libmp3lame, there's a Makefile.MSVC file that is compatible with the nmake command on Windows. If we run it and inspect the output, we can see that there are a few different defines used on Windows, but all the same sources files.

Let's adjust the gyp file to reflect that:

diff --git a/libmp3lame.gyp b/libmp3lame.gyp
index de2984d..bb4b010 100644
--- a/libmp3lame.gyp
+++ b/libmp3lame.gyp
@@ -38,6 +38,16 @@
       'PIC',
       'HAVE_CONFIG_H'
     ],
+    'conditions': [
+      ['OS=="win"', {
+        'defines': [
+          'TAKEHIRO_IEEE754_HACK',
+          'FLOAT8=float',
+          'REAL_IS_FLOAT=1',
+          'BS_FORMAT=BINARY',
+        ]
+      }]
+    ],
   },

   'targets': [

Sometimes additional sources files are used (or not used) so in that case you would add a sources block within that conditional.

At this point, Windows should work as well:

C:UsersNathanDesktoplame> svn co http://gyp.googlecode.com/svn/trunk gyp
C:UsersNathanDesktoplame> gypgyp.bat -f msvs -G msvs_version=2010 --depth=. libmp3lame.gyp
C:UsersNathanDesktoplame> msbuild libmp3lame.sln
C:UsersNathanDesktoplame> Debug	est.exe
get_lame_version(): 3.99.5

Excellent!

The finished `gyp` file

So we're done! At this point, we have our completed gyp file:

{
  'variables': { 'target_arch%': 'ia32' },

  'target_defaults': {
    'default_configuration': 'Debug',
    'configurations': {
      'Debug': {
        'defines': [ 'DEBUG', '_DEBUG' ],
        'msvs_settings': {
          'VCCLCompilerTool': {
            'RuntimeLibrary': 1, # static debug
          },
        },
      },
      'Release': {
        'defines': [ 'NDEBUG' ],
        'msvs_settings': {
          'VCCLCompilerTool': {
            'RuntimeLibrary': 0, # static release
          },
        },
      }
    },
    'msvs_settings': {
      'VCLinkerTool': {
        'GenerateDebugInformation': 'true',
      },
    },
    'include_dirs': [
      'include',
      'mpglib',
      'libmp3lame',
      'libmp3lame/vector',
      # platform and arch-specific headers
      'config/<(OS)/<(target_arch)'
    ],
    'defines': [
      'PIC',
      'HAVE_CONFIG_H'
    ],
    'conditions': [
      ['OS=="win"', {
        'defines': [
          'TAKEHIRO_IEEE754_HACK',
          'FLOAT8=float',
          'REAL_IS_FLOAT=1',
          'BS_FORMAT=BINARY',
        ]
      }]
    ],
  },

  'targets': [
    # mpglib
    {
      'target_name': 'mpgdecoder',
      'product_prefix': 'lib',
      'type': 'static_library',
      'sources': [
        'mpglib/common.c',
        'mpglib/dct64_i386.c',
        'mpglib/decode_i386.c',
        'mpglib/interface.c',
        'mpglib/layer1.c',
        'mpglib/layer2.c',
        'mpglib/layer3.c',
        'mpglib/tabinit.c',
      ],
    },

    # liblamevectorroutines
    {
      'target_name': 'lamevectorroutines',
      'product_prefix': 'lib',
      'type': 'static_library',
      'sources': [
        'libmp3lame/vector/xmm_quantize_sub.c'
      ],
    },

    # libmp3lame
    {
      'target_name': 'mp3lame',
      'product_prefix': 'lib',
      'type': 'static_library',
      'sources': [
        'libmp3lame/VbrTag.c',
        'libmp3lame/bitstream.c',
        'libmp3lame/encoder.c',
        'libmp3lame/fft.c',
        'libmp3lame/gain_analysis.c',
        'libmp3lame/id3tag.c',
        'libmp3lame/lame.c',
        'libmp3lame/newmdct.c',
        'libmp3lame/presets.c',
        'libmp3lame/psymodel.c',
        'libmp3lame/quantize.c',
        'libmp3lame/quantize_pvt.c',
        'libmp3lame/reservoir.c',
        'libmp3lame/set_get.c',
        'libmp3lame/tables.c',
        'libmp3lame/takehiro.c',
        'libmp3lame/util.c',
        'libmp3lame/vbrquantize.c',
        'libmp3lame/version.c',
        'libmp3lame/mpglib_interface.c',
      ],
      'dependencies': [
        'lamevectorroutines',
        'mpgdecoder',
      ],
      'direct_dependent_settings': {
        'include_dirs': [
          'include',
          'mpglib',
          'libmp3lame',
          'libmp3lame/vector',
          # platform and arch-specific headers
          'config/<(OS)/<(target_arch)'
        ],
      },
    },

    # test program that prints the version number
    {
      'target_name': 'test',
      'type': 'executable',
      'dependencies': [ 'mp3lame' ],
      'sources': [ 'test.c' ]
    },
  ]
}

Are we done yet?

No still not done... now we have to add it as a dependency to our node.js native addon.

Following the model of node.js core, I'm gonna place libmp3lame with our gyp stuff in deps/lame of the node-lame repo. After that I can add it as a dependency to the main binding.gyp file like so:

{
  'targets': [
    {
      'target_name': 'bindings',
      'sources': [
        'src/bindings.cc',
        'src/node_lame.cc'
      ],
      'dependencies': [
        'deps/lame/libmp3lame.gyp:mp3lame'
      ]
    }
  ]
}

The binding.gyp file remains nice and simple. NOW we're done. Grab a pint!