For years, NVDA has used Py2exe to package Python code into something that is executable on a system that doesn't have Python installed. For Python 2.7, we have been at Py2exe 0.6.9 (introduced in 2008) for a long time. For Python 3, a new version has been introduced in 2014 which looks like a rewrite. The last official release (0.9.2.2) was introduced in 2014.
Starting with Python 3.6 (end 2016) however, Python contains a backwards incompatible change of the byte code format that is not supported by the official py2exe.
Note that many parts of the build process of NVDA are handled by SCons. This includes:
- Compiling, linking and code signing all the additional libraries, such as NVDAHelper, liblouis, minhook, ESPeak
- Compiling ESpeak dictionaries, CLDR data and gettext translations
- Converting documentation to HTML using TXT2Tags, see #8734
In summary, Scons deals with everything that is required to run NVDA from source, Py2exe is not involved in that process. Py2exe gets called as soon as a distribution of NVDA is created. The portable distribution that Py2exe creates is then packed into a launcher using NSiS.
Requirements
Before deciding on what packaging tool to use, this section lists the requirements for a packaging tool, based on the work that is now performed by Py2exe.
-
Byte compile every source file into a *.pyc or *.pyo file. Note that for Python 3, only the *pyc files remain.
-
Byte compile every Python dependency (comtypes, pyserial, etc.) required to run NVDA.
-
Collect Python extension modules (*.pyd files)
-
Bundle the several byte compiled files into a library.zip file in the main folder of the distribution.
-
Exclude several unneeded modules
-
Create executables from certain source files:
source destination(s) description nvda.pyw nvda_noUIAccess.exe, nvda_uiAccess.exe Main executables (see below) nvda_eoaProxy.pyw nvda_eoaProxy.exe Used to integrate in Windows 7 EOA center nvda_slave.pyw nvda_slave.exe Used for add-on installation from shell, elevated rights during installation, etc. -
Bundle a manifest with the executables. Theoretically, this can also be done using SCons, though in contrast to what I thought earlier, there is no code within the current scons structure of NVDA that can be reused for this.
-
Make sure that the UIAccess flag is added to the manifest of nvda_uiAccess.exe and nvda_eoaProxy.exe. Strictly spoken, this is code that has been added to our subclass of build_exe.py2exe in order to inject the UIAccess flag into the manifest created by py2exe. However, droppign py2exe in favour of another tool might force us to completely rewrite this logic.
-
Embed the NVDA logo into the executables as icon.
-
Set several version info flags in the version info resource of the executables.
-
Collect several system dll's not available on every system. This includes visual C++ redistributables.
-
Collect data files created using SCons, such as several libraries, language files, documentation, etc.
Alternatives
According to research and ideas from others, there are four packaging/freezing options that can be used with Python 3. The following sections provide short descriptions along with pros and cons.
Py2exe fork by @albertosottile
See https://github.com/albertosottile/py2exe
Thankfully, @albertosottile contacted us in #8375 (comment), commenting that he picked up Py2exe where it has been abandoned in 2014. This resulted into a release of Py2exe 0.93.0. This version is said to work with Python 3.6 and 3.7.
Pros
- Stick to the same tool as used before. Though heavily changed under the hood, it still promises to be mostly compatible with older setup files. According to the original readme:
It is planned to achive full compatibility with the setup-scripts for Python 2; however this is probably not yet the case.
- Probably the least efford if @albertosottile's fork proves to be working.
Cons
- A look at the code of Py2exe for Python 3 seems to reveal that the support to create manifests is not yet implemented, that is, there is lots of commented out code that seems to origin from the Python 2 version. Having said that, the logic to embed resource files in the executable still seems to be available and an example setup files shows an alternative method to embed the manifest. It's just not compatible with the current approach in our setup.py.
- @albertosottile's fork of this project is pretty new. Some details:
- It is not yet listed on PyPI. Note the version number
- It has only 3 watchers, 5 stars and 2 forks on github
- It has an open issue regarding wxPython support.
Requirements scheme
| Feature | supported |
|---|---|
| Create multiple executables | Yes |
| Bundle manifest | Yes, but differently than before, might require pr to upstream |
| UIAccess | No explicit support, see manifest |
| Embed logo | Yes |
| Set version info | Yes |
| Collect system dll's | Yes |
| Collect external Python dependencies and *.pyd files | Yes |
| Collect data files | Yes |
| Library file | Yes |
PyInstaller
PyInstaller is an alternative tool to build executables. It also supports byte code encryption, though that seems not very useful in the case of NVDA.
Pros
- This tool supports most functionality we need out of the box, including stuff that's now handled by scons, such as:
- Code signing
- Full automatic support for CRTs
- Manifests with UAC
- It is very actively maintained.
Cons
-
Building multiple executables at once seems to be broken
-
It does not integrate very well with distutils. It looks like the PyInstaller documentation suggests using a batch file to write down command line commands. Alternatively, you can use a spec file. Writing a setup.py for your package seems unsupported. Having said that, we're now calling a python interpretter from scons to execute the setup script, so basically, we're using a command line call already. This is therefore no show stopper, just a change of approach.
-
More importantly, I've seen ridiculous tracebacks with one of my test applications referring to pyinstaller:
Traceback (most recent call last): File "main.py", line 1, in <module> File "c:python36libsite-packagesPyInstallerloaderpyimod03_importers.py", line 631, in exec_module exec(bytecode, module.__dict__) File "core.py", line 5, in <module> File "c:python36libsite-packagesPyInstallerloaderpyimod03_importers.py", line 631, in exec_module exec(bytecode, module.__dict__)It looks like every import is somehow redirected to pyInstaller. I'm pretty sure this will cause issues with our use of comtypes com interface modules, globalPlugins, etc. which rely on appending paths to the module path variable. Showing raw paths to the python directory also looks a bit cheap, but I recall that with py2exe, we also had to work around this.
Requirements scheme
Here is the official features page from PyInstaller
| Feature | supported |
|---|---|
| Create multiple executables | No, broken in v3.0, see pyinstaller/pyinstaller#1527 |
| Bundle manifest | Yes |
| UIAccess | Yes |
| Embed logo | Yes |
| Set version info | Yes |
| Collect system dll's | Yes |
| Collect external Python dependencies and *.pyd files | Yes |
| Collect data files | Yes |
| Library file | No, seems to require a custom implementation using scripting according to features section, but no information found in documentation so far |
cx_Freeze
This is an alternative that integrates well with distutils. Note that it is far from compatible with py2exe, so setup.py would still require an overhaul. Running cx_Freeze also results into a package structure that is very similar to the one py2exe creates, including a librar .zip file in the lib sub directory. The default settings do not create a zip file, but that can be customized easily.
cx_Freeze is hosted on GitHub. We can include it as a separate submodule. Though it requires some c code to be compiled, that shouldn't be that much of a problem and takes less than 20 seconds, so that doesn't justify an inclusion in misc deps in my opinion.
Pros
- We're using it for a @BabbageCom Python project including wx and comtypes, and it works pretty well for our aims. So, I do have experience with it.
- It can replace paths shown in tracebacks
- Can include MSVC redistributable automatically
- Can include additional files in the library zip file
Cons
-
No official build supporting Python 3.7, we'd have to build from source.
-
There is no built-in support for manifests and setting the UI Access privilege level. We will probably have to implement this manually, either by expanding setup.py or by building the base executables ourselves.
-
By default, cx_freeze assumes that all python stuff is in a lib sub directory (i.e. in lib/library.zip) rather than in the root of the folder where the executables live. This is not easily customisable and seems to be hard-coded behaviour. We're using the lib folder for our x86 libraries, storing the python library in there might be a bit ugly.
-
When there is a startup error, cx_Freeze shows an error message box including the name of cx_Freeze itself. This might be confusing for users.
-
The last commit for the project dates from september 2018. There are 260 open issues, including two from myself:
Both issues haven't gotten any attention.
Requirements scheme
| Feature | supported |
|---|---|
| Create multiple executables | Yes |
| Bundle manifest | No, should probably be embedded in base executables |
| UIAccess | No |
| Embed logo | Yes |
| Set version info | Yes |
| Collect system dll's | Yes |
| Collect external Python dependencies and *.pyd files | Yes |
| Collect data files | Yes |
| Library file | Yes |
Nuitka
Suggestion by @ctoth in #8375 (comment)
Nuitka chooses a completely different approach. Instead of compiling the Python code to byte code and bundling a loader with it, it converts all code to c and then compiles it using VC++ and links it against the Python library. It also claims to be faster than CPython.
PROS
- Nuitka might improve performance.
Cons
- As all python code has to be converted to C++ and then compiled, creating a distribution of NVDA is going to be a time-consuming process. We might be able to provide pre-compiled obj files for static dependencies (such as wx and comtypes). This requires some investigation, to find out and is currently undocumented.
- It looks like Nuitka doesn't support changing the path of modules. I'm not even sure whether it would be possible to make Nuitka import python code bundled in add-ons. This is something to find out if we decide to explore this direction further.
- THere is a huge runtime difference between running NVDA from source and from a distribution.
Requirements scheme
| Feature | supported |
|---|---|
| Create multiple executables | Yes |
| Bundle manifest | No |
| UIAccess | No |
| Embed logo | Yes |
| Set version info | No |
| Collect system dll's | No |
| Collect external Python dependencies and *.pyd files | Unsure |
| Collect data files | No |
| Library file | No |
HO to proceed?
Based on the above outline, I propose the following order of investigation of our options:
- Devote time to investigate using the Py2exe fork by @albertosottile. He seems to be pretty interested in our possible efforts to freeze NVDA with his fork. It might also be the least effort on our end. However, it would help to know what his goals are with the project, and whether he's willing to maintain it in the near and far future. @albertosottile: Would you be able to elaborate on this?
- Try to get in touch with the PyInstaller developers to point them at the main cons listed above. The most major question would be whether there would be support for addons and custom comtypes com interfaces.
- Try to get in touch with @anthony-tuininga to ask him about whether he's intending to continue the active development of cx_Freeze.
- Based on the outcome of 2 and 3 when 1 fails, decide between PyInstaller and cx_Freeze. In the current state, cx_Freeze has my preference, but that preference may change.
Note that option 1, 2 and 3 could be worked on in parallel. I'm tempted to consider Nuitka being out of scope for now.