[Airflow] ImportError: No module named lockfile.pidlockfile

When setting up a clean install of airflow on ubuntu 14.04, I ran into the following error:

vagrant@vagrant-ubuntu-trusty-64:~$ airflow
[2016-07-19 15:37:41,839] {__init__.py:36} INFO - Using executor SequentialExecutor
[2016-07-19 15:37:41,912] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/Grammar.txt
[2016-07-19 15:37:41,929] {driver.py:120} INFO - Generating grammar tables from /usr/lib/python2.7/lib2to3/PatternGrammar.txt
Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 5, in <module>
    from airflow.bin.cli import CLIFactory
  File "/usr/local/lib/python2.7/dist-packages/airflow/bin/cli.py", line 17, in <module>
    from daemon.pidfile import TimeoutPIDLockFile
  File "/usr/local/lib/python2.7/dist-packages/daemon/pidfile.py", line 18, in <module>
    from lockfile.pidlockfile import PIDLockFile
ImportError: No module named lockfile.pidlockfile

This seems to be because Airflow includes python-daemon >= 2.1.1 as a dependency, but not lockfile. It seems that some time before 2.1.1 python-daemon removed TimeoutPIDLockFile and instead decided just to use lockfile.

Uninstalling python-daemon and reinstalling fixed the issue, as python-daemon included a lockfile dependency.

Not sure why PyPi wouldn't have pulled this down when resolving Airflow's python-daemon dependency, but figured I'd open this issue in case other run into it in the future. It might make sense for Airflow to explicitly require lockfile.

Tim added a comment - 15/Aug/16 16:30

I am having this same issue. Unfortunately uninstalling python-daemons and reinstalling does not work.

Tim added a comment - 15/Aug/16 18:07 - edited

I was using apt-get package manager for python-daemon. When switching to pip, the uninstall and reinstall worked.

 

Sergio Herrera added a comment - 07/Oct/16 10:06

Airflow has the dependency with python-daemon (currently 2.1.1), and that version depends on lockfile>=0.10 (latest 0.12.2). The problem is that python-daemon installation (through pip) doesn't carry its transitive dependency. I tried several times with virtual machines and virtual environments for testing the issue. Also I have upgraded my Airflow installation from 1.6.2 to 1.7.1.3 and it fails with this problem.
The problem is that Airflow doesn't depends on lockfile directly (or I cannot find the direct dependency), but I can make a PR with this change if necessary.