This forum is shutting down! Please post new discussions at community.naturalcapitalproject.org

Compiling Invest under Linux Ubuntu 14.04

Hi
I'm trying to compile Invest under Ubuntu 14.04 and:

- installed all required dependencies either by using the central repositories (and ubuntugis repositories) or by using pip install

  • GDAL
  • shapely
  • numpy
  • scipy
  • pyqt4 (if running a model user interface)
  • matplotlib
  • bs4
  • pyAMG
  • cython
  • setuptools
  • h5py
  • dbfpy
  • poster
  • pygeoprocessing
  • natcap.versioner
  • rtree

- cloned the repository by running "hg clone https://bitbucket.org/natcap/invest"

- navigated to the copy of the repository by running "cd invest"

- checked if all depencies are correctly installed by running "paver test" and got:

---> pavement.check
All's well.

- Finally I run "paver build" but got:
Build failed: Unknown task: build

paver was installed by running sudo apt-get install python-paver (installs version 1.2.1.1)

Fells like I'm missing something obvious. Can anyone help?

Thanks

Tagged:

Comments

  • RichRich Administrator, NatCap Staff
    Is it possible you're at a different branch tip?  The latest stable version should be at 'develop'.  Can you 'hg up develop' and see if 'paver build' works then?  If that was the issue you might also want to check 'paver check' first in case some dependancies got changed from the revision you're on.
  • Thanks @Rich, that was it.

    - I run this:

    cd invest
    hg up develop

    - Then run paver check and installed all the missing dependencies:

    fd@fd-Lenovo:~/invest$ paver check
    file /usr/bin/rpm is not owned by any package
    ---> pavement.check
    Checking binaries
    Found hg            : /usr/bin/hg
    Found git           : /usr/bin/git
    Found svn           : /usr/bin/svn
    Found make          : /usr/bin/make
    Found pdflatex      : /usr/bin/pdflatex
    Found pandoc        : /usr/bin/pandoc
    Found fpm           : /usr/local/bin/fpm

    Checking python packages
    Python package OK: gdal 1.11.2 (meets gdal<2.0,>=1.9.0)
    Python package OK: h5py 2.5.0 (meets h5py>=2.3.0)
    Python package OK: matplotlib 1.3.1 (meets matplotlib)
    Python package OK: nose 1.3.7 (meets nose)
    Python package OK: numpy 1.10.4 (meets numpy)
    Python package OK: paramiko 1.16.0 (meets paramiko)
    Python package OK: paver 1.2.4 (meets paver)
    Python package OK: pip 8.0.3 (meets pip>=6.0.0)
    Python package OK: poster 0.8.1 (meets poster)
    Python package OK: pyamg 3.0.2 (meets pyamg>=2.2.1)
    Python package OK: pygeoprocessing 0.3.0a13 (meets pygeoprocessing>=0.3.0a13)
    Python package OK: PyQt4 4.10.4 (meets PyQt4)
    Python package OK: Pyro4 4.41 (meets Pyro4)
    Python package OK: pyyaml 3.11 (meets pyyaml)
    Python package OK: scipy 0.17.0 (meets scipy>=0.14.0)
    Python package OK: setuptools 20.2.2 (meets setuptools>=8.0)
    Python package OK: shapely 1.5.13 (meets shapely)
    Python package OK: sphinx 1.3.1 (meets sphinx==1.3.1)
    Python package OK: virtualenv 14.0.6 (meets virtualenv>=12.0.1)
    Python package OK: wheel 0.24.0 (meets wheel)

    Checking natcap namespace
    OK natcap.versioner==0.3.2 (/usr/local/lib/python2.7/dist-packages/natcap.versioner-0.3.2-py2.7.egg/natcap/versioner) installed as expected
    All's well.

    - Finally run "paver build > buildlog.txt" but the compilation fails. Here is the log. There seems to be something wrong with libyaml. Not sure if there is something else wrong. Any hints?

    Thanks

  • jdouglassjdouglass Administrator, NatCap Staff
    Hmm this is interesting.  It looks like the setup for pygeoprocessing is requiring the reinstall of pyyaml even though you already have it installed.  I'll take a look at it.

    In any case, is there any chance that you installed the ubuntu python-yaml package to satisfy the pyyaml requirement?  If so, you should be able to compile pyyaml from source by apt-get installing libyaml-dev and then re-running paver build.  libyaml-dev will install the development headers needed to be able to compile pyyaml from source.

    Let us know how this goes!
    James
  • @jdouglass, I installed libyaml-dev via apt-get install and then re-run paver build (after deleting everything and re-downloading the source code as above). It still fails, but the log is different. 

    Error message: "Could not find .egg-info directory in install record for pyyaml (from natcap.versioner>=0.2.4"
  • jdouglassjdouglass Administrator, NatCap Staff
    Interestingly enough and despite the .egg-info note in the logfile, it looks like pyyaml and the rest of the natcap.invest installation to the virtual env might have actually succeeded.  The part that worries me a little is that the binary-generation appears to be failing.  The goal of the binaries is to create a distributable directory and compressed installer (in this case, it'll be a .deb, which can be installed on your computer via `dpkg -i`) that should be standalone.  If you'd like to get the distributable binaries to work as expected, could you take a look at https://pythonhosted.org/PyInstaller/#linux and verify that you have ldd and objdump available?

    So, it's also worth noting that you might not actually need to build binaries here.  If your goal is to be able to run the InVEST models on your local machine, I would instead recommend installing the natcap.invest python package via `sudo python setup.py install` from the invest repo.  This will install the python package as well as an 'invest' command-line executable that you can use to launch the applications.  You already have all of the python dependencies installed, so installing this should be near trivial!
  • @jdouglass, I run sudo python setup.py and it seems to have worked. The log ends with:

    Processing dependencies for natcap.invest==3.3.0a1.post260+nc34320c819ba
    Finished processing dependencies for natcap.invest==3.3.0a1.post260+nc34320c819ba

    and see no error messages. How can I launch applications...? 

    I'm a little bit out my depth (it's first time I try compile anything), but I since I managed to get this far I would like see if I can create the .deb installers. How can I check if ldd and objdump are available?

    Thanks for the help


  • jdouglassjdouglass Administrator, NatCap Staff
    Ok great, it sounds like the invest python package installed as expected!  You should now be able to test that the `invest` entry-point is available by:

    $ invest --help

    Which should print some info about the available command-line parameters to the invest application.  To list the available models, type:

    $ invest --list

    And then to launch a model, you would type `invest <modelname>`.  For example, this would launch the Carbon Storage and Sequestration model:

    $ invest carbon


    RE: how to test if ldd and objdump are installed, `which ldd` and `which objdump` should suffice!  If the paths don't appear when you type the appropriate `which` command, they probably aren't installed.  If they need to be installed, ldd can be installed through `apt-get install libc-bin`, and objdump can be installed through `apt-get install binutils`.

  • Thanks @jdouglass. Invest 3.3.0 seems to be working on Ubuntu 14.04. I tested the carbon model (invest carbon) and it produces the same results as Invest 3.2.0 installed on Windows 10.

    I also tried the Scenic Quality model but I'm getting a python error (see attached log). Is this related to my installation or is it a bug introduced in Invest 3.3.0?


    About the the.deb installers:

    fd@fd-Lenovo:~$ which ldd
    /usr/bin/ldd
    fd@fd-Lenovo:~$ which objdump
    /usr/bin/objdump
    fd@fd-Lenovo:~$ 
  • jdouglassjdouglass Administrator, NatCap Staff
    The Scenic Quality model is quite buggy at the moment (hence the unstable label in the UI and at the CLI).  We've got a massively improved version of the model in the works, however, that should be released in the next few months and aims to address a slew of issues with the current version of the model.  I'd recommend avoiding this model until we can complete and release the new version.

    I've spent a goodly amount of time looking into getting builds to work on an ubuntu 14.04 instance, but I just don't think I have anything else I can offer at the moment that might help.  The linux builds are definitely a very early prototype, and have not been tested on many systems, but we'll be sure to test it out on the latest ubuntu LTS release once we have a more reliable build.  If you are able to get it to work on your machine, though, I'd love to hear what you did so we can make this work better for future users!
  • I'm happy that it's a bug and not an issue with my installation!

    Invest is working fine on my system and if anyone asks I can show them the instructions you provided here.

    I would suggest adding a link to this thread to the compilation instructions. It might help other Linux Ubuntu/Mint users in the future.

    Thanks a lot for the help!
  • jdouglassjdouglass Administrator, NatCap Staff
    Of course!  I'm glad we were able to get everything working on your computer.  I'll be sure to add a note to the README linking to our discussion here.

    Thanks!
  • Hi @jdouglass

    I'm trying to use the instructions you provided above to install Invest on a fresh Ubuntu 16.04 installation. I did the following:


    hg clone https://bitbucket.org/natcap/invest

    cd invest

    hg up develop

    paver check

    First, I was surprised the list that by paver check returened did not include pygeoprocessing or natcap.versioner but I assumed they were still necessary. I installed most of the required dependencies but I'm getting an error related to "libgeos" I don't understand (please see below). Am I missing something?

    Thanks

    fd@fd-XPS:~/invest$ paver check
    file /usr/bin/rpm is not owned by any package
    ---> pavement.check
    Checking binaries
    Found hg            : /usr/bin/hg
    Found git           : /usr/bin/git
    Found svn           : /usr/bin/svn
    Found make          : /usr/bin/make
    Found pdflatex      : /usr/bin/pdflatex
    Found pandoc        : /usr/bin/pandoc
    Found fpm           : /usr/local/bin/fpm

    Checking python packages
    Python package OK: cython 0.24.1 (meets cython)
    WARNING: GDAL 2.1.0 is installed but gdal<2.0,>=1.11.2 is required.  You may need to upgrade your development headers along with the python package.
    Python package OK: matplotlib 1.5.1 (meets matplotlib)
    Python package OK: mock 2.0.0 (meets mock)
    Python package OK: nose 1.3.7 (meets nose)
    Python package OK: numpy 1.11.1 (meets numpy>=1.11.0)
    Python package OK: paramiko 2.0.2 (meets paramiko)
    Python package OK: paver 1.2.1 (meets paver)
    ERROR: Package not found: pip>=6.0.0
    Python package OK: poster 0.8.1 (meets poster)
    Python package OK: pyamg 2.2.1 (meets pyamg>=2.2.1)
    Trying `CDLL(libgeos_c.so.1)`
    09/05/2016 15:14:36  shapely.geos       DEBUG        Trying `CDLL(libgeos_c.so.1)`
    Library path: 'libgeos_c.so.1'
    09/05/2016 15:14:36  shapely.geos       DEBUG        Library path: 'libgeos_c.so.1'
    DLL: <CDLL 'libgeos_c.so.1', handle 245ab30 at 7f38639d46d0>
    09/05/2016 15:14:36  shapely.geos       DEBUG        DLL: <CDLL 'libgeos_c.so.1', handle 245ab30 at 7f38639d46d0>
    Trying `CDLL(libc.so.6)`
    09/05/2016 15:14:36  shapely.geos       DEBUG        Trying `CDLL(libc.so.6)`
    Library path: 'libc.so.6'
    09/05/2016 15:14:36  shapely.geos       DEBUG        Library path: 'libc.so.6'
    DLL: <CDLL 'libc.so.6', handle 7f389051d9b0 at 7f3890460e90>
    09/05/2016 15:14:36  shapely.geos       DEBUG        DLL: <CDLL 'libc.so.6', handle 7f389051d9b0 at 7f3890460e90>
    Python package OK: pygeoprocessing 0.3.0a17 (meets pygeoprocessing>=0.3.0a17)
    Python package OK: PyQt4 4.11.4 (meets PyQt4)
    Python package OK: Pyro4 4.47 (meets Pyro4>=4.41)
    Python package OK: rtree 0.8.2 (meets rtree>=0.8.2)
    Python package OK: scipy 0.18.0 (meets scipy>=0.14.0)
    Python package OK: setuptools 20.7.0 (meets setuptools>=8.0)
    Python package OK: shapely 1.5.13 (meets shapely)
    Python package OK: sphinx 1.3.1 (meets sphinx==1.3.1)
    Python package OK: virtualenv 15.0.3 (meets virtualenv>=12.0.1)
    Python package OK: wheel 0.29.0 (meets wheel)

    Checking natcap namespace
    OK natcap.versioner==0.4.2 (/usr/local/lib/python2.7/dist-packages/natcap.versioner-0.4.2-py2.7.egg/natcap/versioner) installed as expected
    Warnings found; Builds may not work as expected



  • jdouglassjdouglass Administrator, NatCap Staff
    It makes sense that you would need to install pygeoprocessing and natcap.versioner ... cloning the InVEST source tree only fetches the InVEST source code.

    Package management in python is kind of an interesting subject.  If you run `python setup.py install` on the InVEST source tree, dependencies will not be installed because setuptools doesn't do dependency resolution.  If you run `pip install natcap.invest`, on the other hand, pip will inspect the listed dependencies for InVEST and will install those as needed.

    Those DEBUG messages are just extra information about where the libgeos binary is located and it looks like it was able to import as expected.  As far as I can tell, it looks like you should be good to go to use InVEST!
  • @jdouglass


    Great, the debug scared me a little. Invest is indeed working well.

    Thanks again!


Sign In or Register to comment.