This line installs all requirements for pip first. After that you can install pip. With pip you can install Scrapy.
1 | sudo apt-get install build-essential libssl-dev libffi-dev libxml2-dev libxslt1-dev python-dev python-libxml && sudo apt-get install python-pip && sudo pip install Scrapy |
This should avoid errors like:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | Downloading/unpacking lxml Running setup.py egg_info for package lxml /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url' warnings.warn(msg) Building lxml version 3.1.2. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib warning: no files found matching '*.txt' under directory 'src/lxml/tests' Installing collected packages: lxml Running setup.py install for lxml /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url' warnings.warn(msg) Building lxml version 3.1.2. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib building 'lxml.etree' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -I/home/admin/.virtualenvs/dev.actualito.com/build/lxml/src/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o src/lxml/lxml.etree.c: In function '__pyx_f_4lxml_5etree__getFilenameForFile': src/lxml/lxml.etree.c:26851:7: warning: variable '__pyx_clineno' set but not used [-Wunused-but-set-variable] src/lxml/lxml.etree.c:26850:15: warning: variable '__pyx_filename' set but not used [-Wunused-but-set-variable] src/lxml/lxml.etree.c:26849:7: warning: variable '__pyx_lineno' set but not used [-Wunused-but-set-variable] src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_4XSLT_18__call__': src/lxml/lxml.etree.c:138273:81: warning: passing argument 1 of '__pyx_f_4lxml_5etree_12_XSLTContext__copy' from incompatible pointer type [enabled by default] src/lxml/lxml.etree.c:136229:52: note: expected 'struct __pyx_obj_4lxml_5etree__XSLTContext *' but argument is of type 'struct __pyx_obj_4lxml_5etree__BaseContext *' src/lxml/lxml.etree.c: In function '__pyx_f_4lxml_5etree__copyXSLT': src/lxml/lxml.etree.c:139667:79: warning: passing argument 1 of '__pyx_f_4lxml_5etree_12_XSLTContext__copy' from incompatible pointer type [enabled by default] src/lxml/lxml.etree.c:136229:52: note: expected 'struct __pyx_obj_4lxml_5etree__XSLTContext *' but argument is of type 'struct __pyx_obj_4lxml_5etree__BaseContext *' src/lxml/lxml.etree.c: At top level: src/lxml/lxml.etree.c:12384:13: warning: '__pyx_f_4lxml_5etree_displayNode' defined but not used [-Wunused-function] gcc: internal compiler error: Killed (program cc1) Please submit a full bug report, with preprocessed source if appropriate. See for instructions. error: command 'gcc' failed with exit status 4 Complete output from command /home/admin/.virtualenvs/dev.actualito.com/bin/python -c "import setuptools;__file__='/home/admin/.virtualenvs/dev.actualito.com/build/lxml/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-asDtN5-record/install-record.txt --single-version-externally-managed --install-headers /home/admin/.virtualenvs/dev.actualito.com/include/site/python2.7: /usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'bugtrack_url' warnings.warn(msg) Building lxml version 3.1.2. Building without Cython. Using build configuration of libxslt 1.1.26 Building against libxml2/libxslt in the following directory: /usr/lib running install running build running build_py copying src/lxml/includes/lxml-version.h -> build/lib.linux-x86_64-2.7/lxml/includes running build_ext building 'lxml.etree' extension gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/usr/include/libxml2 -I/home/admin/.virtualenvs/dev.actualito.com/build/lxml/src/lxml/includes -I/usr/include/python2.7 -c src/lxml/lxml.etree.c -o build/temp.linux-x86_64-2.7/src/lxml/lxml.etree.o src/lxml/lxml.etree.c: In function '__pyx_f_4lxml_5etree__getFilenameForFile': src/lxml/lxml.etree.c:26851:7: warning: variable '__pyx_clineno' set but not used [-Wunused-but-set-variable] src/lxml/lxml.etree.c:26850:15: warning: variable '__pyx_filename' set but not used [-Wunused-but-set-variable] src/lxml/lxml.etree.c:26849:7: warning: variable '__pyx_lineno' set but not used [-Wunused-but-set-variable] src/lxml/lxml.etree.c: In function '__pyx_pf_4lxml_5etree_4XSLT_18__call__': src/lxml/lxml.etree.c:138273:81: warning: passing argument 1 of '__pyx_f_4lxml_5etree_12_XSLTContext__copy' from incompatible pointer type [enabled by default] src/lxml/lxml.etree.c:136229:52: note: expected 'struct __pyx_obj_4lxml_5etree__XSLTContext *' but argument is of type 'struct __pyx_obj_4lxml_5etree__BaseContext *' src/lxml/lxml.etree.c: In function '__pyx_f_4lxml_5etree__copyXSLT': src/lxml/lxml.etree.c:139667:79: warning: passing argument 1 of '__pyx_f_4lxml_5etree_12_XSLTContext__copy' from incompatible pointer type [enabled by default] src/lxml/lxml.etree.c:136229:52: note: expected 'struct __pyx_obj_4lxml_5etree__XSLTContext *' but argument is of type 'struct __pyx_obj_4lxml_5etree__BaseContext *' src/lxml/lxml.etree.c: At top level: src/lxml/lxml.etree.c:12384:13: warning: '__pyx_f_4lxml_5etree_displayNode' defined but not used [-Wunused-function] gcc: internal compiler error: Killed (program cc1) Please submit a full bug report, with preprocessed source if appropriate. See for instructions. error: command 'gcc' failed with exit status 4 ---------------------------------------- Command /home/admin/.virtualenvs/dev.actualito.com/bin/python -c "import setuptools;__file__='/home/admin/.virtualenvs/dev.actualito.com/build/lxml/setup.py';exec(compile(open(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-asDtN5-record/install-record.txt --single-version-externally-managed --install-headers /home/admin/.virtualenvs/dev.actualito.com/include/site/python2.7 failed with error code 1 in /home/admin/.virtualenvs/dev.actualito.com/build/lxml Storing complete log in /home/admin/.pip/pip.log |
The reason for this error might be “out of memory”. Check the last lines of dmesg | tail
Wow, finally some instructions that worked, thank you! I had to add sudo before pip though for permissions. The following is what I ran on fresh ubuntu:trusty install in docker container.
sudo apt-get install build-essential libssl-dev libffi-dev libxml2-dev \
libxslt1-dev python-dev python-libxml2 \
&& sudo apt-get install python-pip \
&& sudo pip install scrapy
Nice! I tried several combinations of the dev libs until this worked.