Skip to content

Conversation

@pks-t
Copy link
Member

@pks-t pks-t commented Jul 10, 2017

Another day, another build infrastructure tweak. So yet again, please do not merge until #4282 is merged.

This PR improves how we build libssh2. In recent commits, they have changed some things regarding their build infrastructure, which would require us to add additional hacks to compile libssh2 directly as part of our own sources. As libssh2 uses CMake, though, we should simply be including it as a subdirectory and be done about it.

Note that this means that there is an additional dynamic library that has to be distributed. I don't know yet whether it is possible to statically link it instead of generating a dynamic one to avoid this, but I'll take another stab at this later. Regardless of this it is definitly the right thing to do, as we do not want to fiddle with their build instructions. Doing so may lead to subtle breakages due to the build environment not matching what the libssh2 project expects, which is critical to avoid in a library directly related to the user's security.

This fixes #4302.

@pks-t
Copy link
Member Author

pks-t commented Aug 25, 2017

Rebased due to #4282. I'm just sitting at the train station and got no Windows VM ready, so I cannot test this right now. So please only merge if you've confirmed this works on Windows. Otherwise I'll try to test this in a few days, hopefully.

@carlosmn
Copy link
Member

carlosmn commented Nov 4, 2017

In this branch, specifying -DEMBEDDED_LIBSSH2_PATH seems to only build libssh2. After building, everything under build/ is just for libssh2.

PS C:\Users\carlos\libgit2\build> cmake .. -DEMBEDDED_LIBSSH2_PATH=../../libssh2 -DBUILD_EXAMPLES=ON
-- Building for: Visual Studio 14 2015
-- The C compiler identification is MSVC 19.0.24215.1
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe
-- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio 14.0/VC/bin/cl.exe -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing:
 OPENSSL_LIBRARIES OPENSSL_INCLUDE_DIR)
-- Could NOT find Libgcrypt (missing:  LIBGCRYPT_LIBRARY LIBGCRYPT_INCLUDE_DIR)
-- Looking for include files windows.h, bcrypt.h
-- Looking for include files windows.h, bcrypt.h - found
-- Looking for include file ntdef.h
-- Looking for include file ntdef.h - not found
-- Looking for include file ntstatus.h
-- Looking for include file ntstatus.h - found
-- Looking for include files windows.h, wincrypt.h
-- Looking for include files windows.h, wincrypt.h - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - not found
-- Looking for include file inttypes.h
-- Looking for include file inttypes.h - found
-- Looking for include file stdlib.h
-- Looking for include file stdlib.h - found
-- Looking for include file sys/select.h
-- Looking for include file sys/select.h - not found
-- Looking for include file sys/uio.h
-- Looking for include file sys/uio.h - not found
-- Looking for include file sys/socket.h
-- Looking for include file sys/socket.h - not found
-- Looking for include file sys/ioctl.h
-- Looking for include file sys/ioctl.h - not found
-- Looking for include file sys/time.h
-- Looking for include file sys/time.h - not found
-- Looking for include file sys/un.h
-- Looking for include file sys/un.h - not found
-- Looking for include file windows.h
-- Looking for include file windows.h - found
-- Looking for include file ws2tcpip.h
-- Looking for include file ws2tcpip.h - found
-- Looking for include file winsock2.h
-- Looking for include file winsock2.h - found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of long long
-- Check size of long long - done
-- Looking for gettimeofday
-- Looking for gettimeofday - not found
-- Looking for strtoll
-- Looking for strtoll - found
-- Looking for snprintf
-- Looking for snprintf - found
-- Looking for poll
-- Looking for poll - not found
-- Performing Test HAVE_O_NONBLOCK
-- Performing Test HAVE_O_NONBLOCK - Failed
-- Performing Test HAVE_FIONBIO
-- Performing Test HAVE_FIONBIO - Failed
-- Performing Test HAVE_IOCTLSOCKET
-- Performing Test HAVE_IOCTLSOCKET - Success
-- Looking for include file arpa/inet.h
-- Looking for include file arpa/inet.h - not found
-- Looking for include file netinet/in.h
-- Looking for include file netinet/in.h - not found
-- Looking for strcasecmp
-- Looking for strcasecmp - not found
-- Looking for _stricmp
-- Looking for _stricmp - found
-- Looking for _snprintf
-- Looking for _snprintf - found
-- Looking for __func__
-- Looking for __func__ - found
-- Looking for __FUNCTION__
-- Looking for __FUNCTION__ - found
--
-- The following features have been enabled:

 * diffie-hellman-group-exchange-sha1 , "new" diffie-hellman-group-exchange-sha1 method

-- The following features have been disabled:

 * Shared library , creating libssh2 as a shared library (.so/.dll)
 * Compression , using zlib for compression
 * "none" cipher
 * "none" MAC
 * Logging , Logging of execution with debug trace

-- The following OPTIONAL packages have not been found:

 * OpenSSL
 * Libgcrypt

-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    EMBEDDED_LIBSSH2_PATH


-- Build files have been written to: C:/Users/carlos/libgit2/build

@pks-t pks-t force-pushed the pks/cmake-embedded-libssh2 branch from 0f3969d to fc2ccf6 Compare November 12, 2017 14:24
@pks-t
Copy link
Member Author

pks-t commented Nov 12, 2017

That is... strange. At least that's why I'm adding the bindir parameter to ADD_SUBDIRECTORY. Will have to test at a later point, as I don't have a Windows machine right now.

@pks-t
Copy link
Member Author

pks-t commented Mar 26, 2018

@carlosmn: have you actually specified -DUSE_EMBEDDED_LIBSSH2=ON when trying this PR?

@pks-t pks-t force-pushed the pks/cmake-embedded-libssh2 branch from fc2ccf6 to dd81940 Compare June 15, 2018 10:08
@pks-t
Copy link
Member Author

pks-t commented Jun 15, 2018

Refactored this PR to not introduce a new variable "USE_EMBEDDED_LIBSSH2". I've also added a new change to allow using an embedded libssh2 not only on Windows systems, but also on all other systems. @tiennou, this might be of interest to you for iOS.

@pks-t pks-t force-pushed the pks/cmake-embedded-libssh2 branch from dd81940 to 77d3953 Compare July 12, 2019 10:33
@pks-t
Copy link
Member Author

pks-t commented Jul 12, 2019

I've rebased this PR on top of latest master to fix conflicts. I've also converted the USE_SSH option to work exactly like USE_HTTPS works, so that one can explicitly choose backends.

@pks-t pks-t force-pushed the pks/cmake-embedded-libssh2 branch 2 times, most recently from c71f845 to 8aba1a2 Compare July 12, 2019 10:56
@pks-t pks-t force-pushed the pks/cmake-embedded-libssh2 branch from 8aba1a2 to ef4a1ce Compare July 21, 2019 10:28
@pks-t
Copy link
Member Author

pks-t commented Aug 1, 2019

Anybody cares enough to review? I'd really like to get this landed :)

@tiennou
Copy link
Contributor

tiennou commented Aug 1, 2019

A Ninja build reported the following (note the detection of v1.8.2 in /usr/local), then failed while --build .-ing.

cmake .. -DUSE_SSH=libssh2_embedded -DLIBSSH2_EMBEDDED_PATH=../../libssh2 -G Ninja
-- The C compiler identification is AppleClang 10.0.1.10010046
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
-- Check for working C compiler: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PkgConfig: /usr/local/bin/pkg-config (found version "0.29.2") 
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - found
-- Found Threads: TRUE  
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM
-- Performing Test HAVE_STRUCT_STAT_ST_MTIM - Failed
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC
-- Performing Test HAVE_STRUCT_STAT_ST_MTIMESPEC - Success
-- Performing Test HAVE_STRUCT_STAT_MTIME_NSEC
-- Performing Test HAVE_STRUCT_STAT_MTIME_NSEC - Failed
-- Performing Test HAVE_STRUCT_STAT_NSEC
-- Performing Test HAVE_STRUCT_STAT_NSEC - Success
-- Performing Test IS_WALL_SUPPORTED
-- Performing Test IS_WALL_SUPPORTED - Success
-- Performing Test IS_WEXTRA_SUPPORTED
-- Performing Test IS_WEXTRA_SUPPORTED - Success
-- Performing Test IS_FVISIBILITY_HIDDEN_SUPPORTED
-- Performing Test IS_FVISIBILITY_HIDDEN_SUPPORTED - Success
-- Performing Test IS_WDOCUMENTATION_SUPPORTED
-- Performing Test IS_WDOCUMENTATION_SUPPORTED - Success
-- Performing Test IS_WNO_MISSING_FIELD_INITIALIZERS_SUPPORTED
-- Performing Test IS_WNO_MISSING_FIELD_INITIALIZERS_SUPPORTED - Success
-- Performing Test IS_WSTRICT_ALIASING_SUPPORTED
-- Performing Test IS_WSTRICT_ALIASING_SUPPORTED - Success
-- Performing Test IS_WSTRICT_PROTOTYPES_SUPPORTED
-- Performing Test IS_WSTRICT_PROTOTYPES_SUPPORTED - Success
-- Performing Test IS_WDECLARATION_AFTER_STATEMENT_SUPPORTED
-- Performing Test IS_WDECLARATION_AFTER_STATEMENT_SUPPORTED - Success
-- Performing Test IS_WSHIFT_COUNT_OVERFLOW_SUPPORTED
-- Performing Test IS_WSHIFT_COUNT_OVERFLOW_SUPPORTED - Success
-- Performing Test IS_WUNUSED_CONST_VARIABLE_SUPPORTED
-- Performing Test IS_WUNUSED_CONST_VARIABLE_SUPPORTED - Success
-- Performing Test IS_WUNUSED_FUNCTION_SUPPORTED
-- Performing Test IS_WUNUSED_FUNCTION_SUPPORTED - Success
-- Performing Test IS_WINT_CONVERSION_SUPPORTED
-- Performing Test IS_WINT_CONVERSION_SUPPORTED - Success
-- Performing Test IS_WFORMAT_SUPPORTED
-- Performing Test IS_WFORMAT_SUPPORTED - Success
-- Performing Test IS_WFORMAT_SECURITY_SUPPORTED
-- Performing Test IS_WFORMAT_SECURITY_SUPPORTED - Success
-- Performing Test IS_WNO_DOCUMENTATION_DEPRECATED_SYNC_SUPPORTED
-- Performing Test IS_WNO_DOCUMENTATION_DEPRECATED_SYNC_SUPPORTED - Success
-- Looking for futimens
-- Looking for futimens - found
-- Checking prototype qsort_r for HAVE_QSORT_R_BSD - True
-- Checking prototype qsort_r for HAVE_QSORT_R_GNU - False
-- Looking for qsort_s
-- Looking for qsort_s - not found
-- Looking for clock_gettime in rt
-- Looking for clock_gettime in rt - not found
-- Checking for module 'libssh2'
--   Found libssh2, version 1.8.2
--   Resolved libraries: /usr/local/lib/libssh2.dylib
-- Found OpenSSL: /usr/local/lib/libcrypto.dylib (found version "1.1.1b")  
-- Looking for EVP_aes_128_ctr
-- Looking for EVP_aes_128_ctr - found
-- Looking for include file unistd.h
-- Looking for include file unistd.h - found
-- Looking for include file inttypes.h
-- Looking for include file inttypes.h - found
-- Looking for include file stdlib.h
-- Looking for include file stdlib.h - found
-- Looking for include file sys/select.h
-- Looking for include file sys/select.h - found
-- Looking for include file sys/uio.h
-- Looking for include file sys/uio.h - found
-- Looking for include file sys/socket.h
-- Looking for include file sys/socket.h - found
-- Looking for include file sys/ioctl.h
-- Looking for include file sys/ioctl.h - found
-- Looking for include file sys/time.h
-- Looking for include file sys/time.h - found
-- Looking for include file sys/un.h
-- Looking for include file sys/un.h - found
-- Looking for include file windows.h
-- Looking for include file windows.h - not found
-- Looking for include file ws2tcpip.h
-- Looking for include file ws2tcpip.h - not found
-- Looking for include file winsock2.h
-- Looking for include file winsock2.h - not found
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of long long
-- Check size of long long - done
-- Looking for gettimeofday
-- Looking for gettimeofday - found
-- Looking for strtoll
-- Looking for strtoll - found
-- Looking for snprintf
-- Looking for snprintf - found
-- Looking for memset_s
-- Looking for memset_s - found
poll use is disabled on this platform
-- Looking for socket
-- Looking for socket - found
-- Looking for select
-- Looking for select - found
-- Looking for inet_addr
-- Looking for inet_addr - found
-- Performing Test HAVE_O_NONBLOCK
-- Performing Test HAVE_O_NONBLOCK - Success
-- The following features have been enabled:

 * nanoseconds, whether to use sub-second file mtimes and ctimes
 * threadsafe, threadsafe support
 * diffie-hellman-group-exchange-sha1, "new" diffie-hellman-group-exchange-sha1 method
 * Logging, Logging of execution with debug trace

-- The following OPTIONAL packages have been found:

 * OpenSSL

-- The following features have been disabled:

 * debugpool, debug pool allocator
 * tracing, tracing support
 * Shared library, creating libssh2 as a shared library (.so/.dll)
 * Compression, using zlib for compression
 * "none" cipher
 * "none" MAC

-- Found mbedTLS:
--   version 2.16.1
--   TLS: /usr/local/lib/libmbedtls.dylib
--   X509: /usr/local/lib/libmbedx509.dylib
--   Crypto: /usr/local/lib/libmbedcrypto.dylib
-- Found Security /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework
-- Looking for SSLCreateContext in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework
-- Looking for SSLCreateContext in /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/Security.framework - found
-- Found CoreFoundation /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks/CoreFoundation.framework
-- Found PCRE: /usr/local/lib/libpcre.dylib  
-- Looking for regcomp_l
-- Looking for regcomp_l - found
-- http-parser version 2 was not found or disabled; using bundled 3rd-party sources.
-- Performing Test IS_WIMPLICIT_FALLTHROUGH_1_SUPPORTED
-- Performing Test IS_WIMPLICIT_FALLTHROUGH_1_SUPPORTED - Failed
-- Found ZLIB: /usr/lib/libz.dylib (found version "1.2.11") 
-- Performing Test IS_WNO_IMPLICIT_FALLTHROUGH_SUPPORTED
-- Performing Test IS_WNO_IMPLICIT_FALLTHROUGH_SUPPORTED - Success
-- Looking for iconv_open
-- Looking for iconv_open - not found
-- Found Iconv: -L/usr/lib -liconv
-- Found PythonInterp: /usr/local/bin/python (found version "2.7.16") 
-- Enabled features:
 * nanoseconds, whether to use sub-second file mtimes and ctimes
 * threadsafe, threadsafe support
 * diffie-hellman-group-exchange-sha1, "new" diffie-hellman-group-exchange-sha1 method
 * Logging, Logging of execution with debug trace
 * SSH, SSH transport using embedded libssh (../../libssh2)
 * HTTPS, using SecureTransport
 * SHA, using CollisionDetection
 * regex, using system regcomp_l
 * http-parser, http-parser support (bundled)
 * zlib, using system zlib
 * ntlmclient, NTLM authentication support for Unix
 * iconv, iconv encoding conversion support

-- Disabled features:
 * debugpool, debug pool allocator
 * tracing, tracing support
 * Shared library, creating libssh2 as a shared library (.so/.dll)
 * Compression, using zlib for compression
 * "none" cipher
 * "none" MAC
 * SPNEGO, SPNEGO authentication support

-- Configuring done
-- Generating done
-- Build files have been written to: /Users/tiennou/Projects/libgit2/build-ssh2
[130/590] Building C object src/CMakeFiles/git2internal.dir/transports/ssh.c.o
FAILED: src/CMakeFiles/git2internal.dir/transports/ssh.c.o 
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc -DHAVE_QSORT_R_BSD -DSHA1DC_CUSTOM_INCLUDE_SHA1_C=\"common.h\" -DSHA1DC_CUSTOM_INCLUDE_UBC_CHECK_C=\"common.h\" -DSHA1DC_NO_STANDARD_INCLUDES=1 -D_FILE_OFFSET_BITS=64 -Isrc -I../src -I../include -I../deps/http-parser -I../deps/ntlmclient -isystem ../src/libssh2 -iframework /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/System/Library/Frameworks -D_GNU_SOURCE  -Wall -Wextra -fvisibility=hidden -fPIC -Wdocumentation -Wno-missing-field-initializers -Wstrict-aliasing -Wstrict-prototypes -Wdeclaration-after-statement -Wshift-count-overflow -Wunused-const-variable -Wunused-function -Wint-conversion -Wformat -Wformat-security -Wno-documentation-deprecated-sync -g -D_DEBUG -O0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk   -std=gnu90 -MD -MT src/CMakeFiles/git2internal.dir/transports/ssh.c.o -MF src/CMakeFiles/git2internal.dir/transports/ssh.c.o.d -o src/CMakeFiles/git2internal.dir/transports/ssh.c.o   -c ../src/transports/ssh.c
../src/transports/ssh.c:11:10: fatal error: 'libssh2.h' file not found
#include <libssh2.h>
         ^~~~~~~~~~~
1 error generated.
[139/590] Building C object src/CMakeFiles/git2internal.dir/util.c.o
ninja: build stopped: subcommand failed.

-isystem ../src/libssh2 ? That ought to be -isystem ../../libssh2/include, but I'm not sure what went wrong…

@pks-t
Copy link
Member Author

pks-t commented Aug 2, 2019

Thanks for testing, @tiennou.

A Ninja build reported the following (note the detection of v1.8.2 in /usr/local), then failed while --build .-ing.

The detection is perfectly normal and is part of the auto-detection similar to how the HTTPS selection works. As you've set USE_SSH=libssh2_embedded it's not used in the end.

-isystem ../src/libssh2 ? That ought to be -isystem ../../libssh2/include, but I'm not sure what went wrong…

Right, I see that, too. Didn't notice earlier because I also had system libssh2 available and thus it still found the correct header. Will investigate.

Due to recent changes to some required defines in libssh2, we now fail
to build the library correctly. This results from the fact that we're
manually including libssh2 sources and its include directory into our
own build infrastructure, which may not be configured as required by
libssh2.

As libssh2 is driven by CMake as well, we can simply include it as a
subdirectory and stop caring for its configuration. Instead, we will
now just link against the generated library and be done with it, without
caring about the library internals.
Following the HTTPS backend selection, let's make the SSH backend
selection more generic. Instead of choosing "-DUSE_SSH=(ON|OFF)", one
can now also set it to an explicit backend. Currently, only two backends
"libssh2" and "libssh2_embedded" are supported.
When setting the STRINGS property for cache variables, CMake GUI will
present a dropdown box so that the developer may choose one of supplied
values. Use this feature for HTTPS, SHA1 and SSH backends, which all
support a fixed set of values, only.
@pks-t
Copy link
Member Author

pks-t commented Sep 21, 2019

Fixed the issue @tiennou noticed.

@pks-t pks-t force-pushed the pks/cmake-embedded-libssh2 branch from ef4a1ce to da35922 Compare September 21, 2019 14:23
@pks-t
Copy link
Member Author

pks-t commented Oct 10, 2019

@tiennou: btw, this pull request probably impacts your #5253.

@tiennou
Copy link
Contributor

tiennou commented Nov 5, 2019

Sorry, forgot to say that I don't mind, and both have been dwarfed by your #5284 since 😜. I usually rebase on top of whatever master is reflexively, only to stop if there's any conflicts… If this one goes it quickly, I'll have to adapt the other on top anyways.

@pks-t
Copy link
Member Author

pks-t commented Nov 5, 2019

Sorry, forgot to say that I don't mind, and both have been dwarfed by your #5284 since .

Indeed. I guess it's going to take some time until that one lands, though, as I still need to see how to make it backwards compatible with older versions of CMake.

Base automatically changed from master to main January 7, 2021 10:09
@ethomson
Copy link
Member

I don't want to embed libssh2. Going to close this; let's stop embedding that library entirely.

@ethomson ethomson closed this Dec 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Build failure on Win32 with embedded libssh2 newer than 1.6.0

4 participants