Skip to content

Makefile: some cleanups / added VERBOSE option - #4438

Merged
danmar merged 8 commits into
cppcheck-opensource:mainfrom
firewave:dmake
Sep 9, 2022
Merged

Makefile: some cleanups / added VERBOSE option#4438
danmar merged 8 commits into
cppcheck-opensource:mainfrom
firewave:dmake

Conversation

@firewave

@firewave firewave commented Sep 4, 2022

Copy link
Copy Markdown
Collaborator

CPPFLAGS is the variable for preprocessor options - see http://porthos.ist.utl.pt/docs/make/make_97.html.

A cleanup and fixes to the MinGW/Cygwin detection will follow in a separate PR.

We should use implicit rules instead of specifying our own. Something to look at for later.

@firewave

firewave commented Sep 4, 2022

Copy link
Copy Markdown
Collaborator Author

MinGW CI:

COMSPEC not found
ComSpec found
PATH=C:\ProgramData\chocolatey\lib\mingw\tools\install\mingw64\bin;C:\Program Files\MongoDB\Server\5.0\bin;C:\aliyun-cli;C:\vcpkg;C:\cf-cli;C:\Program Files (x86)\NSIS\;C:\tools\zstd;C:\Program Files\Mercurial\;C:\hostedtoolcache\windows\stack\2.7.5\x64;C:\cabal\bin;C:\\ghcup\bin;C:\tools\ghc-9.4.2\bin;C:\Program Files\dotnet;C:\mysql\bin;C:\Program Files\R\R-4.2.1\bin\x64;C:\SeleniumWebDrivers\GeckoDriver;C:\Program Files (x86)\sbt\bin;C:\Program Files (x86)\GitHub CLI;C:\Program Files\Git\bin;C:\Program Files (x86)\pipx_bin;C:\npm\prefix;C:\hostedtoolcache\windows\go\1.17.13\x64\bin;C:\hostedtoolcache\windows\Python\3.7.9\x64\Scripts;C:\hostedtoolcache\windows\Python\3.7.9\x64;C:\hostedtoolcache\windows\Ruby\2.5.9\x64\bin;C:\tools\kotlinc\bin;C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.345-1\x64\bin;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\ProgramData\kind;C:\Program Files\Eclipse Foundation\jdk-8.0.302.8-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\ProgramData\Chocolatey\bin;C:\Program Files\Docker;C:\Program Files\PowerShell\7\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\dotnet\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\;C:\Program Files (x86)\Microsoft SQL Server\110\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\130\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\140\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\150\DTS\Binn\;C:\Program Files (x86)\Microsoft SQL Server\160\DTS\Binn\;C:\Program Files\OpenSSL\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\CMake\bin;C:\ProgramData\chocolatey\lib\maven\apache-maven-3.8.6\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\nodejs\;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\GitHub CLI\;c:\tools\php;C:\Program Files (x86)\sbt\bin;C:\SeleniumWebDrivers\ChromeDriver\;C:\SeleniumWebDrivers\EdgeDriver\;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files\Amazon\SessionManagerPlugin\bin\;C:\Program Files\Amazon\AWSSAMCLI\bin\;C:\Program Files (x86)\Google\Cloud SDK\google-cloud-sdk\bin;C:\Program Files (x86)\Microsoft BizTalk Server\;C:\Program Files\LLVM\bin;C:\Users\runneradmin\.dotnet\tools;C:\Users\runneradmin\.cargo\bin;C:\Users\runneradmin\AppData\Local\Microsoft\WindowsApps
WINNT found
MSYSTEM=

Cygwin CI:

COMSPEC not found
ComSpec found
PATH=C:\cygwin\bin;C:\Program Files\MongoDB\Server\5.0\bin;C:\aliyun-cli;C:\vcpkg;C:\Program Files (x86)\NSIS\;C:\tools\zstd;C:\Program Files\Mercurial\;C:\hostedtoolcache\windows\stack\2.7.5\x64;C:\cabal\bin;C:\\ghcup\bin;C:\tools\ghc-9.4.2\bin;C:\Program Files\dotnet;C:\mysql\bin;C:\Program Files\R\R-4.2.1\bin\x64;C:\SeleniumWebDrivers\GeckoDriver;C:\Program Files (x86)\sbt\bin;C:\Program Files (x86)\GitHub CLI;C:\Program Files\Git\bin;C:\Program Files (x86)\pipx_bin;C:\npm\prefix;C:\hostedtoolcache\windows\go\1.17.13\x64\bin;C:\hostedtoolcache\windows\Python\3.9.13\x64\Scripts;C:\hostedtoolcache\windows\Python\3.9.13\x64;C:\hostedtoolcache\windows\Ruby\3.0.4\x64\bin;C:\tools\kotlinc\bin;C:\hostedtoolcache\windows\Java_Temurin-Hotspot_jdk\8.0.345-1\x64\bin;C:\Program Files (x86)\Microsoft SDKs\Azure\CLI2\wbin;C:\ProgramData\kind;C:\Program Files\Microsoft\jdk-11.0.12.7-hotspot\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Windows\System32\OpenSSH\;C:\Program Files\dotnet\;C:\ProgramData\Chocolatey\bin;C:\Program Files\Docker;C:\Program Files\PowerShell\7\;C:\Program Files\Microsoft\Web Platform Installer\;C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\;C:\Program Files\Microsoft SQL Server\150\Tools\Binn\;C:\Program Files\OpenSSL\bin;C:\Strawberry\c\bin;C:\Strawberry\perl\site\bin;C:\Strawberry\perl\bin;C:\ProgramData\chocolatey\lib\pulumi\tools\Pulumi\bin;C:\Program Files\TortoiseSVN\bin;C:\Program Files\CMake\bin;C:\ProgramData\chocolatey\lib\maven\apache-maven-3.8.6\bin;C:\Program Files\Microsoft Service Fabric\bin\Fabric\Fabric.Code;C:\Program Files\Microsoft SDKs\Service Fabric\Tools\ServiceFabricLocalClusterManager;C:\Program Files\nodejs\;C:\Program Files\Git\cmd;C:\Program Files\Git\mingw64\bin;C:\Program Files\Git\usr\bin;C:\Program Files\GitHub CLI\;c:\tools\php;C:\Program Files (x86)\sbt\bin;C:\SeleniumWebDrivers\ChromeDriver\;C:\SeleniumWebDrivers\EdgeDriver\;C:\Program Files\Amazon\AWSCLIV2\;C:\Program Files\Amazon\SessionManagerPlugin\bin\;C:\Program Files\Amazon\AWSSAMCLI\bin\;C:\Program Files\Microsoft SQL Server\130\Tools\Binn\;C:\Program Files\LLVM\bin;C:\Users\runneradmin\.dotnet\tools;C:\Users\runneradmin\.cargo\bin;C:\Users\runneradmin\AppData\Local\Microsoft\WindowsApps
WINNT found
MSYSTEM=

@chrfranke
Could you please provide the make VERBOSE=1 output for your Cygwin installation? Neither my local one (using "Cygwin Terminal") nor the CI build are successfully detecting Cygwin so the workarounds are never applied.

@chrfranke

Copy link
Copy Markdown
Contributor

@chrfranke Could you please provide the make VERBOSE=1 output for your Cygwin installation? ...

$ printenv | grep -i ^comspec
COMSPEC=C:\WINDOWS\system32\cmd.exe

$ uname -o
Cygwin

$ make VERBOSE=1
WINNT not found
uname_S=CYGWIN_NT-10.0-19044-WOW64
make: 'cppcheck' is up to date.

The Cygwin detection is broken. It is guarded by ifndev COMSPEC ... endif which does not make much sense as current Windows 10 versions set COMSPEC and not ComSpec.

Build works anyway because the workaround CPPFLAGS+=-U__STRICT_ANSI__ is no longer needed since the Makefile fix in 21d992c, see related discussion in PR #4229.

Using $(findstring /cygdrive/,$(PATH)) to detect Cygwin is also unreliable, because /cygdrive/ could be reconfigured, a commonly used value is /mnt/ .

Reliable ways to detect Cygwin are for example:

ifeq ($(shell uname -o),Cygwin)
  CYGWIN=1
endif

or

ifeq ($(shell uname -s | sed 's/_.*$$//'),CYGWIN)
  CYGWIN=1
endif

Note that this detection is wrong if Cygwin is used to build a non-Cygwin version of cppcheck with the MinGW-w64 toolchain but fortunately this could be overridden:

 make CXX=x86_64-w64-mingw32-g++ WINNT=1 CYGWIN=

(If Cygwin is installed, MSYS is not needed!)

In general, typical cross compile scenarios like MinGW-w64 builds on Linux are not handled by the Makefile as it only detects the build platform but not the target platform.

@firewave

firewave commented Sep 4, 2022

Copy link
Copy Markdown
Collaborator Author

@chrfranke
Awesome. Thanks a lot for the thorough and fast investigation. But I only just needed you to verify my assumption. 😁

So the Cygwin detection is broken, the workarounds are not being used at all and the application "just works". So we can just bin them along with the detection. Great.

I have tested MinGW (cmd shell), MSYS2 (own shell) and Cygwin (own shell) locally and finally have those fixed up as well.

Now I just need to do some final tests with the _GLIBCXX_DEBUG defines and then this whole mess is finally sorted out. And we still have a whole development cycles time for people coming out of the weeds and tell us it no longer works and we can handle it better now.

I will put those changes into a follow-up PR. I will CC you on it.

I will also look into cross-compiling since with ccache in place another additional CI build should not impact the turnout times too much anymore.

FYI ComSpec is set when you use cmd as the shell.

@chrfranke

Copy link
Copy Markdown
Contributor

Cygwin detection may still make sense due to the stack size workaround which apparently only affects certain use cases:

ifdef CYGWIN
    ...
    # Set the flag to address compile time warnings
    # with tinyxml2 and Cygwin.
    CPPFLAGS+=-U__STRICT_ANSI__       <=== remove this
    
    # Increase stack size for Cygwin builds to avoid segmentation fault in limited recursive tests.
    CXXFLAGS+=-Wl,--stack,8388608     <=== keep this
endif # CYGWIN

FYI ComSpec is set when you use cmd as the shell.

Indeed, sorry for the noise. I forgot that the Cygwin DLL unconditionally converts certain variables to uppercase, see environ.cc.
This was also the case for older versions of the MSYS DLL which is a fork of Cygwin. I didn't test newer versions because Cygwin also supports MinGW-w64 builds.

Regarding cross compiling, the following logic may be a first start: If CXX has the form CPU-MFR-OPSYS-g++ [OPTIONS] or contains the option --target=..., assume cross compiling. Then disable all automatic platform checks. Settings like WINNT=1 or CYGWIN=1 should be set manually in the make command line then.

A more elaborate approach would be to guess the target platform from CXX. Here an example POC Makefile. Run make test to see what happens:

CXX = g++
UNAME_S := $(shell uname -s 2>/dev/null)

# Platform         uname -s          uname -o       BUILD_HOST
# ============================================================
# Debian, ...      Linux             GNU/Linux      linux
# Cygwin           CYGWIN_<WINVER>   Cygwin         cygwin
# MSYS             MINGW32_<WINVER>  Msys           mingw32
# Debian kFreeBSD  GNU/kFreeBSD?     GNU/kFreeBSD?  kfreebsd
# FreeBSD          FreeBSD           FreeBSD        freebsd
# MacOS/Darwin     Darwin            ?              darwin

BUILD_HOST := $(shell echo "$(UNAME_S)" | \
  tr A-Z a-z | sed -e 's/_.*$$//' -e 's/^gnu\///' \
)

# x86_64 target  CXX for cross-compiler                     TARGET_HOST
# =====================================================================
# Linux          x86_64-linux-gnu-g++                       linux
# Cygwin         x86_64-pc-cygwin-g++                       cygwin
# MinGW-w64      x86_64-w64-mingw32-g++                     mingw32
# FreeBSD 13     clang++ --target=x86_64-unknown-freebsd13  freebsd
# MacOS/Darwin   ?                                          ?

ifneq ($(findstring -,$(CXX)),)
  TARGET_HOST := $(shell echo "$(CXX)" | \
    sed -e 's/^.*--target=\([^ ]*\).*$$/\1/' \
        -e 's/-linux-gnu/-gnu-linux/' -e 's/bsd[0-9]*/bsd/' | \
    sed -n -e 's/^ *[^- ][^- ]*-[^- ][^- ]*-\([^- ][^- ]*\).*$$/\1/p' \
  )
endif
ifeq ($(TARGET_HOST),)
  TARGET_HOST := $(BUILD_HOST)
endif

all:
	@echo "uname -s:    '$(UNAME_S)'"
	@echo "CXX:         '$(CXX)'"
	@echo "BUILD_HOST:  '$(BUILD_HOST)'"
	@echo "TARGET_HOST: '$(TARGET_HOST)'"

test:
	@echo "=== Linux build ==="
	$(MAKE) UNAME_S=Linux
	@echo "=== Cygwin build ==="
	$(MAKE) UNAME_S=CYGWIN_NT-10.0-19044 
	@echo "=== MinGW-w64 build ==="
	$(MAKE) UNAME_S=MINGW32_NT-10.0-19044 
	@echo "=== MinGW-w64 build on Cygwin ==="
	$(MAKE) UNAME_S=CYGWIN_NT-10.0-19044 CXX=x86_64-w64-mingw32-g++
	@echo "=== MinGW-w64 build on Linux ==="
	$(MAKE) UNAME_S=Linux CXX=x86_64-w64-mingw32-g++
	@echo "=== FreeBSD build on Linux ==="
	$(MAKE) UNAME_S=Linux CXX='clang++ --target=x86_64-unknown-freebsd13'

@firewave

firewave commented Sep 5, 2022

Copy link
Copy Markdown
Collaborator Author

Cygwin detection may still make sense due to the stack size workaround which apparently only affects certain use cases:

As it seems that the workaround was never used (at least not for several years - seems like the Cygwin workarounds were targets at a decade-old version) I see no point in keep it without having an actual case.

FYI ComSpec is set when you use cmd as the shell.

Indeed, sorry for the noise. I forgot that the Cygwin DLL unconditionally converts certain variables to uppercase, see environ.cc. This was also the case for older versions of the MSYS DLL which is a fork of Cygwin. I didn't test newer versions because Cygwin also supports MinGW-w64 builds.

No noise at all. Thanks for the pointer - something to consider for more tests.

That was just an FYI since I didn't know that and the comments indicate that this is used as a Windows detection when it actually was a shell detection.

Regarding cross compiling, the following logic may be a first start: If CXX has the form CPU-MFR-OPSYS-g++ [OPTIONS] or contains the option --target=..., assume cross compiling. Then disable all automatic platform checks. Settings like WINNT=1 or CYGWIN=1 should be set manually in the make command line then.

Thanks a lot again for that awesome research. I was thinking more in terms of existing cross-compile use cases. So it appears there might be none and we should not be adding features to the Makefile.

Makefile is not supporting anything it should. That would be CMake...which isn't complete either (yet). But without understanding what the non-CMake systems are doing it is not possible to make CMake properly covers that.

So for cross-compiling I would point people to CMake instead. It's already necessary for the packaging builds which do not use system dependencies (which are not 100% correct as I just stumbled into with macos). And it seems it is already working since there are packages for other architectures. And if something doesn't work the packagers usually point that out fast and I also try to give them heads up when something changes so we don't have to fix post-release.

I understand people have preferences and there's reasons to keep Makefile but (as already stated to death) we just have too many build systems.

@danmar

danmar commented Sep 9, 2022

Copy link
Copy Markdown
Collaborator

So for cross-compiling I would point people to CMake instead.

That is fine for me.

@danmar
danmar merged commit 9542a7b into cppcheck-opensource:main Sep 9, 2022
@firewave
firewave deleted the dmake branch September 9, 2022 17:18
@firewave

firewave commented Sep 9, 2022

Copy link
Copy Markdown
Collaborator Author

So for cross-compiling I would point people to CMake instead.

That is fine for me.

Great. I will try to improve the documentation as well when I look into it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants