X11 not being linked into static library

So I am building a tiny "game engine" in C++, using cmake to build it, and I am making it as a static library. In windows it works perfectly, but in linux I need to link X11.

Now this wouldn't be a problem, I can just add add_link_options(-lX11) in CMakeLists.txt, and mylib.a will build fine, but when I go and use my mylib.a, and link it into a C++ program, I also need to link X11 again using the add_link_options command.

I am trying to do something similar to what raylib does, and I also want to do linking and including the library how raylib does it (in raylib you only need to link libraylib.a, and everything else is taken care of), but in my case, every time I want to link mylib.a into a C++ project, I also need to link X11.

If I make mylib to be shared (mylib.so), it works without the need to link X11 in the second project as well, but I would like to have a static version of the library working as well.

Here's my CMakeLists.txt for my library:

cmake_minimum_required(VERSION 3.0.0)
project(ntt VERSION 0.1.0)

set(CMAKE_CXX_STANDARD 20)
set(CMAKE_CXX_STANDARD_REQUIRED TRUE)
set(CMAKE_C_STANDARD 17)
set(CMAKE_C_STANDARD_REQUIRED TRUE)

add_link_options(-lX11)

# GLFW config (this one I copied from raylib directly)
set(BUILD_SHARED_LIBS OFF)
set(GLFW_BUILD_EXAMPLES OFF)
set(GLFW_BUILD_DOCS OFF)
set(GLFW_BUILD_TESTS OFF)
set(GLFW_INSTALL OFF)
add_subdirectory("${CMAKE_SOURCE_DIR}/src/glfw")
add_compile_definitions(_GLFW_X11)

# Generate source list
file(GLOB SOURCES "${CMAKE_SOURCE_DIR}/src/*.cpp")
list(APPEND SOURCES "${CMAKE_SOURCE_DIR}/src/glad/src/glad.c")
list(APPEND SOURCES $<TARGET_OBJECTS:glfw>)

# Create the library
add_library(ntt ${SOURCES})
set_target_properties(ntt PROPERTIES
    PUBLIC_HEADER "${CMAKE_SOURCE_DIR}/include/Engine.hpp"
)

# Add include directories
target_include_directories(
    ntt
        PUBLIC
            "${CMAKE_SOURCE_DIR}/include"
        PRIVATE
            "${CMAKE_SOURCE_DIR}/src/glad/include"
            "${CMAKE_SOURCE_DIR}/src/glfw/include"
)

And here's CMakeLists.txt for a project that uses my library, and doesn't work (unless I add the add_link_options(-lX11)):

cmake_minimum_required(VERSION 3.0.0)
project(TestCpp VERSION 0.1.0)

# I want to not need to have this line here
# add_link_options(-lX11)

add_executable(TestCpp main.cpp)

target_include_directories(TestCpp PUBLIC "${CMAKE_SOURCE_DIR}/include")
target_link_libraries(TestCpp PUBLIC "${CMAKE_SOURCE_DIR}/libntt.a")

When I try to build the second project I get a bunch of errors like this: /usr/bin/ld: /media/hdd/Development/C-C++/test/libntt.a(x11_monitor.c.o): in function `_glfwPollMonitorsX11': x11_monitor.c:(.text+0x3e8): undefined reference to `XFree' (they all come from places like x11_window, or x11_monitor, and all of them are abount missing references to different X11 functions used by glfw)

I know I must be missing something really easy, but I just don't know what. I am not new to C++, but this is the first time I've made an actual proper library, and I'm also pretty new to cmake (so pls excuse bad practices you might see in my CMakeLists).

I mostly relied on Microsoft's Visual Studio build system until recently, and the only reason I switched to cmake was to be able to port my code to more than one platform(windows) without jumping through the weird hoops that VS makes you jump through when configuring the project, and now I'm starting to like it, but every once in a while I bump into something like this.

So, in short: X11 links properly with -lX11, but I would like to only have the user link my library, and not have to bother with also linking X11 manually.

With raylib you only need to do target_link_libraries(mytarget PUBLIC raylib.a) and that links X11 and opengl and everything else needed, but in my case you also need to add_link_options(-lX11), and I want to not have to do that.