Repository navigation
Conversation
|
I suggest you rebase and see if CI will become green. There are compiler errors that might have been fixed. |
a90349e to
c24476b
Compare
|
Rebased onto current main, it applied cleanly and donate_cpu_lib_test.py still passes locally. Let's see what CI says now. Separately, could one of you create a Trac ticket for this? I don't have access, and I'll put the number in the title once it exists. |
dmcppcheck
left a comment
There was a problem hiding this comment.
AI review
The fix is incomplete when Python has no tarfile.data_filter, which is the case it is meant for. The new realpath check runs on the member from the loop, but extraction still calls tf.extract(member.name, ...). Extracting by name looks up the last member with that name in the archive, so a harmless first entry can pass the check while a later entry with the same name, such as a hardlink or symlink that points outside temp, gets extracted. Hardlinks also defeat the realpath check of later members: realpath cannot tell that a file inside temp is a hardlink to a file outside it, and a later regular-file member that writes through that path writes to the outside file. On Python versions with data_filter (3.12 here), the filter still catches this. The fix is to pass the member object to tf.extract. Apart from that I found no new problems.
Comments
| # | Comment | By | Where | Status |
|---|---|---|---|---|
| 1 | extract(member.name) extracts the last member with that name, not the one that was checked | AI review | tools/donate_cpu_lib.py:409 |
🆕 New |
@Nussu06, please fix, resolve or reply to the open and new comments: #1.
Testing
- I did not build
cppcheckortestrunner. The PR only changestools/donate_cpu_lib.pyand its Python test, so no C++ code is affected. PYTHONPATH=tools python3 -m pytest -Werror -q test/tools/donate_cpu_lib_test.pyinpr/with Python 3.12.3: 3 passed.- The same test file run against
base/:test_unpack_package_outside_tempfails as expected, and the other 2 pass. - I checked how name lookup behaves with duplicate member names, which is what the finding depends on:
# scratch/dup.py
import io, tarfile, tempfile, os
d = tempfile.mkdtemp(); p = os.path.join(d, 'x.tar')
with tarfile.open(p, 'w') as tf:
for c in (b'first\n', b'second\n'):
i = tarfile.TarInfo('a.c'); i.size = len(c); tf.addfile(i, io.BytesIO(c))
with tarfile.open(p) as tf:
for m in tf:
tf.extract(m.name, d, filter='data')
print(open(os.path.join(d, 'a.c')).read().strip())
breakpython3 -I dup.py prints second. On the first loop iteration, tf.extract(m.name) extracted the second member, not the one the loop was on.
- I did not reproduce a full escape from
temp. The impact on Python versions withoutdata_filtercomes from reading the code: the check runs on one member while another member with the same name is extracted.
Automated review of c24476b by Claude. The pull request was built and tested locally.
| continue | ||
| try: | ||
| tf.extract(member.name, temp_path) | ||
| tf.extract(member.name, temp_path, **extract_args) |
There was a problem hiding this comment.
AI review
The safety check above validates member, but tf.extract(member.name, ...) goes through getmember(name), which returns the last member with that name in the archive. A tarball can contain a harmless pkg/a.c first and a later pkg/a.c that is a hardlink or symlink pointing outside temp. The first entry passes the check, and extracting it creates the later, unchecked link. The later entry is then skipped, but by that point the link has already been created.
With filter='data' the filter still rejects the link, but on Python versions without tarfile.data_filter (the case this PR protects) nothing stops it. A hardlink is also invisible to the realpath check, so a later regular-file member that writes through that path (for example through a directory symlink that stays inside temp) writes into the file outside temp.
Passing the member object fixes this:
tf.extract(member, temp_path, **extract_args)It would be worth adding a duplicate-name case to test_unpack_package_outside_temp.
There was a problem hiding this comment.
Good catch, extracting by name picks up the last entry with that name, so the check and the extraction could be looking at different members. It now passes the member object to tf.extract. I added test_unpack_package_duplicate_name, which has a regular pkg/sym.c followed by a same-named symlink to a file outside temp, and the same for a hardlink. It fails with the old code on Python 3.10.11 (no data_filter) and passes with the fix on 3.10 and 3.11. The Trac ticket is still open on my side, if someone can create one I'll put the number in the title.
dmcppcheck
left a comment
There was a problem hiding this comment.
AI review
On Python versions without tarfile.data_filter, which this PR is meant to protect, a symlink that points outside temp can still be created. The symlink target is resolved against os.path.dirname(member_path), but member_path has already gone through realpath. If the archive first creates pkg/inc.h as a symlink to a deep path inside temp, a second pkg/inc.h symlink to ../../outside is checked against that deep directory and looks inside temp. tarfile then replaces the link at temp/pkg/inc.h, so the new link resolves relative to temp/pkg and points outside. Later members that write through the link are still skipped. But cppcheck follows directory symlinks (stat in cli/filelister.cpp), so donate-cpu would analyse files outside the package, and their results would be uploaded. I found no other new problems, and the earlier comment about extracting by name is fixed.
Comments
| # | Comment | By | Where | Status |
|---|---|---|---|---|
| 1 | extract(member.name) extracts the last member with that name, not the checked one | AI review | general | ✅ Fixed |
| 2 | Symlink target is resolved from the realpath of an existing link, so a replacing symlink can point outside temp | AI review | tools/donate_cpu_lib.py:386 |
🆕 New |
@Nussu06, please fix, resolve or reply to the open and new comments: #2.
Testing
- I did not build
cppcheckortestrunner, because the PR only changestools/donate_cpu_lib.pyand its Python test. PYTHONPATH=tools python3 -m pytest -q test/tools/donate_cpu_lib_test.pyinpr/with Python 3.12.3: 4 passed. They also pass withtarfile.data_filterdeleted, to simulate an older Python.- I reproduced the symlink escape with Python 3.12 and
data_filterremoved:
# scratch/esc.py — run: python3 -I esc.py /work/pr/tools nofilter
import io, os, sys, tarfile, tempfile
sys.path.insert(0, sys.argv[1])
if len(sys.argv) > 2:
del tarfile.data_filter # simulate Python without extraction filters
import donate_cpu_lib as d
w = tempfile.mkdtemp(); out = os.path.join(w, 'outside'); os.mkdir(out)
open(os.path.join(out, 'secret.c'), 'w').write('int secret;\n')
tgz = os.path.join(w, 'p.tgz')
def add(tf, name, link=None):
i = tarfile.TarInfo(name); data = b'int x;\n'
if link: i.type = tarfile.SYMTYPE; i.linkname = link; data = b''
else: i.size = len(data)
tf.addfile(i, io.BytesIO(data))
with tarfile.open(tgz, 'w:gz') as tf:
add(tf, 'pkg/main.c')
add(tf, 'pkg/inc.h', 'a/b/c') # inside temp
add(tf, 'pkg/inc.h', '../../outside') # checked relative to temp/pkg/a/b, created in temp/pkg
tp, found = d.unpack_package(w, tgz)
l = os.path.join(tp, 'pkg', 'inc.h')
print('link:', os.readlink(l) if os.path.islink(l) else None, '->', os.path.realpath(l))
print('listing via temp:', os.listdir(l) if os.path.isdir(l) else None)Output without the filter:
extracted 3 of 3 files (skipped 0)
link: ../../outside -> /tmp/tmpbyf2t84d/outside
listing via temp: ['secret.c']
With filter='data' (the normal 3.12 run), the second link is rejected and temp/pkg/inc.h stays a/b/c.
Automated review of 998d32a by Claude. The pull request was built and tested locally.
| member_path = os.path.realpath(os.path.join(temp_path, member.name)) | ||
| link_path = None | ||
| if member.issym(): | ||
| link_path = os.path.realpath(os.path.join(os.path.dirname(member_path), member.linkname)) |
There was a problem hiding this comment.
AI review
member_path has already gone through realpath. If an earlier member created a symlink with the same name (for example pkg/inc.h -> a/b/c), os.path.dirname(member_path) is temp/pkg/a/b, not temp/pkg. tarfile's makelink unlinks the old link and creates the new one at temp/pkg/inc.h, so its relative target resolves from temp/pkg.
Example: a second member pkg/inc.h -> ../../outside passes this check, because from the deep directory it resolves to temp/outside. On disk it points to work_path/outside. I reproduced this with data_filter removed: all 3 members were extracted, and os.listdir(temp/pkg/inc.h) listed the files in the outside folder. cppcheck follows directory symlinks when it scans temp (stat in cli/filelister.cpp), so files outside the package would be analysed and reported.
Resolve the link target from the real parent directory of the unresolved destination instead:
parent = os.path.realpath(os.path.dirname(os.path.join(temp_path, member.name)))
link_path = os.path.realpath(os.path.join(parent, member.linkname))This case could be added to test_unpack_package_duplicate_name.
There was a problem hiding this comment.
Right, realpath on the destination follows the earlier pkg/inc.h link, so the target was resolved from temp/pkg/a/b instead of temp/pkg. The link target is now resolved from the realpath of the unresolved parent directory, as you suggested. I added the pkg/inc.h -> a/b/c followed by pkg/inc.h -> ../../outside case to test_unpack_package_duplicate_name. The test file passes on 3.10.11 (no data_filter) and 3.11, but my Windows box can't create symlinks, so that new case only really exercises the code on the Linux CI runs. A Trac ticket is still needed for the title, could one of you create it?
dmcppcheck
left a comment
There was a problem hiding this comment.
AI review
On Python versions without tarfile.data_filter, a symlink that points outside temp can still be created. The check resolves the link target with realpath when the link is extracted. Components of the target that don't exist yet are then normalised as plain text, but a later member can turn them into a symlink. For example, pkg/l.h -> d.h/../../outside is checked while d.h doesn't exist yet, so it looks like temp/outside. A later pkg/d.h -> . passes its own check. After both are extracted, l.h resolves to work_path/outside. With filter='data' (Python 3.12.3 here) the link is rewritten and stays inside temp. I found no other new problems. The two earlier findings (extracting by name, and resolving the target through an existing link) are fixed.
Comments
| # | Comment | By | Where | Status |
|---|---|---|---|---|
| 1 | extract(member.name) extracts the last member with that name, not the checked one | AI review | general | ✅ Fixed |
| 2 | Symlink target resolved through realpath of an existing same-named link can point outside temp | AI review | tools/donate_cpu_lib.py:386 |
✅ Fixed |
| 3 | Symlink target checked with realpath before later members exist; a later symlink can redirect it outside temp | AI review | tools/donate_cpu_lib.py:388 |
🆕 New |
@Nussu06, please fix, resolve or reply to the open and new comments: #3.
Testing
- I did not build
cppcheckortestrunner: the PR only changestools/donate_cpu_lib.pyand its Python test. PYTHONPATH=tools python3 -m pytest -q test/tools/donate_cpu_lib_test.pyinpr/with Python 3.12.3: 4 passed.- Escape through a symlink that is created later. I deleted
tarfile.data_filterto simulate an older Python:
# scratch/esc2.py — run: python3 -I esc2.py /work/pr/tools nofilter
import io, os, sys, tarfile, tempfile
sys.path.insert(0, sys.argv[1])
if len(sys.argv) > 2:
del tarfile.data_filter
import donate_cpu_lib as d
w = tempfile.mkdtemp(); out = os.path.join(w, 'outside'); os.mkdir(out)
open(os.path.join(out, 'secret.c'), 'w').write('int secret;\n')
tgz = os.path.join(w, 'p.tgz')
def add(tf, name, link=None):
i = tarfile.TarInfo(name); data = b'int x;\n'
if link: i.type = tarfile.SYMTYPE; i.linkname = link; data = b''
else: i.size = len(data)
tf.addfile(i, io.BytesIO(data))
with tarfile.open(tgz, 'w:gz') as tf:
add(tf, 'pkg/main.c')
add(tf, 'pkg/l.h', 'd.h/../../outside') # d.h does not exist yet -> checked as temp/outside
add(tf, 'pkg/d.h', '.') # inside temp
tp, found = d.unpack_package(w, tgz)
l = os.path.join(tp, 'pkg', 'l.h')
print('link:', os.readlink(l) if os.path.islink(l) else None, '->', os.path.realpath(l))
print('listing via temp:', os.listdir(l) if os.path.isdir(l) else None)Output without the filter:
extracted 3 of 3 files (skipped 0)
link: d.h/../../outside -> /tmp/tmp62_2fkdw/outside
listing via temp: ['secret.c']
Output with the filter (python3 -I esc2.py /work/pr/tools): the link is rewritten to ../outside -> .../temp/outside, so nothing escapes.
Automated review of 904ee65 by Claude. The pull request was built and tested locally.
| if member.issym(): | ||
| # resolve from the directory the link is created in, not through an existing link with the same name | ||
| link_parent = os.path.realpath(os.path.dirname(os.path.join(temp_path, member.name))) | ||
| link_path = os.path.realpath(os.path.join(link_parent, member.linkname)) |
There was a problem hiding this comment.
AI review
This realpath only resolves the link target as it is on disk right now. Components that don't exist yet are normalised as plain text, but a later member of the archive can make one of them a symlink. Example:
pkg/l.h -> d.h/../../outside # d.h missing: checked as temp/outside, passes
pkg/d.h -> . # temp/pkg, passes
After extraction, temp/pkg/l.h resolves to work_path/outside. I reproduced this with tarfile.data_filter deleted: all 3 members were extracted, and os.listdir(temp/pkg/l.h) listed the outside folder. cppcheck follows directory symlinks when it scans temp, so files outside the package would be analysed.
Checking each member when it is extracted can't catch this, because the target can change after the check. There are two options. One is to reject symlinks whose linkname is absolute or has a .. component. The other is a pass after extraction that removes every symlink under temp whose realpath is outside temp_path. That pass needs to loop until nothing more is removed, because removing one link can change what another resolves to.
unpack_package in tools/donate_cpu_lib.py only skips archive members whose name starts with '/' or '..', so a package tarball containing pkg/../../outside/x.c, or a symlink pkg/inc.h pointing at another directory followed by pkg/inc.h/x.c, writes source files outside work_path/temp when donate-cpu.py, test-my-pr.py or compare-valueflow-options.py unpack it on Python before 3.14 (3.14 refuses these by default, but the uncaught OutsideDestinationError aborts the run). Each member's destination, and the target of symlink and hardlink members, is now resolved with realpath and skipped unless it stays inside the temp folder, and filter='data' is passed where tarfile supports it; links that stay inside the package, such as include/a.h -> ../src/a.h, are still extracted. Added test_unpack_package_outside_temp, which fails on main with Python 3.9 and 3.12 and passes with the fix on 3.9, 3.12 and 3.14, and bumped CLIENT_VERSION to 1.4.1. I don't have Trac access, so could you create a ticket for this? I'll update the title once there is a number.