Skip to content

Switch to LTRData.DiscUtils fork #145

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
gfs merged 14 commits into from
Jan 30, 2024
Merged

Switch to LTRData.DiscUtils fork #145

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
14 commits
Select commit Hold shift + click to select a range
19c6057
Switch nuget package to LTRData.DiscUtils
Jan 18, 2024
60c14d8
Add regression test for corrupt WIM
Jan 20, 2024
35358b5
Include WIM test file in output
Jan 20, 2024
e5f8b32
Add DMG test cases
Jan 20, 2024
a53ce45
Update to latest DiscUtils with WIM fix
Jan 24, 2024
992f694
Handle missing WIM image
Jan 24, 2024
6cb991a
Update pipeline templates.
gfs Jan 25, 2024
cff057a
Small error handling changes in WimExtractor.
gfs Jan 25, 2024
45a2dab
More descriptive error message for failure to read file from wim
gfs Jan 25, 2024
e59e966
Update Dependencies (SharpCompress)
gfs Jan 25, 2024
de92670
Remove unused Workflow Files
gfs Jan 25, 2024
a73b079
Implement DMG Extractor
gfs Jan 26, 2024
8490639
Update README.md
gfs Jan 26, 2024
6ed0118
Update DiscUtils with Path.Combine fix
Jan 27, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 0 additions & 26 deletions .github/workflows/publish-code-coverage.yml
View file
Open in desktop

This file was deleted.

31 changes: 0 additions & 31 deletions .github/workflows/publish-wasm-to-gh-pages.yml
View file
Open in desktop

This file was deleted.

6 changes: 3 additions & 3 deletions Pipelines/recursive-extractor-pr.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ resources:
- repository: templates
type: git
name: SecurityEngineering/OSS-Tools-Pipeline-Templates
ref: refs/tags/v1.1.0
ref: refs/tags/v1.1.1

variables:
BuildConfiguration: 'Release'
Expand All @@ -27,13 +27,13 @@ stages:
parameters:
jobName: 'lib_dotnet_test_windows'
dotnetVersions: ['6.0.x','7.0.x','8.0.x']
vmImage: 'oss-tools-win2022_1es-managed'
vmImage: 'win2022-image-base'
projectPath: 'RecursiveExtractor.Tests/RecursiveExtractor.Tests.csproj'
- template: dotnet-test-job.yml@templates
parameters:
jobName: 'cli_dotnet_test_windows'
dotnetVersions: ['6.0.x','7.0.x','8.0.x']
vmImage: 'oss-tools-win2022_1es-managed'
vmImage: 'win2022-image-base'
projectPath: 'RecursiveExtractor.Cli.Tests/RecursiveExtractor.Cli.Tests.csproj'

- stage: SDL
Expand Down
8 changes: 4 additions & 4 deletions Pipelines/recursive-extractor-release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ resources:
- repository: templates
type: git
name: SecurityEngineering/OSS-Tools-Pipeline-Templates
ref: refs/tags/v1.1.0
ref: refs/tags/v1.1.1

variables:
BuildConfiguration: 'Release'
Expand All @@ -32,13 +32,13 @@ stages:
parameters:
jobName: 'lib_dotnet_test_windows'
dotnetVersions: ['6.0.x','7.0.x','8.0.x']
vmImage: 'oss-tools-win2022_1es-managed'
vmImage: 'win2022-image-base'
projectPath: 'RecursiveExtractor.Tests/RecursiveExtractor.Tests.csproj'
- template: dotnet-test-job.yml@templates
parameters:
jobName: 'cli_dotnet_test_windows'
dotnetVersions: ['6.0.x','7.0.x','8.0.x']
vmImage: 'oss-tools-win2022_1es-managed'
vmImage: 'win2022-image-base'
projectPath: 'RecursiveExtractor.Cli.Tests/RecursiveExtractor.Cli.Tests.csproj'

- stage: SDL
Expand Down Expand Up @@ -87,7 +87,7 @@ stages:
displayName: Code Sign, Generate Hashes, Publish Public Releases
pool:
name: 'OSS-Tools-1ESPool'
vmImage: 'oss-tools-win2022_1es-managed'
vmImage: 'win2022-image-base'
steps:
- task: UseDotNet@2 # For ESRP. Do not use variable.
inputs:
Expand Down
32 changes: 16 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,27 +7,28 @@ Recursive Extractor is a Cross-Platform [.NET Standard 2.0 Library](#library) an
| | | |
|-|-|-|
| 7zip+ | ar | bzip2 |
| deb | gzip | iso |
| rar^ | tar | vhd |
| vhdx | vmdk | wim* |
| xzip | zip+ | |
| deb | dmg** | gzip |
| iso | rar^ | tar |
| vhd | vhdx | vmdk |
| wim* | xzip | zip+ |

<details>
<summary>Details</summary>
<br/>
* Windows only<br/>
+ Encryption Supported<br/>
^ Rar version 4 Encryption supported<br/>
^ Encryption supported for Rar version 4 only<br/>
** Limited support. Unencrypted HFS+ volumes with certain compression schemes.
</details>

# Variants

## Command Line
### Installing
1. Ensure you have the latest [.NET SDK](https://dotnet.microsoft.com/download).
2. run `dotnet tool install -g Microsoft.CST.RecursiveExtractor.Cli`
2. Run `dotnet tool install -g Microsoft.CST.RecursiveExtractor.Cli`

This adds `RecursiveExtractor` to your path so you can run it directly from the shell.
This adds `RecursiveExtractor` to your path so you can run it directly from your shell.

### Running
Basic usage is: `RecursiveExtractor --input archive.ext --output outputDirectory`
Expand Down Expand Up @@ -57,7 +58,7 @@ Run `RecursiveExtractor --help` for more details.
</details>

## .NET Standard Library
Recursive Extractor is available on NuGet as [Microsoft.CST.RecursiveExtractor](https://www.nuget.org/packages/Microsoft.CST.RecursiveExtractor/). Recursive Extractor targets netstandard2.0+ and the latest .NET, currently .NET 6.0 and .NET 7.0.
Recursive Extractor is available on NuGet as [Microsoft.CST.RecursiveExtractor](https://www.nuget.org/packages/Microsoft.CST.RecursiveExtractor/). Recursive Extractor targets netstandard2.0+ and the latest .NET, currently .NET 6.0, .NET 7.0 and .NET 8.0.

### Usage

Expand All @@ -77,8 +78,7 @@ foreach(var file in extractor.Extract(path))
<details>
<summary>Extracting to Disk</summary>
<br/>
This code adapted from the Cli extracts the contents of given archive located at `options.Input`
to a directory located at `options.Output`, including extracting failed archives as themselves.
This code adapted from the Cli extracts the contents of given archive located at `options.Input` to a directory located at `options.Output`, including extracting failed archives as themselves.

```csharp
using Microsoft.CST.RecursiveExtractor;
Expand Down Expand Up @@ -166,12 +166,12 @@ catch(OverflowException)
RecursiveExtractor protects against [ZipSlip](https://snyk.io/research/zip-slip-vulnerability), [Quines, and Zip Bombs](https://en.wikipedia.org/wiki/Zip_bomb).
Calls to Extract will throw an `OverflowException` when a Quine or Zip bomb is detected and a `TimeOutException` if `EnableTiming` is set and the specified time period has elapsed before completion.

Otherwise, invalid files found while crawling will emit a logger message and be skipped. RecursiveExtractor uses NLog for logging.
Otherwise, invalid files found while crawling will emit a logger message and be skipped. You can also enable `ExtractSelfOnFail` to return the original archive file on an extraction failure.

## Notes on Enumeration

### Multiple Enumeration
You should not iterate the Enumeration returned from the `Extract` and `ExtractAsync` interfaces multiple times, if you need to do so, convert the Enumeration to the collection of your choice first.
You should not iterate the Enumeration returned from the `Extract` and `ExtractAsync` interfaces multiple times, if you need to do so, convert the Enumeration to an in memory collection first.

### Parallel Enumeration
If you want to enumerate the output with parallelization you should use a batching mechanism, for example:
Expand Down Expand Up @@ -208,7 +208,7 @@ while (moreAvailable)
```

### Disposing During Enumeration
If you are working with a very large archive or in particularly constrained environment you can reduce memory/file handle usage for the Content streams in each FileEntry by disposing as you iterate.
If you are working with a very large archive or in particularly constrained environment you can reduce memory and file handle usage for the Content streams in each FileEntry by disposing as you iterate.

```csharp
var results = extractor.Extract(path);
Expand All @@ -217,7 +217,7 @@ foreach(var file in results)
using var theStream = file.Content;
// Do something with the stream.
_ = theStream.ReadByte();
// The stream is disposed here from the using statement
// The stream is disposed here by the using statement
}
```

Expand All @@ -229,11 +229,11 @@ If you are having trouble parsing a specific archive of one of the supported for

# Dependencies

Recursive Extractor uses a number of libraries to parse archives.
Recursive Extractor aims to provide a unified interface to extract arbitrary archives and relies on a number of libraries to parse the archives.

* [SharpZipLib](https://github.com/icsharpcode/SharpZipLib)
* [SharpCompress](https://github.com/adamhathcock/sharpcompress)
* [DiscUtils](https://github.com/discutils/discutils)
* [LTRData/DiscUtils](https://github.com/LTRData/discutils)

# Contributing

Expand Down
4 changes: 2 additions & 2 deletions RecursiveExtractor.Blazor/RecursiveExtractor.Blazor.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@
</PropertyGroup>

<ItemGroup>
<PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly" Version="8.0.0" />
<PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly.DevServer" Version="8.0.0" PrivateAssets="all" />
<PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly" Version="8.0.1" />
<PackageReference Include="Microsoft.AspNetCore.Components.WebAssembly.DevServer" Version="8.0.1" PrivateAssets="all" />
<PackageReference Include="System.Net.Http.Json" Version="8.0.0" />
<PackageReference Include="Tewr.Blazor.FileReader" Version="3.3.2.23201" />
</ItemGroup>
Expand Down
4 changes: 2 additions & 2 deletions RecursiveExtractor.Cli.Tests/RecursiveExtractor.Cli.Tests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,8 @@

<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
<PackageReference Include="MSTest.TestAdapter" Version="3.1.1" />
<PackageReference Include="MSTest.TestFramework" Version="3.1.1" />
<PackageReference Include="MSTest.TestAdapter" Version="3.2.0" />
<PackageReference Include="MSTest.TestFramework" Version="3.2.0" />
<PackageReference Include="coverlet.collector" Version="6.0.0">
<PrivateAssets>all</PrivateAssets>
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
Expand Down
6 changes: 4 additions & 2 deletions RecursiveExtractor.Tests/ExtractorTests/ExpectedNumFilesTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,8 @@ public static IEnumerable<object[]> ArchiveData
new object[] { "EmptyFile.txt", 1 },
new object[] { "TestDataArchivesNested.Zip", 54 },
new object[] { "UdfTest.iso", 3 },
new object[] { "UdfTestWithMultiSystem.iso", 3 }
new object[] { "UdfTestWithMultiSystem.iso", 3 },
new object[] { "HfsSampleUDCO.dmg", 2 }
};
}
}
Expand Down Expand Up @@ -73,7 +74,8 @@ public static IEnumerable<object[]> NoRecursionData
new object[] { "TestData.wim", 3 },
new object[] { "EmptyFile.txt", 1 },
new object[] { "TestDataArchivesNested.Zip", 14 },
new object[] { "UdfTestWithMultiSystem.iso", 3 }
new object[] { "UdfTestWithMultiSystem.iso", 3 },
new object[] { "HfsSampleUDCO.dmg", 2 }
};
}
}
Expand Down
1 change: 1 addition & 0 deletions RecursiveExtractor.Tests/ExtractorTests/MiniMagicTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ public class MiniMagicTests : BaseExtractorTestClass
[DataRow("TestData.vhdx", ArchiveFileType.VHDX)]
[DataRow("TestData.wim", ArchiveFileType.WIM)]
[DataRow("Empty.vmdk", ArchiveFileType.VMDK)]
[DataRow("HfsSampleUDCO.dmg", ArchiveFileType.DMG)]
[DataRow("EmptyFile.txt", ArchiveFileType.UNKNOWN)]
public void TestMiniMagic(string fileName, ArchiveFileType expectedArchiveFileType)
{
Expand Down
1 change: 1 addition & 0 deletions RecursiveExtractor.Tests/ExtractorTests/MiscTests.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ public async Task ExtractFlatFileAsync(string fileName, bool requireTopLevelToBe
[DataRow("TestDataCorrupt.tar", true, 1, 1)]
[DataRow("TestDataCorrupt.tar.zip", false, 0, 2)]
[DataRow("TestDataCorrupt.tar.zip", true, 0, 2)]
[DataRow("TestDataCorruptWim.zip", true, 0, 0)]
public void ExtractCorruptArchive(string fileName, bool requireTopLevelToBeArchive, int expectedNumFailures, int expectedNumFiles)
{
var extractor = new Extractor();
Expand Down
36 changes: 30 additions & 6 deletions RecursiveExtractor.Tests/RecursiveExtractor.Tests.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,12 @@

<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
<PackageReference Include="DiscUtils.Btrfs" Version="0.16.13" />
<PackageReference Include="DiscUtils.HfsPlus" Version="0.16.13" />
<PackageReference Include="DiscUtils.SquashFs" Version="0.16.13" />
<PackageReference Include="DiscUtils.Xfs" Version="0.16.13" />
<PackageReference Include="MSTest.TestAdapter" Version="3.1.1" />
<PackageReference Include="MSTest.TestFramework" Version="3.1.1" />
<PackageReference Include="LTRData.DiscUtils.Btrfs" Version="1.0.36" />
<PackageReference Include="LTRData.DiscUtils.HfsPlus" Version="1.0.36" />
<PackageReference Include="LTRData.DiscUtils.SquashFs" Version="1.0.36" />
<PackageReference Include="LTRData.DiscUtils.Xfs" Version="1.0.36" />
<PackageReference Include="MSTest.TestAdapter" Version="3.2.0" />
<PackageReference Include="MSTest.TestFramework" Version="3.2.0" />
<PackageReference Include="NLog.Extensions.Logging" Version="5.3.8" />
<PackageReference Include="Newtonsoft.Json" Version="13.0.3" />
</ItemGroup>
Expand Down Expand Up @@ -110,6 +110,12 @@
<None Update="TestData\TestDataArchives\100trees.7z">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\DmgSampleFolderCompressed.dmg">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\DmgSampleFolderReadOnly.dmg">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\Empty.vmdk">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
Expand All @@ -122,9 +128,24 @@
<None Update="TestData\TestDataArchives\EncryptedWithPlainNames.rar4">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\HfsSample.dmg">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\HfsSampleUDCO.dmg">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\HfsSampleUlmo.dmg">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\Lorem.txt">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\Shared.dmg">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\SharedDmg.zip">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\sysvbanner_1.0-17fakesync1_amd64.deb">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
Expand Down Expand Up @@ -176,6 +197,9 @@
<None Update="TestData\TestDataArchives\TestDataCorrupt.tar.zip">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\TestDataCorruptWim.zip">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
<None Update="TestData\TestDataArchives\TestDataEncrypted.7z">
<CopyToOutputDirectory>PreserveNewest</CopyToOutputDirectory>
</None>
Expand Down
Binary file added RecursiveExtractor.Tests/TestData/TestDataArchives/HfsSampleUDCO.dmg
View file
Open in desktop
Binary file not shown.
Binary file added RecursiveExtractor.Tests/TestData/TestDataArchives/TestDataCorruptWim.zip
View file
Open in desktop
Binary file not shown.
8 changes: 4 additions & 4 deletions RecursiveExtractor.sln
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio Version 16
VisualStudioVersion = 16.0.30309.148
# Visual Studio Version 17
VisualStudioVersion = 17.8.34408.163
MinimumVisualStudioVersion = 10.0.40219.1
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "RecursiveExtractor", "RecursiveExtractor\RecursiveExtractor.csproj", "{A7F7492B-60E0-468C-B267-BA60EC131E86}"
EndProject
Expand All @@ -15,9 +15,9 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "Solution Items", "Solution
EndProject
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "RecursiveExtractor.Blazor", "RecursiveExtractor.Blazor\RecursiveExtractor.Blazor.csproj", "{18D0803C-052E-4338-9162-F2DB8F8E51E2}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "RecursiveExtractor.Cli", "RecursiveExtractor.Cli\RecursiveExtractor.Cli.csproj", "{443B4E50-9AAF-436E-B3DF-644F782AF9B6}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "RecursiveExtractor.Cli", "RecursiveExtractor.Cli\RecursiveExtractor.Cli.csproj", "{443B4E50-9AAF-436E-B3DF-644F782AF9B6}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "RecursiveExtractor.Cli.Tests", "RecursiveExtractor.Cli.Tests\RecursiveExtractor.Cli.Tests.csproj", "{F37B314B-F641-4336-BCD6-BC5B85BEC5DB}"
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "RecursiveExtractor.Cli.Tests", "RecursiveExtractor.Cli.Tests\RecursiveExtractor.Cli.Tests.csproj", "{F37B314B-F641-4336-BCD6-BC5B85BEC5DB}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Expand Down
1 change: 1 addition & 0 deletions RecursiveExtractor/Extractor.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ public void SetDefaultExtractors()
SetExtractor(ArchiveFileType.VMDK, new VmdkExtractor(this));
SetExtractor(ArchiveFileType.XZ, new XzExtractor(this));
SetExtractor(ArchiveFileType.ZIP, new ZipExtractor(this));
SetExtractor(ArchiveFileType.DMG, new DmgExtractor(this));
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
SetExtractor(ArchiveFileType.WIM, new WimExtractor(this));
Expand Down
Loading