-
Notifications
You must be signed in to change notification settings - Fork 3
chore: Added tools project for internal tasks + updated README #114
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,3 @@ | ||
// Global using directives | ||
|
||
global using MagicBytesValidator.Services; |
12 changes: 12 additions & 0 deletions
12
MagicBytesValidator.Tools/MagicBytesValidator.Tools.csproj
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,12 @@ | ||
<Project Sdk="Microsoft.NET.Sdk"> | ||
|
||
<PropertyGroup> | ||
<ImplicitUsings>enable</ImplicitUsings> | ||
<OutputType>exe</OutputType> | ||
</PropertyGroup> | ||
|
||
<ItemGroup> | ||
<ProjectReference Include="..\MagicBytesValidator\MagicBytesValidator.csproj" /> | ||
</ItemGroup> | ||
|
||
</Project> |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
namespace MagicBytesValidator.Tools; | ||
|
||
public class Program | ||
{ | ||
public static void Main(string[] args) | ||
{ | ||
var argument = args.FirstOrDefault()?.ToLower() ?? "help"; | ||
|
||
switch (argument) | ||
{ | ||
case "table": | ||
PrintTypesTable(); | ||
break; | ||
case "help": | ||
PrintHelp(); | ||
break; | ||
default: | ||
Console.Error.WriteLine($"Unknown option '{argument}'. Maybe try using 'help'?"); | ||
break; | ||
} | ||
} | ||
|
||
private static void PrintTypesTable() | ||
{ | ||
Console.WriteLine(); | ||
Console.WriteLine("|FileType|Extensions|MIME Types|"); | ||
Console.WriteLine("|-|-|-|"); | ||
|
||
foreach (var fileType in (new Mapping()).FileTypes) | ||
{ | ||
var mimeTypeList = string.Join(", ", fileType.MimeTypes); | ||
var extensionList = string.Join(", ", fileType.Extensions); | ||
|
||
Console.WriteLine($"|{fileType.GetType().Name.ToUpper()}|{extensionList}|{mimeTypeList}|"); | ||
} | ||
} | ||
|
||
private static void PrintHelp() | ||
{ | ||
Console.WriteLine(""" | ||
|
||
Usage: dotnet run -- [tool] | ||
|
||
Available tools: | ||
|
||
- table | ||
Prints a markdown table containing all known file types, including extensions and MIME types. | ||
Useful for updating the project README file. | ||
|
||
- help | ||
Shows this help | ||
|
||
(traperto GmbH, 2024) | ||
"""); | ||
} | ||
} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,20 @@ | ||
# MagicByteValidator.Tools | ||
|
||
## About this project | ||
|
||
This is a helper project for internal tasks, e. g. updating the main [README.md](../README.md). | ||
**It is not meant to be used by non-traperto developers.** | ||
If you want to debug your file types or sample files, use the [MagicBytesValidator.CLI](../MagicBytesValidator.CLI/) | ||
project. | ||
|
||
## Usage | ||
|
||
```shell | ||
dotnet run --project MagicBytesValidator.Tools -- [tool] | ||
``` | ||
|
||
### Available tools | ||
|
||
- _table_: Prints a markdown table containing all known file types, including extensions and MIME types. | ||
Useful for updating the project README file. | ||
- _help_: Shows help page of this project. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -1,6 +1,6 @@ | ||
# MagicBytesValidator | ||
Recognize filetypes from `Streams` or `IFormfiles` using mime types or file extensions and validate them against the magic bytes according to the filetypes. | ||
The existing `FileTypes` can be expanded in various ways. | ||
Recognize file types from `Stream`s or `IFormFile`s using MIME types or file extensions and validate them against the magic bytes according to the file types. | ||
The existing `IFileType`s can be expanded in various ways. | ||
|
||
### How to install? | ||
|
||
|
@@ -21,19 +21,19 @@ dotnet add package MagicBytesValidator --version 2.0.2 | |
|
||
### How to use it? | ||
|
||
- Create new instances of the validator & providers: | ||
- Create new instances of the validator and providers: | ||
```c# | ||
var validator = new MagicBytesValidator.Services.Validator(); | ||
var formFileTypeProvider = new MagicBytesValidator.Services.Http.FormFileTypeProvider(); | ||
var streamFileTypeProvider = new MagicBytesValidator.Services.Streams.StreamFileTypeProvider(); | ||
``` | ||
|
||
- Find a filetype by extension or mimetype: | ||
- Find a file type by extension or MIME type: | ||
```c# | ||
var pngFileType = validator.Mapping.FindByExtension("png"); | ||
var pdfFileType = validator.Mapping.FindByMimeType("application/pdf"); | ||
``` | ||
- Determine & validate a filetype by uploaded IFormFile: | ||
- Determine and validate a file type by uploaded IFormFile: | ||
```c# | ||
var fileType = await formFileTypeProvider.FindValidatedTypeAsync(formFile, null, CancellationToken.None); | ||
``` | ||
|
@@ -43,12 +43,12 @@ var fileType = await formFileTypeProvider.FindValidatedTypeAsync(formFile, null, | |
var fileType = await streamFileTypeProvider.TryFindUnambiguousAsync(fileStream, CancellationToken.None); | ||
``` | ||
|
||
- Check a file with its stream and filetype: | ||
- Check a file with its stream and file type: | ||
```c# | ||
var isValid = await validator.IsValidAsync(memoryStream, fileType, CancellationToken.None); | ||
``` | ||
|
||
#### Expand the filetype mapping | ||
#### Expand the file type mapping | ||
|
||
- Get mapping: | ||
```c# | ||
|
@@ -62,52 +62,51 @@ var mapping = formFileTypeProvider.Mapping; | |
var mapping = new MagicBytesValidator.Services.Mapping(); | ||
``` | ||
|
||
- Register a single Filetype: | ||
- Register a single `FileByteFilter`: | ||
```c# | ||
mapping.Register( | ||
new FileByteFilter( | ||
"traperto/trp", // mime type | ||
"traperto/trp", // MIME type | ||
new[] { "trp" } // file extensions | ||
) { | ||
// magic byte sequences | ||
StartsWith([ | ||
0x78, 0x6c, 0x2f, 0x5f, 0x72, 0x65 | ||
]) | ||
.EndsWith([ | ||
.EndsWith([ | ||
0xFF, 0xFF | ||
]) | ||
} | ||
) | ||
``` | ||
|
||
- FileTypes with specific offset checks: | ||
- `FileByteFilter`s with specific offset checks: | ||
```c# | ||
mapping.Register( | ||
new FileByteFilter( | ||
"traperto/trp", // mime type | ||
"traperto/trp", // MIME type | ||
new[] { "trp" } // file extensions | ||
) { | ||
// magic byte sequences | ||
Specific(new ByteCheck(512, [0xFD])); | ||
} | ||
) | ||
``` | ||
ByteCheck allows for negative offset values to look for a specific offset counting from the end of file | ||
`ByteCheck` allows for negative offset values to look for a specific offset counting from the end of file. | ||
|
||
- Register a list of filetypes: | ||
```c# | ||
mapping.Register(listOfFileTypes); | ||
``` | ||
|
||
You can also create variants of `IFileType` and register them by passing the Assembly of the new FileTypes, e.g. | ||
`mapping.Register(typeof(CustomFileType).Assembly);`. This will register all FileTypes of the given Assembly that are also | ||
not abstract and have an empty constructor! | ||
You can also create variants of `IFileType` and register them by passing the Assembly of the new `IFileType`s, e.g. | ||
`mapping.Register(typeof(CustomFileType).Assembly);`. This will register all `IFileType`s of the given assembly that are also not abstract and have an empty constructor. | ||
|
||
```c# | ||
public class CustomFileType : FileTypeWithStartSequences | ||
{ | ||
public CustomFileType() : base( | ||
"traperto/trp", // mime type | ||
"traperto/trp", // MIME type | ||
new[] { "trp" }, // file extensions | ||
new[] { // magic byte sequences | ||
new byte[] { 0x74, 0x72, 0x61, 0x70, 0x65, 0x72, 0x74, 0x6f } | ||
|
@@ -123,54 +122,65 @@ _mapping.Register(assembly); | |
|
||
### CLI | ||
There's a CLI tool (_MagicBytesValidator.CLI_) which can be used to determine | ||
MIME types for a local file by calling the following command. | ||
MIME types for a local file by calling the following command: | ||
|
||
```shell | ||
dotnet run --project MagicBytesValidator.CLI -- [PATH] | ||
``` | ||
|
||
This can be useful when debugging or validating newly added FileTypes. | ||
|
||
### List of Filetypes | ||
|
||
| Mimetype | Extension | Magicbytes (decimal) | | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This list was outdated af. Therefore I dared to update it ^^ |
||
|-------------------------------------------------|--------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------| | ||
| audio/x-pn-realaudio-plugin | rpm | 237 171 238 219 | | ||
| application/octet-stream | bin<br />file<br />com<br />class<br />ini | <ul><li>83 80 48 49</li><li>201</li><li>202 254 186 190</li></ul> | | ||
| video/3gpp | 3gp | 102 116 121 112 51 103 | | ||
| image/x-icon | ico | 0 0 1 0 | | ||
| image/gif | gif | <ul><li>71 73 70 56 55 97</li><li>71 73 70 56 57 97</li></ul> | | ||
| image/tiff | tif<br />tiff | <ul><li>73 73 42 0</li><li>77 77 0 42</li></ul> | | ||
| image/jpeg | jpg<br />jpeg<br />jpe | <ul><li>255 216 255 219</li><li>255 216 255 224 0 16 74</li><li>70 73 70 0 1</li><li>255 216 255 238</li><li>105 102 0 0</li></ul> | | ||
| image/png | png | 137 80 78 71 13 10 26 10 | | ||
| video/ogg | ogg<br />ogv | 79 103 103 83 | | ||
| audio/basic | snd<br />au | <ul><li>56 83 86 88</li><li>65 73 70 70</li></ul> | | ||
| application/dsptype | tsp | 77 90 | | ||
| text/plain | txt | <ul><li>239 187 191</li><li>255 254</li><li>254 255</li><li>255 254 0 0</li></ul> | | ||
| application/zip | zip | 80 75 3 4 | | ||
| application | docx<br />xlsx | 80 75 7 8 | | ||
| application/vnd.oasis.opendocument.presentation | odp | 80 75 7 8 | | ||
| application/vnd.oasis.opendocument.spreadsheet | ods | 80 75 7 8 | | ||
| application/vnd.oasis.opendocument.text | odt | 80 75 7 8 | | ||
| audio/mpeg | mp3 | 73 68 51 | | ||
| image/bmp | bmp | 66 77 | | ||
| audio/x-midi | midi<br />mid | 77 84 104 100 | | ||
| application/msword | doc<br />dot | 208 207 17 224 161 177 26 255 | | ||
| application/msexcel | xlx<br />xla | 208 207 17 224 161 177 26 255 | | ||
| application/mspowerpoint | ppt<br />ppz<br />pps<br />pt | 208 207 17 224 161 177 26 225 | | ||
| application/gzip | gz | 31 139 | | ||
| video/webm | webm | 26 69 223 163 | | ||
| application/rtf | rtf | 123 92 114 116 102 49 | | ||
| text/tab-separated-values | tsv | 71 | | ||
| video/mpeg | mpg<br />mpeg<br />mpe | <ul><li>71</li><li>0 0 1 186</li><li>0 0 1 179</li></ul> | | ||
| video/mp4 | mp4 | <ul><li>102 116 121 112 105 115 111 109</li><li>102 116 121 112 109 112 52 50</li><li>102 116 121 112 77 83 62 86</li></ul> | | ||
| image/x-portable-bitmap | pbm | 80 49 10 | | ||
| image/x-portable-graymap | pgm | 80 50 10 | | ||
| image/x-portable-pixmap | ppm | 80 51 10 | | ||
| application/pdf | pdf | _multiple_ | | ||
### List of file types | ||
|
||
|FileType|Extensions|MIME Types| | ||
|-|-|-| | ||
|AIF|aif, aiff, aifc|audio/x-aiff| | ||
|BIN|bin, file, com, class, ini|application/octet-stream| | ||
|BMP|bmp|image/bmp| | ||
|CAB|cab|application/vnd.ms-cab-compressed, application/x-cab-compressed| | ||
|DOC|doc, dot|application/msword| | ||
|DOCX|docx|application/vnd.openxmlformats-officedocument.wordprocessingml.document| | ||
|DXR|dxr, dcr, dir|application/x-director| | ||
|EXE|exe, com, dll, drv, pif, qts, qtx , sys, acm, ax, cpl, fon, ocx, olb, scr, vbx, vxd, mui, iec, ime, rs, tsp, efi|application/x-dosexec, application/x-msdos-program| | ||
|GIF|gif|image/gif| | ||
|GZ|gz|application/gzip| | ||
|ICO|ico|image/x-icon| | ||
|JPG|jpg, jpeg, jpe, jif, jfif, jfi|image/jpeg| | ||
|MIDI|midi, mid|audio/x-midi| | ||
|MP3|mp3|audio/mpeg| | ||
|MP4|mp4|video/mp4| | ||
|MPG|mpg, mpeg, mpe, m2p, vob|video/mpeg| | ||
|ODP|odp|application/vnd.oasis.opendocument.presentation| | ||
|ODS|ods|application/vnd.oasis.opendocument.spreadsheet| | ||
|ODT|odt|application/vnd.oasis.opendocument.text| | ||
|OGV|ogv, ogg, oga|video/ogg| | ||
|PBM|pbm|image/x-portable-bitmap| | ||
|PDF|pdf|application/pdf| | ||
|PGM|pgm|image/x-portable-graymap| | ||
|PNG|png|image/png| | ||
|PPM|ppm|image/x-portable-pixmap| | ||
|PPT|ppt, ppz, pps, pot|application/mspowerpoint, application/vnd.ms-powerpoint| | ||
|PPTX|pptx|application/vnd.openxmlformats-officedocument.presentationml.presentation| | ||
|RAR|rar|application/vnd.rar, application/x-rar-compressed| | ||
|RPM|rpm|application/x-rpm, application/x-redhat-package-manager| | ||
|RTF|rtf|application/rtf| | ||
|SND|snd, au|audio/basic| | ||
|SVG|svg, svgz|image/svg+xml| | ||
|SWF|swf|application/x-shockwave-flash| | ||
|3GP|3gp|video/3gpp| | ||
|TIF|tif, tiff|image/tiff| | ||
|TSV|ts, tsv, tsa, mpg, mpeg|video/mp2t| | ||
|TXT|txt|text/plain| | ||
|WEBM|mkv, mka, mks, mk3d, webm|video/webm| | ||
|XLS|xls, xla|application/msexcel| | ||
|XLSX|xlsx|application/vnd.openxmlformats-officedocument.spreadsheetml.sheet| | ||
|XML|xml|application/xml, text/xml| | ||
|Z|z|application/x-compress| | ||
|ZIP|zip|application/zip, application/x-zip-compressed| | ||
|
||
### What is the licence? | ||
|
||
MIT License | ||
[MIT License](./LICENSE) | ||
|
||
``` | ||
▓▓ ▓▓▓▓▓▓▓▓▓▓ | ||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Couldn't run the tests in the terminal otherwise.