-
Notifications
You must be signed in to change notification settings - Fork 8.1k
Refactor the tokenizer #7509
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Refactor the tokenizer #7509
Changes from all commits
Commits
Show all changes
74 commits
Select commit
Hold shift + click to select a range
f3b0413
Merge pull request #1 from PowerShell/master
vexx32 c0c1248
Add tokenizer logic for recognising numeric tokens with 'u' suffix
vexx32 37802f0
inline very temporary variables
vexx32 dc9dc4c
Add typesuffix trait to uppercase and lowercase 'u'
vexx32 88377e4
Add case to get ulong vs uint depending on total value of number
vexx32 567809f
Fix some small things codefactor complained about
vexx32 34812ba
Add type mappings for `uint` and `ulong` accelerators
vexx32 47287fb
Add cases for new type accelerators
vexx32 da25c79
Add parser tests for uint literals
vexx32 8491b10
Fix typos in tests
vexx32 cc90f72
fix ulong parsing
vexx32 ee9c641
Fix silly error
vexx32 0d66604
Remove unneeded cast
vexx32 3c56a94
Adding ushort accelerator as well (maybe unnecessary?)
vexx32 5146db0
Update accelerator tests
vexx32 27ed4f0
Fix type conversion
vexx32 ace51dc
Remove stray characters breaking hash literal
vexx32 cdae19e
Parse as UInt64 directly
vexx32 a4c18aa
Small whitespace rule resolution
vexx32 dae82c1
Fix weird whitespace alignment discrepancy
vexx32 bad1533
Attempt #2
vexx32 d9bfbc8
Get alignment back to more or less how it was initially
vexx32 ebb795e
This is where they should be. I think
vexx32 3738191
Nope it was a full tab, I lied.
vexx32 b67b98e
Screw it, keep it Normal:tm:
vexx32 7a6b44e
Add code path to mirror `L` long token codepaths
vexx32 625a223
Move token detection after real check
vexx32 27f1457
Add tests for rounding on token parse
vexx32 8595619
Convert `suffix` across the board to string
vexx32 415a129
Add behaviour to account for new suffix behaviour
vexx32 beba14f
Add case for nonsensical type suffixes and don't accept parse
vexx32 05bd457
Changes for CodeFactor
vexx32 29da202
Add parser tests for 'ul' suffix
vexx32 4a28fa9
Oh look, I totally forgot the type syffix on this test.
vexx32 5024265
Remember: double->(u)int type casts in this manner
vexx32 f2b365c
Add NumberSuffixFlags
vexx32 6b0ff63
replace suffix references' types
vexx32 ba850c8
Change string/char checking for suffixes into flag checking
vexx32 26c7565
Fix switch syntax
vexx32 1b0439b
Fix enum name reference
vexx32 7985774
Add enum summary comments
vexx32 8da4969
Fix switch casing to not permit unknown two-char suffixes
vexx32 fa4e985
Do not allow `dl` or `ll` suffixes
vexx32 4633871
Fixes for CodeFactor ♥
vexx32 e646ea8
Remove public designation from NumberSuffixFlags
vexx32 28666a6
Fix some typos in my code comments
vexx32 09b85f1
Add `s` as a permissibly type suffix
vexx32 e55282f
add parse cases for `s` type tokens
vexx32 e42556b
Add missing ushort case to tokenizer
vexx32 9e84f1c
TryGetNumberValue -> change param to ReadOnlySpan<char>
vexx32 feb15bc
Shift if series -> if/else series
vexx32 a1f43d9
Swap Convert.ChangeType() for Convert.To[Type]
vexx32 bf8b731
Add missing tests for invalid tokens
vexx32 8bd503c
Remove superfluous comment block
vexx32 eb47fd3
Fix tests to operate properly
vexx32 523385e
Slight tidy
vexx32 6c729c6
Begin adding helper utils for conversions
vexx32 22498e0
Replace throwable methods with non-throwing
vexx32 7c04d68
Eliminate throwing conversions from real number parsing
vexx32 4fc3990
Shorten temp variable names
vexx32 ba93604
More helper functions!
vexx32 9da5a0e
Finish rebuilding the tokenizer with helper functions.
vexx32 cdb8bce
Fix the parsing. I think.
vexx32 b9a74f6
Refactor integer parsing to cast instead of continually calling TryParse
vexx32 4bf3435
Refactor multiplier -> switch statement
vexx32 5782443
Small fixes for CodeFactor
vexx32 781fb7f
Fix sign bits in hex readings
vexx32 2159c9a
Add some sign bit checks to the hex literal parsing
vexx32 d9d265b
Add logic to enable use of sign bits properly... I think.
vexx32 d3618c7
Remove busted logic for int hex sign bits that wasn't doing anything
vexx32 f1411d5
Let huge hex values parse as bigint
vexx32 4a21a1e
Commented out the test failing because the huge number can now be parsed
vexx32 5d2a39b
Allow hex numbers to specify long values without suffix
vexx32 25913b4
Add case for dealing with unsigned ints
vexx32 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.