Skip to content
Navigation Menu
Sign in
Products
Platform
GitHub
Repository hosting, code review, and collaboration
Copilot
AI coding agent for tasks, projects, and automated work
Build & Ship
Actions
CI/CD and automation
Code Quality
Quality analysis, fixes, and enforcement
Codespaces
Cloud development environments
Secure
Code Security
Vulnerability detection and remediation
Secret Protection
Secret scanning and leak prevention
All Features
Developers
Learn
Documentation
Training
Certifications
Build
Build with Copilot
Customize Copilot
Integrations
Explore
Trending repos
GitHub Sponsors
Developer articles
Changelog
Changelog
Your source for the latest features, improvements, and fixes across GitHub products, all in one place.
Businesses
By Stage
Enterprises
Team
Startups
Nonprofits
By Industry
Healthcare
Financial services
Manufacturing
Government
See all industries
Evaluate GitHub
Why GitHub
Customer Stories
Business insights
Trust Center
Add-ons
GitHub Advanced Security
Enterprise-grade code security
Copilot for business
Enterprise-grade AI features
Premium Support
Enterprise-grade 24/7 support
Resources
Learn & Connect
Documentation
Resource center
Blog
Events & webinars
Community Forum
Customer Stories
Integrations
Support
Customer support
Training
GitHub Status
Company
About
Partners
Changelog
GitHub Universe
GitHub Universe
The event for developers, by developers — talks, workshops, and what’s next for GitHub.
Pricing
Search
/
Sign in
Sign up
Appearance settings
You signed in with another tab or window.
Reload
to refresh your session.
You signed out in another tab or window.
Reload
to refresh your session.
You switched accounts on another tab or window.
Reload
to refresh your session.
Dismiss alert
{{ message }}
unsigned-R
scancode-toolkit
Repository navigation
Code
Pull requests
Actions
Projects
Security and quality
Insights
More
items
Commits
Breadcrumbs
History for
scancode-toolkit
tests
licensedcode
data
tokenize
on
develop
User selector
All users
Datepicker
All time
Commit history
Commits on Jan 3, 2022
Add new filter for false positive license lists
Show description for b9ad65f
pombredanne
committed
b9ad65f
View commit details
Copy full SHA for b9ad65f
View code at this point
Browse repository at this point
Commits on Dec 14, 2021
Throw InvalidRule when key phrase is not closed
Show description for 4510855
mrombout
committed
4510855
View commit details
Copy full SHA for 4510855
View code at this point
Browse repository at this point
Add unit tests to key_phrase_tokenizer
Show description for 65384ab
mrombout
committed
65384ab
View commit details
Copy full SHA for 65384ab
View code at this point
Browse repository at this point
Commits on Apr 19, 2021
Update query_tokenizer tests
Show description for db8a535
pombredanne
committed
db8a535
View commit details
Copy full SHA for db8a535
View code at this point
Browse repository at this point
Commits on Nov 17, 2020
Format license test YAML files for consistency
Show description for d12fd40
pombredanne
committed
d12fd40
View commit details
Copy full SHA for d12fd40
View code at this point
Browse repository at this point
Commits on Jul 2, 2019
Do not treat "id" as a stopword
Show description for 727cb2e
pombredanne
committed
727cb2e
View commit details
Copy full SHA for 727cb2e
View code at this point
Browse repository at this point
Commits on Jun 3, 2019
Add tfidf to license set match ranking
Show description for 2fb0265
pombredanne
committed
2fb0265
View commit details
Copy full SHA for 2fb0265
View code at this point
Browse repository at this point
Add new and improved license rules and tests
Show description for e1e2fc7
pombredanne
committed
e1e2fc7
View commit details
Copy full SHA for e1e2fc7
View code at this point
Browse repository at this point
Commits on Mar 12, 2019
Add new license detection rules and update tests
Show description for a6e9482
pombredanne
committed
a6e9482
View commit details
Copy full SHA for a6e9482
View code at this point
Browse repository at this point
Commits on Sep 14, 2018
Return correct line numbers for long lines #1176
Show description for e0ba4be
pombredanne
committed
e0ba4be
View commit details
Copy full SHA for e0ba4be
View code at this point
Browse repository at this point
Commits on Apr 9, 2018
Exclude underscore. Remove rule tokenizer #817
Show description for acf2e43
pombredanne
committed
acf2e43
View commit details
Copy full SHA for acf2e43
View code at this point
Browse repository at this point
Commits on Oct 2, 2017
Treat underscore as a part of a token
Show description for 8e03d42
pombredanne
committed
8e03d42
View commit details
Copy full SHA for 8e03d42
View code at this point
Browse repository at this point
New test file for expected tokenization
Show description for 33bd5b4
pombredanne
committed
33bd5b4
View commit details
Copy full SHA for 33bd5b4
View code at this point
Browse repository at this point
Commits on Jun 29, 2017
Use simpler tokenizing regex for license scan #655
Show description for 436a6b4
pombredanne
committed
436a6b4
View commit details
Copy full SHA for 436a6b4
View code at this point
Browse repository at this point
Commits on Oct 21, 2016
New and improved license detection tests
Show description for 57c9d39
pombredanne
committed
57c9d39
View commit details
Copy full SHA for 57c9d39
View code at this point
Browse repository at this point
Commits on Mar 11, 2016
#86 New detection approach supporting approximate detection
Show description for b666fc4
pombredanne
committed
b666fc4
View commit details
Copy full SHA for b666fc4
View code at this point
Browse repository at this point
Loading
You can’t perform that action at this time.