@@ -466,13 +466,13 @@ The performance of git-filter-branch is glacially slow; its design makes it
466466impossible for a backward-compatible implementation to ever be fast:
467467
468468* In editing files, git-filter-branch by design checks out each and
469- every commit as it existed in the original repo. If your repo has 10\^5
470- files and 10\^5 commits, but each commit only modifies 5 files, then
471- git-filter-branch will make you do 10\^10 modifications, despite only
472- having (at most) 5*10^5 unique blobs.
469+ every commit as it existed in the original repo. If your repo has
470+ 10\^5 files and 10\^5 commits, but each commit only modifies 5
471+ files, then git-filter-branch will make you do 10\^10 modifications,
472+ despite only having (at most) 5*10^5 unique blobs.
473473
474474* If you try and cheat and try to make git-filter-branch only work on
475- files modified in a commit, then two things happen
475+ files modified in a commit, then two things happen
476476
477477 ** you run into problems with deletions whenever the user is simply
478478 trying to rename files (because attempting to delete files that
@@ -481,39 +481,41 @@ files modified in a commit, then two things happen
481481 user-provided shell)
482482
483483 ** even if you succeed at the map-deletes-for-renames chicanery, you
484- still technically violate backward compatibility because users are
485- allowed to filter files in ways that depend upon topology of
486- commits instead of filtering solely based on file contents or names
487- (though this has not been observed in the wild).
484+ still technically violate backward compatibility because users
485+ are allowed to filter files in ways that depend upon topology of
486+ commits instead of filtering solely based on file contents or
487+ names (though this has not been observed in the wild).
488488
489489* Even if you don't need to edit files but only want to e.g. rename or
490- remove some and thus can avoid checking out each file (i.e. you can use
491- --index-filter), you still are passing shell snippets for your filters.
492- This means that for every commit, you have to have a prepared git repo
493- where those filters can be run. That's a significant setup.
494-
495- * Further, several additional files are created or updated per commit by
496- git-filter-branch. Some of these are for supporting the convenience
497- functions provided by git-filter-branch (such as map()), while others
498- are for keeping track of internal state (but could have also been
499- accessed by user filters; one of git-filter-branch's regression tests
500- does so). This essentially amounts to using the filesystem as an IPC
501- mechanism between git-filter-branch and the user-provided filters.
502- Disks tend to be a slow IPC mechanism, and writing these files also
503- effectively represents a forced synchronization point between separate
504- processes that we hit with every commit.
490+ remove some and thus can avoid checking out each file (i.e. you can
491+ use --index-filter), you still are passing shell snippets for your
492+ filters. This means that for every commit, you have to have a
493+ prepared git repo where those filters can be run. That's a
494+ significant setup.
495+
496+ * Further, several additional files are created or updated per commit
497+ by git-filter-branch. Some of these are for supporting the
498+ convenience functions provided by git-filter-branch (such as map()),
499+ while others are for keeping track of internal state (but could have
500+ also been accessed by user filters; one of git-filter-branch's
501+ regression tests does so). This essentially amounts to using the
502+ filesystem as an IPC mechanism between git-filter-branch and the
503+ user-provided filters. Disks tend to be a slow IPC mechanism, and
504+ writing these files also effectively represents a forced
505+ synchronization point between separate processes that we hit with
506+ every commit.
505507
506508* The user-provided shell commands will likely involve a pipeline of
507- commands, resulting in the creation of many processes per commit.
508- Creating and running another process takes a widely varying amount of
509- time between operating systems, but on any platform it is very slow
510- relative to invoking a function.
509+ commands, resulting in the creation of many processes per commit.
510+ Creating and running another process takes a widely varying amount
511+ of time between operating systems, but on any platform it is very
512+ slow relative to invoking a function.
511513
512514* git-filter-branch itself is written in shell, which is kind of slow.
513- This is the one performance issue that could be backward-compatibly
514- fixed, but compared to the above problems that are intrinsic to the
515- design of git-filter-branch, the language of the tool itself is a
516- relatively minor issue.
515+ This is the one performance issue that could be backward-compatibly
516+ fixed, but compared to the above problems that are intrinsic to the
517+ design of git-filter-branch, the language of the tool itself is a
518+ relatively minor issue.
517519
518520 ** Side note: Unfortunately, people tend to fixate on the
519521 written-in-shell aspect and periodically ask if git-filter-branch
@@ -546,51 +548,55 @@ easily corrupt repos or end up with a mess worse than what you started
546548with:
547549
548550* Someone can have a set of "working and tested filters" which they
549- document or provide to a coworker, who then runs them on a different OS
550- where the same commands are not working/tested (some examples in the
551- git-filter-branch manpage are also affected by this). BSD vs. GNU
552- userland differences can really bite. If lucky, error messages are
553- spewed. But just as likely, the commands either don't do the filtering
554- requested, or silently corrupt by making some unwanted change. The
555- unwanted change may only affect a few commits, so it's not necessarily
556- obvious either. (The fact that problems won't necessarily be obvious
557- means they are likely to go unnoticed until the rewritten history is in
558- use for quite a while, at which point it's really hard to justify
559- another flag-day for another rewrite.)
551+ document or provide to a coworker, who then runs them on a different
552+ OS where the same commands are not working/tested (some examples in
553+ the git-filter-branch manpage are also affected by this).
554+ BSD vs. GNU userland differences can really bite. If lucky, error
555+ messages are spewed. But just as likely, the commands either don't
556+ do the filtering requested, or silently corrupt by making some
557+ unwanted change. The unwanted change may only affect a few commits,
558+ so it's not necessarily obvious either. (The fact that problems
559+ won't necessarily be obvious means they are likely to go unnoticed
560+ until the rewritten history is in use for quite a while, at which
561+ point it's really hard to justify another flag-day for another
562+ rewrite.)
560563
561564* Filenames with spaces are often mishandled by shell snippets since
562- they cause problems for shell pipelines. Not everyone is familiar with
563- find -print0, xargs -0, git-ls-files -z, etc. Even people who are
564- familiar with these may assume such flags are not relevant because
565- someone else renamed any such files in their repo back before the person
566- doing the filtering joined the project. And often, even those familiar
567- with handling arguments with spaces may not do so just because they
568- aren't in the mindset of thinking about everything that could possibly
569- go wrong.
570-
571- * Non-ascii filenames can be silently removed despite being in a desired
572- directory. Keeping only wanted paths is often done using pipelines like
573- `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`. ls-files will
574- only quote filenames if needed, so folks may not notice that one of the
575- files didn't match the regex (at least not until it's much too late).
576- Yes, someone who knows about core.quotePath can avoid this (unless they
577- have other special characters like \t, \n, or "), and people who use
578- ls-files -z with something other than grep can avoid this, but that
579- doesn't mean they will.
580-
581- * Similarly, when moving files around, one can find that filenames with
582- non-ascii or special characters end up in a different directory, one
583- that includes a double quote character. (This is technically the same
584- issue as above with quoting, but perhaps an interesting different way
585- that it can and has manifested as a problem.)
565+ they cause problems for shell pipelines. Not everyone is familiar
566+ with find -print0, xargs -0, git-ls-files -z, etc. Even people who
567+ are familiar with these may assume such flags are not relevant
568+ because someone else renamed any such files in their repo back
569+ before the person doing the filtering joined the project. And
570+ often, even those familiar with handling arguments with spaces may
571+ not do so just because they aren't in the mindset of thinking about
572+ everything that could possibly go wrong.
573+
574+ * Non-ascii filenames can be silently removed despite being in a
575+ desired directory. Keeping only wanted paths is often done using
576+ pipelines like `git ls-files | grep -v ^WANTED_DIR/ | xargs git rm`.
577+ ls-files will only quote filenames if needed, so folks may not
578+ notice that one of the files didn't match the regex (at least not
579+ until it's much too late). Yes, someone who knows about
580+ core.quotePath can avoid this (unless they have other special
581+ characters like \t, \n, or "), and people who use ls-files -z with
582+ something other than grep can avoid this, but that doesn't mean they
583+ will.
584+
585+ * Similarly, when moving files around, one can find that filenames
586+ with non-ascii or special characters end up in a different
587+ directory, one that includes a double quote character. (This is
588+ technically the same issue as above with quoting, but perhaps an
589+ interesting different way that it can and has manifested as a
590+ problem.)
586591
587592* It's far too easy to accidentally mix up old and new history. It's
588- still possible with any tool, but git-filter-branch almost invites it.
589- If lucky, the only downside is users getting frustrated that they don't
590- know how to shrink their repo and remove the old stuff. If unlucky,
591- they merge old and new history and end up with multiple "copies" of each
592- commit, some of which have unwanted or sensitive files and others which
593- don't. This comes about in multiple different ways:
593+ still possible with any tool, but git-filter-branch almost
594+ invites it. If lucky, the only downside is users getting frustrated
595+ that they don't know how to shrink their repo and remove the old
596+ stuff. If unlucky, they merge old and new history and end up with
597+ multiple "copies" of each commit, some of which have unwanted or
598+ sensitive files and others which don't. This comes about in
599+ multiple different ways:
594600
595601 ** the default to only doing a partial history rewrite ('--all' is not
596602 the default and few examples show it)
@@ -609,8 +615,8 @@ don't. This comes about in multiple different ways:
609615 "DISCUSSION" section of the git filter-repo manual page for more
610616 details.
611617
612- * Annotated tags can be accidentally converted to lightweight tags, due
613- to either of two issues:
618+ * Annotated tags can be accidentally converted to lightweight tags,
619+ due to either of two issues:
614620
615621 ** Someone can do a history rewrite, realize they messed up, restore
616622 from the backups in refs/original/, and then redo their
@@ -623,71 +629,74 @@ to either of two issues:
623629 restored from refs/original/ in a previously botched rewrite).
624630
625631* Any commit messages that specify an encoding will become corrupted
626- by the rewrite; git-filter-branch ignores the encoding, takes the original
627- bytes, and feeds it to commit-tree without telling it the proper
628- encoding. (This happens whether or not --msg-filter is used.)
632+ by the rewrite; git-filter-branch ignores the encoding, takes the
633+ original bytes, and feeds it to commit-tree without telling it the
634+ proper encoding. (This happens whether or not --msg-filter is
635+ used.)
629636
630637* Commit messages (even if they are all UTF-8) by default become
631- corrupted due to not being updated -- any references to other commit
632- hashes in commit messages will now refer to no-longer-extant commits.
633-
634- * There are no facilities for helping users find what unwanted crud they
635- should delete, which means they are much more likely to have incomplete
636- or partial cleanups that sometimes result in confusion and people
637- wasting time trying to understand. (For example, folks tend to just
638- look for big files to delete instead of big directories or extensions,
639- and once they do so, then sometime later folks using the new repository
640- who are going through history will notice a build artifact directory
641- that has some files but not others, or a cache of dependencies
642- (node_modules or similar) which couldn't have ever been functional since
643- it's missing some files.)
638+ corrupted due to not being updated -- any references to other commit
639+ hashes in commit messages will now refer to no-longer-extant
640+ commits.
641+
642+ * There are no facilities for helping users find what unwanted crud
643+ they should delete, which means they are much more likely to have
644+ incomplete or partial cleanups that sometimes result in confusion
645+ and people wasting time trying to understand. (For example, folks
646+ tend to just look for big files to delete instead of big directories
647+ or extensions, and once they do so, then sometime later folks using
648+ the new repository who are going through history will notice a build
649+ artifact directory that has some files but not others, or a cache of
650+ dependencies (node_modules or similar) which couldn't have ever been
651+ functional since it's missing some files.)
644652
645653* If --prune-empty isn't specified, then the filtering process can
646- create hoards of confusing empty commits
654+ create hoards of confusing empty commits
647655
648656* If --prune-empty is specified, then intentionally placed empty
649- commits from before the filtering operation are also pruned instead of
650- just pruning commits that became empty due to filtering rules.
657+ commits from before the filtering operation are also pruned instead
658+ of just pruning commits that became empty due to filtering rules.
651659
652660* If --prune-empty is specified, sometimes empty commits are missed
653- and left around anyway (a somewhat rare bug, but it happens...)
661+ and left around anyway (a somewhat rare bug, but it happens...)
654662
655663* A minor issue, but users who have a goal to update all names and
656- emails in a repository may be led to --env-filter which will only update
657- authors and committers, missing taggers.
664+ emails in a repository may be led to --env-filter which will only
665+ update authors and committers, missing taggers.
658666
659667* If the user provides a --tag-name-filter that maps multiple tags to
660- the same name, no warning or error is provided; git-filter-branch simply
661- overwrites each tag in some undocumented pre-defined order resulting in
662- only one tag at the end. (A git-filter-branch regression test requires
663- this surprising behavior.)
668+ the same name, no warning or error is provided; git-filter-branch
669+ simply overwrites each tag in some undocumented pre-defined order
670+ resulting in only one tag at the end. (A git-filter-branch
671+ regression test requires this surprising behavior.)
664672
665673Also, the poor performance of git-filter-branch often leads to safety
666674issues:
667675
668- * Coming up with the correct shell snippet to do the filtering you want
669- is sometimes difficult unless you're just doing a trivial modification
670- such as deleting a couple files. Unfortunately, people often learn if
671- the snippet is right or wrong by trying it out, but the rightness or
672- wrongness can vary depending on special circumstances (spaces in
673- filenames, non-ascii filenames, funny author names or emails, invalid
674- timezones, presence of grafts or replace objects, etc.), meaning they
675- may have to wait a long time, hit an error, then restart. The
676- performance of git-filter-branch is so bad that this cycle is painful,
677- reducing the time available to carefully re-check (to say nothing about
678- what it does to the patience of the person doing the rewrite even if
679- they do technically have more time available). This problem is extra
680- compounded because errors from broken filters may not be shown for a
681- long time and/or get lost in a sea of output. Even worse, broken
682- filters often just result in silent incorrect rewrites.
683-
684- * To top it all off, even when users finally find working commands, they
685- naturally want to share them. But they may be unaware that their repo
686- didn't have some special cases that someone else's does. So, when
687- someone else with a different repository runs the same commands, they
688- get hit by the problems above. Or, the user just runs commands that
689- really were vetted for special cases, but they run it on a different OS
690- where it doesn't work, as noted above.
676+ * Coming up with the correct shell snippet to do the filtering you
677+ want is sometimes difficult unless you're just doing a trivial
678+ modification such as deleting a couple files. Unfortunately, people
679+ often learn if the snippet is right or wrong by trying it out, but
680+ the rightness or wrongness can vary depending on special
681+ circumstances (spaces in filenames, non-ascii filenames, funny
682+ author names or emails, invalid timezones, presence of grafts or
683+ replace objects, etc.), meaning they may have to wait a long time,
684+ hit an error, then restart. The performance of git-filter-branch is
685+ so bad that this cycle is painful, reducing the time available to
686+ carefully re-check (to say nothing about what it does to the
687+ patience of the person doing the rewrite even if they do technically
688+ have more time available). This problem is extra compounded because
689+ errors from broken filters may not be shown for a long time and/or
690+ get lost in a sea of output. Even worse, broken filters often just
691+ result in silent incorrect rewrites.
692+
693+ * To top it all off, even when users finally find working commands,
694+ they naturally want to share them. But they may be unaware that
695+ their repo didn't have some special cases that someone else's does.
696+ So, when someone else with a different repository runs the same
697+ commands, they get hit by the problems above. Or, the user just
698+ runs commands that really were vetted for special cases, but they
699+ run it on a different OS where it doesn't work, as noted above.
691700
692701GIT
693702---
0 commit comments