Tags: wrye-bash/wrye-bash
Tags
314 Mini-Release for MO2 [2025/04/05] [Various community members] A small release, mostly to work around a very annoying MO2 problem that causes various Python filesystem APIs to fail when Python 3.12 applications are run through the VFS. The workaround is to revert to Python 3.11 for now, a real fix has to be made on the MO2 side. We originally wanted this to be a 313.1, but enough refactoring and small features had landed on dev that we could no longer call it 313.1. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### MO2 Workaround (#689) The whole reason for this small release. We quickly got reports that WB 313 did not work with MO2 after releasing it. Turns out the reason is the upgrade to Python 3.12 (#669). Or more accurately, this MO2 issue is the cause: ModOrganizer2/modorganizer#2174 0a3e933 refactored the code slightly in preparation for workarounds and hacks, but we did not fully understand the problem at the time and it turns out there's nothing we can do to fix this on our side. ----------------------------------------------------------------------- ### Refreshes (#353) The refresh refactoring continued in 5778fe3, where data structures were finalized and UIList refreshes got further refactored. All this was prompted by regressions in 313. In 0c7fcbd, load order operations and refreshes got reworked further. This was a complex endeavor, see that commit for more details. ----------------------------------------------------------------------- ### Renames (#580) In 098836b, we once again tackled the complex beast that is renames. The fixes wanted in #580 are done with this. Further work could be done by extracting some of the complex logic into dedicated data structures and classes to encapsualte it, but that did not make its way into 314. ----------------------------------------------------------------------- ### Translations (#500) Lastly, we have some translation updates in 29fcbdb. Many thanks to the translators! ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Utumno, @Infernio, @thiagojramos, @xsSplater, @TamilNeram and many more who GitHub's contribution tracker doesn't list.
313 Translations rework [2024/12/21] [Various community members] This one kind of ballooned out of control again, with >390 commits included :P A lot of this is because of translations, however, which was one of the main themes this release. It's still smaller than 311, however, which is also because midway into its development, I have gone on an extended break from WB development. I reserve the right to return at any time, however :P All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### Python 3.12 (#669) The development began with 3c2837c, which upgraded the codebase to Python 3.12. This turned out be quite a journey (see the commit message for more information). ----------------------------------------------------------------------- ### FileInfo(s) (#336) Remember how I talked about #336 "returning with a vengeance" in the 312 commit, but then there was only a single commit mentioned in that section? Whoops. The reason is that all that work ended up only getting merged in 313, starting with 24fb7fb, which got rid of some ancient, deprecated APIs on FileInfo. See also the refreshes section down below, with which this topic is tightly coupled and hence shares many commits. fb769f0 moved operations and APIs from DataStore to ListInfo subclasses. In d42d769, we refactored ESM swapping and worked towards dropping the table attribute from DataStore, which was then done in ca48e96. Lots of code was simplified and the complexities of persistance here laid bare. This was a necessity for tackling the refreshes. 0a69959 tackled renames, which are still a difficult subject. More work to be done (see #580), but this was more about tackling #336, hence why it's in this section. This issue was closed in c6761b5, but see the refresh section below for more details on that merge. ----------------------------------------------------------------------- ### Localization (#55, #500, #677) This started back in 311, continued throughout 312 and concluded here in 313. Our translations are completely overhauled and done through Weblate now. This is what I picked as the title of the release, since so much effort got poured into it, not only by developers but also by translators. Work in 313 began with a86c2e1, which is the followup to 2442285 from 311. This commit edited our translatable strings to improve their translatability by getting rid of wtxt formatting, dropping leading and trailing spaces, replacing '%' specifiers with named '%(...)' specifiers and improving various miscellaneous problems like concatenations that make translating harder. 394cea7 continued this topic by beginning the transition to Weblate. A POT (PO Template) file was introduced, meaning we could now use more standardized infrastructure like msgmerge to handle updating our translations. We wrote some special porting code to handle substitutions like %s -> %(...)s (named specifiers that we now prefer for their improved translatability, see also #55) and switched all our translation files to UTF-8 (they were a jumble of different encodings before). The second part of the Weblate transition occurred in 4dc8570. This brought us significant architectural improvements like translations being compiled at buildtime rather than runtime, which meant we could stop shipping our vendored dependencies (msgfmt and pygettext from the Python upstream) at runtime. And, of course, it set up the CI infrastructure for an automatic flow of translatable strings to Weblate and updated translations back into the WB repo. Part 3 arrived directly afterwards in dea7ce0. Various small fixes to the CI setup made their way in here. In addition, we improved our script architecture significantly. All scripts now log in one place (scripts/log/*), use the standard Python logging API and feature --help text. Finally, the build script got massively simplified by dropping its brittle git stashing. Instead, we explicitly exclude the files we don't want to include in the built releases. fd36189 fixed a couple issues with our translatable strings like double spaces or usage of '...' instead of the Unicode ellipsis '…'. add5b8f then improved our l10n generation script to automatically add the 'python-format' flag to appropriate translations, which significantly improved Weblate's handling of our translations, making it highlight and warn about translations that don't properly handle the format specifiers (along with other cool features, like Ctrl+1/2/3 etc. to directly insert format specifiers from the source string into the translated string you're working on). 50e5b02 saw a few more small fixes and improvements to the translatable strings, but most importantly brought the first new languages in the form of Swedish by David Brofalk and Turkish by @TheMalteseFalcone. In ded5321, more translatable strings got edited to make translations easier (e.g. by making the ==Last== marker translatable) and pulled in a new Ukrainian translation by @IllusiveMan196 along with various translation updates. More translation updates were pulled in with da619a7. In c8380e1, @xsSplater finished the Russian translation of WB. e8b8665 brought in even more translation updates. The last big merge of this release was e648960, which introduced a new Tamil translation by @TamilNeram and updated some other languages. ----------------------------------------------------------------------- ### App Launchers (#570) There's still a lot to do here, but we got started refactoring the app launchers (the little icons on the status bar that launch other applications, e.g. LOOT). dbdccd5 saw saw a load of miscellaneous cleanup and refactoring in app_buttons.py. ----------------------------------------------------------------------- ### Refreshes (#265, #353) One of the other big goals of 313 besides the translation rework was refactoring the refresh system. We want to eventually move to event-based refreshes (see #265), but to do that, we need a rock-solid system for tracking refreshes. This is closely related to #336 (see above), because in order to track refreshes properly, we need to track files properly. bcf08e3 began work here by reworking the way we calculate changes to the load order. It also introduced RefrData (adapted from BAIN's RefreshInfo) to centralize all the various bits of refresh info (updated/added/deleted files). fb769f0 centralized refreshes by moving various operations from DataStore to subclasses of ListInfo. Along the way, a few edge case bugs were fixed and the code got a major facelift. In eea41fa, the BSA load order calculations were refactored heavily. The result is much more accurate APIs, a cache and support for Morrowind BSA load order. That also paves the way to the BSA tab (#233). Finally, in c6761b5, the FileInfo(s) APIs got a few more improvements, refreshes got centralized further and the final phase of the refresh refactoring can be begun in earnest. And the first part of the final phase of #353 also managed to sneak in at the last minute in 222954b, where we can, at last, pass the RefrDatas introduced by the merges mentioned above to RefreshUI, which should signifcantly reduce the amount of GUI redrawing we have to do when refreshing. ----------------------------------------------------------------------- ### Game support improvements @sibir-ine contributed many improvements to our out-of-box game support, especially for Starfield, along with some fixes and updates for the record definitions. Many thanks! 906e14a restored support for the disc/retail versions of older games, which we accidentally broke when we rewrote Steam game detection back in 20dd955. Whoops! ----------------------------------------------------------------------- ### Skyrim VR ESLs (#673) Skyrim VR does not support ESLs out of the box. However, a third-party SKSE plugin was created that backports ESL support to Skyrim VR. So in 313, we added support for these. This is interesting, because it's the first time we have a truly dynamically supported game feature, where we need to check for the existence of the SKSE plugin to know whether or not the game supports ESLs. c815d10 includes some preliminary support by fixing a bug where WB would write out BPs with a header version that SSE introduced, but Skyrim VR does not understand (without the third-party plugin). ca361bd fully implemented support for ESLs in Skyrim VR. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525, #680) More FO4 record definition work landed in d6d53c4. This merge introduced some new tools for building record definitions and implemented 7 new record types. I honestly miss writing the FO4 record definitions, it's very methodical and lets me just turn my brain off and listen to music, so perhaps look forward to more of these from me in 314 :) The FO4 Next Gen Update also happened during 313's development, so we added support for it in cb24caf. ----------------------------------------------------------------------- ### Starfield updates (#681, #688) We still barely support Starfield, but some improvements have been made to WB's support for it. c54aa91 added support for saves created by Starfield v1.9. 6ef81f4 featured some preliminary work on Starfield support, including work towards v1.14 save files, reverting much of the Overlay support (thanks Bethesda...), reverting the CCC file workaround, etc. With the newer Starfield updates comes a new plugin type, medium plugins (plus there's the blueprint plugins, and the older overlay plugins that Bethesda broke). It became clear that our current, mostly ad hoc, handling of plugin flags would no longer scale. So in 6229baa, we tackled this mess by encapsulating all that complexity in a new class, PluginFlag. The rest of the outstanding Starfield support for 313 was also implemented in this merge. As a side effect, a lot of refresh complexity was reduced, so this is also under #353 (see refresh section above). ----------------------------------------------------------------------- ### Persistent language setting (#26) The oldest still-open issue in the WB repo was closed in this version. Issue #26 was about making the language switching option in the settings dialog persistent. There was a good reason why this issue was open for so long, however: WB's settings are stored per game, but language has to be set before a game is chosen. This fundamental problem made this issue infeasible for a long time. However, with all the translation improvements and updates that landed in 313, we decided to tackle this. We settled on introducing a new TOML file for early boot settings, which also permitted us to introduce another long-requested feature: remembering the last-selected game and defaulting to that in the game selection dialog. All these changes and improvements landed in 50af459. ----------------------------------------------------------------------- ### Load order (#578) Lots of random refactorings up above improved our load order handling, but a concerted effort to improve it even further was made as well in issue #578. The first major merge related to this was 7345863, which massively reduced the complexity of _games_lo.py and moved the static data (e.g. the CCC fallbacks) to game/, where it belongs. Somewhat born out of this and the launcher refactoring mentioned above, ini_files.py got a major facelift in e5dcd17. The entire parsing logic got rewritten and lots of complexity disappeared. More load order refactoring happened in 3587801, where we cleaned up pinned plugins handling to pave the way for better Starfield support (see section above). ----------------------------------------------------------------------- ### Customizable checkbox colors (#511) Not done yet, but @lojack5 did great work towards this goal in f962cc9, where the checkbox PNG images were replaced with dynamically generated SVGs. This already has the advantage of scaling perfectly on high DPI displays, but will also allow us to make their colors customizable in the future. As a bonus, the game icons also got turned into SVGs for high DPI scaling. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Utumno, @Infernio, @sibir-ine, @lojack5, @TheMalteseFalcone, @xsSplater, @robertgk2017, @cottondisciple, @jbostrus, @IllusiveMan196, @Evan555alpha, @TheBronn, @TamilNeram, @JonySnowball, @ernieIzde8ski, @Ortham and many more who GitHub's contribution tracker doesn't list.
312.1 Bug fixes for 312 [2024/01/07] [Infernio] This release fixes a few critical bugs and regressions that made their way into 312. There's potentially a couple more fixups coming for FO4, but they're all related to Bashed Patch building, which is experimental in FO4, so I'm fine with including those only in the WIP builds.
312 Starfield [2024/01/04] [Various community members] On the commit side, we managed to keep this one smaller (~250 commits rather than the ~500 commits that made up 311). On the complexity side, we probably failed, introducing a couple huge refactorings. A lot of Fallout 4 refactoring and patcher porting, Starfield support, native Linux support and refactoring WB's refresh handling are the biggest contributors to line count and commit number. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### FileInfo(s) (#336) #336 came back with a vengeance this time around. It started with fbb1925, which reworked the AFile API to cover installers and netted us some very nice performance gains and simplifications of refresh APIs - we'll some more work on refreshes later on. ----------------------------------------------------------------------- ### Native Linux Support (#243) Since Windows crapped out on me recently, I decided to ditch it and use Linux full-time. As a result, native Linux support for WB suddenly became a much more pressing issue. 312 improves it to the point where we now mark it as supported, with only some QOL issues remaining (and launchers, but those are a complete WIP anyways, see the later Launchers section in this commit message). There were a ton of random commits that made up this improved support. Basically, whenever I noticed something broken or in need of improvement, I would end up committing the fix, which means the commits are scattered like crazy. Nevertheless, here are some highlights: - e86e939 reworked temporary files handling, but this deserves its own section, which it will get below. - 1993f9b reworked parent handling in wx menus. We were always using Link.Frame.show_popup_menu, but this broke the Bashed Patch menus on Linux entirely. This turned into a mini-refactoring under #190 to track down and do this properly. - b762cc6 made the date and time dialog work on Linux (at the cost of no longer using locale-specific date/time formatting for it). - 20dd955 rewrote Steam game detection entirely to avoid the Windows registry. This was prompted by Starfield not creating a registry key at all, but I was planning to do this anyways, because Linux obviously does not have a registry and because it means users no longer need to launch the game once to generate a registry key. - 61d4d87 is what prompted me to actually mark Linux as supported. This merge added: - Proton detection, meaning the out of the box experience on Linux is now comparable to the one on Windows (in terms of it simply detecting your installed games with no manual intervention needed). - A warning when the Data folder is mounted case-sensitively. - Various fixes for race conditions and more, exposed by Linux's filesystems (perhaps specifically by btrfs?). - Functional Ctrl+Tab handling for cycling through WB's tab and some other misc improvements. - Also worth mentioning here is the File Operations (#241) section, see below for more on that. Linux was then finally marked as supported in c855882. Shortly before 312 was released, @BGazotti also contributed a whole host of small fixes and improvements for WB's Linux support in 00381da. Many thanks! ----------------------------------------------------------------------- ### Temporary Files (#665) This was born out of three needs: - On Linux, the global temporary directory (/tmp) is generally mounted in RAM. This means one can easily run out of space here when e.g. extracting a giant couple BSAs. And even worse, if the system has no swap configured, it can completely lock up when this happens. Wrye Bash should, in fact, *not* lock up the entire system. - We can get a decent speed boost by making sure the temporary directory we're using sits on the same device/filesystem as the Data folder. That way, the OS only has to rename some file paths rather than copying data around. - And lastly, our temporary file APIs were all over the place. There were three distinct types of temp file handling, and various pieces of code seemingly randomly chose which one to use: - bass.getTempDir/newTempDir/rmTempDir - Path.temp and Path.untemp - Just use Path.baseTempDir/tempDir or even tempfile directly and do it completely manually See the commit that introduced this refactoring (e86e939) for a full breakdown of the problems these APIs had. So we designed a new API that can cover all use cases and makes it hard to get wrong. Because, as it turns out, correct temporary file handling is *hard*. And another huge advantage of this design is that it will work with multiple instances of WB running in parrallel, which is an eventual goal. See the aforementioned e86e939 for the full lowdown. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525) The refactoring towards full FO4 Bashed Patch support is still ongoing. The goal was to get it done for 312, but then Starfield released and took the title of this version, plus we didn't want to drag out the release of 312 even further. Still, 312 brings the BP for FO4 very far. Work began in 2327ef4, which cleaned up header flags and implemented the next ten record types. Next up, 548bce5 tackled two difficult record types (NPC_ and OMOD, see the referenced commit for details on their problems) and implemented the next seven record types. With so many record types done, it was time to test them properly. To that end, d941cae ported the first batch of patchers over. In total, we now have 20/33 patchers available for FO4, though not all of them support all the record types they could support yet, simply because those record types aren't implemented yet. 28fb000 continued the records refactoring, though with the added complication that now, each time we implement a record type that an already-ported patcher can target, we also add support for that record type to the patcher in question. In that vein, this merge implements the next thirteen record types and adds them to patchers where appropriate. ----------------------------------------------------------------------- ### Starfield (#667) The star of the show (no pun intended). Note that this is early support, meaning that we don't actually support reading or writing plugins for Starfield yet. The main reason for that is Starfield's... *poor* design, when it comes to the plugin format. You can see the merge commit for more details (ec30f02), but basically, Bethesda clearly did not take moddability into account when designing the plugin format for Starfield. Unless they *drastically* rework the engine before releasing the Creation Kit, the Bashed Patch might not happen, period. ----------------------------------------------------------------------- ### wx (#190) It never ends (okay, maybe soon, but no guarantees). ea96e99 at least took some good steps towards closing off #190 for good by decoupling bosh (the data model) from balt/gui (the GUI backend). An important step towards #600 (headless mode) as well. Some more work landed in 170ad99, where @Utumno introduced gui.Lazy, a wrapper class for components that need to lazily create their native widgets. This led to heavy refactoring of the status bar and its buttons, taking down lots of dodgy code in the process. Also contained in that merge is a sub-merge, bd0a897, which utilized the new lazy classes to really tackle WB's image handling. Especially useful in preparation for the high DPI improvements that will be elaborated on later in this commit message and the launchers (see Launchers section below). ----------------------------------------------------------------------- ### Overlay Plugins (#668) The actual reason to be excited for Starfield's potential is shown in 9d21b40. Overlay plugins are a new type of plugin that does not take up a plugin slot at all, meaning you can have an unlimited number of them(!), with the downsides that such plugins can't contain new records (since they don't have a plugin slot to place the new records into) and must have at least one master. Unfortunately, due to the aforemnentioned massive problems with Starfield's plugin format, overlay plugins and ESLs aren't usable right now. Let's hold out hope that Bethesda can fix this, if only so the ~1 day of engineering I put into supporting them won't go to waste :P But jokes aside, the refactoring in this merge is very much worthwhile on its own. It makes supporting multiple mergeability checks for one game possible (since Starfield supports both ESLs and Overlay plugins - well, in theory it does) and thus opens the door for #596. ----------------------------------------------------------------------- ### Refreshes (#265, #353) We already mentioned fbb1925, which tackled refreshes from the perspective of BAIN and AFile. But midway through 312's development, I wanted to try my hand at implementing a new tab, namely the xSE Plugins tab (#456). That made me run headfirst into a wall, namely the fact that BAIN only knows about plugins and INI tweaks when it comes to trackin and updating other data stores. BSAs were kind of in there too, but not fully. My new tab needed BAIN to keep proper track of it as well, which meant refactoring. That bloomed into a very nice merge in c6ec399, which took down a bunch of old and rusty APIs (e.g. RefreshUIMods, saveListRefresh, _is_ini_tweak, etc.). The refactoring eventually culminated in us centralizing cross-tab communication via bass.Store and taking down a few hundred lines of really annoying, duplicate code. Some followup refactorings grew out of this as well. 70fe061 took down lots of complicated code related to ghost handling, paving the way for much more refresh handling work to come in 313+. Similarly, b8d9e0a refactored ModInfos.refresh, slowly unraveling the complicated stack of refreshes we've got going on. 387c9df tackled refreshes from the perspective of load order handling, attempting to simplify the latter significantly. One big goal here was to preserve as much information about the changes in load order throughout any given LO operation as possible, since that is crucial in order to properly refresh only the changed files - think elegance *and* performance. ----------------------------------------------------------------------- ### File Operations (#241) d897347 reworked our file operations backend significantly, prompted by two things: - Linux support, since we can't delegate to ifileoperation like we do on Windows (duh). - The last couple bits of FOMOD support (fingers crossed). Specifically, it turned out that StarUI, a Starfield mod, would fail to install correctly in WB when run via FOMOD. The reason turned out to be that it wanted to install a single file to two destinations. You can see the linked commit for all the details. BAIN did not have much trouble handling this, since it generally stores everything as destination -> source dicts, but our file operations backends did not support this at all. As part of this commit, we also introduced support for reflinks/file cloning to Wrye Bash. This currently has to be done via a less-than-ideal third-party depedency. An upstream request to the CPython project has stalled. As a result, we only support reflinks/file cloning on Linux and macOS filesystems right now (not a major problem, since ReFS has shown no signs of making it to mainstream Windows deployment yet). But what are reflinks? They behave like regular file copies, except that the backing data is shared between both copies. This is generally enabled by copy-on-write (COW) filesystems, since this feature is pretty much implemented for free on such filesystems. This lets all of WB's copy operations become much faster on Btrfs, XFS, ZFS, APFS, etc. ----------------------------------------------------------------------- ### Auto-Splitting the BP (#533) A long-standing feature request (open for more than three years at this point) has been addressed in 312: automatically splitting the Bashed Patch into multiple files once it ends up with too many masters for a single file. The advantage is obvious, especially compared to the previous behavior (throwing all work away at the end). 6ef2198 introduced the feature, though it turned out to be simultaneously much less work than I expected *and* much more complicated than I expected. Which is weird. ----------------------------------------------------------------------- ### Scattered Packages (#670) I have had this idea for more than three years, at least since I completed work on #380 back in 2020. The last big problem with FOMOD support in Wrye Bash was that a whole bunch of weirdly packaged FOMODs could not be installed in BAIN without a silly workaround (creating an empty plugin and putting it somewhere in the package to trick BAIN into thinking it's a simple package, then using its FOMOD support to remap files and thereby skip the empty plugin, which won't be referenced by the FOMOD's ModuleConfig). The plan was to model the properly, then see what happens. That meant creating an entirely new type of BAIN package. Previously, we had three types: - Simple packages, which contain some files to be placed in the Data folder. - Complex packages, which contain multiple sub-packages. Each sub-package behaves like a simple package. - Invalid packages, whose layout BAIN can't make sense of. Granted, from an end-user perspective, there are more types of 'packages': - Simple/complex packages, which behave like simple packages but have some folders in between the package root and the actual files that need to be placed in the Data folder. In WB's code, these are treated as simple packages. - Markers, which don't exist in the filesystem and are only an aid for users to better organize their packages. In WB's code, these aren't technically packages at all. 759055c introduces a new type of package: - Scattered packages, which have an unrecognized layout, but also come with instructions that tell BAIN how to make sense of the *scattered* mess of files that it otherwise can't parse - hence the name. Combined with some more refactoring to make BAIN recognize scattered packages by the presence of an 'fomod' folder and the aforementioned multi-destination file installation support, this finally tackles the last remaining FOMOD issues (again, fingers crossed). ----------------------------------------------------------------------- ### High DPI (#511) One big goal for 313 is #511, i.e. the ability to customize checkbox colors. But this is much more than just that - it's also about simplifying the resources for checkboxes by using a single SVG instead of a ton of different PNGs, making it possible to have more overlays over the icons (the current implementation of the wizard overlay is just terrible, it's literally a matter of copy-pasting all the checkbox icons, adding a wand over them and then loading all of them manually. This won't scale if we want to, e.g., add an FOMOD overlay). And, of course, this is also about using an SVG to make these scale properly on screens with >100% scaling. To that end, ff276b1 addresses high DPI handling of images. Yes, we already had support for high DPI images via SVGs since #557 in 313, but it turns out that implementation had lots of holes - most notably at 200% scaling, where wxPython decided to scale all our icons *again*, making them absolutely massive and really ugly. With the aforementioned commit, nothing should stop us anymore from tackling #511 in 313. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BGazotti and many more that GitHub's contribution tracker doesn't list.
311 Fallout 4 preparation [2023/05/28] [Various community members] I wanted this release to be smaller for once. 500 commits and nearly a year later, mission failed :( Hopefully we'll manage to keep the next one smaller. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### Fallout 4 (#482, #525, #650) The first major change in 311 was the beginning of the road towards proper Fallout 4 records support and hence a fully function Bashed Patch in FO4. - 00c9ee8 implemented a key piece of the records infrastructure, NVNM. It also dropped some hacks to further pave the way. The first two FO4 record types were implemented. - dc6311a refactored CTDA and implemented the next seven record types. - 641bc73 tackled the single biggest challenge of the entire undertaking: VMAD. As a bonus, if you enjoy rants about stupid design decisions, see c5b5d8a. The next ten record types were also implemented. - 28578a6 moved some record types that are identical between Skyrim and FO4 to brec and implemented the next eighteen records. - This was immediately followed by bb7e5ae, which slightly refactored GameInfo.init() but mostly just implemented the next twelve record types. - 3d8b23e implemented nine more record types. - 9bfb6de implemented two more. Also, @BeermotorWB began the porting process by bringing Tweak Settings to FO4 in 57b2443. There are about sixty FO4 record types left to implement. This will be concluded in 312. I chose this part for the title of the release ("Fallout 4 preparation") since it's what occupied my brain during most of 311's development. ----------------------------------------------------------------------- ### Python 3 (#618, #619, #644) Since moving to Python 3 in 310 (see #460), we've been taking full advantage of everything py3 offers us. Some example commits: - 8de5f40 cleaned up one of the most annoying files to edit, common_subrecords.py. - cc198c9 shaved hundreds of lines off by replacing verbose nested OrderedDict instantiations with regular, now-ordered dicts. - fd1d4d9 took advantage of ordered dicts to drop duplication between the records list we used to keep track of records in a Mob* class and the id_records dict we used for fast random access by FormID. Now we only need the dict. - In bbb8fa2, @lojack5 used Python 3's type annotations to refactor our flags handling. We also upgraded to Python 3.10 in 7b3573c and to Python 3.11 in 69e7e3a. We also optimized a bunch of central code for e.g. the BP, FName and Path to take advantage of changes like 3.11's adaptive interpreter, which gave pretty good speedups. We've also been doing what we call 'fdusting' - replacing any random pieces of code that look like a + '.esp' with f-strings, i.e. f'{a}.esp'. Another (made-up) example: 'Loaded %d images' % len(images) becomes f'Loaded {len(images)} images'. This has been happening as a sort of "if you're editing in the vicinity, fix any you touch" thing, so spread out over tons of commits. F-strings are one of the best things about py3 :) ----------------------------------------------------------------------- ### Records (#480, #647) The eternal #480 rears its head again, as it does every release. First off, every single merge mentioned in the FO4 section above also falls under this section. In addition to that, we are approaching setDefault. Uprooting that will give us major speed boosts, simplify code, fix lots of random "the BP is editing this value for some reason" bugs (all of those are due to defaults), etc. One major blocker was FormID handling (see section "FormIDs (#637)" below), now cleaned up. Some other commits related to records refactoring in 311: - 28e0734 reworked defaults handling, making defaults explicit where they are needed and eliminating them from most structures in the records code. - fd1d4d9, already mentioned in the "Python 3 (#618, #619, #644)" section above, reduced duplication in the record_groups code by getting rid of the 'self.records' list in favor of just the (thanks to py3, now ordered) 'self.id_records' dict. - 71520f7 massively reduced code duplication in the various games' init() methods and the __slots__ definitions for all the record types for a total of nearly 1000 LOC removed! - bbb8fa2 by @lojack5, already mentioned in the "Python 3 (#618, #619, #644)" section above, took advantage of Python 3's type hints to refactor our flags handling. The biggest merge of this development cycle also falls in this category. In a truly herculean merge (d2152aa), @Utumno tackled record_groups, turning it from a TODO-riddled mess that didn't really know if it wanted to be Oblivion-only or game-agnostic into a modern API that will serve us well for the future (e.g. for FO4, which makes QUST a new top-level group). Seriously, this refactoring was so complex, its merge has two sub-merges. ----------------------------------------------------------------------- ### FormIDs (#637) One of the major refactoring goals of 311 was tackling FormIDs. We used to repesent them as either integers (called "short FormIDs") or tuples of strings and integers (called "long FormIDs"). Note that the latter became tuples of FNames and integers in 310 (see section "FName (#543)" in the 310 release commit. This was achieved in 4964710, which is once again an entire branch squashed down to a commit so as to not break dev. They are now stored as a class instead (brec.FormId). This required careful engineering and testing to make sure we don't kill BP performance. For the full story, see the linked commit. ----------------------------------------------------------------------- ### Localization (#55, #500) We're trying to replace all '%s' specifiers with more useful '%(something)s' specifiers. This is still nowhere close to being finished, but 2442285 began the process. Once this is done, we'll want to rethink how we allow people to contribute localizations - see #500 for more info. ----------------------------------------------------------------------- ### Epic Games Store & GOG (#646, #648) No release is complete without adding support for new games, and 311 is no exception. Bethesda released most of their games on GOG and some on the Epic Games Store, so we added support. See the linked issues and the referencing commits for more information. The biggest difficulty was adding support for yet *another* method of game detection (for the Epic Games Store). Still to come is refactoring the game detection on the Steam side, since some of these rereleases will actually overwrite the registry key used by the Steam versions, making it impossible for WB to reliably detect the installation of both versions. This will also pave the way for Proton-based game detection on Linux (see "Linux (#243)" section below). ----------------------------------------------------------------------- ### Nexus integration (#459) 5e688fd prepared us for proper Nexus integration, which will be one of the big feature goals of 312, by merging a nexus.py file that wraps the Nexus API into the source tree so that keeping it up to date is easier. ----------------------------------------------------------------------- ### FileInfos (#336) Some refactoring work on FileInfos and related APIs occurred as well, mostly on the INIs. See 22c8c5e, 2495972 and b3e4116 for details. ----------------------------------------------------------------------- ### wx (#190) That's right, more of this. - 6f2bd1d finally de-wx's the mighty UIList (which backs all of WB's tabs) by making it inherit from our PanelWin instead of wx.Panel. Turns out it was that easy (I'm sure it had nothing to do with the metric tons of refactoring that had been done towards this in 310, 309, 308 and 307). - f7dd4dc decouples us from wx with regards to the default, builtin art from wx we used to use. We instead now use SVGs (mostly from Font Awesome - don't worry, I was careful with the licensing), which also gives us some consistency across platforms. - d1290a5 begins the fight against balt.ListBoxes, an absolutely horrible class that not only had a really weird API but also featured terrible usability. I wrote a mini-novel on this refactoring, see the linked commit. The second part arrived later, in 15444ca. - 6659eb3 features some good old de-wx'ing of dialogs for opening/saving/choosing files. Also included is some refactoring of status bar-related classes. ----------------------------------------------------------------------- ### Usability (#625, #643, #645, #652, #656) Back on my bullshit, aka trying to make Wrye Bash more usable. For 311, we have: - bfdee3e, which added a Ctrl+S shortcut (and matching link for discoverability) to make frequent saving easier. - 47ef378, which aimed to improve the INI Edits tab's usability by making it more consistent with the other tabs (e.g. adding Alt+Click support, opening tweaks via Enter or double click, etc.). - effcf83, which added links for LO undo/redo to improve their discoverability. - 486a640, which restricted the red background color to timestamp conflicts since it kept confusing people. See the relevant issue (#656) for some more background info on this change. - 941fb25, which was born out of me going "hey, I've never used the Fit Contents and Fit Header options before, let's try that", really liking how much easier it made adjusting columns for Master lists, then noticing that it applied *globally* and so nuked my carefully adjusted columns on the Mods tab. Instead, each setting now only applies to the tab/list you enable it on, allowing you to use it for just the lists where it helps you. - d1290a5, which was already mentioned in "wx (#190)". ListBoxes was just no good for usability. They didn't remember sizes, always started out being sized wrong and don't even wrap text correctly when you do resize them. Replacing them with custom classes allowed us to massively improve the usability of all kinds of popups and dialogs throughout WB. The second part of this refactoring arrived later, in 15444ca. ----------------------------------------------------------------------- ### Image handling (#366, #557) As an offshoot of #190, we have the venerable #366. Image handling is still very much a work in progress, but f7dd4dc brings us a lot closer. Fleshing out the image API and making it ready for SVGs, it also replaces many of our somewhat crusty images with SVGs that can scale to any size we need. It also further decouples us from wx (see the second entry in the "wx (#190)" section up above). ----------------------------------------------------------------------- ### File operations (#241) In b5bb441, @lojack5 changed our file operations backend on Windows to use the newer IFileOperation API instead of the old SHFileOperation API. This was no easy task, as pywin32's support for IFileOperation is broken for some reason and so @lojack5 had to write a wrapper from scratch. ----------------------------------------------------------------------- ### Update checking (#663) Come 312, we want to replace our nine(!) Nexus page with a single one in the Modding Tools section. The LOOT team recently took the same step. The main reason is simply because updating nine pages for every release *sucks*. Even moreso if the release gets detected as a virus. However, the problem with moving to a single page is discoverability. The Nexus team have indicated that they're interested in adding the ability for pages in the Modding Tools section to indicate that they support certain games, so that they'll show up in the search results for searches on those games' pages, but that's not implemented yet. To that end, we implemented an update check. Once 312 is released on the new Nexus page, we can stop updating the other pages and direct users to the new one via the update popup. Of course, the update check can be disabled in the settings and a manual version of the check is also accessible from there. You can also adjust how frequently the update check happens (the default is once every hour). ----------------------------------------------------------------------- ### Linux (#243) Near the end of 311's development, I bought an NVME SSD and took the opportunity to redo my dual boot environment. After backing up and restoring the Windows part of my dual boot, Windows promptly decided to ruin my day by first breaking Firefox and Thunderbird, then breaking the entire Start menu. Since I can't really get work done when I have to use stupid workarounds like "Windows+R > explorer.exe or wt.exe > launch the app I actually want to launch", I said "screw it" and moved entirely to Linux. Born out of that is the last part of 311: - 1d6db12 fixed a smorgasbord of random problems when using WB on Linux. - 1fe6d2d made BAIN wizard and FOMOD images work on Linux. - 1581953 fixed the GTK webview refusing to load when we have a file with '&' in its filename. - 1d07fc3 implemented sending files to the recycle bin on Linux. Most of these apply to our (even more WIP) macOS support as well. ----------------------------------------------------------------------- There were many, many, *many* other changes - again, 500 commits. One person who hasn't been mentioned so far is @sibir-ine, who contributed lots of minor improvements to WB's out-of-the-box usability (e.g. default BAIN directories) and tirelessly helped with monitoring the Discord server and recording the reported bugs on our project board. Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @sibir-ine, @lojack5, @BeermotorWB and many more that GitHub's contribution tracker doesn't list.
310 Python 3 upgrade [2022/07/24] [Infernio, Lojack, sibir, Utumno] 310 is the culmination of years of work, finally moving us to Python 3 (3.9 to be exact - we can't upgrade to 3.10+ yet, see #619). All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### New BP Tweaks (#602) The first major change in 310 was a contribution by @sibir-ine, who added six new settings tweaks in a423f34. @sibir-ine also contributed further patcher and tweak updates throughout 310's development, especially targeting consistency of patched records and subrecords across games. Some of the relevant commits: - 41ecb03 & 4ea307f: FO3/FNV patcher updates - 6513e3d: TES5 patcher updates - 80f2338: Improvements to tweaks - fe70f2d: New settings tweak Many thanks for the great contributions (and endless help with the Discord server)! ----------------------------------------------------------------------- ### Python 3 (#460) This is of course the highlight of the release. The merge that finally completed this goal is f0a803b, but we had to squash the (lengthy) branch that completed the upgrade down to one commit so as to not break dev. See that merge's commit message for more details. There is still some followup work to do (and a bunch that was already done throughout the rest of 310), but this is such a massive step forward it's hard to overstate. Hacks gone, bugs fixed, dependencies upgraded, new stdlib tools starting to get employed (see e.g. 43fc54c, which allowed us to get rid of our bundled lzma.exe in favor of py3's stdlib lzma module) - and of course the main benefit being that we're no longer stuck on obsolete technology. Massive thanks to everyone who has contributed to this herculean effort over the past four releases! ----------------------------------------------------------------------- ### Locale crashes (#610) One of the biggest remaining issues in WB post-wx3-upgrade were the locale crashes. This proved to be a wild goose chase. Every time we thought we'd solved it for good, some user with a different unusual setup came along and proved us wrong. The core problem is that wxWidgets, wxPython and Python itself are all trying to handle locales, and all of them end up fighting over it. We initially thought we'd finally fixed it in 2ebc752, part of the py3 merge, but that *still* didn't do the trick. 0e93496 seems to have finally fixed it, but at the cost of *completely disregarding the user's locale* and always setting the C locale. Far from an ideal solution, if you can even call it that. wxPython 4.2.0, which will be based on wxWidgets 3.2.x, should hopefully help here, when it finally (it's been *two years* since the last wxPython release!) gets a release. ----------------------------------------------------------------------- ### Skyrim AE (#616) Midway through 310's development, Skyrim Anniversary Edition was released. As this is really just an update to the executable and an optional batch purchase of Creation Club plugins, adding support was easy. 937b182 completed the task. However, an unexpected annoyance arose from this. All of the Creation Club plugins included in the AE bundle are localized, and it turns out WB's boot process did not love this. Which is why f789663 was necessary. This merge significantly improves the boot performance in AE (~50% on my machine, meaning an improvement from 2s down to 1s). ----------------------------------------------------------------------- ### Windows Store versions of FO3/FNV (#617) Quick and easy to support. Note that Microsoft has *once again* changed how Windows Store games are installed, and 310 does not support this new method yet. See #639. ----------------------------------------------------------------------- ### FName (#543) After the py3 upgrade mentioned above, the next biggest beast lurking in 310 is #543. What started as an innocent enough idea (bolt.Path is pretty heavyweight and unnecessary for use cases like masters and filenames, which never have path separators in them - would be great if we could replace all that with a more appropriate and lightweight class) quickly ballooned into a massive refactoring of much of the codebase, eventually culminating in one hell of a rabbit hole whose event horizon spanned many merges and three WB releases. See 308 and 309's release commits for notes on #543 progress back then. Reiterating all of this here would take hundreds of not thousands of words, so this section will only focus on #543 during 310's development. 15b979f reduced Path usages, replaced some LBYL patterns with EAFP ones and wrapped common patterns as preparation for the final #543 merge. Some more refactoring, especially regarding BAIN, happened shortly afterwards in 97768fd. The next few commits (specifically c55b60d and 9c25544) mostly focused on renames and some minor refactoring in preparation for further attacks on Path and its usages. In 8555285, many more Path usages were pruned. And in 9a0db27, the converters part of the codebase (the code that handles BCFs) received significant touchups. More explicit Path usages gone, more LBYL -> EAFP refactoring and improvements to this somewhat ancient and dusty API. And then it was time for the big one. b3b730c introduced bolt.FName, the ultimate result of all this effort. This gave performance and memory usage improvements, improved code style in many places, etc. But the main benefit, as mentioned all the way back in the 308 release notes, is that this paves the way for further work on #480 and related issues in 311+. ----------------------------------------------------------------------- ### Preliminary macOS support (#622) f085cf0 introduced some WIP support for macOS. Just like with Linux, this does not mean you can actually use WB for that OS yet, but it's an eventual goal. The immediate benefit of adding partial support for other operating systems is the resulting refactoring and isolation of Windows-specific code. Some more refactoring towards this goal landed in 14c7639, which refactored our archives handling (used for reading and writing archives, e.g. .zip and .7z files) to be OS-agnostic. ----------------------------------------------------------------------- ### Skyrim patchers (#151) As of 802423a, the last patcher (barring FormID Lists, which will need rewriting - see #497) has been ported to Skyrim (to be exact, it was Tweak Names). This finally brings the eight-year project of porting patchers to Skyrim to an end. ----------------------------------------------------------------------- ### FOMODs (#627, #603 and #593) Ever since 307, where FOMOD support was first added to WB in #380 (see 307 release notes for commit links), we've been steadily improving BAIN's support for FOMODs. 310 continues this theme. d636b5b implemented the last few parts of the FOMOD schema that hadn't been implemented yet (foseDependency, fommDependency, alwaysInstall and installIfUsable). Next up, 870c0ed implemented schema validation using lxml. bc6d1c0 resolved a long-standing feature request by adding a link to capture the output of running an FOMOD installer into a new project. And finally, 2c603d6 added support for generating an empty ModuleConfig when creating a new project via BAIN. ----------------------------------------------------------------------- ### Usability (#628) Continuing the fight for better usability (aka me sticking search bars everywhere), the doc browser got a bit of a makeover in c7e555c. On top of small improvements like "Forget Doc" reselecting a document after being clicked, the doc browser now also contains a dropdown menu listing all your load order's plugins and has a search bar to help you narrow those plugins down. fdb34dd cleaned up the Installers context menu even further (this started way back in 307) by moving archive-specific links into an 'Archive..' submenu that only appears for archives and similarly moving project-specific links into a 'Project..' submenu. WB has support for various keyboard shortcuts. Not that you'd ever know that unless you read the docs (wish more people did that to be honest, but oh well...). To help with this discoverabilty issue, b8d3e8d added keyboard hints next to links that perform the equivalent functionality. And while we're on the topic of keyboard shortcuts, 70ca87e and d8fdd57 added some new ones. Told you this is just me sprinkling search bars all over WB: 03122cc added search bars to every list patcher in the BP dialog. That means tweakers ("Tweak X" patchers) and importers ("Import X" patchers, including "Leveled Lists" and "FormID Lists") now have a search bar you can use to filter their sources. Especially useful for Tweak Settings, which has more than *eighty* tweaks in Oblivion! There are also, as per usual, a bunch more minor commits improving random bits of WB with regards to usability in 310 that aren't mentioned here. ----------------------------------------------------------------------- ### Patchers (#312) Yep, still going on (and will be going on for a while longer...). e3b93d1 refactored a lot of Oblivion-specific constants and code, especially stuff related to MGEF handling. This stood in the way of extending WB to handle more games in the future, since no one was sure how exactly to update these constants (or if they even needed updating for other games). ----------------------------------------------------------------------- ### Records (#480) And where #312 dwells, #480 is not far behind. One of the big goals for 311+ will be FormIDs handling, and 2e3e5ee functions as a prelude in that direction. Especially nice as it showcases how Python 3 (specifically its ordered dicts) can help in refactoring. ----------------------------------------------------------------------- ### Debug mode (or lack thereof) (#638) We were getting mighty sick of people not understanding how to properly report bugs and volunteers having to spend time over and over explaining the same thing. So in c9a5036, we changed WB's debug mode so that it's always enabled in standalone builds. That's great and all, but debug mode has always had one problem: if you enable it, you lose the "native" error reporting mechanism provided by the environment you're running WB in. Most notably the wxPython error window that would otherwise pop up if you hit a traceback. So in d66da27, we dropped debug mode entirely and always enabled debug printing. To solve the problem of the native error reporting mechanism getting disabled, we now copy the debug print output to both the native stdout and to the BashBugDump. That way reporting is as easy as possible: users get notified of a traceback and the BashBugDump already contains full debug prints. The reason not to do this in the past was the overhead of deprint, but since fab8b90 deprint is pretty much as fast as a regular print (all the overhead came from usage of the inspect APIs, which statted the source/bytecode files). ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @BeermotorWB, @warmfrost85, @GandaG and many more that GitHub's contribution tracker doesn't list.
309 Python 3 preparation, pt. 2 [2021/06/01] [Infernio, Lojack, Utumno] 309 continues where 308 left off, containing the last bits of refactoring and transition code needed before we can move to Python 3. All mentioned commits are authored by @Utumno or @Infernio unless otherwise mentioned. ----------------------------------------------------------------------- ### Python 3 (#460 and #597) Way too many commits for me to go in depth on all of them here, so a short list of important commits follows: - d22469b replaced all remaining usages of `items()` and `values()` with the `iter*` or `view*` equivalent, as needed by 2to3 to successfully convert them to the Python 3 versions. - c120e18 audited all usages of `open()` to ensure that they either open in binary mode or pass in an encoding. Interestingly, Python 3.10 will finally contain an option to emit a warning whenever this is violated: https://docs.python.org/3.10/whatsnew/3.10.html#optional-encodingwarning-and-encoding-locale-option - e823570 got rid of the last couple places that were causing bytestrings to end up in our settings. - f2f1140 added code to convert bytestrings in existing settings files to unicode strings. ----------------------------------------------------------------------- ### Patchers (#312 and #494) The eternal war continues in bb52329, which split the 'Race Records' patcher into several patchers. This was absolutely essential to porting the patcher to other games (which was done in the follow-up merge 2c4ce17). The race patcher was incredibly complex, containing as much functionality as *seven* smaller patchers packed into one. Most of the now-split patchers could use more refactoring to shrink them and make them more maintainable, but this is already a huge step. There are several side-wars being fought as part of the eternal patcher war. One of those is the war against records, and another one is the war against parsers. The last major advancement on this front was a merge near the end of 307's development that created a proof of concept design for an ABC (abstract base class) for the parsers. 9840c3b and bdd19a1 take this much further. In the process it has unearthed just how unfinished and rough the parsers really are, including tons of previously undiscovered bugs. This is still not done, but already a massive improvement. ----------------------------------------------------------------------- ### Enderal SE (#582) Enderal: Forgotten Stories: Special Edition is one hell of a title. Adding support for it was mostly straightforward, the only really interesting thing is that some fairly delicate untangling of load order properties was needed. First merge was done in 9a9981b, with a follow-up commit implementing the more difficult parts in 4298fc1. ----------------------------------------------------------------------- ### wx (#190) Some minor de-wx'ing happened in 5de85b8. Nothing particularly exciting, most de-wx'ing work is now focused on shrinking balt by moving its code into basher and/or gui. The same goes for b4290e6 as well. 7386e12 is another commit thinning balt and refactoring gui code further (this time targeting file selection dialogs). ----------------------------------------------------------------------- ### Game refactoring (#584) Just in time for the Windows Store merge detailed below, we refactored the way that game-specific about patchers is stored in a7d7300. ----------------------------------------------------------------------- ### Windows Store support (#585) Just as we were getting ready to do some extensive testing on Python 3 and perhaps do a release soon, Microsoft bought Bethesda and decided to drop new versions of four Bethesda games (TES3, TES4, TES5 & FO4) on us. These proved to be an absolute pain to support due to them being Windows Store apps, which were clearly not designed to be moddable: - Finding the paths at which the games are installed is pretty involved. If we had access to UWP APIs it might be much easier, but we don't. As a result, we basically have to emulate what those APIs do by jumping through the registry ourselves. - The game EXEs are locked down hard. That's also the reason why script extenders and mod managers like MO2 will not work with these versions of the games. - Launching the games has to be done through a special command (since you can't even access the EXEs). Nevertheless, @lojack5 and @Infernio managed to add full support for these games to Wrye Bash. BAIN, Bashed Patches, load order management, etc. - it all works. ----------------------------------------------------------------------- ### Plugin Checker rework (#83) A long-standing feature request was to make the plugin checker aware of record type collisions (where two records have the same FormID, but different record types - this can cause anything from unexpected behavior to CTDs). We took that much further and made the plugin checker scan for all kinds of problems in 6aa9bba, including deleted navmeshes, deleted references, deleted base records, HITMEs and even added a heuristic detection for injected record collisions. The Plugin Checker also contains full explanations for all of these problems, so don't worry if you don't know what some of those terms mean. ----------------------------------------------------------------------- ### Fallout 3/NV feature parity (#468 and #594) As of 309, Wrye Bash finally has feature parity with valda's FO3/FNV ports. The aforementioned merges that split and ported the Race Records patcher were the second to last step, and the final step was 8977e43, which ported the Tweak Names patcher. We even gained another (mostly) FO3/FNV-specific feature that valda's versions lack in the form of d363475, which allows BAIN to extract `.fomod` files, similar to how it can already handle extracting `.omod` files. ----------------------------------------------------------------------- ### LOOT v0.17 compatibility 02435d3 added support for features introduced in libloot v0.16.1, which will become usable in the masterlists once libloot v0.17 is released. ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @lojack5, @sibir-ine, @valda and many more that GitHub's contribution tracker doesn't list.
308 Python 3 preparation [2021/03/03] [Infernio, Utumno] 308 is a release almost entirely focused on getting us closer to running on Python 3. Nearly all refactoring here has the end goal of making that transition easier, even if that may not be immediately apparent. All commits mentioned here were authored by @Infernio or @Utumno. ----------------------------------------------------------------------- ### Records (#480) 308 development began with the merge of Part '2.5', which was already finished in 307, but had not been merged yet for lack of testing. This was done in bfae14e. 7e67f63 introduced a significant refactoring of subrecord handling, bringing all sorts of improvements like usage of AOT unpacker construction via struct.Struct, centralized strings handling (see also #460 below), removal of lots of special cases, etc. It also acts as a basis for most other records work that happened in 308 and will happen in 309+. A huge improvement came in the form of 0848872, which finally got rid of the horrible ModFile.__getattr__. This hack (let's call it what it is) caused fun, surprising behavior when debugging and trying to extend ModFile's APIs every time. It also stood in the way of #460, because py3 does not like getattr with a bytestring. d7bd007 was a merge that sprang from the goal of refactoring getNumRecords, but soon grew to encompass larger refactorings under #312, #460 and #480. One final key merge in 308 was af16a5d, which focused on the setDefault API. Wrye Bash mostly copied defaults from xEdit, but it turns out that in many cases we don't need or want to have defaults (e.g. for performance - see also next section). Dropping them entirely is not feasible, however, since we do need to write out some new records (e.g. the TES4 header, GMSTs, etc.). This merge is a step in the direction of making defaults None unless otherwise specified. ----------------------------------------------------------------------- ### Performance (mostly #563) One of my personal pet peeves is how slow many of Wrye Bash's operations still are. 309 will tackle more in this direction. These are not just nice to haves either, they can cause serious usability problems. See issue #563 for some information on the '0.1s threshold' and how important it is to usability of an application. Specific improvements in 308 are: - c0b638b improved performance when viewing INI tweaks and target INIs on the INI Edits tab. - 9946a83 cached the target INIs on the INI Edits tab to reduce system calls. - 3d09aeb was the main merge, significantly improving the performance of most load order operations. See the merge and its commit message for details on the improvements. - 8507350 was a small commit that fixed one of my biggest problems with 307: the incessant flicker every time you tried using the global menu. - 4a48ae9 dropped backwards compatibility with 306 settings (see #376), speeding up BAIN exit significantly. - bd73ea6 made CIstr construction much faster, which made many BAIN operations (e.g. dragging and dropping, startup, exit) faster. ----------------------------------------------------------------------- ### wx (#190) Some more de-wx'ing took place in f707b59 and 7645e40. In 5dbda51, we upgraded to wxPython 4.1, mostly for the high DPI code that was added in wxWidgets 3.1.x. We were originally OK with keeping bash.py was a file that directly used wxPython APIs, but high DPI (see #555 section below) forced us to find a way to de-wx even it. This was accomplished in e899dc8, which also laid the groundwork for showing our app icons on boot popups (which we did in 538ff2a). One major open problem with de-wx'ing was the wizards API. The problem is that the wx.adv.Wizard class wants instances of wx.adv.WizardPage to be passed along, but we want to encapsulate those instances entirely. In addition, our hacky usage of 'dummy pages' did not at all fit with wx's intended use case of the wizards API. All this came to a head when wxPython 4.1 broke the Back button in our wizards, likely because of our hacky usage with 'dummy' pages. Instead of monkeypatching and trying to fix it, we threw out the entire wx.adv.Wizard API and wrote our own, based on de-wx'd components, in a3e6c4d. With this merge, the biggest roadblock to fully de-wx'ing Wrye Bash is now gone. ----------------------------------------------------------------------- ### Patchers (#312) The first massive and extremely impactful merge of 308 was b25727b, known as 'tweak pooling'. This is an entirely different implementation of tweaks in the Bashed Patch that brings all sorts of improvements: - Thousands of lines of boilerplate code are gone - The pickle files we used to store injection FormIDs are gone - Tweaks are now almost entirely static classes (finished off a couple merges later) - Several tweakers have been ported to newer games - Significant performance improvements because much fewer records now have to be copied into the BP The second merge focusing on patchers was ac72709, in which the names of patcher classes were decoupled from the keys used to store and load them from/to BP config pickles. This allowed us to rename them to all fit one consistent scheme. Another notable commit that was part of this merge is eaeb831, which brings us much closer to being able to absorb Import Cells into _APreserver as well as fixing several issues with Import Cells. 8a9b581 was a small followup commit that unified scanOrder and editOrder into a single patcher_order variable. f3b6578 followed up on tweak pooling to make all tweaks entirely static. This brings with it huge benefits like potential for deduplication of similar tweaks, making it much easier to add new ones, etc. 9a1cff7 was a relatively small merge under #312, #460 and #480 that focused on renames and using some of the APIs introduced in earlier #312 and #480 merges to refactor more code. ----------------------------------------------------------------------- ### CBash (#530) After having been deprecated in 307, CBash was removed entirely in 120234c. See that commit for reasoning on why, but the short version is that it was unmaintained, cint was poorly designed and the CBash patcher implementations were extremely buggy. ----------------------------------------------------------------------- ### GPLv3 (#531) Mostly a formality, but we used the 'or any later version' clause of our GPLv2 or later license to upgrade to GPLv3 or later in 3721668. We were effectively already on GPLv3 because of the dependencies we used, some of which would only work if the combination of WB source code + dependency is licensed under the GPLv3. This is just making it official. ----------------------------------------------------------------------- ### Morrowind (#479) a5d40e1 was a merge that introduced decoded record classes for Morrowind. We're still nowhere close to actually reading and writing Morrowind plugins, but this is a step in the right direction. The main reason of course is so that all the records refactoring has to take into account Morrowind support when designing APIs. ----------------------------------------------------------------------- ### 64bit (#481) In 30dd357, we finally upgraded Wrye Bash to 64bit after it had sat around on nightly for a long, long time. The main reason for the hesitation was that some people reported cryptic tracebacks on launch (python-lz4 was failing to import). It turned out in the end that the fix for this was installing the MSVC 2010 x64 redistributable, which we now do as part of our installer. 64bit Python 2 gives us somewhere between a 10%-30% speedup for most operations that we tested, most notably the Bashed Patch. ----------------------------------------------------------------------- ### Python 3 (#460) As mentioned above, Python 3 preparation is the focus of this release. The main hurdle here turned out to be the amount of low-level bytestring handling that Wrye Bash does. As a result, refactoring these parts of the code was a key goal, which resulted in patchers (#312) and records (#480) refactoring becoming the most important open issues. Everything described in those sections above applies to this section too. Furthermore, we performed many targeted changes to make a Python 3 upgrade more feasible and to catch errors early. Several commits here were originally created by @GandaG in 307: - 4b4cc01 cherry-picked a bunch of backwards-compatible changes made by 2to3. - 8e0fa26 dropped Path usages in bash.ini settings handling (see also the section on #543 below). - 567f153 and 09e1633 replaced several usages of keys() and iterkeys() with equivalent expressions that work identically in py3 (e.g. next(iter(a)) intead of a.keys()[0]). - 83162d7 was a minor merge focusing mostly on getting rid of __getattr__ and __setattr__ and its cousins. They are slower on Python 3 than simple getattr/setattr. - b337e98 was a major merge focusing on cooperating with 2to3 to help it do better at its job. We also created a fork of 2to3 to push this concept even further with specialized fixers. - a75dbc6 was originally a large commit by @GandaG that shrank over the course of 308's development. It focused on dropping map/filter/range usages in favor of list comprehensions or xrange. Prefixing of strings was also an ongoing task, worked on in many commits: - cb6b9ce - 2486397 - 83162d7 - 1a29a01 - 20432e6 - 42e5795 - 52ba47b ----------------------------------------------------------------------- ### DataStore keys (#543) With Python 3 on the way and after a bunch of performance profiling (see the performance section above), it became clear that bolt.Path stood in the way of both. Thus, #543 was born - a project to make DataStore (our class for handling file collections like mods, BSAs, INI tweaks, etc.) use strings instead of Paths as keys. This turned out to be a gigantic rabbit hole, leading to a ton of refactoring: - 74e0524 was a merge containing a bunch of housekeeping, but the most important change was a reduction in the usage of the '.s' property on Paths. - 129e098 renamed tons of 'path' occurences to make tracking down Path instances easier. - 5d81acd was a big merge focusing on following the DataStore key rabbit hole down to its event horizon. The resulting refactoring improved the names of many variables, locals, etc. and brought us much closer to absorbing BAIN structures, Paths instances, etc. ----------------------------------------------------------------------- ### BSA load order (#534, #546) We had several scattered attempts to discern the order in which BSAs load all over the codebase. 6aec0ce centralized this handling and gave us several key improvements in the process, including the ability for BAIN to show conflicts with BSAs loaded via INIs, e.g. the vanilla BSAs (#546) and correct handling of the special sVrResourceArchiveList INI setting in Skyrim VR (#534). ----------------------------------------------------------------------- ### The People tab (#548) Was removed in 308. We used to keep it around for the purpose of keeping the data structures generic enough, but with the ongoing refactoring, it turned out to be too much of a maintenance burden, so it had to go. ----------------------------------------------------------------------- ### High DPI (#555) With the upgrade to wxPython 4.1, nothing was stopping us from finally adding high DPI support to Wrye Bash, which we did in e899dc8. This is still not perfect, the most notable issue is that we do not have high resolution versions of our icons, but it's better than the blurry mess Windows displays us as otherwise. 12571a6 was a small followup commit that marked our installer as high DPI-aware too. ----------------------------------------------------------------------- ### env (#258) Running Wrye Bash on Linux is a long-term goal that's very important to us - not just for personal usage, but also for code quality. One important part of this is abstracting over OS differences, for which we have our env module. Unfortunately, env in 307 was in effect a completely Windows-specific module that could only run on Linux if you commented out large chunks of it. All that was fixed in 19461b3, a merge that split env.py and windows.py into an env package. Wrye Bash can now launch of Linux and even build a BP - though it is still far from usable, especially BAIN. ----------------------------------------------------------------------- ### Renaming (#580) Renaming is a highly complex thing. Near the end of 308's development, one big branch focusing on this area was merged in 5a985d8. It fixed issues with renaming screenshots and installers, significantly improved our rename APIs and added the ability to rename .bak files on the Saves tab. We're not quite done with refactoring here, which is why #580 is not yet closed, but this will hopefully be the last time we have to deal with renaming issues (plus renaming is not really a good idea in many cases - e.g. renaming a mod will cause serious problems if you're using it on an existing save and power users can do it anyways by just opening the Data folder and editing the mod in there). ----------------------------------------------------------------------- Massive thanks to everyone who contributed to this release, including: @Infernio, @Utumno, @GandaG, @lojack5, @Gavvers, @LordNyriox, @Sharlikran, @Arthmoor and many more that GitHub's contribution tracker doesn't list.
PreviousNext