Well well, followup:

FYI, due to importing RT tickets into Phabricator on 2014-12-17 and due to some tickets originally created either in Bugzilla or in RT being access restricted, there is still a number of non-open tickets filed before 2014-12-17 without a closedEpoch value in the DB.
However the vast majority got fixed today (which might not necessarily fix any statistics if they do not rely on querying the closedEpoch value but e.g. on transaction log values or such).

Done. Thanks everyone!

Database fiddler hero welcome here with production access. Unassigning since I'm not.

The performance impact would be the same since the query plan uses the primary key. It would be just more verbose.

@Pppery at the moment the export followed T107254#9028797 so I'm quite sure we are already covered.

In T107254#9778251, @Pppery wrote:

Make sure you handle the situation where a ticket is closed on Bugzilla, and then later reopened on Phabricator (and thus is open now). The result is that its closedDate would be null and should not be set.

Make sure you handle the situation where a ticket is closed on Bugzilla, and then later reopened on Phabricator (and thus is open now). The result is that its closedDate would be null and should not be set.

To any RelEng hero:

LGTM from a quick glance

Oh, thanks Dzahn :3

Ignore my last comment. We can obtain that information using Arcanist and Conduit APIs 🌈 🦄

Can somebody please run this SQL command in production?

In T107254#9765585, @Pppery wrote:

appears to be truncated.

And there's no reason to ignore mike.lifeguard - he was a valid bugzilla contributor, albeit now inactive.

Actually it looks like this bug was already reported and fixed above and I was following aklapper's quote of an old version of the SQL. But even the most recent version of the SQL (https://gitlab.wikimedia.org/valeriobozzolan/yet-another-bugzilla-parser/-/blob/master/data/T107254-migration-commands.sql) appears to be truncated.

Bug 1189/T3189 was open at the time Bugzilla was migrated to Phabricator. It was closed years later. There's nothing to do here.

Or, the scaper can surely ignore edits done by mike.lifeguard+bugs :D ihih

In T107254#9764885, @Pppery wrote:

UPDATE phabricator_maniphest.maniphest_task SET closedEpoch = 1247766128 WHERE closedEpoch IS NULL AND id = 3189; -- in date 2009-07-16 17:42:08 UTC BugZilla ID 1189 closed by user mike.lifeguard+bugs with status REOPENED

This doesn't make sense - a ticket can't be closed with status REOPENED.

This doesn't make sense - a ticket can't be closed with status REOPENED.

There are a few entries that say:

UPDATE phabricator_maniphest.maniphest_task SET closedEpoch = 1247766128 WHERE closedEpoch IS NULL AND id = 3189; -- in date 2009-07-16 17:42:08 UTC BugZilla ID 1189 closed by user mike.lifeguard+bugs with status REOPENED

This doesn't make sense - a ticket can't be closed with status REOPENED.

In T107254#9710354, @valerio.bozzolan wrote:

https://gitlab.wikimedia.org/valeriobozzolan/yet-another-bugzilla-parser/-/blob/5ef4c7d2f640f856f581a53fa35fde5c354f7f0b/data/T107254-migration-commands.sql

In T107254#9710082, @hashar wrote:

As for CLOSED, I guess we did that to prevent further comments and/or to indicate the task resolution got verified. Then.. I cant quite remember how we used Bugzilla 15/20 years ago :D

In T107254#9710082, @hashar wrote:

I cant quite remember how we used Bugzilla 15/20 years ago

In T107254#9710628, @Krinkle wrote:

@valerio.bozzolan The SQL statements and comments appear in contradiction. For every comment that says "Skipped bug X - state was REOPENED" there is in fact a real update statement that does exactly what the comment says it doesn't. Is this intentional?

@valerio.bozzolan The SQL statements and comments appear in contradiction. For every comment that says "Skipped bug X - state was REOPENED" there is in fact a real update statement that does exactly what the comment says it doesn't. Is this intentional?

Here our generated SQL patch candidate ready for review:

Note: we have dump of Bugzilla data at https://dumps.wikimedia.org/other/bugzilla/ , as static html file and a database dump (without emails)

And... the script now takes seconds instead of weeks. Now we have enough extra Watts to shutdown a small nuclear plant somewhere 🌈

In T107254#9706654, @Aklapper wrote:

And if it would work on a static HTML dump like ours, instead of a real BZ instance.

In T107254#9706866, @valerio.bozzolan wrote:

Have anybody already checked the file mentioned in the description, that is, maybe an useful starting point?
https://gitlab.wikimedia.org/valeriobozzolan/yet-another-bugzilla-parser/-/blob/master/data/bugzilla.csv

Anyway a required step is to take P49618 and make a nice phabricator.csv.

Have anybody already checked the file mentioned in the description, that is, maybe an useful starting point?

@valerio.bozzolan I admit I didn't try to fully understand those 400 lines which seem to cover way way more stuff than needed here. :D Like why there is an array with statuses when we only care about last Status=Resolved etc. Or why care about ACTION_WITH_LINKS etc. (Also, stuff like WORKSFORME and REMIND and LATER and --- are no statuses but resolutions.) And if it would work on a static HTML dump like ours, instead of a real BZ instance.

Small note

Had another quick look and managed to get down to have only <table>...</table> left in the local bug activity page. But then interesting problem is dealing with the rowspan: No date cell in the Status change row when several actions were performed at once and 'Status change' is not listed as first row)

In T107254#9702401, @Dzahn wrote:

git clone https://gitlab.wikimedia.org/repos/sre/miscweb/bugzilla.git

git clone https://gitlab.wikimedia.org/repos/sre/miscweb/bugzilla.git

Note: we have dump of Bugzilla data at https://dumps.wikimedia.org/other/bugzilla/ , as static html file and a database dump (without emails)

To summarize: Would need to perform 57008 times the following steps:

• take the $PhabID
• subtract 2000 from the $PhabID to get the $BzID
• pull https://static-bugzilla.wikimedia.org/show_activity.cgi?id=$BzID (e.g. $data = file_get_contents('https://fooo');)
• scrape the last (!!) line where What==Statusand Added==RESOLVED to get the When value from that line - maybe https://nyamsprod.com/blog/extracting-data-from-html-table-in-php/ or such could be handy, because need to also handle HTML table rowspan etc
• convert the When value to epoch to get $epoch (e.g. echo strtotime("2014-11-21 00:15:49 UTC");)
• UPDATE phabricator_maniphest.maniphest_task SET closedEpoch = $epoch WHERE closedEpoch IS NULL AND id = $PhabID;

I obviously do not care who set the closed status (we cannot match non-existing accounts!), and I obviously do not care about creating fake transactions in the DB, but would only set that one closedEpoch column value, if at all.

Got it. In this case I'll just untag Collab.

This is not resolved; it would require altering our DB

Closing based on the last comment.

I sincerely do not remember what I was doing here :D will retry to do something during the next WMHack. I will continue from this point:

Let's join this lovely pastebin with our scraped stuff