Raw webrequest partitions for 2014-12-04T16/1H not marked successful
Closed, ResolvedPublic
Actions

Assigned To

Authored By

	QChris
	Dec 25 2014, 2:28 PM

Description

None of the webrequest partitions for 2014-12-04T16/1H have been marked successful.

What happened?

Related Objects
Search...

Status	Assigned	Task
Resolved	Ottomata	T72085 Raw webrequest partitions that were not marked successful
Resolved	Ottomata	T72087 Kafka partition leader elections causing a drop of a few log lines
Resolved	Ottomata	T71667 Kafka broker analytics1021 not receiving messages every now and then
Resolved	QChris	T85312 Raw webrequest partitions for 2014-12-04T16/1H not marked successful

Event Timeline

QChris created this task.Dec 25 2014, 2:28 PM

QChris raised the priority of this task from to Medium.

QChris updated the task description. (Show Details)

QChris added a project: Analytics-Clusters.

QChris added subscribers: Unknown Object (MLST), QChris, Milimetric and 2 others.

Analytics1021 lost it's partition leader role at that time.

Since this time, it affected all hosts us for a long time and produced many duplicates,
I'll dedupe the partitions.

Here are the stats before deduping:

Affected period: 2014-12-04T16:22:36--2014-12-04T16:35:10
Duplicates: 7478541
Missing: 4220371

Bits

Affected period: 2014-12-04T16:22:36--2014-12-04T16:35:10
Duplicates: 1826504
Missing: 1017094

Host	Start of issue	End of issue
cp1056.eqiad.wmnet	2014-12-04T16:22:39	2014-12-04T16:32:33
cp1057.eqiad.wmnet	2014-12-04T16:22:37	2014-12-04T16:32:37
cp1069.eqiad.wmnet	2014-12-04T16:22:39	2014-12-04T16:35:09
cp1070.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:51
cp3019.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:26:51
cp3020.esams.wikimedia.org	2014-12-04T16:22:36	2014-12-04T16:34:19
cp3021.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:35:10
cp3022.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:34:42
cp4001.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:32:21
cp4002.ulsfo.wmnet	2014-12-04T16:22:37	2014-12-04T16:33:51
cp4003.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:53
cp4004.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:32:35

Text

Affected period: 2014-12-04T16:22:37 --2014-12-04T16:35:10
Duplicates: 1777980
Missing: 1196547

Host	Start of issue	End of issue
amssq31.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq32.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:09
amssq34.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:09
amssq35.esams.wmnet	2014-12-04T16:22:38	2014-12-04T16:35:10
amssq36.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq37.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:26:51
amssq38.esams.wmnet	2014-12-04T16:22:38	2014-12-04T16:35:09
amssq39.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:09
amssq40.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:26:50
amssq41.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq42.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:26:51
amssq43.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq44.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq45.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:09
amssq46.esams.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:50
amssq47.esams.wmnet	2014-12-04T16:22:38	2014-12-04T16:35:10
amssq48.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq49.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq50.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:09
amssq51.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq52.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq53.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:35:10
amssq54.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq55.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq56.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:35:10
amssq57.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq58.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:10
amssq59.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:35:10
amssq60.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:35:09
amssq61.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:26:50
amssq62.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:26:51
cp1052.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:32:15
cp1053.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:32:06
cp1054.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:53
cp1055.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:54
cp1065.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:51
cp1066.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:53
cp1067.eqiad.wmnet	2014-12-04T16:22:39	2014-12-04T16:26:51
cp1068.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:52
cp4008.ulsfo.wmnet	2014-12-04T16:22:39	2014-12-04T16:32:24
cp4009.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:35:10
cp4010.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:54
cp4016.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:32:53
cp4017.ulsfo.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:09
cp4018.ulsfo.wmnet	2014-12-04T16:22:37	2014-12-04T16:26:54

Mobile

Affected period: 2014-12-04T16:22:37--2014-12-04T16:35:09
Duplicates: 164165
Missing: 193326

Host	Start of issue	End of issue
cp1046.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:51
cp1047.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:51
cp1059.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:32:14
cp1060.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:52
cp3011.esams.wikimedia.org	2014-12-04T16:22:39	2014-12-04T16:34:02
cp3012.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:32:54
cp3013.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:26:52
cp3014.esams.wmnet	2014-12-04T16:22:38	2014-12-04T16:35:09
cp4011.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:33:17
cp4012.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:31:37
cp4019.ulsfo.wmnet	2014-12-04T16:22:37	2014-12-04T16:34:00
cp4020.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:53

Upload

Host	Start of issue	End of issue
cp1048.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:52
cp1049.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:35:08
cp1050.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:52
cp1051.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:34:25
cp1061.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:35:08
cp1062.eqiad.wmnet	2014-12-04T16:22:39	2014-12-04T16:34:33
cp1063.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:26:34
cp1064.eqiad.wmnet	2014-12-04T16:22:38	2014-12-04T16:32:42
cp3003.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:35:09
cp3004.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:09
cp3005.esams.wikimedia.org	2014-12-04T16:22:36	2014-12-04T16:35:09
cp3006.esams.wikimedia.org	2014-12-04T16:22:38	2014-12-04T16:35:10
cp3007.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:09
cp3008.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:26:51
cp3009.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:09
cp3010.esams.wikimedia.org	2014-12-04T16:22:37	2014-12-04T16:35:10
cp3015.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:09
cp3016.esams.wmnet	2014-12-04T16:22:36	2014-12-04T16:35:09
cp3017.esams.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:09
cp3018.esams.wmnet	2014-12-04T16:22:36	2014-12-04T16:26:52
cp4005.ulsfo.wmnet	2014-12-04T16:22:37	2014-12-04T16:34:46
cp4006.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:35:10
cp4007.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:35:09
cp4013.ulsfo.wmnet	2014-12-04T16:22:38	2014-12-04T16:34:38
cp4014.ulsfo.wmnet	2014-12-04T16:22:37	2014-12-04T16:32:34
cp4015.ulsfo.wmnet	2014-12-04T16:22:37	2014-12-04T16:35:06

Affected period: 2014-12-04T16:22:36--2014-12-04T16:35:10
Duplicates: 3709892
Missing: 1813404

After deduplication, the affected period went down from ~13 minutes to the ~5 minutes of 2014-12-04T16:22:36--2014-12-04T16:26:55
No more duplicates.
But of course still the 4220371 missing log lines (worth ~30 seconds of total traffic).

QChris closed this task as Resolved.Dec 25 2014, 10:45 PM

QChris claimed this task.

QChris added a subtask: T72087: Kafka partition leader elections causing a drop of a few log lines.

QChris removed a subtask: T72087: Kafka partition leader elections causing a drop of a few log lines.

Raw webrequest partitions for 2014-12-04T16/1H not marked successfulClosed, ResolvedPublicActions

Description

Related ObjectsSearch...

Event Timeline

Raw webrequest partitions for 2014-12-04T16/1H not marked successful
Closed, ResolvedPublic
Actions

Related Objects
Search...