-
-
Notifications
You must be signed in to change notification settings - Fork 831
Event logging client for (Modern) Event Platform #3648
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from 12 commits
Commits
Show all changes
56 commits
Select commit
Hold shift + click to select a range
66c7de5
Event logging client for (Modern) Event Platform
bearloga ab60d39
add EPCStorageManager + tests
tonisevener c3350cb
initial work on EPCNetworkManager
tonisevener b351b0f
finish NetworkManager, add tests
tonisevener 6ad7cdf
Remove stream cc-ing feature from Event Platform Client
bearloga 96a554b
finished integration + a couple of sample calls
tonisevener d57ab1d
Update EPC for actual stream configs format
bearloga 0065eac
some renames and code cleanup
tonisevener 5dd634d
Merge branch 'event-platform-client' into epc-integration
tonisevener 08b799e
undo warning fix
tonisevener 0f496a2
add some test logging
tonisevener 8d8eea8
Merge branch 'main' into event-platform-client
tonisevener 431870f
More documentation for EPC & helpers, fix stream config processing
bearloga 47dae16
Event Platform Client: sync ID generation algo to MediaWiki's
bearloga 40cfa5f
Event Platform Client: some more polish
bearloga 6799de1
Merge branch 'main' into event-platform-client
joewalsh 8d18452
Fix tests, deviceID was renamed to installID
joewalsh 079d807
T260382: Edit History Compare instrumentation with EPC
bearloga 267984a
EditHistoryCompareFunnel: remove extraneous EPC log call
bearloga 675e056
[Minor] EditHistoryCompareFunnel: remove unnecessary comparison
bearloga a8d78c0
EPC: clarification about domain, absoluteString => host
bearloga a563d4c
Merge branch 'main' into event-platform-client
joewalsh 0699c3e
MEP test events update
bearloga 9be85f1
EPC: Begin stashing storage mng facilities for later
bearloga 5553d47
Rest of EPCStorageManager clean-up
bearloga 70d420c
EPC: More cleaning up/finalizing
bearloga 739336c
EPC: Consolidation of networking into main library
bearloga e1a33e1
Update EventPlatformClientLibrary.swift
bearloga 1efb428
Store body data in the buffer instead of an NSDictionary
joewalsh a3b74cb
codable EPC WIP
joewalsh d5e811c
Codable events
joewalsh 8e46326
Events should conform to EventInterface
joewalsh 2c0d4bc
Use the value from popFirst instead of checking isEmpty
joewalsh 8afbb4c
Encode data on a background queue
joewalsh 41c36e1
Use Codable for stream configurations
joewalsh 8133bce
Fix missing completion call in doPeriodicWork
joewalsh aa95f15
Use stream instead of String as the key
joewalsh ea52390
Remove Dictionary+JSON
joewalsh c5f057a
Merge branch 'main' into event-platform-client
joewalsh 06a842e
EPC: Final v1 clean-up, switch to production stream config endpoint
bearloga b1f831f
Merge branch 'event-platform-client' of github.com:wikimedia/wikipedi…
bearloga 9635b93
Merge branch 'event-platform-client' of github.com:wikimedia/wikipedi…
joewalsh 83ddec5
Remove test streams and schema
joewalsh c9a586b
Restore & update documentation, remove `data` property from the paylo…
joewalsh a68097f
Match prior log message
joewalsh bc63eec
Move `schema` to a static requirement on `EventInterface` so it's def…
joewalsh ceac06b
Log errros from EPC POSTs
joewalsh f336aab
Fix functionality added by bodyData parameter
joewalsh 479c577
Restore central definition of schemas, keep per-Event static schema d…
joewalsh 5c6e5ee
Add error handling to catch when the service rejects an event for not…
joewalsh a1eb285
Fix append - use event data instead of stream data
joewalsh 991940b
Use concrete EventRequest type for event buffers
joewalsh 6c86ad9
Remove old TODO from EditHistoryCompareFunnel
bearloga 0299b52
Merge pull request #3682 from wikimedia/codable_epc
joewalsh 20cf1be
Merge branch 'main' into event-platform-client
joewalsh 62a9eb3
Remove client-facing date param, record the date synchronously
joewalsh File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
140 changes: 140 additions & 0 deletions
140
WMF Framework/Event Platform Client/EPCNetworkManager.swift
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,140 @@ | ||
|
|
||
| import Foundation | ||
|
|
||
| class EPCNetworkManager: EPCNetworkManaging { | ||
|
|
||
| private let storageManager: EPCStorageManaging | ||
| private let session: Session | ||
| private let operationQueue: OperationQueue | ||
|
|
||
| init(storageManager: EPCStorageManaging, session: Session = Session.shared) { | ||
| self.storageManager = storageManager | ||
| self.session = session | ||
|
|
||
| operationQueue = OperationQueue() | ||
| operationQueue.maxConcurrentOperationCount = 1 | ||
| } | ||
|
|
||
| func httpPost(url: URL, body: NSDictionary) { | ||
| storageManager.createAndSavePost(with: url, body: body) | ||
| } | ||
|
|
||
| func httpDownload(url: URL, completion: @escaping (Data?) -> Void) { | ||
bearloga marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| httpDownload(url: url, attempt: 0, maxAttempts: 5, attemptDelay: 2, completion: completion) | ||
bearloga marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
| } | ||
|
|
||
| private func httpDownload(url: URL, attempt: Int, maxAttempts: Int, attemptDelay: TimeInterval, completion: @escaping (Data?) -> Void) { | ||
|
|
||
| guard attempt < maxAttempts else { | ||
| completion(nil) | ||
| return | ||
| } | ||
|
|
||
| let task = session.dataTask(with: url) { (data, response, error) in | ||
|
|
||
| let failureBlock = { | ||
| dispatchOnMainQueueAfterDelayInSeconds(attemptDelay) { | ||
| self.httpDownload(url: url, attempt: attempt + 1, maxAttempts: maxAttempts, attemptDelay: attemptDelay, completion: completion) | ||
| } | ||
| } | ||
bearloga marked this conversation as resolved.
Outdated
Show resolved
Hide resolved
|
||
|
|
||
| guard let httpResponse = response as? HTTPURLResponse, | ||
| let data = data else { | ||
| failureBlock() | ||
| return | ||
| } | ||
|
|
||
| guard httpResponse.statusCode == 200 else { | ||
| failureBlock() | ||
| return | ||
| } | ||
|
|
||
| guard error == nil else { | ||
| failureBlock() | ||
| return | ||
| } | ||
|
|
||
| completion(data) | ||
| } | ||
| task?.resume() | ||
| } | ||
|
|
||
| func httpTryPost(_ completion: (() -> Void)? = nil) { | ||
| let operation = AsyncBlockOperation { (operation) in | ||
|
|
||
| self.storageManager.deleteStalePosts() | ||
| let postItems = self.storageManager.fetchPostsForPosting() | ||
|
|
||
| self.postItems(postItems) { | ||
| operation.finish() | ||
| } | ||
| } | ||
|
|
||
| operationQueue.addOperation(operation) | ||
| guard let completion = completion else { | ||
| return | ||
| } | ||
| let completionBlockOp = BlockOperation(block: completion) | ||
| completionBlockOp.addDependency(operation) | ||
| operationQueue.addOperation(completion) | ||
| } | ||
|
|
||
| private func postItems(_ items: [EPCPost], completion: @escaping () -> Void) { | ||
|
|
||
| let taskGroup = WMFTaskGroup() | ||
|
|
||
| var completedIDs = Set<NSManagedObjectID>() | ||
| var failedIDs = Set<NSManagedObjectID>() | ||
|
|
||
| for item in items { | ||
| let moid = item.objectID | ||
| guard let urlAndBody = storageManager.urlAndBodyOfPost(item) else { | ||
| failedIDs.insert(moid) | ||
| continue | ||
| } | ||
| taskGroup.enter() | ||
| let userAgent = item.userAgent ?? WikipediaAppUtils.versionedUserAgent() | ||
| submit(url: urlAndBody.url, payload: urlAndBody.body, userAgent: userAgent) { (error) in | ||
| if let error = error { | ||
| if error != .network { | ||
| failedIDs.insert(moid) | ||
| } | ||
| } else { | ||
| completedIDs.insert(moid) | ||
| } | ||
| taskGroup.leave() | ||
| } | ||
| } | ||
|
|
||
| taskGroup.waitInBackground { | ||
| if (completedIDs.count == items.count) { | ||
| DDLogDebug("EPCNetworkManager: All records succeeded") | ||
| } else { | ||
| DDLogDebug("EPCNetworkManager: Some records failed") | ||
| } | ||
| self.storageManager.updatePosts(completedIDs: completedIDs, failedIDs: failedIDs) | ||
| completion() | ||
| } | ||
| } | ||
|
|
||
| private func submit(url: URL, payload: NSDictionary, userAgent: String, completion: @escaping (EventLoggingError?) -> Void) { | ||
|
|
||
| var request = session.request(with: url, method: .post, bodyParameters: payload, bodyEncoding: .json) | ||
| request.setValue(userAgent, forHTTPHeaderField: "User-Agent") | ||
| let task = session.dataTask(with: request, completionHandler: { (_, response, error) in | ||
| guard error == nil, | ||
| let httpResponse = response as? HTTPURLResponse, | ||
| httpResponse.statusCode / 100 == 2 else { | ||
| if let error = error as NSError?, error.domain == NSURLErrorDomain { | ||
| completion(EventLoggingError.network) | ||
| } else { | ||
| completion(EventLoggingError.generic) | ||
| } | ||
| return | ||
| } | ||
| completion(nil) | ||
| }) | ||
| task?.resume() | ||
| } | ||
|
|
||
| } | ||
8 changes: 8 additions & 0 deletions
8
WMF Framework/Event Platform Client/EPCPost+CoreDataClass.swift
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
|
|
||
| import Foundation | ||
| import CoreData | ||
|
|
||
| @objc(EPCPost) | ||
| public class EPCPost: NSManagedObject { | ||
|
|
||
| } |
19 changes: 19 additions & 0 deletions
19
WMF Framework/Event Platform Client/EPCPost+CoreDataProperties.swift
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,19 @@ | ||
|
|
||
| import Foundation | ||
| import CoreData | ||
|
|
||
|
|
||
| extension EPCPost { | ||
|
|
||
| @nonobjc public class func fetchRequest() -> NSFetchRequest<EPCPost> { | ||
| return NSFetchRequest<EPCPost>(entityName: "EPCPost") | ||
| } | ||
|
|
||
| @NSManaged public var body: NSObject? | ||
| @NSManaged public var recorded: Date? | ||
| @NSManaged public var posted: Date? | ||
| @NSManaged public var failed: Bool | ||
| @NSManaged public var url: URL? | ||
| @NSManaged public var userAgent: String? | ||
|
|
||
| } |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.