Skip to content
/ machine-learning-and-data-processing Public
forked from asavinov/machine-learning-and-data-processing

A collection of resources on machine learning, data processing and related areas

0 stars 7 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

learning-materials/machine-learning-and-data-processing

BranchesTags
 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

19 Commits
README.md
README.md
 
 

Repository files navigation

A collection of resources on machine learning, data processing and related areas

Analysis of different types of data

Time series and forecasting

Books and articles:

Implementations:

Stock price forecasting:

Spatial data

Audio and sound

General audio processing libraries:

Pitch trackers:

Other

Books:

Text NLP

Lists:

Video

Images

Graphs, RDFs etc.

Graph stores:

Databases:

Visualizations and dashboards:

  • graphviz

Data and knowledge engineering

Feature engineering

Lists:

Time series:

Feature extraction:

Feature selection:

Hyper-parameter optimization:

Feature stores

Links:

AutoML

AutoML libs:

  • AutoWeka
  • MLBox
  • auto-sklearn
  • TPOT
  • HpBandSter
  • AutoKeras

Systems and companies:

  • DataRobot
  • DarwinAI
  • H2O.ai Dreverless AI
  • OneClick.ai

AI, data mining, machine learning algorithms

Resources:

Algorithms:

  • XGBoost, CatBoost, LightGBM
  • https://github.com/spotify/annoy
    • Approximate Nearest Neighbors
    • USP: ability to use static files as indexes, share index across process, that is, in-memory and efficient and multi-process
    • e.g, music recommendation in Spotify, simlar images (for labeling etc.)

Data integration, ETL, job management, web scrapping

ETL:

Stream processing:

Web scrapping

Machine learning training, deployment etc.

Data labeling

Model management

TBD

Visualization and VA

Dashboards:

Publishing notebooks (from github etc.):

Asynchronous data processing

What is asynchronous data processing

TBD

Reactive programming

Approaches to asynchronous programming:

  • Callback model:

    • A callback function is provided as part of an asynchronous call
    • The call is non-blocking and the source program continues execution
    • It is not possible to await for the return (it is essentially done by the callback function)
    • The callback function can be viewed as a one-time listener for a return event, that is, it represents code which consumes the result
    • The source code where the call is made and the consumer of the result are in different functions and cannot share (local) context
    • The callback function may make its own asynchronous calls which leads to a "callback hell"

  • Future/promise:

    • An asynchronous call is made as usual but return a special wrapper object
    • A callback function is not specified and is not used
    • The returned result is consumed by the code which follows the call (as opposed to its use in a separate callback function)
    • The future/promise is supposed to be awaited. Awaiting denotes a point where we say that the next instruction needs the result
    • The awaiting point is like a one-time single-value listener for the result where the program execution is suspended until the return event is received

Links:

Reactive streaming

  • Listeners:

    • A callback function is registered and then automatically called for each incoming event
    • Callback functions are (normally) called only sequentially, that is, next event can be processed only when the previous event has been processed by the previous callback invocation. Callbacks are not executed concurrently.
    • The result of a callback invocation is frequently needed because it is executed by the event producer

  • Reactive streams:

    • It is a graph of producers and consumers
    • Consumers and producers are not supposed to communicate in a free manner by sending messages to each other where a sender knows the address(s) of the receivers
    • An element declares which messages it produces but it is unaware of who will subscribe to and consume its messages, and how they will be used (in contrast to the actor model)
    • An element must know what kind of messages it needs and explicitly subscribe to specific producers - messages will not come automatically just because somebody wants to send them to us
    • For data processing, reative streams provide a number of operators which can be applied to an input stream(s) and produce an output stream
    • Links:

  • Actor model:

    • Each element has an identifier (address, reference) which is used by other actors to send messages
    • A sender (producer) must know what it wants to do when it sends messages and it must know its destination
    • Receiving elements can receive messages from any other element and their task is to respond according to their logic (as expected by the sender)
    • Actors receive messages without subscribing to any producer (as opposed to the actor model where you will not receive anything until you subscribe to some event producer)
    • Each actor has a handler (callback) which is invoked for processing incoming messages
    • Actors are supposed to have a state and frequently it is why we want to define different actors
    • Links:

Event loops vs. threads

  • Both a thread task and an event loop task are executed until finished, that is, the code to execute is provided as a procedure
  • Thread tasks are dispatched by the system (not application) while dispatching logic of event tasks is part of the application
  • At each moment, there is a fixed number of threads concurrently executed by one process. The number of concurrently executed event loop tasks is not limited.
  • Thread tasks are (automatically) switched at the instruction level and the dispatcher is unaware of the needs of this thread or the application. Event loop tasks are switched at the level of logical application units depending on what this application needs.
  • In a multi-thread application, we need to manage the threads ourselves, e.g., by creating and deleting them. In an event loop application, the tasks (starting, suspending, finishing) is managed by the event loop manager.
  • In an event loop application, tasks specify dependencies on other tasks, and these points are used while dispatching the execution of tasks. Threads cannot declare dependencies on the results provided by other tasks. If we need some external result, then the thread has to wait. This logic has to be implemented manually and the system dispatcher is unaware of these dependencies.

Event loops:

Async networking libraries

  • (5.1k) https://github.com/gevent/gevent coroutine - based Python networking library. "systems like gevent use lightweight threads to offer performance comparable to asynchronous systems, but they do not actually make things asynchronous"

    • greenlet to provide a high-level synchronous API
      • on top of the libev or libuv event loop (like libevent)

  • (909) https://github.com/eventlet/eventlet concurrent networking library for Python

    • epoll or kqueue or libevent for highly scalable non-blocking I/O

  • (9.5k) https://github.com/aio-libs/aiohttp Asynchronous HTTP client/server framework for asyncio and Python

  • (3.8k) https://github.com/twisted/twisted Event-driven networking engine written in Python.

    • Twisted projects variously support TCP, UDP, SSL/TLS, IP multicast, Unix domain sockets, many protocols (including HTTP, XMPP, NNTP, IMAP, SSH, IRC, FTP, and others), and much more.
    • Twisted supports all major system event loops:
      • select (all platforms),
      • poll (most POSIX platforms),
      • epoll (Linux),
      • kqueue (FreeBSD, macOS),
      • IOCP (Windows),
      • various GUI event loops (GTK+2/3, Qt, wxWidgets)

Async web frameworks

Utilities

Retry libraries:

Libraries, utilities, tools

Python

Tools

  • Data structures:

    • https://github.com/mahmoud/boltons boltons should be builtins
    • https://github.com/pytoolz/toolz List processing tools and functional utilities (replaces itertools and functools)
    • Zict: Composable Mutable Mappings
    • HeapDict: a heap with decrease-key and increase-key operations
    • sortedcontainers: Python Sorted Container Types: SortedList, SortedDict, and SortedSet

  • Networking:

    • certifi: A carefully curated collection of Root Certificates for validating the trustworthiness of SSL certificates while verifying the identity of TLS hosts
    • urllib3: HTTP library with thread-safe connection pooling, file post support, sanity friendly, and more

Other:

  • click: creating beautiful command line interfaces
  • chardet

Formats, persistence and serialization

Authentication, Authorization, Security

Identity and Access Management

Secrets management, encryption as a service, and privileged access management. A secret is anything that you want to tightly control access to, such as API keys, passwords, certificates, and more.

Policy Enforcement Point, Identity And Access Proxy (IAP), Zero-Trust Network Architecture, i.e. a reverse proxy in front of your upstream API or web server that rejects unauthorized requests and forwards authorized ones to your server.

Other links:

Linux and OS

Platform and servers

Load balancing:

Dockerized automated https reverse proxy:

Discussions:

Service registry and orchestrator:

  • etcd
  • consul

Logging and tracing

Computing

Books:

Other resources

Data sources

Books

About

A collection of resources on machine learning, data processing and related areas

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

No packages published