Download to read offline
Uploaded byQt
Machine learning meets embedded development
The document discusses the integration of machine learning with embedded development through the collaboration of The Qt Company and Ekkono Solutions, focusing on optimizing model deployment in various applications. It addresses challenges like scalability, cost reduction, and the need for an integrated toolchain to facilitate the development of smart, context-aware IoT devices. Ekkono's approach emphasizes edge machine learning capabilities, empowering devices to learn and operate autonomously while reducing reliance on cloud connectivity.
More Related Content
Similar to Machine learning meets embedded development
Machine learning meets embedded development
- 1. Machine Learning meets Embedded Development TheQt Company Ekkono 09 June 2021
- 2. Machine Learning meetsEmbedded Development 15 June 2021 © The Qt Company 2 SPEAKERS MICHELE ROSSI Business Development, Qt Venture The Qt Company AMIT NAINAWAT Pre-Sales Engineer The Qt Company SIMON HEDSTRÖM Machine Learning Engineer Ekkono Solutions
- 3. A growing complexityin the embedded development market MACHINE LEARNING MEETS EMBEDDED DEVELOPMENT UI / UX Designer Machine Learning / Data Scientist Expert Developers Product managers / Management Qt & Ekkono 1. Faster GoToMarket: • as a company I want to validate and integrate the optimal model faster in my Qt application 2. Scalability: • As a company I want to have the freedom to port and reuse my application output across all different use cases I have 3. CAPEX & OPEX costs: • Investing in a DIY toolchain is a huge investment in terms of initial effor and maintenance burden • As a company I want to reduce as much as possible my Bill of Material, and limit the interaction between a device and cloud platform Industry Pain Points
- 4. A unified toolchainto create and maintain your application Qt Design Studio Qt Creator QML and Model UI Asset Desktop MCU WEB MPU Mobile Deploy DESIGN – MODEL – TEST - DEPLOY ML Model Test
- 5. Closing the gapbetween data scientists, and embedded stakeholders 15 June 2021 5 THE PROBLEM WE ARE LOOKING TO SOLVE How we help companies: • To deploy and integrate ML application in productions: we support you on optimizing the model you need • The IP is yours; we enable you to deploy your knowledge in your application • Deployment cross platform: desktop, mobile, edge • By using incremental learning, we enable Offline training - no need to use cloud infrastructure to train models, saving connectivity costs • Reducing the Bill of Material implementing the concept of Virtual Sensors Desktop MCU WEB MPU Mobile
- 6. How to tryQt & Ekkono in your commercial application 15 June 2021 6 BOOKING AN APPOINTMENT WITH US Available for commercial trials or commercial customers only • Reaching out your local Qt contact • Reaching out your local Ekkono contact • New to Qt? Write to us: https://www.qt.io/contact-us/sales-contact-request
- 7. FROM CONNECTED TOSMART
- 8. who does EdgeMachine Learning Ekkono is a Swedish Software Company
- 9. Ekkono’s Edge Machine Learning Exactly,we domachine learningandnotjustinference attheedge,whichmeansthatwecan learnindividualuse andsuper-localconditions.We evendothisonreallysmalldeviceswithsensorsize microcontrollerunits. Sensor layer Embedded layer Communication & Control Layer
- 10. A comprehensive toolbox tosupport implementation and integration Supports ML techniques, including decision trees, random forest, and neural networks Limited data science experience required to deploy advanced edge machine learning Incremental learning at the edge, onboard devices Learning on streaming sensor data Supports execution of pre-trained models Software Development Kit Ekkono’sSDKisacomprehensivetoolboxtosupportimplementationand integration,100%softwareand totallyplatformagnostic.ThecoreisaC++ library. TheproductisdesignedfordevelopersandtheAPI offersbindingstoC#and Python.
- 11. ProductOffering CloudhostedMLworkbenchforrunningEdge,Crystal,Primer • CreatesolutionswithdraganddropcodeSnippets • Evaluation,FeatureSelection,AutoML,plottingetc. •ExportmodelstoabinaryformattoloadthemfromC++ EkkonoStudio • A libraryforEdgeML,compiledtothetargetplatform • Edge -Trainingandinference,Req.C++14compatiblecompiler • Crystal-SubsetofEdge features(Req.C99) • Primer- AutoML,FeatureselectionandDataWrangling (C++14,normallynotrunontheedgedevices) • Extensivetutorialsanddocumentation EkkonoSDK Ekkono Studio
- 12. Howit Works Product withSensors Edge Machine Learning Predictive, Self-Configuring and Context-Aware
- 13. Added Smartness Ekkono meanscognition,andthatis whatwe add to IoT.We make connectedthingssmart by embeddingadvancededgemachine learning– thatruns onboardtheconnecteddevice. This empowersIoTto realizeitstrue potential,where companiessaveandmake moneythrough predictivemaintenance,automation,performance optimization,self-configuration,intuitive products,andnew data-drivenbusinessmodels. University Research The result of seven years of research at the University of Borås, Sweden. A lightweight machine learning engine that can run on small hardware platforms, close to the data source, i.e. the sensors, on the device, where it can see and process all data, in real-time, and take instant actions. This reduces network load, make things less dependent on connectivity and improves data integrity. Self-Learning Devices This enables individual learning per device. A vehicle learns the climate and traffic in which it operates, a machine learns its surroundings, and a robot mower learns your specific garden. This opens up for a lot of new features, and it makes Ekkono’s edge machine learning a powerful complement to your cloud solution, as it feeds good, enriched, individualized and relevant data. Smart & Sustainable IoT holds the promise of everything becoming smart – machines, vehicles, cities and devices. Reality is that most of them are still just connected. Smartness is capped at uploading raw data to a big-data haystack and showing historical averages for the entire installed base. With Ekkono you can deliver on IoT’s promise of making things genuinely smart. Ekkono’s Uniqueness
- 14. Sensor virtualization o Costsaving on sensors that are expensive or difficult to install. Condition monitoring o Cost savings on scheduled check-ups and spot checking. Predictive alarming o Predict failures ahead of time. Predictive maintenance o Cost savings on time spent compared to other types of maintenance, e.g. calendar-based maintenance. o Maximizing the useful life of components and equipment. o Less unplanned stops. Device modeling o Create your digital twin of device or component. o Summarization of device use patterns. Predictive control o Cost saving through better utilization of resources. o Customer retainment through the transferable and personalized configurations. o Performance optimization and autotuning. Use Case Examples Anomaly detection Change detection Efficiency estimation Dynamic thresholding Predicted exceeding of threshold Usage analytics Condition-based maintenance Maintenance demand classification Remaining useful life Model predictive control Controller imitation learning Sensor replacement Sensor imitation Sensor forecasting Scenario simulation Remaining range S en s o r v irtu alizatio n C o n ditio nm o n ito rin g P redic tiv e a larm in g P redic tiv e m a in ten a n c e D ev ic e m o delin g P redic tiv e c o n tro l Increased demand on domain expertise for formulating and solving the machine learning problem Variations of the general use cases Typical use cases enabled by Ekkono Value created when using Ekkono’s software
- 15. Ekkono and Qt DataDrivenDecisionSupportSystemsrequiresaccuratemodelsanduserinteintuitiverfaces tobetrustedandacceptedby usersandthisiswhatEkkonoandQtbringtothefingertips ofembeddeddevelopersin anaccessibleformat.
- 16. Ekkono & QtUse Cases Anomaly detection Change detection Efficiency estimation Dynamic thresholding Predicted exceeding of threshold Usage analytics Condition-based maintenance Maintenance demand classification Remaining useful life Model predictive control Controller imitation learning Sensor replacement Sensor imitation Sensor forecasting Scenario simulation Remaining range S en s o r v irtu alizatio n C o n ditio nm o n ito rin g P redic tiv e a larm in g P redic tiv e m a in ten a n c e D ev ic e m o delin g P redic tiv e c o n tro l Variations of the general use cases Data Driven Decision Support Systems requires accurate models and intuitive user interfaces to be trusted and accepted by users and this is the sweet spot for combining Ekkono & Qt. Machine learning models can be created, tested and validated in Ekkono Studio and then loaded, hooked up to sensor and the user interface in Qt Design Studio. All but the four use cases in the two top levels of the stair to the right are related to decision support and could be managed within Qt Design Studio. More complex use cases can be done by adding additional C++ logic in the backend.
- 17. Integrating ML intoQt workflows
- 18. Software Architecture Front end Back end Ekkono’s librarysits in the back end – processing sensor data, training and running machine learning models. Before deployment, the models are created, validated and tested in Ekkono Studio. Qt’s framework facilitates creation of the embedded application leveraging Ekkono – Qt Design Studio enables creation of intuitive user interfaces and connecting it the model which in turn can be hooked up to sensors. Through Qt Add-on libraries sensors and remote devices is also easily be easily handled. Ekkono Studio
- 19. Design Studio Integration Qt hasa plug-in that allows you to load Ekkono models into Qt Design Studio. The models can be hooked up to sensors in the graphical interface. Ekkono also supports incremental training of models instead of just model inference. This applies to both supervised learning regression models as well as unsupervised on device learning models. Supported Ekkono Models • Linear regression • Multilayer perceptron • Random forest • Decision trees • Ekkono change detector* • Ekkono anomaly detector* *unsupervised learning algorithms
- 20. Use Case Example Anomaly Detection Ekkono hasan algorithm for multivariate anomaly detection. It calibrates on a given window size and then starts reporting an anomaly score between 0 (nothing anomalous) and 1 (very anomalous). model = edge.ModelFactory.create_anomaly_detector(data_pipeline_template, calibration_window_size = 250) anomaly_score = model.score(instance) #calibration is done automatically with the score-call Each time the sensors are read from Calibration on first 250 observations
- 21. Why Ekkono +Qt? Ifyouarealready working with Ekkono Qt can providetheplatform thatis readyto catch the outputfrom a solution with Ekkono. It can visualizeit,feedit back intothe systemasa parameter,or communicateas a connectedservice. Ifyou arealready working with Qt If you arealreadyusingQt andarelooking toenablemachinelearning capabilitiesonyourplatform thenconsiderthe collaborationwith Ekkono. Youcan scaledown tomicrocontrollers.With a singleadd-onyoucan communicatewith thestate-of-the-artedge machinelearninglibrary.On top of it youcanbuildapplicationsthataddsmartfeaturesto yourproduct.Qt + Ekkono reducesthenumberof stepsinvolvedandallowsyoutoiterateon conceptsfaster. Ifyou areworking with neither Ekkono andQt offerthe smoothestmachine learningto front-endintegration onthemarket. Seamlessdeploymentwith littleeffort.Ekkono isbuilt to be embeddedandhasnoexternaldependencies,Qt providesthe full eco system thatfully supportsinteractionswithEkkono.
- 22.
- 23. Design Development UI specificationProduct implementation Deployment Deployand test Design Studio Value proposition Traditional UI development workflow
- 24. Design Development Deployment DesignStudio Value proposition Designer & Developer tools
- 25. Design Development Deployment DesignStudio Value proposition Enhanced UI development workflow Deployand test Prototype on real device
- 26. Design Interaction Designer Developer DesignStudio Value proposition Enhanced UI development with ML workflow ML Modeling Deploy and test Prototype on real device Machine learning Engineer
- 27. 27 UI Design UIPrototyping + ML integration Project Development Design Studio Value proposition Qt designer – ML - Developer demonstration ML Modeling
- 28. How to tryQt & Ekkono in your commercial application 15 June 2021 28 BOOKING AN APPOINTMENT WITH US Available for commercial trials or commercial customers only • Reaching out your local Qt contact • Reaching out your local Ekkono contact • New to Qt? Write to us: https://www.qt.io/contact-us/sales-contact-request
- 29. Q&A The future iswritten with Qt www.qt.io
Editor's Notes
- #2 Target audience Knows nothing or little about Qt Technical and/or C-level, we should accommodate both Decision makers Intention of presentation Overall: Pitch the Qt values from different perspectives to different target groups How: High-level explanation of what Qt is Challenges and how Qt can solve these Show how the design-develop-deploy workflow is supported by Qt Compare with competition and point at Qt’s strengths Customer success stories
- #5 Design tool bridges: Import your UI designs from Photoshop and Sketch to 2D Qt QML Scene editors: Fine-tune your designs to pixel-perfection Side-by-side visual and code editor: Modify your designs visually or with QML - Qt's easy to use declarative language 3D editor: is now much improved from 1.4. Visual flow editor: Preview of what we’re working on, demo to follow later on. Basically allows fast prototyping inside the Qt Design Studio. Dynamic layouts: Adapt your UI to any screen Components: Qt turns your assets into QML components that can be reused in different projects. Qt has also Ready-made and customizable buttons, switches, dials Timeline-based animations: With fully customizable easing curves makes breathing life into your designs with animations simple Built-in and customizable visual effects: Fancy up your graphic designs Live on-device UI previews: See how your changes affect the UI directly on your target device Now I am going to show you the Qt toolchain from design to deploy, it is called designer developer workflow. Qt have developed a plugins for Abobe photoshop and Apple Sketch, this plugin is called Qt bridge. By using Qt bridge an Artwork from photoshop or from Sketch can be imported into the Qt design studio, during the import process Qt design studio generates the reusable QML code. Later designers can work on this auto generated QML code and finally add the user experience; graphical part is validated by the designer as they visualize the final GUI in Qt design studio without deploying on the real HW target. Once visual validation of UI/UX is done by UX engineers, code is passed to the developer. Developer uses the Qt Creator where they 0work on the backend connection, write C++ logic and integration code for the GUI. Qt Creator is tool for the software development life cycle, ie you can design, develop, debug, optimize and finally deploy your application on target HW like desktop, MCU, embedded targets, etc..
- #24 Hello again, First, I would like to discuss the usual design development workflow. It's not the designer who builds the design - it's the developer Designer shares graphic assets and images - as photo or background The job of the designer is to communicate the specs, screen and animation to give a more precise idea of what is needed But the developer builds the user interface from scratch by checking the graphical assets provided by the designer. Until development is not finished output of application on the final device is not know. Validation of design is done at very later stage… Each new change in design will add workload on the developers…. Late validation of design in development cycle, responsibility of design validation is shifted to developers are real issues ….
- #25 These issues are due to limitation of tools Because tools are not common between designer and developers. Except some assets, the developer can’t reuse the work of the designer Much of this design works goes the trash It's a very big waste of time This makes it more difficult to achieve satisfactory graphics quality It is a source of tension between the Design and development teams
- #26 Qt is already solving these pain points by the enhanced UI development work flow. 1 - QT creates a QT bridge on the four main design tools: Photoshop, Figna, Sketch (IOS) and soon Xd - These Bridges allow to export the work of the Designer on DS 2 – Import is "pixel perfect" –And DS allows now designer to add a large proposal of effects and animation 3 The difference between us and other tools is imports and animation are interpreted in QML language - it is a descriptive language both usable by the designer and the developer 4 - No need to wait for the full deployment of the application on the device - the preview allows the designer to verify and validate Once we enabled the designer developer to work in enhanced UI workflow. We realized there are more to do in this workflow. One part we were exploring with our partner Ekkno is how to bring machine learning modules and integrate machine learning in UI development flow.
- #27 Thanks for Ekkono integration in Qt Design studio and Qt creator now both designer and developer can also work in the machine learning modules. Goal is to easy the integration of a machine learning module, which is created by the ekkon studio. Machine learning engineer works on the Ekkano studio and generated the target machine learning module. That module can then easily integrate UI application, connect input and out to the module.. Is done in design studio by dragging and dropping as any other graphical widget. Once input and oputput connection are made, DS enable interaction engineers to test the UI with machine learning module. Once test on UI and machine learning modules are done project can be passed to developers. Developer can connect the machine learning module now to the real sensors data and deploy the I with machine learning module.
- #28 I will present an application : UI design and machine learing algorithm integration. I will use a photoshop artwork for UI design, that artwrok will be imported to Qt deisgn studio for UI deisgn creation. I will use the electrical motor module generated by my dear collague Simon in Qt Design studio. I will quickly show a simpile animation, rotation of rotator. Then I will show how can we make integration of electrial moter module generated by the simon. I have some dummy data file, I will use dummy data file for the input of the machine learning and display the predicated value. I will first run my project UI + machine learing on the Windows PC and finally I will also run the same project on the Rpi for the prototyping purpose.