Clear Linux OS - Architecture Overview

Describes how Clear Linux OS is designed, highlighting core features, operating models, and foundational tools that are key to understanding how the distro operates.

1. Clear Linux* OS Architecture Overview Patrick McCarty - Software Engineer *Other names and brands may be claimed as the property of others

2. © 2018 Intel Corporation. Intel, the Intel logo, Intel Inside, the Intel Inside logo, Intel Experience What’s Inside, The Intel Experience What’s Inside logo, and Xeon are trademarks of Intel Corporation in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. Intel technologies’ features and benefits depend on system configuration and may require enabled hardware, software or service activation. Learn more at intel.com , or from the OEM or retailer. Intel processors of the same SKU may vary in frequency or power as a result of natural variability in the production process. For more complete information about performance and benchmark results, visit www.intel.com/benchmarks . The cost reduction scenarios described are intended to enable you to get a better understanding of how the purchase of a given Intel based product, combined with a number of situation-specific variables, might affect future costs and savings. Circumstances will vary and there may be unaccounted-for costs related to the use and deployment of a given product. Nothing in this document should be interpreted as either a promise of or contract for a given level of costs or cost reduction. Intel does not control or audit third-party benchmark data or the web sites referenced in this document. You should visit the referenced web site and confirm whether referenced data are accurate. Optimization Notice: Intel's compilers may or may not optimize to the same degree for non-Intel microprocessors for optimizations that are not unique to Intel microprocessors. These optimizations include SSE2, SSE3, and SSSE3 instruction sets and other optimizations. Intel does not guarantee the availability, functionality, or effectiveness of any optimization on microprocessors not manufactured by Intel. Microprocessor-dependent optimizations in this product are intended for use with Intel microprocessors. Certain optimizations not specific to Intel microarchitecture are reserved for Intel microprocessors. Please refer to the applicable product User and Reference Guides for more information regarding the specific instruction sets covered by this notice. Notice Revision #20110804. No computer system can be absolutely secure. Intel® Advanced Vector Extensions (Intel® AVX)* provides higher throughput to certain processor operations. Due to varying processor power characteristics, utilizing AVX instructions may cause a) some parts to operate at less than the rated frequency and b) some parts with Intel® Turbo Boost Technology 2.0 to not achieve any or maximum turbo frequencies. Performance varies depending on hardware, software, and system configuration and you can learn more at http://www.intel.com/go/turbo . Available on select Intel® processors. Requires an Intel® HT Technology-enabled system. Your performance varies depending on the specific hardware and software you use. Learn more by visiting http://www.intel.com/info/hyperthreading . § Configurations: The testing was done on Based on fourth-generation Intel Xeon E5-2699 v4 @2.20 GHz processor with 22 cores, 55 MB LLC and 62 GB memory 16 1G hugepages . The testing was conducted in OPNFV Pharos testbed on Pod 12 by VSPERF community engineers Intel, the Intel logo are trademarks of Intel Corporation or its subsidiaries in the U.S. and/or other countries. *Other names and brands may be claimed as the property of others. © Intel Corporation Legal Disclaimer

3. Agenda Clear Linux* OS Overview Performance optimizations Use-case focused bundles Stateless OS design Telemetry Updates *Other names and brands may be claimed as the property of others

4. Clear Linux* OS Overview Optimized for IA Rolling release distribution Average of 9 releases per week Developer-focused *Other names and brands may be claimed as the property of others

5. Performance Optimizations Optimize the entire stack Compiler flags Westmere baseline Haswell tuned Optimized libraries selected at runtime based on available CPU features Performance patches to packages Example optimized package: https://github.com/clearlinux-pkgs/opencv/blob/master/opencv.spec Program using OpenCV* AVX2-enabled CPU dynamic linker libopencv_*.so (base) libopencv_*.so (avx2) *Other names and brands may be claimed as the property of others

6. Use-Case Focused Bundles Bundles provide use-case driven functionality to end user Dependencies resolved at build time on server, not at install or runtime Similar to package groups in other distros Vertically vs horizontally integrated os-core os-core-update network-basic webserver openssl python-basic application-server kvm-host ansible iproute2 virt-manager scm-server cloud-control *Other names and brands may be claimed as the property of others

7. TRADITIONAL OS User Data System Configuration Operating System CLEAR LINUX* OS User Data System Configuration Operating System Stateless OS provides functional and secure default configuration in /usr Defaults can be overridden or modified in /etc and the home directory Wiping /etc and /var performs a "factory reset", restoring OS default configs *Other names and brands may be claimed as the property of others

8. Stateless – example Default telemetrics.conf from operating system in /usr record_expiry=1200 spool_max_size=5120 spool_process_time=900 rate_limit_enabled=true record_burst_limit=1000 record_window_length=15

9. Stateless – example Default telemetrics.conf from operating system in /usr Custom configuration in /etc record_expiry=1200 spool_max_size=5120 spool_process_time=900 rate_limit_enabled=true record_burst_limit=1000 record_window_length=15 record_expiry=1200 spool_max_size=5120 spool_process_time=900 rate_limit_enabled=false record_burst_limit=1000 record_window_length=15

10. Stateless – example Default telemetrics.conf from operating system in /usr Custom configuration in /etc record_expiry=1200 spool_max_size=5120 spool_process_time=900 rate_limit_enabled=true record_burst_limit=1000 record_window_length=15 record_expiry=1200 spool_max_size=5120 spool_process_time=900 rate_limit_enabled=false record_burst_limit=1000 record_window_length=15

11. Telemetry Opt-in telemetry solution Lightweight client service Client-side probes send records to help debug software anomalies. Probes avoid collecting personally identifiable information and records comply with Privacy Policy*. Records are analyzed and displayed in a developer-oriented format on the telemetry server. * https://www.intel.com/content/www/us/en/privacy/intel-privacy-notice.html

12. Probe Probe Probe … telemprobd libtelemetry telempostd Server Telemetry – Client architecture

13. Updating All installed bundles are updated at once Entire system update (one OS version) QA is done on the entire OS release at once Proportional updates Auto-update on by default

14. Update content created by mixer tool Upstream Sources Bundle definitions Bundle A data Bundle B data Bundle C data Full chroot Update Creator Update Creator Update Artifacts Swupd clients... s wupd clients... Mixing

15. Mixing – Update artifacts Manifests MANIFEST 24 # OS tooling/content format version: 21260 # OS Version this manifest describes previous: 21220 # Previous change to this manifest at this OS version filecount: 13624 # Number of files in the manifest timestamp: 1520706949 # Epoch of creation contentsize: 811403622 # Size, in bytes, of this bundle (not accounting for included bundles) includes: os-core # Bundle included by this bundle F... 0437fc1556fdfe08ee8cfa492094e5c11a86b7b793213767d4f5697d9b437b36 21080 /usr/bin/c_hash F... 4fdebd92c2ad33ad063c8de973b4eafa35d800ff70abe75644172ae6d0b81436 21080 /usr/bin/corelist < 13622 more entries > Manifest of Manifests (MoM) M... 39be958b03625d0507222996f167de279bc2edaec9a1ff45a86f3cdfac83ca6a 21080 desktop-autostart M... 3ac656e9bdb43871f5345cf71c866a67a58d3ce0a2a085efb8e703be4dd3d753 21080 desktop-locales M... 1dbd2354eb2cbf47a871a4d70fc5cee0dc0e6df2c940b03ab6d5ac2edbad594d 21080 dhcp-server

16. Manifest.MoM signed, verification cascades through SHA256 hashes hashes verified hashes verified Manifest.MoM Manifest.MoM.sig Manifest.os-core Manifest.go-basic Manifest.shells Manifest.editors Manifest.desktop ... /usr /usr/bin/bash /usr/bin/cat /usr/bin/chmod /usr/bin/rm ... /usr/bin/chown /usr/bin/chroot Mixing – Security

17. Mixing – Update artifacts Packs Delta-packs (from version x to y, content difference between versions) Binary deltas Zero-packs (from version 0, complete content of bundle) Full files (for fallback) Compressed full files available for download if pack download/extraction fails

18. NON-ATOMIC ms-s duration ATOMIC Download + verify MoM Download + extract packs Apply delta files and stage Verify pack contents with manifests Rename staged to final Update Artifacts Updating – Client Operation

19. Clear Linux bundles and content Mixer Update Artifacts s wupd clients... User bundles and content Creating Custom Mixes Useful for teams that want to provide their own content on top of Clear Linux* OS content for development, testing, etc. *Other names and brands may be claimed as the property of others

20. mixin Useful for individual users that want to add their own content User adds package Mixer Local Artifacts Upstream Artifacts Merge s wupd client Side-loading Custom Content

21. Clear Linux* OS Rolling release security updates Stateless OS design Performance focused Use-case optimized bundles Fast, secure, and reliable updates *Other names and brands may be claimed as the property of others

22. Contact details Patrick McCarty pmccarty on # clearlinux ( freenode ) More resources: Project site: clearlinux.org Forum: community.clearlinux.org Git repos: github.com/clearlinux github.com/clearlinux-pkgs