Software Engineering Blog - Gilles Legoux (glegoux)

Top Five Ways to Beat Blue Monday

2024-01-10T00:00:00+00:00

Knowing that the year’s most depressing day is soon can be upsetting. No recurrent day will see everyone feeling depressed. However, periodic factors can contribute to defining the social climate of a society, where statistically, a significant proportion of the population suffers from depression compared to other moments.

Let’s understand the Blue Monday effect and how to go over it:

📜 History
🧮 Formulae
📊 Analysis
🪄 Tips

📜 History

Blue Monday is a day recognized as the most depressing day of the year, mainly because of the return to work and routine after a weekend and the end of the festivities of the previous year. It is the third Monday of January.

It was initially created by psychologist Dr. Cliff Arnall in 2005 when he developed the concept as part of a marketing campaign for the now-defunct British company Sky Travel. It’s a fantastic example of how an advertising campaign can become embedded in cultural traditions.

🧮 Formulae

Dr. Cliff Arnall suggested a function that models how depressing each day of the year is. By finding the date on which a mathematical formula based on economic, psychological, and climatological factors, then validated by his observations, takes its maximum value on Blue Monday:

Depression formula — not proven scientifically 🤓 [source]

Where the parameters are the weather (W), the debt (D), the monthly salary (d), the time since Christmas (T), the time since failing our New Year’s resolutions (Q), low motivational levels (M), and the feeling of the need to take action (Na).

Later, Dr. Cliff Arnall created the “Yellow Day”, a celebration of the happiest day of the year to counterbalance the saddest day. This time, he was in cahoots with an ice cream manufacturer, Wall’s Ice Cream, where the period around the 20th of June would be a perfectly happy day thanks to the pleasant temperature, more daylight hours, and holiday perspectives. On a similar approach, the formula is:

Happiness formula — not proven scientifically 🤓 [source]

Where the parameters are the outdoor activities (O) in full connection with nature (N) and also the social interactions (I), the childhood and other positive memories (S), multiplied by the temperature (T), and divided holiday excitement (P).

Several versions of these mathematical formulae exist. Each one gives periodic dates because they are founded on pseudo-periodic coupled parameters. However, the scientific community considers them as baseless pseudoscience. And they are right. In addition, the mental health of a population can be influenced by many unpredictable phenomena like an economic crisis, an epidemic, or a natural catastrophe. Even if a mathematical formula and/or these annual periodicities were proven, they would be disturbed by it. But time slices exist where the mental health of a whole population is affected. It isn’t easy to quantify. Only trends can be observed.

👀 Read more

Read the rest of the article …

Ubuntu: apt-key is deprecated

2023-07-29T00:00:00+00:00

apt-key is deprecated from Ubuntu 22.04 LTS (Jammy Jellyfish) for security flaws: don’t use it anymore whatever your Linux environment. Instead of using it, consider the command gpg now. Then move your public keys relative to your APT package manager to a new location with the binary format OpenPGP, and specify the option Signed-By (signed-by) for all your used package repositories.

Let’s follow this plan together to properly use your secure package management and see if you are not infected by these security flaws:

🔒Understand the security flaws
💡 Explore its workstation environment for apt and apt-key
🔨 Use directly the gpg command to manage your new keys
➡️ Migrate your current trusted public security keys to the correct format and location
✅ Check the migration and stop the useapt-key, /etc/apt/trusted.gpg, /etc/apt/trusted.gpg.d/
❔Questions
🚀 Go further

🔒 Understand the security flaws

apt-key is a sub-command of apt to secure the management packages with asymmetric security keys. As you know, package management is a sensible area for security; it is an access door for an attacker to install malicious packages. That’s why when a package is installed, cryptography is used to validate that the installed package is really the one wishes (see Debian Wiki: SecureApt).

Security risks in package management [built by author]

Here are the milestones of the lifecycle of a package until be installed:

The package is produced on a build machine;
The package is signed on the build machine with a private key;
The signed package is uploaded to a repository (download mirror);
The client system downloads the package (with its package dependencies before, if needed) thanks to the metadata and the usage of its package manager from an authorized repository (or from cache);
The signature of the package is verified (with its package dependencies before, if needed) with the associated public key;
If the signature is valid, the package is installed.

The developers of Debian and Ubuntu realized that the apt-key command has had flaws for a long time (see Security tracker: package apt and particularly CVE-2011–3374). While apt-key was not guaranteed to work people continue to use it, that’s why now this command line has been marked as deprecated (see Fully deprecate apt-key, schedule removal for Q2/2022 (ee284d59) · Commits · APT Developers / apt · GitLab) and will be deleted. Support for the function ends with Debian 11 (Bullseye) and Ubuntu 22.04 (Jammy Jellyfish).

The problem is to move all tech ecosystems because numerous documentation or blog posts are not up-to-date on this aspect. Even tech giants like Google: Linux Software Repositories, where the usage of apt-key is still referenced in August 2023 (after the publication of this article in July 2023). After raising the alert, the correction was made in February 2024 (here).

The home page of the Google Linux Software Repositories is not up-to-date with the usage of apt-key [source]

Let’s understand an attack scenario among several possible ones. Even if, by default, APT checks the validity of the signature of each package to install, and if it is invalid, a warning message is printed, and the package is not installed, this is not sufficient because APT also needs to know which trusted public GPG key belongs to which package repository.

👀 Read more

Read the rest of the article …

History of the Generative AI

2023-04-16T00:00:00+00:00

Generative AI is the state of the art of machine learning and has recently seen significant advancements with the emergence of AI ChatBots with high performance.

Generative AI for all types of content [built by author]

Let’s discover its history, from simple textual machine learning models specialized to universal large language models at a vast scale:

💬 AI ChatBots
🔥 From NLP to Multimodal GAI ChatBots based on LLM
🧪LLMs Timeline: Closed-source vs. Open-source
🎯 From Specialized Model to Generalized ML Model
📈 Scaling Laws & Performance
🏁 Technological and Economic Race
🚀 Go Further with Limited & Augmented LLMs

💬 AI ChatBots

Building generalist AI chatbots is the most complex challenge for a technology that simulates human intelligence. It is the base of the artificial human assistant. They are experiencing dazzling success with the release of ChatGPT, a new step that has been overcome after Google Home and Alexa by AWS. The technical component behind these products has disrupted and improved the ecosystem considerably, although not perfect yet (perhaps it will never be).

The interest in chatBots followed the one for deep learning from 2010. Before, it was reserved for a restrained audience. Still, it exploded in November 2022 with the release of chatGPT open to the general public with a ramp-up of 100M Monthly Active Users (MAUs) after two months, a worldwide record (4.5x faster than TikTok) and 1.6B MAUs now. At the same time, machine learning and deep learning trends have continued growing in the background for ten years.

Google Trends from 2010 [source]

Similar web for chat.openai.com [source]

To have an order of magnitude, the human brain has, on average, 100 billion neurons and 100 trillion synapses. The estimated resources used for chatGPT+ (paying version of chatGPT) use deep learning models based on GPT-4 with an estimated model size close to these values (digits kept private by OpenAI). But a human neuron/synapse is much more powerful than a deep learning neuron/synapse. The gap is still important, above on reasoning tasks on unknown problems requiring many resources.

Human brain vs. Deep learning [adapted from source]

🔥From NLP to Multimodal GAI ChatBots based on LLM

Thanks to Large Language Models (LLMs) working with Generative AI, chatBots continue getting closer to the objective of simulating human intelligence. The NLP AI models have progressively migrated from simple parsing to complex processing on the language structure, from shallow learning to deep learning, from a single-language vocabulary of words to a vocabulary of tokens with multi-language embeddings, from an RNN (LSTM) to the Transformer with multi-head attention layers approach.

NLP timeline with the Deep Learning [source]

Finally, by mixing other types of data (like images, audio, …), NLP models became multi-modal generative models where varied instructions (in input) give varied results (in output).

👀 Read more

Read the rest of the article …

From Hadoop to the modern data stack

2022-12-14T00:00:00+00:00

Hadoop at Criteo

by Anthony Rabier, Staff Site Reliability Lead Engineer at Criteo
and William Montaz, Senior Staff Site Reliability Engineer at Criteo.

Criteo migrated from Hadoop 3 to Hadoop 2, with a lot of patches and in an awesome way by migrating first the runtime to Hadoop 3 for a big progressively without downtime for Hadoop YARN and HDFS without any intervention of the development teams, then by migrating the Hadoop projects for Spark, Flink, and Hadoop Map Reduce. So a vanilla distribution of Hadoop 3 has been created, running with compiled projects in Hadoop 2 and 3 with retro compatibility and tricks developed and merged by Criteo in the core project of Hadoop.

Migration from Hadoop 2 to Hadoop 3

Garmadon is an agent java deployed with all JVM processes running on the Criteo Hadoop cluster to do the metrology jobs for Spark, Flink, and Hadoop Map Reduce … You can build generic Grafana dashboards and create data lineage between your dataset and audit all operations on HDFS.

Garmadon, Java agent for the Metrology on Hadoop

Data processing at Criteo

by Miguel Liroz, Senior Staff Site Reliability Lead Engineer at Criteo
and Raphael Claude, Staff Site Reliability Engineer at Criteo

Datadoc is an internal tool used by Criteo to browse the data catalog and to provide a data catalog on each dataset.

Datadoc [built by author]

BigDataFlow is an internal tool used by Criteo written in Scala to schedule data processing jobs from an extended SQL query.

BigDataFlow extended SQL [built by author]

👀 Read more

Read the rest of the article …

How to write a README?

2022-12-10T00:00:00+00:00

A README is the best place to start documenting a project. A README is normally a single-page document describing your computer project’s key aspects. This document can be read in a few minutes maximum. It is the open door for a newcomer who wants an overview of the fundamental aspects of the project.

Having a README for a computer project is a common enough thing for the code repositories. Still, the template defining what should contain this README is not standard and often depends on the sensibility of the software documentalist. Avoiding gaps in content and format allows you to align the shared knowledge of your projects in the best common possible way.

The rules for writing a README file will be described with a generic template and a pragmatic methodology agnostic to the tech stack and the project type you use.

First, let’s begin defining the type of project that you can have.

A project is a code repository; it can be composed of either one module (simple project) or several modules (complex project). This project is in an ecosystem composed of dependencies between modules of the same project or other projects. A module can be :

An application: It is deployed at runtime with an executable. No other module depends on it except optionally at runtime.
A library: It’s used by other modules that depend on it for their execution and/or build.

For example, a type of project can be a code library in (Python, Scala, or Java …), an SDK, a web app with a UI and/or an API, a machine learning model, a data processor, a set of scripts, an OPS configuration module or a scheduled job …

All projects have not necessarily needed a README respected rigorous template; it can use a degraded version in function efforts that you want to provide. Only mature projects where you want a high level of quality need it, that is to say, for a project used by the community in an intensive or critical way.

Practical guidelines

Here are explained the practical guidelines that a README should respect.

For a simple project:

The README should be a simple text file called README..
It should be written in coherent syntax (like Markdown or reStructuredText).
There should be a README at the root of the project.
According to the README variant you choose (see sections below), you need to adhere to the variant structure.

For a complex project:

The README at the root must have links to the ones of each submodule, with a sentence to describe the role of each module
It should have a README for each module
The README of each module should respect the guidelines of a simple project

Template

Here are the sections that README should contain.

Each section can contain direct or indirect information with links and should contain a title section with format H1 (i.e., ‘#’ with markdown). A section can be optional and required in function of the type of modules.

[Project Title]* [*: required]
- High-level description (with a demo link if applicable)
- It should indicate if it is a library or an application- It should indicate who is the owner of this project
Quick start* [*: required]
- How to set up the development environment, get the repository, install tech stack, …
- Link to a demo
- How to run the project
- How to run the tests
Usage [optional only if relevant and helpful]
- How to use the project (ex. API description)
- Links to help resources
- How to contribute to the project (coding style, code of conduct, …)
Main structure* [*: required]
- Link(s) to other “sub-README(s)” (only if the module has submodules)
- Links to dependencies
- Describe the file architecture of the repository
Architecture [optional only if relevant and helpful]
- Explanation of the design with a diagram and a description …
Deployment** [**: required only for a deployed application]
- Link to the deployment configuration
- Link to this history of the deployment with the changelog and the releases
- Link to the deployed application
Monitoring** [**: required only for a deployed application]
- Links to monitoring dashboards, …
References [required]
- Links to the documentation about specifications, improvements, or maintenance…
- Other useful links
License
- Link to the license and copyright with the general conditions

👀 Read more

Read the rest of the article …

Aerospike: Real-time database for the online ads ecosystem

2022-11-29T00:00:00+00:00

Aerospike can replace at the same time 2 cache layers, an in-memory layer, and a persistent memory layer like Memcached and Couchbase, respectively, with awesome performance.

Especially explore the usages of this distributed key-value store for the e-commerce catalogs indispensable in the online ads ecosystem to build ad campaigns, recommend products to the right audience, and display the ads for the publishers.

As a member of catalog teams at Criteo, I was at the last meetup of Scale Warriors Aerospike, where the CTO (and founder at Aerospike) was present Srini V. Srinivasan to share new and existing features of Aerospike, and where Criteo presented its plan and strategy to migrate to Aerospike.

Aerospike

Aerospike is a key-value NoSQL database with flash memory and in-memory open source. The high-level ecosystem of Aerospike is deeply integrated with tools of the Big data ecosystem for streaming, batch, and SQL usages with Kafka, Spark, and Trino with a Java client:

The Aerospike Real-time Data Platform [source]

Aerospike is a distributed key-value store. You can access the data with a simple API:

Get(key) -> byte[]
Set(key, byte[], ttl)

The goal is to have small objects from a few bytes to 4 MiB, with low latency, 1 ms on average, and 5 ms at the 99th percentile.

Aerospike is first optimized for the OLTP paradigm because it is a KV Store. Still, you can easily do OLAP queries for other usages for more exotic queries to scan the data thanks to the data models layer for Documents, SQL, Graphs, or Time Series.

Aerospike clusters at Criteo

A DKVS plays a central role in online ads. With this storage, the infrastructure can answer to the small-time budgets required for each business use case: to bid a request for the auction of RTB (< 100 ms) or to render the ads (< 250 ms).

Time budget for the online ads [source]

The tech stack is diversified to respond to the needs of online ads, and a DKVS is insufficient. It is only a part of the ecosystem:

Data stack at Criteo [source]

The data stored in the KV stores must be persisted and loaded in memory to access long-term and quick readings.

👀 Read more

Read the rest of the article …

Learning PySpark with Google Colab

2022-11-15T00:00:00+00:00

Learning Apache Spark with a quick learning curve is challenging. Discover distributed computation and machine learning with PySpark, with several tutorials until building your movie recommendation engine.

Links: GitHub | Tutorials quick start | Dataset

Let’s discover how to use PySpark on Google Colab with accessible tutorials.

As a teaching fellow with David Diebold about Systems, paradigms, and algorithms for Big Data for the international Master IASD (graduate degree M2) for the French Dauphine Paris University member of the PSL University, I needed to organize sessions of tutorials for the students on the distributed computation with Apache Spark.

Fast, flexible, and developer-friendly, this data-distributed processing framework has become one of the world’s most significant. Before teaching the features provided by Spark, we had to choose which language and platform our learners could run the tutorials we prepared. We chose the tech stack: PySpark with Google Colab.

PySpark vs. Spark

PySpark allows interaction with Spark in Python. It gives a better learning curve than Spark (written originally in Scala). Even though it is less performant for a production world using Py4J to interact with the JVM of Spark, it gives sufficient performance (sometimes close to Spark with Java/Scala) to experiment with distributed data science and machine learning.

PySpark is an interaction of Spark with Python [built by author]

PySpark with Python remains largely the preferred language for Notebooks.

Percent of commands on the Databricks platform in each Spark language [source]

The support of PySpark is already excellent and continues to be improved with the future version of Spark 3+.

Python is easier to learn than Scala and has a mature ecosystem for applied mathematics.

For these reasons, we preferred teaching PySpark over Spark. Moving from one to another is easy, only the cost for the learners to become familiar with the Scala/Java ecosystem for advanced use.

👀 Read more

Read the rest of the article …

The Apache Software Foundation for the open-source

2022-11-13T00:00:00+00:00

ASF is one of the larger foundations for open-source projects in the tech world. The official website https://apache.org/ is exhaustive of useful information.

Metrics about the Apache Foundation [source]

Apache Organization

The organization of ASF is complex enough, here are useful like to understand how it works:

Apache community

See https://community.apache.org/

Apache Way: Core values

It is a set of methodologies and values designed to enable disparate communities of volunteers to work collaboratively over time.

See https://www.apache.org/theapacheway/

Apache News

ApacheCon is a conference that is a must for anyone who works deeply with Apache-based software programs.

ApacheCon [source]

See the old conferences: https://archive.apachecon.com/

See all the Apache events:

https://events.apache.org/event/

Apache Blog posts:

Apache Reports:

https://www.apache.org/foundation/reports.html

Apache sponsors

See https://apache.org/foundation/thanks

👀 Read more

Read the rest of the article …

Apache Spark: Ecosystem overview with Apache Hadoop YARN and HDFS

2022-11-02T00:00:00+00:00

Apache Spark: ecosystem overview with Apache Hadoop YARN and HDFSApache Spark™ is a multi-language engine for executing data engineerinApache Spark: ecosystem overview with Apache Hadoop YARN and HDFSg, data science, and machine learning on single-node machines or clusters. It is largely used in the industry for distributed computation of data at scale.

This project is open-source under the Apache software foundation, created by the founders of the company Databricks that provides cluster and support for Spark.

Let deep dive on the architecture of Apache Spark on YARN in a distributed ecosystem of containers and Java VMs.

This architecture need to address the computation and the storage at high scale to manipulate big volume of data with an efficient processing.

Table of contents

Spark ecosystem
Spark architecture through Hadoop
Spark engine for computation and storage
Go with Spark in production
Example of Spark application at Scale
Go further
What’s next
Thanks to
Glossary
References

Let’s start with the basics, you can jump straight to the sections that interest you, otherwise see vertically how this whole ecosystem works.

Spark ecosystem

Data pipelines

Spark allows building data pipeline in batch mode and the streaming mode.

Lambda architecture with Spark [source]

Here, you see a lambda architecture, but to associate online and offline processing, you can have a kappa or zeta architecture on your data pipeline. You will see it in another article.

Spark vs Spark Streaming [built by author]

The streaming mode of Spark is more micro-batch approach. Other framework like Kafka Streaming, Faust Streaming or Apache Flink are more used to process data streams. Spark is forceful to process data batch overall.

APIs

Mainly, Spark can run with several APIs, one for each following programming language Scala/Java, Python and R, respectively with Spark, PySpark and SparkR.

Different types of Spark engines [built by author]

In practice, Spark and PySpark are more used than SparkR. For the experimentation, PySpark is more helpful, and for the performances Spark is better.

Connectors

Spark has many connectors to write to different types of storage.

Apache Spark’s ecosystem of connectors [source]

Here, we are going to speak a bit more about the connection with Apache Hadoop HDFS.

Spark SQL

In addition, you can also run Spark with SQL. With a SQL request, you can also access to heterogeneous data then combine them in memory:

Spark SQL overview [source]

The Catalyst & Tungsten engines allows optimizing the execution plan of the query:

Catalyst [source]

PySpark

In the Python driver program, SparkContext uses Py4J to launch a JVM and create a JavaSparkContext. Py4J is only used on the driver for local communication between the Python and Java SparkContext objects; large data transfers are performed through a different mechanism.

👀 Read more

Read the rest of the article …

My tech blog on Medium

2022-11-01T00:00:00+00:00

I will continue writing blog posts here, but they will mainly be published on Medium under the user @glegoux. The goal is always to improve the experience of readers and my experience of writing. After having re-evaluated the market of online editors, the Medium platform is mature and becoming more and more popular, being one of the leaders among the editors of tech blogs now. It provides a full WYSIWYG user interface when editing online, with various formatting options provided as the user edits. It allows publishing blog posts (called stories) to capture large audiences and communities.

With the evolution of the practices and tools of online blogging these last few years, I wanted to cross thresholds in my tech writer experience: be more efficient and effective in writing content and gain more visibility by publishing content. It requires allocating less time to maintaining editor tools and more time to write and publish content on other websites than here. So, I needed to choose between continuing to use only this blog, starting to use only Medium, using both, or something else. I chose to use both:

The most of blog posts will be published on Medium first, with always a partial version here (or inversely)
The other ones will remain only in this blog.

Migration from glegoux.com to Medium

After evaluating and testing Medium, here are the pros and cons of using Medium vs. my blog:

✅ Pros	❌ Cons
• Medium editor is quicker and handier in updating a post with simple assets like text and images, more intuitive than YAFIYGI, and with instant updates without delay during the deployment of new content like with Jekyll used for my blog • Medium has advanced features of collaboration with co-writers, claps, peer comments, and followers, not all available on my blog natively and smoothly • Medium decorates the post with an auto-estimation of reading, no need to fill manually, and an auto-listen generation not available on my blog natively • Medium offers stats with custom metrics (reads, etc.) for each story more relevant than using a third-party solution like Google Analytics used on my blog	• Medium has partial propriety of the written content and limits the number of readings for non-members, while my blog is free and unlimited open • The Medium content has a pre-defined style of design. It is not highly customizable like the content on my blog • The Medium features remain less advanced in pushing varied content like dynamic technical diagrams and mathematics formulae. While some workarounds exist, my blog allows it

✅ Pros

❌ Cons

• Medium editor is quicker and handier in updating a post with simple assets like text and images, more intuitive than YAFIYGI, and with instant updates without delay during the deployment of new content like with Jekyll used for my blog
• Medium has advanced features of collaboration with co-writers, claps, peer comments, and followers, not all available on my blog natively and smoothly
• Medium decorates the post with an auto-estimation of reading, no need to fill manually, and an auto-listen generation not available on my blog natively
• Medium offers stats with custom metrics (reads, etc.) for each story more relevant than using a third-party solution like Google Analytics used on my blog

• Medium has partial propriety of the written content and limits the number of readings for non-members, while my blog is free and unlimited open
• The Medium content has a pre-defined style of design. It is not highly customizable like the content on my blog
• The Medium features remain less advanced in pushing varied content like dynamic technical diagrams and mathematics formulae. While some workarounds exist, my blog allows it

For all these reasons, the right choice is a consensus by keeping the best of these 2 worlds. Also, I prefer remaining independent for the long term, and https://glegoux.com is not only a tech blog but a hub of centralized online activities.

Don’t worry. I will keep maintaining this website and the open-source template behind it. To avoid breaking changes, the past blog posts will remain accessible without breaking their web links.

However, it seems time-consuming to synchronize the content between Medium and this blog bi-directionally. At best, I can only export and import a blog post to and from Medium by copying/pasting the content manually and by adapting the past content. Because the automatic Medium feature for external synchronization has been closed, and no other mature tools allow it, the custom extension of Mardown used here forces potential adaptations. So, the synchronization in updating a blog post from Medium becomes a nightmare to maintain. In contrast, I want to use Medium editor more, which is quicker and more handy to publish followed content. So, as I explained, some complete articles will only be on Medium or exclusively on this blog (with often a second partial version). When I want to have one blog post at both locations, instead of having a full post everywhere, I will use bidirectional web links cross-websites between the partial post and full post (to avoid divergences and too heavy maintenance in case of multiple evolutive versions with there are editions after the original publication).

🙏 Thanks for your understanding and your readings.

Secret Santa

2021-11-23T00:00:00+00:00

Secret Santa 🎅 is a traditional event in December with family, friends, or office colleagues, in which each guest gives a gift to another guest without knowing who will give him/her a gift.

Could you imagine that you want to organize it? As an independent, technical, and meticulous organizer, you could automate the drawing of pairs (donor, recipient) to work even for an odd number of guests. You could implement an algorithm (given later) that generates a directed cyclic graph where the vertices are guests and the edges represent the relationship (donor -> recipient).

But you want to increase the secrecy, you choose to draw lots for the identity of the recipients at the last moment. In this case, the guests do not know neither for whom the gift they buy will be. In addition, it counters the effect of a no-show, in which a guest might not receive the pre-chosen gift during the event.

So the identities of gifts (ideally all different), donors, and receivers remain secret until the last moment. However, several problems could reduce the surprise effect. Let us understand it thanks to applied mathematics in probability, statistics, and combinatorics.

⟲Self-selection problem

Let’s begin with the first problem: to whom does each guest give his/her gift? Let’s suppose each guest has well his/her gift for a unique other guest. Of course, the draw could have been done before and kept secret, as mentioned previously. But you want to do the draw at the last moment and by revealing simultaneously who gives his/her gift to whom. That is to say, you write the name of each guest on a different piece of paper for each, and mix it in a Christmas hat. Then, each one takes a piece of paper randomly. Finally, all of them, at the same time, unfold the piece of paper and discover the name of the guest to whom he/she gives his/her gift.

Will a guest draw his/her name at random? If this is the case, the surprise effect will be broken for him/her, and what would imply redoing the draw or exchanging gifts? What we want to avoid.

This concrete problem can be reformulated: Is there at least a fixed point in the permutation of a set with $n$ elements? Where $n$ is the number of guests. It is the famous hat-check problem.

The probability that at least one guest draws his/her name is the inverse of the probability that no one drew his/her name, that is to say:

\[p_n = 1 - \dfrac{\mathcal{D}_{n}}{n!} = 1 - u_n\]

$\mathcal{D}_{n}$ is the number of derangements, that is to say, the number of permutations without fixed points, knowing that $n!$ is the number of permutation with $n$ elements. You can prove with the inclusion-exclusion generalized principle that:

\[\mathcal{D}_{n} = n! \sum_{k=0}^{n} \dfrac{(-1)^k}{k!}\]

According to the Taylor-Maclaurin development, we have:

\[\displaystyle\lim_{n\mapsto +\infty}u_n = \displaystyle\lim_{n\mapsto +\infty}\sum_{k=0}^{n}\dfrac{(-1)^k}{k!} = \dfrac{1}{e}\]

with $e \approx 2.71828…$. In addition, this suite $(u_n)_{n\in \mathbb{N}}$ has a rate of convergence quick:

Inverse of the probability that a guest draws randomly his/her name

So if the number of guests is greater than or equal to 4, you have around $0.63\%$ of chance, to have at least one guest who drew their name, that is a bit less than 2 out of 3 chances:

\[p_{n\geq 4} \approx \dfrac{2}{3}\]

This simultaneous strategy risks limiting the surprise effect, like the probability that a guest randomly draws his/her name, which becomes high quickly as a function of the number of guests.

However, another sequential strategy could be considered, the same as presented in the introduction, to generate a cyclic-oriented graph (donor -> recipient). The algorithm will be: A guest randomly draws a piece of paper, keeps it, and gives the Christmas hat to The person whose name is on it. Only the first guest could draw her/his name. In this case, the person redraws a piece of paper and then puts the paper back with his/her name. But here, the surprise effect becomes split into two moments, first discovering the identity of donors progressively, and then all the gifts simultaneously, instead of one moment previously.

🎁 Different gifts problem

A second problem will be to know if two guests could bring the same type of gift. Having all gifts different could make it more fun, as the guests did not know for whom the gift they bought would be. These gifts are not personalized. Let’s suppose that you categorized 300 types of gifts.

This is equivalent to another famous problem, the birthday problem. The probability that two people were born on the same day increases quickly depending on the number of people. It becomes almost certain, that is to say, more than $99\%$ of a chance to be valid for a group of 60 people.

More generally, the probability that a group of size $n$, built with replacement and a sequence of independent and uniformly distributed trials from a set of size $N$, contains at least two identical elements:

\[g_n = 1 - \frac{N}{N}\,\frac{N-1}{N}\,\cdots\,\frac{N-n+1}{N} = 1 - 1\,\left(1-\frac{1}{N}\right)\,\cdots\,\left(1-\frac{n-1}{N}\right)\]

It is the inverse of the probability that no element is the same.

Here, $N$ is the number of possible types of Christmas gifts, and $n$ is the number of guests. Except for some guests with a very original gift, suppose that your guest chose a pre-defined list influenced by the constraints and the ads, with a few hundred types of items, let’s say $N=300$:

Probability to have two identical types of gifts among 300 possible types of Christmas gifts

With an approximation $e^{x} \approx 1 +x$ when $x$ is close to $0$, and the sum of young Gauss $\sum_{k=0}^{n}\,k = \frac{n(n+1)}{2}$, you can get:

\[g_n \approx 1 - e^{-\frac{n(n-1)}{2N}}\]

But in any case, you see:

\[\displaystyle\lim_{n\mapsto +\infty}g_n = 1\]

And:

\[g_{n\geq 20} \geq \dfrac{1}{2} \qquad g_{n\geq 60} \approx 1\]

If you have more than 20 guests, we have more than 1 out of 2 chances to have at least two identical types of gifts.

So, I wish you a good Secret Santa 🎄.

Software versioning

2021-11-13T00:00:00+00:00

What is a software version?

A software is a dynamic set of lines of code, that evolves discretely from state to state in function of time. To identify a specific state of a software, a versioning can be used whether it be for a language compiler, a library, a web API, an application, a server, or a client. Of course a versioning can be used more generally for any project, but here I focus only on the software versioning at large.

             ┌─────────┐           ┌─────────┐           ┌─────────┐
             │         │           │         │           │         │
  ──────────►│ State 1 ├──────────►│ State 2 ├──────────►│ State 3 │─────────►
             │         │           │         │           │         │
             └─────────┘           └─────────┘           └─────────┘

Software states

The software versioning is the process to assign a version to a state of a software: a version scheme and a version strategy are chosen to track the successive evolutions. For example, with a version control system as Git, the history of a software is split into commits, where each commit identifier is used as a version.

40551405dc02afe8240024c952fa1de953fc2ff9 Commit message: State 3
3982dcd90b486eab39eb610ec5e7e5963b92ab33 Commit message: State 2
32f0469eb344029bd1ecf5b0626685f2fa795f47 Commit message: State 1
...

Commit logs with commit identifier and message

Here the version scheme is a SHA1-hash, where the version strategy is to assign a new version to each change of code rendered the more atomic than possible. It is a software versioning handy for development team, because each version is unique and points to each different change in the code base. But not very useable for the users of this software, because a given version is not human readable or user friendly, no useful information is given by this complex naming. And each change has a version, what makes too many existing versions with a naming not discriminant, to know which versions are important except the last one.

A software version is a unique identifier to distribute a software also to users. But the development team of a software wishes to reduce the number of supported and maintained versions exposed to their clients, and wants to package these changes to propose a software as a product with coherent features. That is why another software versioning is applied often above these commit identifiers, and the usage of Git tag and/or Git release is privileged.

What are the features of a software version?

As we have seen above, a software can have several versioning systems, for example one used internally by the development team, and one exposed externally to users.

In addition to be unique identifiers, the versions are often ordered to know if a version is more recent than another. So the naming of a version repects a semantics, that contains this order and additional meanings.

The publication of a new version follows a lifecycle with a workflow and a process. This publication produces a distributable format of the software called archive or package. The version is assigned to the archive that is signed by the software provider identity with a hash function on the content of the archive to guarantee that once a particular version is published, this one remains immutable and author of this archive is well the official provider of this software.

A software can have direct and transitive dependencies that are also a versioning, and can be also a dependency of another software. So, the question is raised of the compatibility between the different versions.

There are two directions to describe compatibility between 2 versions of a same software:

Backward compatible means that a newer version can be used successfully in an context where an older version is expected.
Forward compatible means that an older version can be used successfully in an context where a newer version is expected.
Fully compatible means backward compatible and forward compatible.

Backward compatibility is often guaranteed, but forward compatibility is not. Compatibility depends on a context of use, if the use is not exhaustive, it should be noted that compatibility is only guaranteed for this use. Otherwise, implicitly, compatibility is assumed to be true for all possible contexts of use. For example, full compatibility is only possible in sub-context and not on the global context, otherwise the 2 versions would point to an identical state of the software.

How to build a software version?

Breaking change, compatibility, Changelog, releases, alpha, beta, dependencies

For example, Ubuntu versioning, SDK/JRE versioning, …

Example for Java/Scala

A Java/Scala library is compiled to a platform-independent format called JVM bytecode and stored in .class files. These class files are zipped in JAR files for distribution.

References

Time management

2021-04-20T00:00:00+00:00

You could need to optimize the spending of your time for professional or personal activities. I am going to explain what means to me the time, and how you can manage it. I have been inspired by my knowledge, experiences, readings, and listenings, but it is my personal version of time management between math, physics, management, philosophy, and operational research. Without giving a lot of details, time management is also linked to the agile methodologies that are one implementation, like Scrum, Kanban, or Pomodoro.

The time management is the science of the usage of time, a possible definition could be the process of planning and exercising conscious control of time spent on specific activities, especially to increase effectiveness, efficiency, and productivity.

Time Management

The way you can use your time depends on a lot of things, among others, there are your:

context:
- Which resources do you have (knowledge, time, energy, money …)
- What are your features (skills, experiences, character traits …)
- What are your constraints (costs, exigencies, …)
projection:
- What you want to do (ambitions, preferences, choices …)
- What are your goals (objectives, key results, tasks, sub-tasks, projects, missions …)
strategy:
- How do you organize your activities (priorities, hierarchies, dependencies, sizing, partitioning …)
analysis:
- How you know yourself (strong points, weak points, personal will, and effort …)
- How you have a projection & strategy coherent of yourself (clairvoyance, honesty, realism …)
- How you receive a critical opinion positive or negative on yourself (feedback, auto-introspections, questionings, doubts…)

Time Definition

With the meta-physique considerations, the time has wondering features: universal, continuous, unlimited, non-renewable, solid (incompressible/no expansible), perpetually self-generated, no accumulative. Focus is on the timescale of our human life. These features are immutable, or more precisely their changes are negligible. But we can interpret the time differently.

Here I am going to interpret the time like a quantizable object with an arbitrary window in the past, present or future. The present moment is seen like a progressive cursor representing a dynamic instant on the timeline. This present instant plays the role of an irreversible transformer altering the future into the past. The duration of an event is the length of a windows time delimited by 2 instants where the event began and finished. For the following, when I speak about time, I refer to these sliding windows.

Time vs Duration

The time :clock2: is one of our most valuable resources, for me more important than knowledge, money or energy, because all ones depend on time.

It is important to distinguish the absolute time, and the subjective time. The quantity of elapsed time to carry out an activity can be viewed and interpreted subjectively. The same quantity of time measured objectively in seconds could be lived differently by 2 different people esteemed like short or long. Take into account this subjectivity, because time management is more relevant if it is applied individually with your sensibility.

For example, a sharp mind in a particular situation will have the feeling of life in accelerated time by interacting with a less sharp mind, because he compares his thinking speed to that of his interlocutor, and this one is faster, and reciprocally. Another example is when you are doing a passionate activity. The time seems to go faster, because you forget the feeling of elapsed time, and inversely with the boredom.

The absolute time allows you to compare the duration between 2 activities objectively, but when you want to organize or optimize your time, you need to adjust your efforts with your personal consideration of an elapsed time for a given activity. The elapsed time can be interpreted as a sum of done activities. The unit measure of an activity is subjective, and could be called weight, where mixed the quantities of used resources to do the activity (for example the quantities of time, energy, or money). Here more the activity has a light color, the more the activity has a lightweight:

Time usage

The time could be considered as a quantity of spent resources during a duration (itself in another meaning) to do activities now. So the quantity of available time at your disposition does not respect the time features mentioned previously, it is:

personal: you have a personal context and subjective time
discrete: you have a minimum time of reactivity for your frequency sample to measure the elapsed time
limited: you have a life length and resources which are finite
renewable: you can renew some activities in the future
compressible/expansible: you feel elapsed time differently for a given activity in function of time
occasionally generated by you: you make your choices, and your organization to free of time
cumulative: we interpret the time with a quantity now, not only an instant

But the quantity of time is always available immediately.

Concepts

We defined several ways to conceptualize the time, here I can define more precisely the other concepts of time management like the effort and the optimization.

The effort to do an activity is like the way of consummation of a number of resources. Your resources are renewed perpetually and have a potential, the activity has also a potential. You need to create a resource flow, called the intensity of the effort, to do the activity. The awareness to practice an extreme effort in quantity (too much or too few) or quality (too easy or too difficult) slows down the feeling of elapsed time and extends the time. This reduces your endurance in this activity. But the practice and the training allow you to reduce the feeling of effort, you consume less and/or have more resources.

It is important to observe that the optimization of the time usage is in fact the optimization of future projection of the productivity of an activity on a time range. This optimization is under constraints and depends on objectives, called also goals. To achieve these goals one of the available resources to use and save is time.

I use several concepts above without defining them, here is a summary:

Goal: definition of result(s) that you want to get, called also objective.
Task: definition of action(s) that you need to achieve a goal.
Activity: execution of task(s) over time.
Productivity: a measure of your performance (that is a combination of effectiveness and efficiency) of your activities.

Optimization Model

The time management can be seen in the modeling of an objective function to optimize. The model of this objective function is fixed by the definition of your goals (split into tasks) and of the execution plan. Then its optimization is determined by the duality between the minimization of the cost of executions and the maximization of quality of got results for a set of tasks.

In mathematics terms, that is to say optimize the objective function $g \circ f$ on a set of tasks $\bigcup x \subset X$ where:

\[f: X \rightarrow \mathbb{R}^+ \times \mathbb{R}^+ \quad x \mapsto f(x) = (c(x), q(x))\] \[g: Img(f) = \{\,f(x)\,|\,x \in X \} \subset \mathbb{R}^+ \times \mathbb{R}^+ \rightarrow \mathbb{R}^+ \quad y \mapsto g(y) = g(f(x)) = g(c(x), q(x))\]

$c$ is the cost function of executions
$q$ is the quality function of got results
$f$ is the productivity function in terms of efficiency and effectiveness
$g$ is a trade-off function between the minimization of the cost $c$ and maximization of the quality $q$ on subset of $X$

Objective function is a trade-off: cost vs quality of a set of tasks

But the stability, the evolution of the objective function depends on your regularity, your implication, and your progress.

The functions $c$ and $q$ are often linked, and it influences generally each one negatively for the optimization: reducing the cost reduces the quality, and increasing the quality increases the cost. While our optimization consists in reducing the cost, being efficient, and increasing the quality, to be effective. It is often more performant to be firstly effective, than efficient for a given task. Learning to do correctly a task before doing it faster. But it is clear, that being effective and efficient (at the same time and in good proportion) is the final goal for each task.

You can do an analogy with the machine learning algorithm. The execution plan is your model owning hyper-parameters, the tasks are rows of your dataset (training, validation, test) and the maximization of the objection function is the step of learning. With experience in an execution plan, you can predict the speed of execution and the quality of results.

Methodology

After having modeled and defined the concepts of time management, I am interested here by applying it in practice through a methodology. Firstly, even if you can model the time management, in my point of view it is an empirical science. That is to say, you need a methodology to test it, and your knowledge about this science will be provided by your experiences and your observations.

Here the main considered resource is the time, but in practice, the methodology could be more complex you could consider other inter-dependent resources to spend like money or energy.

Context

You need to list and put into perspective your resources, your skills, and your constraints.

Projection

After you need to define what you want to do and build your goals each one with a dateline. Do a first pass of the following section strategy applied to your goals (replace activities with goals). Once your goals are partitioned, hierarchized, sized, prioritized, and sequenced. Your destructed goals should now fill in a coherent structure. Then split each goal into tasks. You can repeat this operation to split each task into smaller activities to pass from a macro timescale to a micro timescale.

Strategy

The strategies are the fact to structure your activities to reach out.

Partition your activities

Partition your activities which have the same type in independent class between them the more possible. Of course, the content of each partition is subjective for some activities, you can classify your activities into these partitions differently in the function of your sensitivities and interests. You can already give an important weight to each partition.

Partitioning

Organize into a hierarchy your activities

Here the goal is to estimate roughly the weight of an activity to have its order of magnitude.

Each goal should be realistic, reachable, measurable with success keys, and sized in difficulty and required time to achieve it. You can play on several scales for your future projections: very long term, long term, middle term, short term, and very short term.

Scaling

Size your activities

Now you can size more precisely each task. You can measure absolutely if it is complicated, you can also measure relatively the ones in relation to the other ones. You need to choose the unit of the * weight* of your activity.

Sizing

Organize the interactions of activities

You need to build a dependency graph of each activity to see if an activity depends on other one(s).

Dependency graph

Your dependency graph must be a set of acyclic graphs.

Prioritize your activities

After you need to rank your activities which ones are the most important ones:

Prioritizing

Sequence your activities

It is you build your plan of execution by sequencing each activity by respecting the previous steps and your other constraints:

Sequencing

You can also optimize these sequencing.

Auto-analysis

Now you can execute your plan. But you need to monitor it, have logs, metrics, and traces, and have dashboards to visualize it. You need to do regular retrospectives and plans. Respectively adjust your strategy in the function of time, and continue to converge to the filling of your goals. Then define new goals or delete goals in order to avoid the shortage of tasks, the overload, or the accumulation of deprecated goals. Then you need to measure your KPI and your productivity. You need to have a feedback loop to build your virtuous circle, and iterate on it.

For Example

Are you a student, a worker, or a retiree? Take the example of a worker, more precisely a professional software engineer full-time working exclusively for a unique company at the beginning of their career. By hoping, that it matches your auto-introspection, else the following could help you to do the same thing for your personal context. I give you here only a partial overview of how it can be done, this example is not necessarily my real usage of my personal time.

Timescale of the human life

I analyze my resources, my constraints, and my features (hard skills and soft skills, …).

Above all, I define what I want to do and my ambitions: following conferences, coding an open-source project, having a side project, becoming a speaker, losing weight, finishing TV series, reading books, following training, giving courses … Each goal should be precise, measurable, and reachable.

I split my activities into 3 partitions:

Life activities: required vital, society or community activities
for example sleep, eat, wash, practice a physical activity, interact with your environment, do the housework, pay the taxes, see your friends …
Work activities: required activities for your job
for example commute, work, do brainstorm, code, participate in a meeting, take a break …
Personal activities: spare and free activities
for example: practice its development personal, relaxation …

I focus on the life activities that compose my lifetime. I let on side work and personal activities, it is the principle.

For example, for the different timescales, I take long: ~hour and short: ~minute. I try to evaluate the time of each life activity:

sleeping: 7 hours per day
washing: 2 times 3 minutes per day for teeth and 15min per day for body
eating: 2 times 20 minutes per day for lunch and dinner, with 10 minutes per day for breakfast
seeing friends: at least one time per week for several hours
practicing sport: 3 hours per week
doing the other tasks: several hours per week (taking out the trash, preparing the meals, doing housework, shopping, do-it-yourself …)

These activities are not really dependent on each other and have the same priority for me. Now I sequence them for each period of a day and of a week.

I have a calendar where I can monitor each timescale and put my tasks, a weekly planning for my deadlines, and a to-do list for daily tasks.

Now I redo the same things for both my personal and work activities to manage my time more effectively.

Go Further

This article is only an introduction to time management applied to oneself by oneself. But the situation can be more complex and applied in all situations. For example, if you think about the time management for other people or projects, in a professional context for the tech sector:

Tech/Team/Dev Leader for a team composed of a group of people
Tech/Project Manager/Director for a department composed of a group of teams
CTO for a R&D pole composed of departments
CEO for a company or a group of companies …

It requires delegating tasks, understanding the personal context of other people, and driving the time management for a group of people at different scales. These organizations can have different structures and different stakes. For this, you must redefine the time management to resolve your new needs for yourself and for other people and/or projects.

To conclude, as time management is a set of meta-concepts learned mainly through self-education practice, here is a paradoxical quote to consider (translated from French):

Dare I set forth here the greatest, the most important, the most useful rule of all education? It is not to save time, but to waste it.

Jean-Jacques Rousseau from Emile, or On Education (1762)

References

MALABOU, Catherine. Le Temps. Hatier, 2019. 80 pages. ISBN: 2-218-71678-X.
Saint-Augustin. Les Confessions. 4 BC. Book XI.
KLEIN, Etienne. Les tactiques de Chronos. Flammarion, 2009. 224 pages. ISBN: 978-2-0812-2305-9.
HAWKING, Stephen. A Brief History of Time. Bantam Book, 2011. Chapter 2: Space and Time. 23 pages. ISBN: 978-0-857-50100-4.
VANDENBERGHE, Lieven. Convex Optimization. Cambridge University Press, 2009. Chapter 1: 16 pages. ISBN: 978-0-521-83378-3.
WINSTON, Wayne L.. Operations Research. Thomson Brooks/Cole, 2004. Chapter 1: 12 pages. ISBN: 0-534-38058-1.
MURPHY, Kevin P.. Machine Learning: A probabilistic Perspective. The MIT Press, 2012. Chapter 1: 24 pages. ISBN: 978-0-262-01802-9.
SUTHERLAND, Jeff. SCRUM. Business Books, 2014. 248 pages. ISBN: 978-1-847-94108-4.
ANDERSON, David J. Kanban: Successful Evolutionary … Blue Hole Press, 2010. 278 pages. ISBN: 978-0-9845214-1-8.
CIRILLO, Francesco. The Pomodoro Technique. 2018. 160 pages. ISBN: 1524760706.
OHNO, Taiichi. Toyota Production System. Synermage, 2020. 150 pages. ISBN: 106992329X.
New York University. Time Management. NYC, January 2023. Link
KOLMAR, Chris. Efficiency vs Effectiveness. Zippia, April 2021. Link

Unix timestamp

2020-11-28T00:00:00+00:00

In fact, 2038 is a particular year when an informatics bug could occur, sometimes referred as the Y2K38 problem. Because an arithmetic overflow on the Unix timestamp risks appearing, what it means, there are insufficient bits in a binary number representation to portray the result of an arithmetic operation. For example, an arithmetic operation is not closed with respect to addition, subtraction, multiplication, or division.

Y2K38 problem is caused by insufficient capacity used to represent a time duration:
the Unix timestamp as a signed 32-bits integer. It could produce a huge chaos.

Unix timestamp

Effectively, the Unix timestamp, used largely in informatics systems, represents the number of elapsed seconds since January 1st, 1970 at midnight in the Coordinated Universal Time (UTC) time standard. This value is incremented every second. And the date 1970-01-01 00:00:00 UTC, usually called epoch date, is a particular date in a lot of informatics system representing the origin time, inherited of the Unix operating system an POSIX standard. The Unix engineers picked this date arbitrarily, because they needed to set a uniform date for the start of time, and New Year’s Day, 1970, seemed most convenient.

As you can see:

$ for s in $(seq 0 3); do date --date "1970-01-01 00:00:00 UTC $s seconds" +%s; done
0
1
2
3

A Unix timestamp is used to known the occurrence date of an informatics event, after 1970. Sometimes other granularity of time like the millisecond, or the nanosecond are used, but traditionally it is the second.

While the trademark today is UNIX, in this article, we use Unix in deference to dmr's wishes, dmr being the username of Dennis MacAlistair Ritchie, famous computer scientist having worked the on Unix operating system.

Arithmetic binary representation overflow

But the arithmetic binary representation of this Unix timestamp is sometimes on 32-bits as a signed integer again as originally at its creation. That is to say, its maximum value is $ 2^{31} - 1 $. The negative values until $ -2^{31} $ (inclusive) are used by convention for representing a duration from 1970-01-01 00:00:00 UTC to an anterior date. Then the maximum duration time supported by a such Unix timestamp will be reach out at 2038-01-19 03:14:07 UTC:

$ date --date "@$((2**31 - 1))" --utc
Tue Jan 19 03:14:07 UTC 2038

The adding of 1 second at this date raises an arithmetic overflow error. For a representation on 32-bits as an unsigned integer. The problem will be the same in 2106:

$ date --date "@$((2**32 - 1))" --utc
Sun Feb  7 06:28:15 UTC 2106

Hopefully, the majority of informatics technology have already migrated to a 64-bits representation. For a signed integer, this problem is rescheduled for around in 290 billions of years, in other words in a quantity quasi-infinity of time, knowing the age of the universe is estimated around 13 billions of years, or around 20 times less. With the approximate computation:

scala> Long.MaxValue/(365*24*3600)/Math.pow(10, 9)
res: Double = 292.471208677

The problem could be considered as resolved. But some informatics system could be bugged by the negligence or the ignorance of this problem from developers and/or maintainers of these informatics systems. The informatics system uses often a lot of dependencies like code libraries, hardware components, or operating systems. Only one of the dependency could be the source of a such bug in the related informatics systems that uses it, because the Unix timestamp are omnipresent on each system layer from low level to high level.

Concretely, it raises an arithmetic overflow error. It could be silent, trigger a coding exception, or have a different behavior. For example for the most programming languages, it will be silent for the programmer but with a different behavior. Even if the test policy of the coding project could catch this kind of error. It is enough rare to have a unit or integration test with the current time or a future time, often the time is fixed to have an exact reproducible test. But it could be observed in the end-to-end test. In any case, the error will appear firstly at the runtime directly in production.

It could produce a chaos, because the consequences of a such error are hard to forecast, with a risk to be destructive. Events can be not synchronized between several systems or just stop working: internet, cars, IoT devices, banks, airplanes, etc … suddenly could no longer work.

Example in Java/Scala

While currently Java/Scala ecosystem is not exposed to this problem in the standard library, with a signed 64-bits representation for an Unix timestamp:

scala> import java.time.Instant; Instant.now().getEpochSecond.isInstanceOf[Long]
res: Boolean = true

A Unix timestamp is well represented by a Long from $-2^{63}$ to $2^{63} - 1$. Let’s study together, what it could happen, if it was a Int.
The range of values will be between $-2^{31}$ and $2^{31} - 1$ for a Int.

scala> Int.MinValue
res: Int = -2147483648
scala> Int.MaxValue
res: Int = 2147483647

In Scala, it is false to express $2^{31}$ with Math.pow(2, 31).intValue that is equal to 2147483647, what it doesn't make sense. Here, you have a odd number instead of an even number, and typically an arithmetic overflow per truncation from Double to Int.

In addition we have:

scala> Int.MaxValue + 1 == Int.MinValue
res: Boolean = true

The adding of 1 second after Tue Jan 19 03:14:07 UTC 2038 gives:

$ date --date "@$((-2**31))" --utc
Fri Dec 13 20:45:52 UTC 1901

Weird, how could we explain that. The error is silent no exception raised, the addition provoking the arithmetic overflow is cyclic on the arithmetic representation. Because, here a Int has a two’s complement binary representation (2C) in Scala. Here the relation between the base 10 and the two’s complement binary representation on 32-bits is:

\[[N]_{10} = [b_{31}\,...\,b_{1}\,b_{0}]_{2C} = - b_{31}\,2^{31} + \sum_{i=0}^{30} b_{i}\,2^{i} \text{ where } b_k \,\in\, \{ 0,1 \}\]

with the notation little endian for the order of the bits, the most significant bit (MSB) $ b_{31} $ is the sign bit and the least significant (LSB) bit $ b_{0} $ the unit bit. It the sign bit is 1, the number is stricly negative. And if the unit bit is 1, the numbder is odd. For example:

def toBinaryRepresentation(i: Int): String =
    i.toBinaryString.reverse.padTo(32, '0').reverse

scala> toBinaryRepresentation(Int.MinValue)
res: String = 10000000000000000000000000000000

scala> toBinaryRepresentation(-1)
res: String = 11111111111111111111111111111111

scala> toBinaryRepresentation(0)
res: String = 00000000000000000000000000000000

scala> toBinaryRepresentation(Int.MaxValue)
res: String = 01111111111111111111111111111111

To conclude the limitations of storage capacity for a value can provoke weird bugs in an informatics ecosystem, it is important to be aware of this kind of behavior when a technology is used.

References

RITCHIE Dennis M., THOMPSON Ken, The UNIX TimeSharing System. Bell Labs, July 1974. Link.
NYGREN Erik, Preparing for Y2038 (ALREADY?!). Akamai Blog, January 2019. Link.
DINAMANI Deepa, Solving the Year 2038 problem in the Linux kernel. opensource.com, January 2019. Link.
THAKUR Vishal, What Is The 2038 Problem?. Science ABC, October 2020. Link.

Geo-location

2020-08-24T00:00:00+00:00

We can model the surface of the earth as a sphere of radius $R$. Even if in reality, the earth is more a kind of ellipsoid with an irregular surface, where each point of its surface owns an altitude in relation to the sea level.

Coordinates on earth with latitude and longitude

Let $p1 = (\text{lat1}, \text{long1})$, $p2 = (\text{lat2}, \text{long2})$ be 2 points on the surface of the earth with a latitude and a longitude given in radian respectively. The latitude has a value between $[-\frac{\pi}{2}, \frac{\pi}{2}]$, and the longitude between $]-\pi, \pi]$. So each point is identified by unique way with this tuple: latitude and longitude. There is at least a great circle of the sphere passing both through $p1$ and $p2$, either an infinity if $p1$, $p2$ and the center of the sphere are aligned in the ambient space, or an unique one. The smallest length of a path joining $p1$ and $p2$ on the earth surface is the size of the minor circular arc of a great circle. A such arc is called a geodesic, that we note $d(p1, p2)$ its size. We put $\alpha \in [0, \pi]$ is the smallest central angle, to avoid all ambiguities, because $p1$ and $p2$ belong to a circle, and define 2 circular arcs: the minor one and the major one, respectively with a size $d(p1, p2)$ and $2\pi R - d(p1, p2)$. If $\alpha = 0$, then $p1$ and $p2$ are confused, and if $\alpha = \pi$, then $p1$ is the antipode of $p2$ (diametrically opposite, and reciprocally).

Smallest distance between 2 points on the earth surface

So:

\[d(p1, p2) = R\;\alpha\]

With the formula of scalar product, we can find $\alpha$, because:

\[p1\,.\,p2 = ||p1||\,||p2||\,\cos(\alpha) = R^2 \cos(\alpha)\]

But also, in the cartesian coordinates system, with $( 0, 0, 0 )$ the center of the earth, we have that a point $p = (\text{lat}, \text{long})$ on the surface of the earth has for cartesian coordinates:

\[p = \begin{pmatrix} R\cos(\text{lat})\cos(\text{long}) \\\ R\cos(\text{lat})\sin(\text{long}) \\\ R\sin(\text{lat}) \end{pmatrix}\]

Indeed, even if it means changing reference points, we can always by symmetry come back to this diagram:

Cartesian coordinates on earth

Then, we have also:

\[\begin{align*} p1\,.\,p2 &= R^2\left(\sin(\text{lat1})\sin(\text{lat2}) + \cos(\text{lat1})\cos(\text{lat2})(\cos(\text{long1})\cos(\text{long2}) + \sin(\text{long1})\sin(\text{long2})\right) \\\ &= R^2\left(\sin(\text{lat1})\sin(\text{lat2}) + \cos(\text{lat1})\cos(\text{lat2})\cos(\text{long1} - \text{long2})\right) \end{align*}\]

So:

\[\cos(\alpha) = \sin(\text{lat1})\sin(\text{lat2}) + \cos(\text{lat1})\cos(\text{lat2})\cos(\text{long1} - \text{long2})\]

By applying the function $\text{arccos}: [-1, 1] \rightarrow [0, \pi]$ the reciprocal bijection of $\cos$ on $[0, \pi]$, we get the smallest center angle $\alpha$, and:

\[\boxed{ d(p1, p2) = R\,\text{arccos}\big(\sin(\text{lat1})\sin(\text{lat2}) + \cos(\text{lat1})\cos(\text{lat2})\cos(\text{long1} - \text{long2})\big) }\]

For a historical reason and numerical computation often the haversine fonction $\text{hav}: \mathbb{R} \rightarrow [0, 1]\quad x \mapsto \sin^2\left(\frac{x}{2}\right)$.

We have $\cos(\alpha) = 1 - 2\text{hav}(\alpha)$, so:

\[1 - 2\text{hav}(\alpha) = \sin(\text{lat1})\sin(\text{lat2}) + \cos(\text{lat1})\cos(\text{lat2}) - 2\text{hav}(\text{long1} - \text{long2})\cos(\text{lat1})\cos(\text{lat2})\]

That is to say:

\[\begin{align*} \text{hav}(\alpha) &= \dfrac{1 - (\sin(\text{lat1})\sin(\text{lat2}) + \cos(\text{lat1})\cos(\text{lat2}))}{2} + \text{hav}(\text{long1} - \text{long2})\cos(\text{lat1})\cos(\text{lat2}) \\\ &= \dfrac{1 - \cos(\text{lat1 - lat2})}{2} + \text{hav}(\text{long1} - \text{long2})\cos(\text{lat1})\cos(\text{lat2}) \end{align*}\]

Thus, we get the harvesine relation:

\[\boxed{ hav(p1, p2) = \text{hav}(\text{lat1} - \text{lat2}) + \text{hav}(\text{long1} - \text{long2})\cos(\text{lat1})\cos(\text{lat2}) }\]

Where $hav: \left(\left[-\frac{\pi}{2}, \frac{\pi}{2}\right] \times ]-\pi, \pi]\right)^2 \rightarrow [0, 1]$ is defined by overloading of $hav$ on $[0, \pi]$ by a change of variables $hav(p1, p2) = hav(\alpha)$, because the couple $(p1, p2)$ defines a unique $\alpha$ (it is not reciprocal).

By using $\text{archav}: [0, 1] \rightarrow [0, \pi]$ the reciprocal bijection of $\text{hav}$ on $[0, \pi]$, we have for $\alpha \in [0, \pi]$:

\[d(p1, p2) = R\,\text{archav}(\text{hav}(\alpha))\]

In addition $\forall x \in [0, 1]$:

\[\text{archav}(x) = 2\,\text{arcsin}(\sqrt{x})\]

And by putting $y = \text{arcsin}(x)$ and $x = \sin(y)$ with $\tan(y) = \sqrt{\dfrac{\sin^2(y)}{1 - \sin^2(y)}}$ we have for $x \neq 1$:

\[\text{arcsin}(x) = \text{arctan}\left(\sqrt{\dfrac{x^2}{1 - x^2}}\right)\]

So:

\[d(p1, p2) = 2\,R\,\text{arctan}\left(\sqrt{\dfrac{\text{hav}(p1, p2)}{1 - \text{hav}(p1, p2)}}\right)\]

In order to have a valid formula for the case where $\text{hav}(p1, p2) = \text{hav}(\alpha) = 1$, that is to say for $\alpha = \pi$, we introduce the function:

\[\text{atan2}: \mathbb{R}^+ \times \mathbb{R}^+ \backslash \left\{0, 0\right\} \quad (x,y) \mapsto \left\{ \begin{array}{ll} \text{arctan}\left(\frac{x}{y}\right) & \text{if } y > 0 \\\ \frac{\pi}{2} & \text{if } y = 0 \end{array} \right.\]

So:

\[\boxed{ d(p1,p2) = 2\,R\,\text{atan2}\left( \sqrt{\text{hav}(p1, p2)}, \sqrt{1 - \text{hav}(p1, p2)} \right) }\]

We can check that the coherence of the formula where $d$ is well a metric distance, it is a positive function, symetric $d(p1, p2) = d(p2, p1)$, and homogeneous $d(p, p) = 0$ respecting the triangle inegality.

In practice, the radius of the earth is $R = 6371\,km$, and the latitude and longitude are given often in decimal degree and not in radian.

GNU privacy guard

2020-05-15T00:00:00+00:00

First, it is important to distinguish: PGP (Pretty Good Privacy), OpenPGP and GnuPG also named GPG (Gnu Privacy Guard). Phil Zimmerman made up PGP in 1991 as a way for people to communicate without eavesdropping. Going further back into the 1990s, there used to be a law in the US that restricted the export of cryptographic technology outside the US until 1992 after the World War II. Due to this patent and license around the source code and specification of the proprietary product PGP, it was not always practical for international use. That’s why the OpenPGP Working Group was formed within the Internet Engineering Task Force (IETF) to define a free standard protocol OpenPGP inspired by PGP with RFC4880 and after the end of the US restriction to allow to everyone to implement a solution. In June 2010 Symantec Corporation bought PGP Inc., but it is acquired by Broadcom Inc in 2019 that owns a propriety implementation of PGP now. GnuPG is a free implementation of the OpenPGP standard very used in the world.

GnuPG proposes services as authentication, confidentiality, compression, compatibility and segmentation with a hybrid cryptosystem combining the efficiency of symmetrical cryptosystem and the convenience of asymmetrical cryptosystem. Particularly, you can:

sign resources (electronic document, Git commit …)
encrypt/decrypt resources (email, sms, message, filesystem partition, stored data …)

References:

Java ecosystem

2020-05-04T00:00:00+00:00

The Java language is initiated by James Gosling, Mike Sheridan and Patrick Naughton is June 1991. After several years, the company Sun Microsystems released the first version of Java. Here its principles:

It must be simple, object-oriented, and familiar.
It must be robust and secure.
It must be architecture-neutral and portable.
It must execute with high performance.
It must be interpreted, threaded, and dynamic.

Java logo

That is to say:

Java is C++ without the guns, clubs and knives.

James Gosling, one of inventors of Java language

Java is a most famous programming language, it is in the top 10 of the languages the most present on the Internet resources and with the highest number of contributors respectively with the TIOBE index that counts the search engine hits, and the octoverse annual report of Github that counts the number of unique contributors of java projects.

But the mascot of Java is Duke that plays the role of a software agent that performed tasks for the user:

Duke the Java mascot

The best resource about Java is surely the Effective Java book with its different editions:

Learning the art of programming, like most other disciplines,
consists of first learning the rules and then learning when to break them.

Joshua Bloch, author of Effective Java

Effective Java book

But Sun Microsystems had an issue a business model, and Oracle acquired Java in January 2010. Now the project has 2 main versions, an open source version with OpenJDK and commercial version with Oracle JDK. These both Java Development Kit are available respectively at https://openjdk.java.net and https://www.oracle.com/java.

There are several JDK vendors among AdoptOpenJDK, Amazon Correto, SAP, Red Hat, Azul, IBM. The most of the vendors have a project based on OpenJDK. The Java Community Process (JCP) provides a Technology Compatibility Kit (TCK) to certify whether a JDK build meets the Java standards. Once a build passes the TCK, it can be referred to as Java SE compatible. The certification process requires a commercial license from Oracle.

You can see, what is the usage of the Java ecosystem in terms of IDE, Java vendors, Java version, Java with the Snyk company survey, or the Jrebel company survey.

To follow the evolution of the ecosystem, you can subscribe to the Java magazine.

References:

Development environment

2020-03-04T00:00:00+00:00

It is a good practice to isolate the used technologies for your development environment, because:

Your operating system could already have an installed version of this technology for its own needs, and you don’t want touch that for not break that.
Your development environment should be close to your production environment, that is not necessary the same as your local environment
Your projects use certainly several versions of the same technology, you don’t want to have conflicts between them

That’s why you need to have a tool in order to build and isolate your environment development, and to can switch between these different environment.

Of course, it depends on which kind of projects you work. You don’t need to have always a full isolated environment by setting up another physical machine (bare metal), a virtual machine or a container. Because, the computer ecosystem is composed of layers.

If you use only a high-level layer for writing applications in Python, Java or Ruby …, you can use a tool of version management as Pyenv, Sdkman and Rvm, it is sufficient. Like that you can use your project language in the correct version for your project, directly on your host machine that you be on Windows, WSL, macOS, or a Linux distribution and regardless of the environment ecosystem … because these languages are cross platform. You need also that related dependencies libraries be installed in dedicated locations, but these tools do that as well.

If you need to have other services like a database, a email service, a message brokers, machine learning platform, perhaps you can use a pre-built staging environment for each service, running on your own machine, or on a cluster with the cloud computing paradigme if the services are too complex to set up or too greedy in resources. Else often, you can use mock-ups for each service respecting the contract of interface.

This problematic is the same for the test environment, or the pre-production environment and prod environment. In addition, for security reason, these environments should be isolated and partitioned with a segregation of permission. It is another subject.

Gafa vs Batx

2019-11-29T00:00:00+00:00

The business world is split into 3 regions with the acronyms: AMER, EMEA, APAC. But the web and computer worldwide market is bi-polaire due to the Internet censorship in China.

Business regions and computer technology areas

So the behaviours of consumers are different according to they are in China or not. This Oligopoly constraints the chinese inhabitants to use the digital chinese services, or to be in the illegality by using a Virtual Network Private (VPN) to use other blacklisted services. So chinese government controls the chinese digital market with its own private companies.

In China we have a gang of four with Baidu, Alibaba, Tencent, Xiaomi (BATX), and for the rest of the world there is the big four tech companies Google, Apple, Facebook, Amazon (GAFA):

BATX & GAFA

Hence, this market has really 2 groups of companies on a B2B or/and B2C position in function of geolocation. Baidu, Alibaba, Tencent, Xiaomi for the APAC hub mainly where the China has a central role in terms of economical weight, and Google, Apple, Facebook, Amazon for the AMER and EMEA hubs.

On the hardware side, Apple and Xiaomi control the market of smartphones, to allow an end user to enter in the digital consummation. On the software side, your digital experience consists of searching with engine of Google or Baidu, buying with e-commerce platform of Amazon or Alibaba, and communicating with social network of Tencent or Facebook, that have respectively instant messaging Messenger & Whatsapp and WeChat. For spending money online, WeChat is in competition with Apple Pay and Google Pay.

The cloud computing and hosting is managed by Google Cloud Platform, or Amazon Web Service (AWS), and in China there is Alibaba Cloud, also known as Aliyun.

After the cloud infrastructure, and the web services, each main actor wants to control the markets of Artificial Intelligence (AI) and of Internet of Things (IoT) what compose the new business opportunities. Of course a lot of companies are forgotten in the artificial classification BATX vs GAFA as (without quoting all other ones). Here are some of them with TikTok too now:

Other big companies

Trace a request path

2019-07-14T00:00:00+00:00

What are the used path of HTTP request generated by your web browser to load a web page between your computer and a server web? I suppose each time DNS resolution is done, that is to say the web browser knows IP server.

Request path from your laptop to web server

Internet is a network of routers. When your web browser is sending one HTTP request your computer uses by default the destination 0.0.0.0 if no one sub-network matches with your original destination IP address in its route table.

$ route -n
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.254   0.0.0.0         UG    600    0        0 wlp6s0

Here the gateway 192.168.1.254 is my Internet box related to network interface wlp6s0.

$ nslookup 192.168.1.254
Server:		127.0.1.1
Address:	127.0.1.1#53
name = bbox.lan.

Your request is sent to box of your Internet Service Provider (ISP), for me it is a French ISP: Bouygues Telecom. Leaving your Local Area Network (LAN) to the Wide Area Network (WAN) of Internet, thanks to Network Address Translation (NAT), your box replaces IP address source in the header of your HTTP request (equals to your private IP, see ifconfig wlp6s0) by its public IP address (see dig +short myip.opendns.com @resolver1.opendns.com), and keeps in memory this matching only for this request in order to it comes from your computer by doing reverse translation.

The Time to live (TTL) of your request allows to know how many maximally your request can cross routers before arriving at destination. It prevents infinite cycle of routers in the network, by default a value of 64 is recommended depending of diameter of Internet. By modifying this value from 1 to N, you can get the IP address of each router on the path of your request: it will be the IP source of the response of your request HTTP.

Time to Live (TTL)

For example, you can get TTL minimum by successive ping commands, see details in n_routers.sh bash script. Each router must decrement TTL value and if TTL is 0, it refuses to transmit message. The minimum TTL is length path in the network to reach the server (by excluding source machine for number of nodes or is directly number of edges). Here one time to live of 4 is insufficient to reach glegoux.com from my machine location:

$ ping -c 1 -t 4 glegoux.com
PING glegoux.com (195.164.48.181) 56(84) bytes of data.
From 10.125.135.205 icmp_seq=1 Time to live exceeded

--- glegoux.com ping statistics ---
1 packets transmitted, 0 received, +1 errors, 100% packet loss, time 0ms

This is valid for outward network path your TTL must be sufficient, you can see your current TTL cat /proc/sys/net/ipv4/ip_default_ttl or with ping -c 1 localhost (for Linux default value is 64), and change that without persistence sudo sysctl net.ipv4.ip_default_ttl=128. Targeted server must have also sufficient TTL for return network path, this value is decremented (until value 56 here, but we don’t know initial value a priori):

$ ping -c 1 glegoux.com
PING glegoux.com (195.164.48.181) 56(84) bytes of data.
64 bytes from 10.125.135.205.rev.poneytelecom.eu (10.125.135.205): icmp_seq=1 ttl=56 time=13.8 ms

--- glegoux.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 13.821/13.821/13.821/0.000 ms

While ping command uses ICMP protocol, the system command traceroute realizes that directly with TCP protocol more easily. If there are stars instead of one IP address it means router refused to give its identity.

There is no rule to guarantee that outward path and return path are identical. This is sometimes useful to get traceroute from both ends of your network. Here I am considering only outward path.

By combining that with free IP locator like :

You can locate request path via Application Programming Interface (API) with the usage of pre-computed databases of geolocalisation of these free IP locators.

It is important to keep in mind that IP locator is not 100% accurate, only 95% for the best ones. And behind one IP address can hide a web proxy and a virtual private cluster preventing to locate really the destination. Thanks to Regional Internet Registry (RIR) and static address IP of ISPs, IP locators can know the location of some IP addresses easily.

Map of Regional Internet Registry

But an constant update and smart analysis of the network is required to locate any IP address (IPv4 or IPv6), because of dynamic IP addresses and moving connected objects for instance in the Internet of Things (IoT) or mobile networks. In addition, this location could not be too precised to respect private life of each Internet surfer, even if this one shares its location via its web browser.

With this Proof of Concept (PoC), you can run get_route.sh glegoux.com to get IP locations in csv file route.csv and see the result via Google Maps (just get your own API token) and a python web server by executing httpserver.sh and by visualizing index.html in your favorite web browser.

Geo-location of request path from Paris to glegoux.com

Java champion

2019-05-24T00:00:00+00:00

Java champion is ambassador of Java ecosystem for the world! It is enough hard to become Java champion.

Here qualities of Java champion:

leader
technical luminary
independent-minded and credible
involved in Java technology
influencer

Currently in the world, there are 282 Java champions in 2019 knowing that in 2012 there are more that 10 millions of Java developers. That is to say about 1 per 30 thousands of Java developers. Here you can see repartition of Java champions in the world:

Java champions in the world (Source: https://github.com/aalmiray/java-champions)

You can write to Java champions:

javachampions@groups.io

References

Lighthouse

2019-04-01T00:00:00+00:00

Lighthouse is an automated tool for improving the quality of web pages by auditing and analyzing your web pages and give you metrics and advices. You can run it against any web page, public or requiring authentication. It creates reports for performance, accessibility, progressive web apps, and more. It is an open-source project written in Javascript.

Logo of Lighthouse

You can run Lighthouse as a Chrome Extension, from the command line, or as a Node module. You give Lighthouse a URL to audit, it runs a series of audits against the page, and then it generates a report on how well the page did. From there, use the failing audits as indicators on how to improve the page. Each audit has a reference doc explaining why the audit is important, as well as how to fix it.

Lighthouse reports

The reports are divided into 4 categories:

Progressive Web App: tests if your app is corect progressive web app.
Performance: measures if the page is loaded quickly.
Accessibility: checks if the content of your page is accessible.
Best Practices: checks if good practices are applied.

The goal is to have a score of 100 for each category and for each web page of your website:

Lighthouse scores by report category

You can export report under HTML or JSON format, then import these format or see directly report due to Lighthouse Viewer (online version). You can install on-premises version directly from the Git repository if Lighthouse Viewer .

Export, import and view Lighthouse report

Lighthouse architecture

The workflow of Lighthouse core is described by the following chart:

Architecture of Lighthouse Core

See details of the architecture.

The project Lighthouse has 5 main components:

lighthouse-cli: allows to call lighthouse services from command line.
lighthouse-core: allows to generate report to measure the quality of your app at some urls.
lighthouse-extension: allows to call lighthouse services from an Google Chrome extension.
lighthouse-logger: allows common logs.
lighthouse-viewer: allows to see report in your web browser or to import reports under JSON or HTML format.

Lighthouse installation

Before installing Lighthouse, we must have and check the versions of your tools:

Google Chrome : google-chrome --version
Node : node --version
NPM : npm --version

For the recent versions of lighthouse, you can use Canary version for Google Chrome.

Google Chrome, Chromium and Canary version

Reminder you in a nutshell that Google Chrome is the Chromium open source project built, packaged, and distributed by Google. You can see a few differences . It has 4 versions Stable, Beta, Dev, Canary for MAC, Windows and Linux for desktops and Android, iOS for smart phones and tablets.

Example

For example, I tested Lighthouse on this blog. Suppose that you have this file urls.txt containing one URL per line:

https://glegoux.com/
https://glegoux.com/blog/
https://glegoux.com/blog/articles/
https://glegoux.com/about/
https://glegoux.com/contact/
https://glegoux.com/contact/email/

Run this bash script to get one report by url:

i=0
for url in $(cat urls.txt); do
  echo -n "--"
  echo "$i ${url}" | tee -a matching_url_report.txt
  echo -n "--"
  lighthouse "${url}" --output=json --output-path="report-$i.json"
  $((i++))
done

Then run this python script lighthouse.py:

python3 lighthouse.py urls.txt reports.csv

See code.

You get this csv file reports.csv. Visualize with matplotlib library:

Charts of scores for a set of URLs

url	date	pwa	performance	accessibility	best-practices
https://glegoux.com/projects/	2017-08-17	45.45	46.12	91.43	69.23
https://glegoux.com/blog/	2017-08-17	45.45	48.71	91.43	69.23
https://glegoux.com/home/	2017-08-17	36.36	36.53	88.57	69.23
https://glegoux.com/snippets/	2017-08-17	45.45	40.88	91.43	61.54
https://glegoux.com/articles/	2017-08-17	45.45	46.12	88.57	69.23
https://glegoux.com/about/	2017-08-17	36.36	40.18	88.57	69.23

References

Kerberos in 5 minutes

2019-03-23T00:00:00+00:00

Kerberos is a protocol of communication to guarantee a part of the security of your information infrastructure. Among the key concepts of security of the CIA triad: Confidentiality, Integrity, and Availability. Kerberos allows setting up communications with confidentiality, integrity and non-repudiation, it is an Authentication and Authorization System (AAS), although the step of authorization is very limited, it is essentially an authentication system.

This article is an introduction to Kerberos ecosystem. It can be interesting for a manager to understand key principles of Kerberos, a non-technical person to have a overview of this technology, or for technical person particularly with DevOps skills to increase action field and go further if necessary.

What is Kerberos?

The goal of Kerberos is to secure a communication channel over UDP and TCP by proving mutually the identity of each entity: transmitter and receiver. Kerberos can also encrypted the communication by exchanging session key securely.

Kerberos allows to secure communications on untrusted networks but where each node is trusted

Massachusetts Institute of Technology (MIT)

Kerberos is built by design from an architecture based on the symmetric key cryptography to authenticate, and to encrypt communication. It requires a trusted third party to guarantee a mutual authentication with a system of secrets and renewable tickets.

Kerberos’ name comes from Greek mythology, the three-headed guard dog of Hades.This technology is invented at MIT in the 1980s, became an IETF Standard in 1993. United States governement classified Kerberos protocol as “Auxiliary Military Equipment” and banned its export until 2000. It continues being used largely today. The important Requests for Comments (RFC) to understand Kerberos are rfc4120 and rfc6649, see all Kerberos RFCs.

The MIT Kerberos Consortium ensures development and lasting quality of the Kerberos project since 2007 with past and present sponsors of Apple, Google, Oracle, Stanford University, Microsoft … but less active now.

Kerberos is a very used in the Hadoop ecosystem to secure a environment Big Data for Spark, Hadoop YARN, Hadoop Mapreduce, or Hadoop Distributed File System… It is used by lot of companies as a Single Sign On solution at large to access to web UIs/APIs, databases, services, servers with SSH, or your local system login of your workstation and not only web HTTP applications, for any operating system. But it is also famous in the Microsoft world by the Active Directory cornerstone of your clusters of Windows servers.

The current widespread implementations are MIT Kerberos, Heimdal, and Active Directory. These are native Kerberos implementations respectively for Linux, macOS, and Windows. But they are built also for other operating systems than their native one. For most usages, these implementations are compatible between them client-server or server-server for Kerberos communication protocol, but the ticket credentials and keytabs are not standardized by the RFCs. Some implementations are diverged on some edge aspects, or propose extra features.

The historical implementation is MIT Kerberos, because it has been written by the founders of Kerberos Steve Miller and Clifford Neuman.The first implementation release was in 1987. New releases are pushblished until present with continuous improvements. The latest MIT Kerberos version is Kerberos V5 written in Language C under MIT License. You can see all available releases. The project is open source project published on a git mirror repository on GitHub by GitHub Organization krb5. There are a online documentation and short guide pdf.

MIT Kerberos

Another interesting implementation of Kerberos is in Java: Apache Kerby. The Java ecosystem integrates very well Kerberos, it is in the standard library. A lot of Java projects use Kerberos.

See Frequently Asked Questions (FAQ).

Pros and cons

You can read this paper: Why is Kerberos a credible security solution?

Pro’s

There is better performance when using of Kerberos. Because it uses symmetric key operations, which are orders of magnitude faster than public key operations used by SSL.
User management is simpler. For example, revoking a user can be done by simply deleting the user from the centrally managed Kerberos KDC (key distribution center). Whereas in SSL, a new certificate revocation list has to be generated and propagated to all servers
Thanks to its ticket system, Kerberos allows temporary access if a key session is stolen.
The passwords is never sent through network (even ciphered, robust against Man In the Middle attack).
Kerberos allows a mutual authentication, when a client is connected to a service. The client knows which is the service and vice versa unlike SSL certificate where the server does not know a priori the client.

Con’s

A brute force attack on Key Distribution Service (KDC) could compromise the entire network, since it contains all the passwords (in an encrypted version, but still). So the security of the KDC is essential.
Kerberos assumes that the host it is running on is safe, so if an attacker can get hold of a client it can steal tickets and during the lifetime of the ticket do the same things a normal user could. On multi-user systems the above point even becomes more important. Meaning that the different users should not be able to access each others files and memory.
Since anybody can request a (TGT) of any user, and that ticket is encrypted with the user’s secret key (password), it is simple to perform an offline attack on this ticket by trying to decrypt it with different passwords. Kerberos 5 introduced to solve this problem.
Kerberos is very dependent on the Domain Name Service (DNS), a service principal should have both: DNS lookup and reverse DNS lookup working. If your DNS Server crashed your authentication system is out.
The set up of Kerberos can be more constraining than another authentication system.

How it works?

Imagine that a client wishes to access to a service through a Kerberos Authentication. The Kerberos authentication is divided into 3 main steps:

Getting a Ticket-Granting Ticket (TGT) from Authentication Server (AS)
Getting a Ticket-Granting Service (TGS) from Ticket-Granting Server (TGS)
Accessing to the Service

Here client (identified as user) and service are already registered in the Kerberos database in the same Realm (understanding same Kerberos perimeter), respectively as User Principal Name (UPN) and Service Principal Name (SPN).

Kerberos authentication architecture

You can use Lightweight Directory Access Protocol (LDAP) as principal database to store users and groups of your organization. Now you have a nice Authentication, Authorization and Accounting System (AAAS) :

Authentication is managed by the KDC to guarantee the identity of an entity.
Authorization is controlled by the Service directly to authorize access (you can also use Access Control List (ACL) for other permissions)
Accounting is allowed by LDAP server.

Kerberos authentication workflow with LDAP

How to test?

I suppose that you are comfortable with docker containers and command-line interface under Linux Operating System as Ubuntu. You can test directly Kerberos ecosystem in few commands.

To do that, just consult this Open Source project kerberos-docker.

Kerberos/Docker icon

This servers cluster is deployed automatically with only one command.

Kerberos architecture on docker for Kerberos/Docker project

Thanks to GSS-API implementation in C, OpenSSH use one method based on Kerberos authentication: gssapi-with-mic. That is a Message Authentication Code (MAC) methods renamed in Message Integrated Code (MIC) for disambiguation MAC address. Thus the suffix with-mic is used. But SSH continue to manage authorization (using Linux-PAM or/and AllowsUser, Allowsgroup,PermitRootLogin in /etc/ssh/sshd_config), and accounting is managed by the Operating System.

The layer of access authorization provided by Kerberos is really minimal and consists in a ~/.k5login file and a few configurations. See details here. By default Kerberos applies a default authorization rule: if host H is in realm R, the Kerberos principal u@R is allowed access to the account u@H. You change the configuration of SSH server with KerberosAuthentication yes to connect directly the service without doing kinit. And you can use also another way to use the PAM module of Kerberos pam_krb5.so thanks to the configuration UsePAM yes.

For developing your own kerberized application, you can read Best Practices for Integrating Kerberos into Your Application.

Go further

To go further, you should know that with Kerberos you can do Delegation of authentication, Servers’ replication and redundancy, Cross Realm Authentication. And it is important to distinguish Kerberos with SSL/HTTPS, SSH or JWT, Radius and Diameter.

Annexes

Key Concepts

Authentication: Action of checking that an entity is really this entity without impersonating another entity.

Authorization: Action of verifying that an entity have sufficient rights to access a system.

Accounting: Management of defined groups and users.

General concepts

Mutual authentication: It is also called two-way authentication, is a process or technology in which both entities in a communications link authenticate each other. In a network environment, the client authenticates the server and vice-versa.

AAS: Authentication and Authorization System.

AAAS: Authentication, Authorization, Accounting/Auditing System.

Client: An entity that can obtain a ticket. This entity is usually either a user or a host.

Host: A computer that can be accessed over a network.

DNS: The Domain Name System is a hierarchical decentralized naming system for computers.

Service: Any program or computer you access over a network. Examples of services include “host” (a host, e.g., when you use telnet and rsh), “ftp” (FTP), “krbtgt” (authentication; cf. ticket-granting ticket), and “pop” (email).

RFC: A Request for Comments (RFC) is a type of publication from the Internet Engineering Task Force (IETF) and the Internet Society (ISOC), the principal technical development and standards-setting bodies for the Internet.

LDAP: Lightweight Directory Access Protocol is a directory service protocol that runs on a layer above the TCP/IP stack. It provides a mechanism used to connect to, search, and modify Internet directories. It is essentially a database that contains information about users and groups in a tree structure

HDFS: Hadoop Distributed File System is a Java-based file system that provides scalable and reliable data storage, and it was designed to span large clusters of commodity servers.

SSO: Single Sign On is a method allowing to an user to access to several applications thanks to one unique authentication.

OpenSSH: Implementation Open Source of Secure SHell (SSH) is a cryptographic network protocol for operating network services securely over an unsecured network.

ACL: Access Control List allows to manage permissions attached to an object.

PAM: A Pluggable Authentication Module allows to standardize security layer dynamically though system of same customizable modules for all applications.

Kerberos concepts

Kerberos: In Greek mythology, the three-headed dog that guards the entrance to the underworld. In the computing world, Kerberos is a network security package that was developed at MIT.

KDC: Key Distribution Center. A machine that issues Kerberos tickets composed of Ticket-Granting Server (TGS), Authentication Server (AS) and database instances.

AS: Authentication Service gives Ticket-Granting Ticket (TGT) to authenticate a client.

AD: Active Directory is a Microsoft product which makes use of a Kerberos implementation.

Keytab: Shortcut for key table a key tab is the file containing one or more keys. A host or service uses a keytab file in much the same way as a user uses his/her password.

Realm: A REALM is in Kerberos terms equivalent to a Domain in Domain Name Server (DNS) by convention in UPPERCASE.

Principal: You can understand that as UNIX username. There 2 types of principles: Service Principal Name (SPN) and User Principal Name (UPN). It is a string that names a specific entity to which a set of credentials may be assigned. It can have an arbitrary number of components: component1/component2/…@REALM The typical format of a typical Kerberos principal is primary/instance@REALM.

primary: The first part of a Kerberos principal. In the case of a user, it is the username. In the case of a service, it is the name of the service. instance: The second part of a Kerberos principal. It gives information that qualifies the primary. The instance may be null. In the case of a user, the instance is often used to describe the intended use of the corresponding credentials. In the case of a host, the instance is the fully qualified hostname. realm: The logical network served by a single Kerberos database and a set of Key Distribution Centers. By convention, realm names are generally all uppercase letters, to differentiate the realm from the internet domain. Ticket: A temporary set of electronic credentials that verify the identity of a client for a particular service (typically 8 or 10 hours of life time).

TGT: Ticket-Granting Ticket. A special Kerberos ticket to guarantee authentication that permits the client to obtain additional Kerberos tickets within the same Kerberos realm given by Authentication Service (AS).

TGS: Ticket-Granting Service or Ticket-Granting Server (depends on context). The TGS exchange is initiated by a client when it wishes to obtain authentication credentials for a given server

GSS-API: Acronym for Generic Security Service Application Program Interface (GSSAPI, also GSS-API) is an application programming interface for programs to access security services.

Session Key: Random number generated by the KDC when it builds a ticket. It is the shared secret that the KDC, the client, and the validating server have in common.

KVNO: It is an acronym for Key version number. Kerberos can hold multiple keys for a principal. To distinguish between those keys every key has version number. The number is incremented by one when the password or encryption is changed.

KRBTGT Account: This is a special account to create Ticket Granting Tickets.

Key salt: The salt is a string concatenated to the password to make sure the string2key function creates a unique encryption key, via a one-way hash. In Kerberos 5 the salt is the complete principal name including the realm. The reason the salt is used is to make sure the outcome of the one way hash is unique across realms.

References

ROBINSON Trevor (eztenia). Kerberos. Canonical Ltd. Ubuntu Article, November 2014. Link: https://help.ubuntu.com/community/Kerberos.
MIGEON Jean. Protocol, Installation and Single Sign On, The MIT Kerberos Admnistrator’s how-to Guide. MIT Kerberos Consortium, July 2008. p 62.
BARRETT Daniel, SILVERMAN Richard, BYRNES Robert. SSH, The Secure Shell: The Definitive Guide, 2nd Edition. O’Reilly Media, June 2009. p. 672. Notes: Chapter 11. ISBN-10: 0596008953, ISBN-13: 978-0596008956
GARMAN, Jason. Kerberos: The Definitive Guide, 2nd Edition. O’Reilly Media, March 2010. p. 272. ISBN-10: 0596004036, ISBN-13: 978-0596004033.
Phillip J. Windley. Digital Identity. O’Reilly Media, August 2005. p. 254.
O’MALLEY Owen, ZHANG Kan, RADIA Sanjay, MARTI Ram, and HARRELL Christopher. Hadoop Security Design. Yahoo! Research Paper, October 2009. p 19. ISBN-10: 0596008783, ISBN-13: 978-0596008789
MATTHIAS Karl, KANE Sean. Docker: Up & Running. O’Reilly Media, June 2015. p. 232. ISBN-10: 1491917571, ISBN-13: 978-1491917572.

Access to a computer from USB key

2019-03-01T00:00:00+00:00

Imagine that you are in one of the following situations:

You have forgotten your password to connect to the personal workspace of your computer, and you want to reset it.
After a wrong manipulation, you can not access the personal data of your computer, and you want to retrieve this access.
You found a computer without supervision for which you are not the owner, but you want to see the file system’s content.

With a bootable USB key on Ubuntu, you can resolve these situations by first mounting the file system of the targeted computer with all permissions, then changing the password of any local user, and finally by connecting to the targeted workspace and exploring the file system.

First, plug your USB drive into the target computer and select your key as the boot device. This will enable the target computer to boot into the Ubuntu system on your USB drive. On a DELL computer, press F12 to access this selection. This information is visible on the welcome screen of your computer’s manufacturer.

DELL welcome screen

Start by choosing the following option:

Advanced startup options

Then select the recovery mode:

Getting started in recovery mode

Launch a root shell:

Choosing the root shell

You access a device as a root user.

If you need to access the internet, select the network entry before. To verify that you have access to the web you can ping 8.8.8.8 on the Google DNS server, then make nslookup 8.8.8.8 to check that the DNS resolution works. If that does not work, add this line nameserver 8.8.8.8 to the file /etc/resolv.conf (this will be deleted when you restart the computer) and restart your service network-manager with service network-manager restart.

For the moment, the file system is only in read only, type the following command to mount the file system in read / write at the root of your key:

mount -o remount, rw /

Now you can copy the file system to an external hard drive, and then you can choose the new user password by typing:

passwd <username>

If you do not know the users on this machine, you can do ls /home. Certainly, a user of this machine corresponds to a directory name in /home/, or even view the file /etc/passwd, or else you can create a user as you are root.

Type in the new password and confirm.

You can shut down the computer.

shutdown -h now

You can now turn on the computer normally and log in with the chosen user and password.

Beware, the morality of this article is that a person physically accessing your computer does not need login/password to access your computer. If you want to protect yourself from this attack, you must encrypt your hard drive.

Well done :smile:!

How to quote sources

2019-02-14T00:00:00+00:00

The standard in force is mainly Afnor Standard Z 44-005, December 1987 (ISO 690). Here is a link of the website of the Ministry of Culture and Communication which presents in a synthetic way the standards in force for the writing of bibliographical references:

http://www.culture.gouv.fr/culture/inventai/telechar/normes.pdf

To write a blog article resources are used and must be specified in a bibliography and / or a sitemap. These groups respectively the paper resources or online like books, articles, videos, research papers, interviews …

In general, these rules are to be respected with a certain freedom, but all known information must be mentioned. Some information is mandatory or conditionally mandatory (ISBN, ISSN for example).

You can find out what an ISBN (International Standard Book Number) is:

https://www.isbn-international.org/en/

Likewise for what is an ISBN (International Standard Book Number):

http://www.issn.org/en/

Two sites are also interesting:

The website of AFNIL (French Agency for International Numbering of Books):

http://www.afnil.org/

The site of the BnF (Libraries National of France):

http://www.bnf.fr/

With these resources:

Get informed and get an ISSN
Get information and get an ISBN

Your references must be in alphabetical order at the end of your article respecting these conventions.

Book

NAME OF THE AUTHOR, First name of the author. Title of the book. Name of publisher, year of publication. Number of pages. Notes. ISBN.

Periodical article

NAME OF THE AUTHOR, First name of the author. Title of the article. Title of the periodical. year of publication. not. paging. ISSN.

Internet resource

This convention is personal. For this blog by default the links given are consulted the day of the creation of the article.

NAME OF THE AUTHOR, First name of the author. Title [online]. Editor’s name. [date of publication or update date of consultation]. Availability and access. (or just a link)

References

Getting started

2019-01-21T00:00:00+00:00

Discover the extended Markdown syntax used on this tech blog, generated from the template 🧪 Jekyll Tech blog, to write articles or to generate static web pages. It is based on Kramdown syntax with GFM parser and a light custom extension for rendering. Besides, this article is written with this syntax. At the top of the web page, click on Source to explore its source code and on Table of contents to browse the supported features. So, you can mix this syntax with existing macro and native HTML syntaxes to write any content. As this blog is highly configurable, you can build your custom macros. Each article also has metadata to inject constants of configurations.

Summary of supported features by the extended Markdown syntax

Read the documentation of the Kramdown syntax. All contents, even if unavailable with the extended Markdown syntax, can be shown thanks to embedded native content: HTML, JS, CSS, or macros: includes and tags of the Liquid template.

Feature name	Comment	Extended Markdown syntax	Existing Macro syntax	Native HTML syntax
Structure
Heading	Organize your content with part, subpart, sub-subpart, etc.	✅	❌	✅
New line & Paragraph	Organize the content in a part	✅	❌	✅
Horizontal Rule	Break between paragraph-level elements	✅	❌	✅
Note	Soon refactored! Add your notes: info, question, warning, error, success, etc.	❌	❌	✅
Textual Content
Enchanced Text	Enhance your text: italics, bold, strikethrough, inline code, highlight, superscript, subscript, etc.	✅	❌	✅
Link	Use absolute and relative links named or not and re-usable	✅	❌	✅
Emoticon	Use emoticons with shortcode or unicode	✅	❌	✅
List	Build an ordered, unordered, definition or task lists and sublists	✅	❌	✅
Table	Arrange the content in rows and columns with an optional header.	✅	❌	✅
Blockquote	Format a special sentence of paragraph to draw attention to the reader	✅	❌	✅
Quote	Soon refactored! Quote your authors	✅	❌	✅
Footnote	Put extra information at the bottom of the article referencing inside content	✅	❌	✅
Abbreviation	Shorten used word or concept by keeping the accessible meaning	✅	❌	✅
Illustration Content
Image	Soon documented!	✅	✅	✅
Video	Soon documented!	❌	✅	✅
Presentation	Soon documented!	❌	✅	✅
Diagram	Soon documented!	❌	✅	✅
Informatics Code
Programming Language	Highlight code: python, java, scala, rust, c#, c++, c, bash, ruby, go, typescript, javascript, html, css ...	✅	❌	✅
Terminal	Highlight content in a terminal	✅	❌	✅
Text File	Highlight text content: plain, json, yaml, xml, ini ...	✅	❌	✅
Mathematics Equation
In-Line Equation	Use LateX syntax for in-line equation	✅	❌	✅
Single-Line Equation	Use LateX syntax for single-line equation	✅	❌	✅
Multi-lines Equation	Use LateX syntax for multi-lines equation	✅	❌	✅
Special Content
HTML	Put native HTML when the Markdown syntax is not available, it can contain Markdown in its markups	❌	❌	✅
Comment	Comment content	❌	❌	✅
Escape	Escape meta or special characters	✅	❌	✅
Others	Create your macros, extend the project or the tools everything is open-source	❌	❌	✅

What is the Markdown syntax? It is a lightweight and popular text metalanguage that is a writing standard. It is often converted into HTML, allowing it to be easily shared on the web :spider_web:. After converting to HTML, it is decorated and animated by CSS style sheets and/or JS scripts. See more details on this Markdown Guide.

Why use Markdown? Because it is very handy to use for the reader and the writer:

Reader: The generated HTML content from the Markdown content is compatible with all web browsers that can render it for the final reader.
Writer: The Markdown content continues to be separated from the rendering. In addition, it is easy to learn and use syntax, with the pros of portability and compatibility, flexibility and customizable, and readability and maintainability. Modify its article so the content is more human-writable/readable and shorter than the HTML code, less expressive, but sufficient for a blog post.

Why use an extended Markdown syntax? Markdown has a basic syntax, that is limited for more advanced use cases (like tables, code blocks, syntax highlighting, footnotes and so one…). This extension is Markdown compatible, so all articles written in Markdown can be used directly in this blog without breaking changes, and largely used by an active community.

How it works? The files having the Markdown extension .md or .markdown are processed to HTML with this processing workflow:

Extended Markdown to HTML

The following sections give examples for each usage to create your content with the right syntax.

Structure

Heading

Make a part, a subpart, and a sub-subpart.

It is used for each section here.

Source code:

# H1
## H2
### H3


H1
======

H2
------

New line & Paragraph

Let 2 trailing spaces have an end of the line, and let an empty line for a new paragraph:

Hello,

World!

Source code:

Hello,␣␣\n
\n
World!\n

Note: \n and ␣ represent respectively a new line character and a space character.

Horizontal rule

To create a horizontal rule, use three or more asterisks ***, dashes ---, or underscores ___ on a line by themselves:

Source code:

---

Note

Add your notes: info, question, warning, error, and success etc.

INFO

Source code:

<div class="info">
INFO
</div>

QUESTION

Source code:

<div class="question">
QUESTION
</div>

WARNING

Source code:

<div class="warning">
WARNING
</div>

ERROR

Source code:

<div class="error">
ERROR
</div>

SUCCESS

Source code:

<div class="success">
SUCCESS
</div>

Textual Content

Enchanced Text

Enhance your text:

Italics: italics or italics
Bold: bold or bold
Strikethrough: ~~strikethrough~~
Bold & Italics: bold italics or bold italics
Inline code: code
Highlight: highlight
Supscript: sup^sup
Subscript: sub_sub

Source code:

- Italics: *italics* or _italics_  
- Bold:	**bold** or __bold__ 
- Strikethrough: ~~strikethrough~~
- Bold & Italics: **_bold italics_** or __*bold italics*__
- Inline code: `code`
- Highlight: <mark>highlight</mark>
- Supscript: sup<sup>sup</sup>
- Subscript: sub<sub>sub</sub>

Link

It works with absolute and relative links:

Simple Link

https://www.glegoux.com

Source code:

<https://www.glegoux.com>

Named Link

glegoux

Source code:

[glegoux](https://www.glegoux.com)

with a title:

glegoux1 (with the title glegoux's Homepage)

Source code:

[glegoux](https://www.glegoux.com "glegoux's Homepage")

Reference-style Link

You can use shortcuts for your links to use that several times on the article or to centralize links:

glegoux1 glegoux2

Source code:

[glegoux1][1] [glegoux2][1]

[1]: https://www.glegoux.com "glegoux's Homepage"

Emoticon

Use emoji, with gemoji see emoji cheat sheet.

For example: with shortcode :smile: :scream: :yum: or unicode directly 😀 😱 😋.

Source code for emoji with shortcode:

:smile: :scream: :yum:

Source code for emoji with unicode:

😀 😱 😋

List

Ordered List

item
item
item

Source code:

item
item
item

Use +, * and - indifferently.

Unordered List

item
- subitem
- …
item
- subitem
- …
item

Source code:

 * item
 * item
 * item

Definition List

item: definition
item: definition

Source code:

item
: definition

item
: definition

Task List

item
item
item

Source code:

- [x] item
- [x] item
- [ ] item

Table

TODO

Blockquote

TODO

Quote

Quote authors:

Talk is cheap. Show me the code.

Linus Torvalds, creator of the Linux operating system

Source code:

> Talk is cheap. Show me the code.

Linus Torvalds, creator of the Linux operating system

Footnote

Footnotes work mostly like reference-style links. A footnote is made of two things: a marker in the text that will become a superscript number; a footnote definition that will be placed in a list of footnotes at the end of the document. A footnote looks like this:

That’s some text with a footnote¹.

Source code:

That's some text with a footnote.[^1]

[^1]: And that's the footnote.

Abbreviation

The HTML specification is maintained by the W3C.

Source code:

The HTML specification is maintained by the W3C.

*[HTML]: Hyper Text Markup Language

Illustration Content

Image

TODO

Video

TODO

Presentation

TODO

Diagram

TODO

Informatics Code

Programming Language

The used syntax highlighter is Rouge, see the list of supported languages and lexers, with python, java, scala, rust, c#, c++, c, bash, ruby, go, typescript, javascript, html, css etc.

print("Hello world!")

Source code:

~~~ python
print("Hello world!")
~~~

print("Hello World!", quote = FALSE)

Source code:

~~~ r
print("Hello World!", quote = FALSE)
~~~

disp('Hello World!')

Source code:

~~~ m
disp('Hello World!')
~~~

System.out.println("Hello world!");

Source code:

~~~ java 
System.out.println("Hello world!");
~~~

println("Hello world!")

Source code:

~~~ kotlin 
println("Hello world!")
~~~

println("Hello world!")

Source code:

~~~ scala 
println("Hello world!")
~~~

Console.WriteLine("Hello world!");

Source code:

~~~ csharp 
Console.WriteLine("Hello world!");
~~~

print("Hello World!");

Source code:

~~~ dart
print("Hello World!");
~~~

print("Hello World!")

Source code:

~~~ swift
print("Hello World!")
~~~

println!("Hello World!");

Source code:

~~~ rust
println!("Hello World!");
~~~

std::cout << "Hello World!";

Source code:

~~~ c++
std::cout << "Hello World!";
~~~

printf("Hello, World!");

Source code:

~~~ c 
printf("Hello, World!");
~~~

echo "Hello world!"

Source code:

~~~ bash 
echo "Hello world!"
~~~

puts "Hello world!"

Source code:

~~~ ruby 
puts "Hello world!"
~~~

import("fmt")

fmt.Println("Hello world!")

Source code:

~~~ go
import("fmt")

fmt.Println("Hello world!")
~~~

print "Hello world!";

Source code:

~~~ php
print "Hello world!";
~~~

print "Hello world!";

Source code:

~~~ perl
print "Hello world!";
~~~

console.log("Hello world!");

Source code:

~~~ typescript
console.log("Hello world!");
~~~

console.log("Hello world!");

Source code:

~~~ javascript
console.log("Hello world!");
~~~

CREATE TABLE helloworld;

Source code:

~~~ sql
CREATE TABLE helloworld;
~~~

<p>Hello world!</p>

Source code:

~~~ html
<p>Hello world</p>
~~~

.hello .world {
  text-align: center
}

Source code:

~~~ css
.hello .world {
  text-align: center
}
~~~

.hello {
  .world {
    text-align: center
  }
}

Source code:

~~~ sass
.hello {
  .world {
    text-align: center
  }
}
~~~

Terminal

Have a dark terminal container:

$ command
result

Source code:

~~~ terminal
$ command
result
~~~

Text File

Highlight your text file content: plain, json, yaml, xml, ini etc.

"Hello World!"

Source code:

~~~
"Hello World!"
~~~

{"message": "Hello World!"}

Source code:

~~~ json
{"message": "Hello World!"}
~~~

hello:
  world: "Hello World!"

Source code:

~~~ yaml
hello:
  world: "Hello World!"
~~~

<?xml version="1.0" encoding="UTF-8"?>
<message>Hello World!</message>

Source code:

~~~ xml
<?xml version="1.0" encoding="UTF-8"?>
<message>Hello World!</message>
~~~

[message]
message="Hello World!"

Source code:

~~~ ini
[message]
message="Hello World!"
~~~

Mathematics Equation

Use LateX syntax, see latex-project.org.

The dollar symbol $ becomes a meta symbol, so there is a conflict with the dollar symbol use the HTML code $ to see the dollar symbol $.

In-line Equation

This equation $\int_0^1 x^2dx = \frac{1}{3}$ is in-line.

Source code:

This equation $\int_0^1 x^2dx = \frac{1}{3}$ is in-line.

Single-line Equation

This equation is single-line and centered:

\[\int_0^1 x^2dx = \frac{1}{3}\]

Source code:

$$\int_0^1 x^2dx = \frac{1}{3}$$

Multi-lines Equation

This equation is multi-line and centered:

\[\displaylines{ \int_0^1 x\,dx = \frac{1}{2} \\ \int_0^1 x^2dx = \frac{1}{3} }\]

Source code:

$$
\displaylines{
\int_0^1 x\,dx = \frac{1}{2} \\
\int_0^1 x^2dx = \frac{1}{3} 
}
$$

Special Content

HTML

TODO

Comment

If you want to put a hidden text not displayed in the article but present in the written source, that is to say, a comment, simply do an HTML comment:

Source code:

<!-- comments -->

Escape

All described multi-lines escapes can work to escape inline and single line characters.

Character by Character

Use a backslash \, see the list of all the characters that can be escaped, or HTML entities like < or >.

Inline and Single Line Characters

Instead of using multiple backslashes, use for multiple characters inline and composing a single line, these syntaxes:

`inline or single line`

But you cannot display ` inside a section delimited by `.

Multi-lines Characters

<code>inline or single line</code>

If you have to display reserved characters such as <, and > within the <code> section, the characters must be escaped using their respective HTML entity: <: <, and >: >. But you cannot escape &.

<pre>multi-lines</pre>

If you have to display reserved characters such as <,>, &, and " within the <pre> section, the characters must be escaped using their respective HTML entity: <: <, >: >, &: &, and ": ".

~~~ multi-lines ~~~

But you cannot to display reserved sequence of characters ~~~ within a section delimited by ~~~.

``` multi-lines ```

But you cannot to display reserved sequence of characters ``` within a section delimited by ```.

Others

TODO

And that’s the footnote. ↩