Software Engineering Blog - Gilles Legoux (glegoux)

Ubuntu: apt-key is deprecated

Sat, 29 Jul 2023 00:00:00 +0000

apt-key is deprecated from Ubuntu 22.04 LTS (Jammy Jellyfish) for security flaws: don’t use it anymore whatever your Linux environment. Instead of using it, consider the command gpg now. Then move your public keys relative to your APT package manager to a new location with the binary format OpenPGP, and specify the option Signed-By (signed-by) for all your used package repositories.

Let’s follow this plan together to properly use your secure package management and see if you are not infected by these security flaws:

🔒Understand the security flaws
💡 Explore its workstation environment for apt and apt-key
🔨 Use directly the gpg command to manage your new keys
➡️ Migrate your current trusted public security keys to the correct format and location
✅ Check the migration and stop the useapt-key, /etc/apt/trusted.gpg, /etc/apt/trusted.gpg.d/
❔Questions
🚀 Go further

🔒 Understand the security flaws

apt-key is a sub-command of apt to secure the management packages with asymmetric security keys. As you know, package management is a sensible area for security; it is an access door for an attacker to install malicious packages. That’s why when a package is installed, cryptography is used to validate that the installed package is really the one wishes (see Debian Wiki: SecureApt).

Security risks in package management [built by author]

Here are the milestones of the lifecycle of a package until be installed:

The package is produced on a build machine;
The package is signed on the build machine with a private key;
The signed package is uploaded to a repository (download mirror);
The client system downloads the package (with its package dependencies before, if needed) thanks to the metadata and the usage of its package manager from an authorized repository (or from cache);
The signature of the package is verified (with its package dependencies before, if needed) with the associated public key;
If the signature is valid, the package is installed.

The developers of Debian and Ubuntu realized that the apt-key command has had flaws for a long time (see Security tracker: package apt and particularly CVE-2011–3374). While apt-key was not guaranteed to work people continue to use it, that’s why now this command line has been marked as deprecated (see Fully deprecate apt-key, schedule removal for Q2/2022 (ee284d59) · Commits · APT Developers / apt · GitLab) and will be deleted. Support for the function ends with Debian 11 (Bullseye) and Ubuntu 22.04 (Jammy Jellyfish).

The problem is to move all tech ecosystems because numerous documentation or blog posts are not up-to-date on this aspect. Even tech giants like Google: Linux Software Repositories, where the usage of apt-key is still referenced in August 2023 (after the publication of this article in July 2023). After raising the alert, the correction was made in February 2024 (here).

The home page of the Google Linux Software Repositories is not up-to-date with the usage of apt-key [source]

Let’s understand an attack scenario among several possible ones. Even if, by default, APT checks the validity of the signature of each package to install, and if it is invalid, a warning message is printed, and the package is not installed, this is not sufficient because APT also needs to know which trusted public GPG key belongs to which package repository.

👀 Read more

Read the rest of the article …

History of the Generative AI

Sun, 16 Apr 2023 00:00:00 +0000

Generative AI is the state of the art of machine learning and has recently seen significant advancements with the emergence of AI ChatBots with high performance.

Generative AI for all types of content [built by author]

Let’s discover its history, from simple textual machine learning models specialized to universal large language models at a vast scale:

💬 AI ChatBots
🔥 From NLP to Multimodal GAI ChatBots based on LLM
🧪LLMs Timeline: Closed-source vs. Open-source
🎯 From Specialized Model to Generalized ML Model
📈 Scaling Laws & Performance
🏁 Technological and Economic Race
🚀 Go Further with Limited & Augmented LLMs

💬 AI ChatBots

Building generalist AI chatbots is the most complex challenge for a technology that simulates human intelligence. It is the base of the artificial human assistant. They are experiencing dazzling success with the release of ChatGPT, a new step that has been overcome after Google Home and Alexa by AWS. The technical component behind these products has disrupted and improved the ecosystem considerably, although not perfect yet (perhaps it will never be).

The interest in chatBots followed the one for deep learning from 2010. Before, it was reserved for a restrained audience. Still, it exploded in November 2022 with the release of chatGPT open to the general public with a ramp-up of 100M Monthly Active Users (MAUs) after two months, a worldwide record (4.5x faster than TikTok) and 1.6B MAUs now. At the same time, machine learning and deep learning trends have continued growing in the background for ten years.

Google Trends from 2010 [source]

Similar web for chat.openai.com [source]

To have an order of magnitude, the human brain has, on average, 100 billion neurons and 100 trillion synapses. The estimated resources used for chatGPT+ (paying version of chatGPT) use deep learning models based on GPT-4 with an estimated model size close to these values (digits kept private by OpenAI). But a human neuron/synapse is much more powerful than a deep learning neuron/synapse. The gap is still important, above on reasoning tasks on unknown problems requiring many resources.

Human brain vs. Deep learning [adapted from source]

🔥From NLP to Multimodal GAI ChatBots based on LLM

Thanks to Large Language Models (LLMs) working with Generative AI, chatBots continue getting closer to the objective of simulating human intelligence. The NLP AI models have progressively migrated from simple parsing to complex processing on the language structure, from shallow learning to deep learning, from a single-language vocabulary of words to a vocabulary of tokens with multi-language embeddings, from an RNN (LSTM) to the Transformer with multi-head attention layers approach.

NLP timeline with the Deep Learning [source]

Finally, by mixing other types of data (like images, audio, …), NLP models became multi-modal generative models where varied instructions (in input) give varied results (in output).

👀 Read more

Read the rest of the article …

From Hadoop to the modern data stack

Wed, 14 Dec 2022 00:00:00 +0000

Hadoop at Criteo

by Anthony Rabier, Staff Site Reliability Lead Engineer at Criteo
and William Montaz, Senior Staff Site Reliability Engineer at Criteo.

Criteo migrated from Hadoop 3 to Hadoop 2, with a lot of patches and in an awesome way by migrating first the runtime to Hadoop 3 for a big progressively without downtime for Hadoop YARN and HDFS without any intervention of the development teams, then by migrating the Hadoop projects for Spark, Flink, and Hadoop Map Reduce. So a vanilla distribution of Hadoop 3 has been created, running with compiled projects in Hadoop 2 and 3 with retro compatibility and tricks developed and merged by Criteo in the core project of Hadoop.

Migration from Hadoop 2 to Hadoop 3

Garmadon is an agent java deployed with all JVM processes running on the Criteo Hadoop cluster to do the metrology jobs for Spark, Flink, and Hadoop Map Reduce … You can build generic Grafana dashboards and create data lineage between your dataset and audit all operations on HDFS.

Garmadon, Java agent for the Metrology on Hadoop

Data processing at Criteo

by Miguel Liroz, Senior Staff Site Reliability Lead Engineer at Criteo
and Raphael Claude, Staff Site Reliability Engineer at Criteo

Datadoc is an internal tool used by Criteo to browse the data catalog and to provide a data catalog on each dataset.

Datadoc [built by author]

BigDataFlow is an internal tool used by Criteo written in Scala to schedule data processing jobs from an extended SQL query.

BigDataFlow extended SQL [built by author]

👀 Read more

Read the rest of the article …

How to write a README?

Sat, 10 Dec 2022 00:00:00 +0000

A README is the best place to start documenting a project. A README is normally a single-page document describing your computer project’s key aspects. This document can be read in a few minutes maximum. It is the open door for a newcomer who wants an overview of the fundamental aspects of the project.

Having a README for a computer project is a common enough thing for the code repositories. Still, the template defining what should contain this README is not standard and often depends on the sensibility of the software documentalist. Avoiding gaps in content and format allows you to align the shared knowledge of your projects in the best common possible way.

The rules for writing a README file will be described with a generic template and a pragmatic methodology agnostic to the tech stack and the project type you use.

First, let’s begin defining the type of project that you can have.

A project is a code repository; it can be composed of either one module (simple project) or several modules (complex project). This project is in an ecosystem composed of dependencies between modules of the same project or other projects. A module can be :

An application: It is deployed at runtime with an executable. No other module depends on it except optionally at runtime.
A library: It’s used by other modules that depend on it for their execution and/or build.

For example, a type of project can be a code library in (Python, Scala, or Java …), an SDK, a web app with a UI and/or an API, a machine learning model, a data processor, a set of scripts, an OPS configuration module or a scheduled job …

All projects have not necessarily needed a README respected rigorous template; it can use a degraded version in function efforts that you want to provide. Only mature projects where you want a high level of quality need it, that is to say, for a project used by the community in an intensive or critical way.

Practical guidelines

Here are explained the practical guidelines that a README should respect.

For a simple project:

The README should be a simple text file called README..
It should be written in coherent syntax (like Markdown or reStructuredText).
There should be a README at the root of the project.
According to the README variant you choose (see sections below), you need to adhere to the variant structure.

For a complex project:

The README at the root must have links to the ones of each submodule, with a sentence to describe the role of each module
It should have a README for each module
The README of each module should respect the guidelines of a simple project

Template

Here are the sections that README should contain.

Each section can contain direct or indirect information with links and should contain a title section with format H1 (i.e., ‘#’ with markdown). A section can be optional and required in function of the type of modules.

[Project Title]* [*: required]
- High-level description (with a demo link if applicable)
- It should indicate if it is a library or an application- It should indicate who is the owner of this project
Quick start* [*: required]
- How to set up the development environment, get the repository, install tech stack, …
- Link to a demo
- How to run the project
- How to run the tests
Usage [optional only if relevant and helpful]
- How to use the project (ex. API description)
- Links to help resources
- How to contribute to the project (coding style, code of conduct, …)
Main structure* [*: required]
- Link(s) to other “sub-README(s)” (only if the module has submodules)
- Links to dependencies
- Describe the file architecture of the repository
Architecture [optional only if relevant and helpful]
- Explanation of the design with a diagram and a description …
Deployment** [**: required only for a deployed application]
- Link to the deployment configuration
- Link to this history of the deployment with the changelog and the releases
- Link to the deployed application
Monitoring** [**: required only for a deployed application]
- Links to monitoring dashboards, …
References [required]
- Links to the documentation about specifications, improvements, or maintenance…
- Other useful links
License
- Link to the license and copyright with the general conditions

👀 Read more

Read the rest of the article …

Aerospike: Real-time database for the online ads ecosystem

Tue, 29 Nov 2022 00:00:00 +0000

Aerospike can replace at the same time 2 cache layers, an in-memory layer, and a persistent memory layer like Memcached and Couchbase, respectively, with awesome performance.

Especially explore the usages of this distributed key-value store for the e-commerce catalogs indispensable in the online ads ecosystem to build ad campaigns, recommend products to the right audience, and display the ads for the publishers.

As a member of catalog teams at Criteo, I was at the last meetup of Scale Warriors Aerospike, where the CTO (and founder at Aerospike) was present Srini V. Srinivasan to share new and existing features of Aerospike, and where Criteo presented its plan and strategy to migrate to Aerospike.

Aerospike

Aerospike is a key-value NoSQL database with flash memory and in-memory open source. The high-level ecosystem of Aerospike is deeply integrated with tools of the Big data ecosystem for streaming, batch, and SQL usages with Kafka, Spark, and Trino with a Java client:

The Aerospike Real-time Data Platform [source]

Aerospike is a distributed key-value store. You can access the data with a simple API:

Get(key) -> byte[]
Set(key, byte[], ttl)

The goal is to have small objects from a few bytes to 4 MiB, with low latency, 1 ms on average, and 5 ms at the 99th percentile.

Aerospike is first optimized for the OLTP paradigm because it is a KV Store. Still, you can easily do OLAP queries for other usages for more exotic queries to scan the data thanks to the data models layer for Documents, SQL, Graphs, or Time Series.

Aerospike clusters at Criteo

A DKVS plays a central role in online ads. With this storage, the infrastructure can answer to the small-time budgets required for each business use case: to bid a request for the auction of RTB (< 100 ms) or to render the ads (< 250 ms).

Time budget for the online ads [source]

The tech stack is diversified to respond to the needs of online ads, and a DKVS is insufficient. It is only a part of the ecosystem:

Data stack at Criteo [source]

The data stored in the KV stores must be persisted and loaded in memory to access long-term and quick readings.

👀 Read more

Read the rest of the article …

Learning PySpark with Google Colab

Tue, 15 Nov 2022 00:00:00 +0000

Learning Apache Spark with a quick learning curve is challenging. Discover distributed computation and machine learning with PySpark, with several tutorials until building your movie recommendation engine.

Links: GitHub | Tutorials quick start | Dataset

Let’s discover how to use PySpark on Google Colab with accessible tutorials.

As a teaching fellow with David Diebold about Systems, paradigms, and algorithms for Big Data for the international Master IASD (graduate degree M2) for the French Dauphine Paris University member of the PSL University, I needed to organize sessions of tutorials for the students on the distributed computation with Apache Spark.

Fast, flexible, and developer-friendly, this data-distributed processing framework has become one of the world’s most significant. Before teaching the features provided by Spark, we had to choose which language and platform our learners could run the tutorials we prepared. We chose the tech stack: PySpark with Google Colab.

PySpark vs. Spark

PySpark allows interaction with Spark in Python. It gives a better learning curve than Spark (written originally in Scala). Even though it is less performant for a production world using Py4J to interact with the JVM of Spark, it gives sufficient performance (sometimes close to Spark with Java/Scala) to experiment with distributed data science and machine learning.

PySpark is an interaction of Spark with Python [built by author]

PySpark with Python remains largely the preferred language for Notebooks.

Percent of commands on the Databricks platform in each Spark language [source]

The support of PySpark is already excellent and continues to be improved with the future version of Spark 3+.

Python is easier to learn than Scala and has a mature ecosystem for applied mathematics.

For these reasons, we preferred teaching PySpark over Spark. Moving from one to another is easy, only the cost for the learners to become familiar with the Scala/Java ecosystem for advanced use.

👀 Read more

Read the rest of the article …

The Apache Software Foundation for the open-source

Sun, 13 Nov 2022 00:00:00 +0000

ASF is one of the larger foundations for open-source projects in the tech world. The official website https://apache.org/ is exhaustive of useful information.

Metrics about the Apache Foundation [source]

Apache Organization

The organization of ASF is complex enough, here are useful like to understand how it works:

Apache community

See https://community.apache.org/

Apache Way: Core values

It is a set of methodologies and values designed to enable disparate communities of volunteers to work collaboratively over time.

See https://www.apache.org/theapacheway/

Apache News

ApacheCon is a conference that is a must for anyone who works deeply with Apache-based software programs.

ApacheCon [source]

See the old conferences: https://archive.apachecon.com/

See all the Apache events:

https://events.apache.org/event/

Apache Blog posts:

Apache Reports:

https://www.apache.org/foundation/reports.html

Apache sponsors

See https://apache.org/foundation/thanks

👀 Read more

Read the rest of the article …

Apache Spark: Ecosystem overview with Apache Hadoop YARN and HDFS

Wed, 02 Nov 2022 00:00:00 +0000

Apache Spark: ecosystem overview with Apache Hadoop YARN and HDFSApache Spark™ is a multi-language engine for executing data engineerinApache Spark: ecosystem overview with Apache Hadoop YARN and HDFSg, data science, and machine learning on single-node machines or clusters. It is largely used in the industry for distributed computation of data at scale.

This project is open-source under the Apache software foundation, created by the founders of the company Databricks that provides cluster and support for Spark.

Let deep dive on the architecture of Apache Spark on YARN in a distributed ecosystem of containers and Java VMs.

This architecture need to address the computation and the storage at high scale to manipulate big volume of data with an efficient processing.

Table of contents

Spark ecosystem
Spark architecture through Hadoop
Spark engine for computation and storage
Go with Spark in production
Example of Spark application at Scale
Go further
What’s next
Thanks to
Glossary
References

Let’s start with the basics, you can jump straight to the sections that interest you, otherwise see vertically how this whole ecosystem works.

Spark ecosystem

Data pipelines

Spark allows building data pipeline in batch mode and the streaming mode.

Lambda architecture with Spark [source]

Here, you see a lambda architecture, but to associate online and offline processing, you can have a kappa or zeta architecture on your data pipeline. You will see it in another article.

Spark vs Spark Streaming [built by author]

The streaming mode of Spark is more micro-batch approach. Other framework like Kafka Streaming, Faust Streaming or Apache Flink are more used to process data streams. Spark is forceful to process data batch overall.

APIs

Mainly, Spark can run with several APIs, one for each following programming language Scala/Java, Python and R, respectively with Spark, PySpark and SparkR.

Different types of Spark engines [built by author]

In practice, Spark and PySpark are more used than SparkR. For the experimentation, PySpark is more helpful, and for the performances Spark is better.

Connectors

Spark has many connectors to write to different types of storage.

Apache Spark’s ecosystem of connectors [source]

Here, we are going to speak a bit more about the connection with Apache Hadoop HDFS.

Spark SQL

In addition, you can also run Spark with SQL. With a SQL request, you can also access to heterogeneous data then combine them in memory:

Spark SQL overview [source]

The Catalyst & Tungsten engines allows optimizing the execution plan of the query:

Catalyst [source]

PySpark

In the Python driver program, SparkContext uses Py4J to launch a JVM and create a JavaSparkContext. Py4J is only used on the driver for local communication between the Python and Java SparkContext objects; large data transfers are performed through a different mechanism.

👀 Read more

Read the rest of the article …

My tech blog on Medium

Tue, 01 Nov 2022 00:00:00 +0000

I will continue writing blog posts here, but they will mainly be published on Medium under the user @glegoux. The goal is always to improve the experience of readers and my experience of writing. After having re-evaluated the market of online editors, the Medium platform is mature and becoming more and more popular, being one of the leaders among the editors of tech blogs now. It provides a full WYSIWYG user interface when editing online, with various formatting options provided as the user edits. It allows publishing blog posts (called stories) to capture large audiences and communities.

With the evolution of the practices and tools of online blogging these last few years, I wanted to cross thresholds in my tech writer experience: be more efficient and effective in writing content and gain more visibility by publishing content. It requires allocating less time to maintaining editor tools and more time to write and publish content on other websites than here. So, I needed to choose between continuing to use only this blog, starting to use only Medium, using both, or something else. I chose to use both:

The most of blog posts will be published on Medium first, with always a partial version here (or inversely)
The other ones will remain only in this blog.

Migration from glegoux.com to Medium

After evaluating and testing Medium, here are the pros and cons of using Medium vs. my blog:

✅ Pros	❌ Cons
• Medium editor is quicker and handier in updating a post with simple assets like text and images, more intuitive than YAFIYGI, and with instant updates without delay during the deployment of new content like with Jekyll used for my blog • Medium has advanced features of collaboration with co-writers, claps, peer comments, and followers, not all available on my blog natively and smoothly • Medium decorates the post with an auto-estimation of reading, no need to fill manually, and an auto-listen generation not available on my blog natively • Medium offers stats with custom metrics (reads, etc.) for each story more relevant than using a third-party solution like Google Analytics used on my blog	• Medium has partial propriety of the written content and limits the number of readings for non-members, while my blog is free and unlimited open • The Medium content has a pre-defined style of design. It is not highly customizable like the content on my blog • The Medium features remain less advanced in pushing varied content like dynamic technical diagrams and mathematics formulae. While some workarounds exist, my blog allows it

✅ Pros

❌ Cons

• Medium editor is quicker and handier in updating a post with simple assets like text and images, more intuitive than YAFIYGI, and with instant updates without delay during the deployment of new content like with Jekyll used for my blog
• Medium has advanced features of collaboration with co-writers, claps, peer comments, and followers, not all available on my blog natively and smoothly
• Medium decorates the post with an auto-estimation of reading, no need to fill manually, and an auto-listen generation not available on my blog natively
• Medium offers stats with custom metrics (reads, etc.) for each story more relevant than using a third-party solution like Google Analytics used on my blog

• Medium has partial propriety of the written content and limits the number of readings for non-members, while my blog is free and unlimited open
• The Medium content has a pre-defined style of design. It is not highly customizable like the content on my blog
• The Medium features remain less advanced in pushing varied content like dynamic technical diagrams and mathematics formulae. While some workarounds exist, my blog allows it

For all these reasons, the right choice is a consensus by keeping the best of these 2 worlds. Also, I prefer remaining independent for the long term, and https://glegoux.com is not only a tech blog but a hub of centralized online activities.

Don’t worry. I will keep maintaining this website and the open-source template behind it. To avoid breaking changes, the past blog posts will remain accessible without breaking their web links.

However, it seems time-consuming to synchronize the content between Medium and this blog bi-directionally. At best, I can only export and import a blog post to and from Medium by copying/pasting the content manually and by adapting the past content. Because the automatic Medium feature for external synchronization has been closed, and no other mature tools allow it, the custom extension of Mardown used here forces potential adaptations. So, the synchronization in updating a blog post from Medium becomes a nightmare to maintain. In contrast, I want to use Medium editor more, which is quicker and more handy to publish followed content. So, as I explained, some complete articles will only be on Medium or exclusively on this blog (with often a second partial version). When I want to have one blog post at both locations, instead of having a full post everywhere, I will use bidirectional web links cross-websites between the partial post and full post (to avoid divergences and too heavy maintenance in case of multiple evolutive versions with there are editions after the original publication).

🙏 Thanks for your understanding and your readings.

Secret Santa

Tue, 23 Nov 2021 00:00:00 +0000

Secret Santa 🎅 is a traditional event in December with family, friends, or office colleagues, in which each guest gives a gift to another guest without knowing who will give him/her a gift.

Could you imagine that you want to organize it? As an independent, technical, and meticulous organizer, you could automate the drawing of pairs (donor, recipient) to work even for an odd number of guests. You could implement an algorithm (given later) that generates a directed cyclic graph where the vertices are guests and the edges represent the relationship (donor -> recipient).

But you want to increase the secrecy, you choose to draw lots for the identity of the recipients at the last moment. In this case, the guests do not know neither for whom the gift they buy will be. In addition, it counters the effect of a no-show, in which a guest might not receive the pre-chosen gift during the event.

So the identities of gifts (ideally all different), donors, and receivers remain secret until the last moment. However, several problems could reduce the surprise effect. Let us understand it thanks to applied mathematics in probability, statistics, and combinatorics.

⟲Self-selection problem

Let’s begin with the first problem: to whom does each guest give his/her gift? Let’s suppose each guest has well his/her gift for a unique other guest. Of course, the draw could have been done before and kept secret, as mentioned previously. But you want to do the draw at the last moment and by revealing simultaneously who gives his/her gift to whom. That is to say, you write the name of each guest on a different piece of paper for each, and mix it in a Christmas hat. Then, each one takes a piece of paper randomly. Finally, all of them, at the same time, unfold the piece of paper and discover the name of the guest to whom he/she gives his/her gift.

Will a guest draw his/her name at random? If this is the case, the surprise effect will be broken for him/her, and what would imply redoing the draw or exchanging gifts? What we want to avoid.

This concrete problem can be reformulated: Is there at least a fixed point in the permutation of a set with $n$ elements? Where $n$ is the number of guests. It is the famous hat-check problem.

The probability that at least one guest draws his/her name is the inverse of the probability that no one drew his/her name, that is to say:

\[p_n = 1 - \dfrac{\mathcal{D}_{n}}{n!} = 1 - u_n\]

$\mathcal{D}_{n}$ is the number of derangements, that is to say, the number of permutations without fixed points, knowing that $n!$ is the number of permutation with $n$ elements. You can prove with the inclusion-exclusion generalized principle that:

\[\mathcal{D}_{n} = n! \sum_{k=0}^{n} \dfrac{(-1)^k}{k!}\]

According to the Taylor-Maclaurin development, we have:

\[\displaystyle\lim_{n\mapsto +\infty}u_n = \displaystyle\lim_{n\mapsto +\infty}\sum_{k=0}^{n}\dfrac{(-1)^k}{k!} = \dfrac{1}{e}\]

with $e \approx 2.71828…$. In addition, this suite $(u_n)_{n\in \mathbb{N}}$ has a rate of convergence quick:

Inverse of the probability that a guest draws randomly his/her name

So if the number of guests is greater than or equal to 4, you have around $0.63\%$ of chance, to have at least one guest who drew their name, that is a bit less than 2 out of 3 chances:

\[p_{n\geq 4} \approx \dfrac{2}{3}\]

This simultaneous strategy risks limiting the surprise effect, like the probability that a guest randomly draws his/her name, which becomes high quickly as a function of the number of guests.

However, another sequential strategy could be considered, the same as presented in the introduction, to generate a cyclic-oriented graph (donor -> recipient). The algorithm will be: A guest randomly draws a piece of paper, keeps it, and gives the Christmas hat to The person whose name is on it. Only the first guest could draw her/his name. In this case, the person redraws a piece of paper and then puts the paper back with his/her name. But here, the surprise effect becomes split into two moments, first discovering the identity of donors progressively, and then all the gifts simultaneously, instead of one moment previously.

🎁 Different gifts problem

A second problem will be to know if two guests could bring the same type of gift. Having all gifts different could make it more fun, as the guests did not know for whom the gift they bought would be. These gifts are not personalized. Let’s suppose that you categorized 300 types of gifts.

This is equivalent to another famous problem, the birthday problem. The probability that two people were born on the same day increases quickly depending on the number of people. It becomes almost certain, that is to say, more than $99\%$ of a chance to be valid for a group of 60 people.

More generally, the probability that a group of size $n$, built with replacement and a sequence of independent and uniformly distributed trials from a set of size $N$, contains at least two identical elements:

\[g_n = 1 - \frac{N}{N}\,\frac{N-1}{N}\,\cdots\,\frac{N-n+1}{N} = 1 - 1\,\left(1-\frac{1}{N}\right)\,\cdots\,\left(1-\frac{n-1}{N}\right)\]

It is the inverse of the probability that no element is the same.

Here, $N$ is the number of possible types of Christmas gifts, and $n$ is the number of guests. Except for some guests with a very original gift, suppose that your guest chose a pre-defined list influenced by the constraints and the ads, with a few hundred types of items, let’s say $N=300$:

Probability to have two identical types of gifts among 300 possible types of Christmas gifts

With an approximation $e^{x} \approx 1 +x$ when $x$ is close to $0$, and the sum of young Gauss $\sum_{k=0}^{n}\,k = \frac{n(n+1)}{2}$, you can get:

\[g_n \approx 1 - e^{-\frac{n(n-1)}{2N}}\]

But in any case, you see:

\[\displaystyle\lim_{n\mapsto +\infty}g_n = 1\]

And:

\[g_{n\geq 20} \geq \dfrac{1}{2} \qquad g_{n\geq 60} \approx 1\]

If you have more than 20 guests, we have more than 1 out of 2 chances to have at least two identical types of gifts.

So, I wish you a good Secret Santa 🎄.