The following is an abstract of some thoughts i had from the RWKV podcast found here: https://www.latent.space/p/rwkv#details

Why Alternatives Matters ?

Transformer architectures are currently at their peak with the AI revolution of 2023. However, in the rush to adopt them due to their recent success, it's easy to ignore alternatives that we can learn from.

As engineers, we should not take a one-size-fits-all approach, using the same hammer for every nail. We should find trade-offs in every solution. Failure to do so would trap us within the limitations of one particular platform, while remaining blissfully unaware of all other alternatives. This could potentially hold back development for years.

This problem is not unique to AI. It's a historical pattern that has repeated itself since ancient times up to the modern day.

A page from the SQL war history...

A notable recent example in software development was the trend towards NoSQL when SQL servers started hitting physical limitations. Startups everywhere made the move to NoSQL for "scale" reasons, despite being nowhere near those scales.

However, over time as eventual consistency and NoSQL management overhead emerged, and hardware capabilities made giant leaps in SSD speed and capacity, we've seen a recent trend back to SQL servers for their simplicity and now sufficient scalability for 90%+ of startups.

Does this mean SQL is better than NoSQL or vice versa? No, it simply means each technology has preferred use cases with pros/cons and learning points that can cross-pollinate among similar technologies.

I hope you view this article as not an attack on transformers, but a healthy appreciation on alternatives. With that, lets get on to the main topic

What is the biggest pain point for existing transformer architecture?

Generally this covers compute, context length, datasets and alignment. For this discussion, we'll focus on compute and context length:

• Quadratic computational costs due to O(N^2) increase per token used/generated. This makes context sizes >100k incredibly expensive. Which impacts inference and training.

• Current GPU shortage exacerbates this issue.

• Context size limits attention, heavily restricting "smart agent" use cases (like smol-dev), forcing workarounds. Larger contexts need fewer workarounds.

So how do we solve this?

Introducing RWKV - a linear transformer / modern large scale RNN

The following summarises the RWKV paper at a high level: here
For a deeper technical dive on RWKV architecture, see the: wiki here

RWKV, along with Microsoft RetNet, are among the first of a new class called "Linear Transformers".

It tackles the above 3 limitations directly by supporting the following

• Linear computational cost regardless of context size.

• Lower requirements allow reasonable token/sec output on CPU (especially ARM) in RNN mode.

• No hard context size limit as an RNN. Any limits in docs are guidelines - you can finetune beyond them.

As we continue to scale up AI models, to large context sizes. From 100k and beyond. The quadratic computational costs starts to exponentially increase.

Linear Transformers, however instead of abandoning the Recurrent Neural Network architecture and their bottlenecks which forced them to be replaced by transformers.

Took lessons learnt from making transformers scalable, and re-engineered RNN's to work similar to transformers, and remove all those bottleneck.

Bringing them back into the race with transformers, in terms of training speeds. Allowing them to both operate efficiently at O(N) cost, while scaling past a billion parameters in training. All while maintaining similar performance levels.

At 14B parameters, RWKV is the largest open-source linear transformer, on par with GPT-NeoX and others on similar datasets (e.g. The Pile).

But what does this mean, in simpler terms?

Pros

• 10x++ cheaper inference/training than transformers at large context sizes

• Can run slowly on very limited hardware in RNN mode

• Similar performance to transformers on same datasets

• No technical context size limit as RNN (infinite context!)

Cons

• Sliding window problem, with lossy memory past a certain point

• Not proven to scale beyond 14B parameters yet

• Less optimised and adopted than transformers

So while RWKV isn't yet at 60B+ parameter scales like LLaMA2, with the right support/resources it has potential to get there at lower cost and larger contexts. Especially as expert models trend towards smaller and efficient.

Consider it if efficiency matters for your use case. But it's not the final solution - healthy alternatives are still key.

Also one last thing i forget to mentioned:
Linear Transformers typically do not use classic Transformer attention, but an approximate equivalent, meaning ...

Attention is Not All You Need

Other alternatives and its benefits we should probably learn from

• Diffusion Models: Slow to train for text, but extremely resilient to multi-epoch training. Figuring out why could help mitigate the token crisis

• Generative Adversarial Networks / Agents: techniques could be used to train desired skillsets, to a specific goal without datasets. Even for text based models

Disclosure: I am a code contributor to the RWKV project, and an active member in the community.

However it does not mean I view RWKV as the final answer, even if in the future where "Linear Transformers" beat out traditional "Transformer" networks in the future, research into alternatives is still important for healthy ecosystem development.

I strongly believe there is no such thing as "one architecture to rule them all"

Additionally, i am aware there has since been developments like TransformerXL to more efficiently scale transformers, somewhere between quadratic cost, and linear cost. However with very limited known implementation of this method in public, it is hard for me to evaluate its pros and cons.

If you ever wanted to know how it feels like to be a senior software engineer, and experience ...

• The joy of coming up with experiments, and testing them to solve problems

• The craftsmanship of building either building small hyper-efficent systems, or

• The pride in building large hyper scale systems.

• All within either a fixed constraint or the lack of it

Factory building games, allows you to experience all of them in a highly compressed timeframe of hours instead of days.

All without the common frustration and pains that comes with programming. (Weird obscure library bugs, lack of documentation, Company politics or bureaucracy in processes, budget ... )

What is a Factory Building Games?

Factory Building Games, or FBG for short, are at its core about
Constructing and expanding a large base of multiple building blocks …

• Connected together to build products using an ever-growing continuous supply of materials …

• To continuously feed a very intensive "resource sink" (the games goal) …

• Which runs automatically without the player interaction for long periods of time

One of the key differences between FBG, and typical resource management games. Is the endless continuous nature of the main production lines, due to the usually infinite scaling nature of the "resource sink". Allowing everything to run "infinitely" without player interaction.

Said "resource sink" may differ from game to game, but it's typically some combination of the following

• Researching new tech, some which are endlessly repeatable

• Sustaining a colony of citizens, that grows in both size, and resource requirements

• A definitive end game goal, with enormous requirements (launch a rocket, build a Dyson sphere, etc)

While this may sound simple enough. Another common feature, is how as the game progresses. It would require more advance products and buildings, that requires other products built by the factory system itself. Which constantly scales up complex combination requirements.

At late game, it is not uncommon for products that are needed for the game goals, to require a process that involves over a dozen parts, and hundreds of steps.

Resulting into a complicated large scale processes, running continuously; Which mirrors some real life factories, hence the FBG title.

This gradual ramp up in complexity encourages constantly scaling up both consumption and production. All while constantly refactoring or updating the factory layout for optimisation, as new "methods of production" is unlocked.

How it makes you a better programmer - by making you practise everything else but code!

To understands the similarities between software development as a programmer, and playing an FBG as a gamer. We start by looking at a typical workflow for a programmer, where each task spans weeks to months

Now contrast to the typical gameplay experience of a FBG.

Notice the huge similarities, and how it compresses all the programmers day to day activities down to hours. While nearly removing all the unpleasant and potentially frustrating bits!

Despite ironically removing coding (one of the most critical things a programmer does). It rapidly speeds up the loop of planning, testing and iterating. At a much faster pace than the real world.

This allow a player to practise systems thinking, which is critical to programmers.

It does so, by forcing the player, to split the problem in game into smaller tasks, while using their existing tools and knowledge to solve them. Before forcing them to do the inverse, in putting together large complex systems, using the same "blueprint" of what they built previously.

This process of making smaller changes, and integrating such changes to the larger system, is part of the day to day of software engineering. (including all its failures and iterations)

Additionally, due to its nature of being a "game" - and the lack of "real life consequences". Most of the negative emotions that come with failure are minimised. This encourages the player to learn quickly through experimentation and failure.

All while avoiding overwhelming the gamer, due to the limited nature of the toolbox in the game. Allowing the player to gain satisfaction, from the mastery of said tools

Hence, despite the exclusion of code. It teaches the gamer/programmer, very essential skills in software development. As it’s worth repeating that …

There is a flip side though - which should come with a warning label. This game genre can be very addictive, due to how well refined the positive feedback loop is.

"It's the one video game (Factorio) that everyone at Shopify can expense”
~ Tobi Lutke, CEO of Shopify

Sounds good, how can I get started?

You can find the full list of FBG games at :
https://github.com/PicoCreator/awesome-list-factory-building-games

However, because the full list can be overwhelming, here are some quick highlights …

Dyson Sphere Program : which is a pure FBG in space, with really nice 3D visuals. Also building a Dyson sphere around the sun, is a cool mega project in itself

Satisfactory : which uses FPS/Fortnite like controls, not recommended if you are not used to such a control scheme - supports multiplayer.

Factorio : which is the most popular in this genre - is only recommended for more experience gamers. In my opinion it's not beginner friendly. However it supports really large scale multiplayer gameplay (50+ players), making it popular with larger teams.

Autonauts : is one of the notable FBG that includes simple programming blocks, for its "automation bots". Adding the code element back into FBG. It does loose however the iconic conveyer belt (to force you to use the bots), while sadly simplifying many things (can be too easy)

If you find yourself struggling with a particular game, try to takes things slow instead of rushing it. Part of the fun is to overcome these challenges.

Also, Some FBG may include other game mechanics like endless combat/war. this can drastically change the style of the game, as such when possible I would recommend "peaceful mode" for new players. To focus on the FBG gameplay.

I did not like the games you recommended, does that make me a bad programmer?

No

There are many good programmers who dun like games

There are many other reasons why one may find joy in programming that is not found in these games.

Most notably

• Personal satisfaction from providing positive impact to users

• Personal satisfaction from solving coding tasks

Similarly there are many other valid reasons why you may dislike a certain part of the game which makes it a dealbreaker. Or even just video games in general.

So while there is some truth that if you enjoy FBG, you probably will enjoy programming (and vice-visa). It is worth reminding that correlation is not causation.

So you be you. There are other hobbies in life, which is both fun, and sharpens your skills.

However, for programmers, who do enjoy playing such games. Good for you, as this is one of the few games which can help both practise your skills, and have fun outside your work-hours.

Separately, For non-programmers who is aspiring to be a programmer, completing an FBG, will help you get a better idea of the ideal workflow of a programmer.

This in itself, is a huge benefit, if you are outside the tech industry, and want to explore if such work is suitable for you

Shameless plug: How can Uilicious.com automate cross browser testing, to make my website development more "Satisfactory"

Is manually testing your websites, prior to production deployment a huge time sink and pain? Find existing website testing tools too complicated for your team to use?

Give uilicious.com a try today, with our easy to use testing commands, which anyone, even manual testers can easily learn.

An example would be the following code snippet for a simple login test

Which would provide the following test results

Making testing user journeys on the internet, a piece of cake 😄
~ Eugene Cheah CTO / founder of Uilicious.com