Background: this is a written long form of the presentation i gave during the ASEAN Digital Minister’s Meeting. Slides are available here

AI is slowly becoming foundational infrastructure for the next era.

The real question is no longer whether AI will be adopted, but:

• who controls it

• how it is adopted

• and who benefits from it economically

I get asked a version of this question very often:

How do we build a local AI ecosystem?
- For economic growth?
- For sovereign AI control, privacy and security?

However, there is a common mental model behind most of these discussions.
Which in my opinion, is wrong.

The wrong pipeline most people are assuming

This is roughly the path taken by OpenAI and other frontier labs.

It is also:

• extremely expensive

• slow (takes years to build up talent and infrastructure)

• optimized for research labs, not local ecosystems

And by the time applications are being built, with commercial value and economical impact. Years would have gone by.

Why this way of thinking fails in practice

The problem with this model, is not that it’s incorrect.

It’s that it assumes you must finish the earlier stages to do the later ones.

This assumption no longer holds. Open source AI has changed the economics.

For example, below - guess which response is from ChatGPT 5.2, and which is from an Open model downloadable today.

Today, strong pre-trained and instruction-tuned models already exist off the shelf. Many of them are competitive with frontier systems for most real-world tasks.

The main exception being on the “frontiers” of AI used in scientific research, and long horizon tasks. Most of which is not needed in most industries such as banking and customer servicing.

In that sense, others have already:

• paid the cost of experimentation

• absorbed the cost of failure

• released their work publicly

You do not need to repeat that journey.

Start where the real commercial value is …

Specifically:

• Applications with existing large models, using custom prompts with RAG systems, proving adoption

• Before subsequently fine-tuning to expand use cases, language support,
  and new domains to have tighter control and alignment

While we cite $10k-1M for these efforts, the later covers isolated larger projects and initiatives with large datasets. For the vast majority of projects we see, this trends towards the $10-100k cost.

That is cheap enough for:

• universities

• startups

• small teams & grassroots efforts

In addition most of the goals for Sovereign AI is to maintain

• Domestic use case & language support

• Privacy & Security

• Reliability & Control

All of these goals are achievable by fine-tuning existing major models instead of pre-training from scratch. And be flexibly deployed either in sensitive private datacenters (for secure govt use), or public cloud (for commercial usage).

Highlighting case studies : Who did not need 100’s of millions for impact

Case Study 1: Sunbird AI - Sunflower AI models

Sunbird AI released Sunflower, Uganda’s first homegrown multilingual AI model designed specifically for local languages. Outperforming Google’s and OpenAI systems in local languages use cases in 24 out of the 31 Ugandan languages.

Built by fine-tuning existing major open source models from ranging from 14 to 32 billion parameters. With models small enough to run on medium to high end notebooks.

With real-world use cases Sunbird AI is being used by : farmers for agricultural guidance in their native languages, educators can translate learning material for students, and healthcare workers can communicate more effectively with patients without language barriers.

Advancing digital inclusion, cultural preservation, and equitable access to technology across Uganda.

Case Study 2: Kissan AI - Dhenu Agriculture model

Kissan AI developed Dhenu (named after the divine bovine goddess), in inida, a language model trained for agricultural and farming use. In English and Hindi

Built by fine-tuning existing major open source models from ranging from 3 to 8 billion in parameters. Which are models that are small enough to run on mid-to-high end phones and tablets.

They allow the usage of AI, where internet connectivity is sparse or even non-existent. Helping farmers improve decision making and productivity.

Applications first, not pre-training or infrastructure first

By starting with application and/or finetuned models, we avoid a common failure mode I see is this:

“We will build a national model first, then later figure out how to use it.”

Which often burns through tens to hundreds of millions of dollars with little to show.

When applications are deployed on the ground, they create direct feedback:

• whether better language support is needed

• whether models need to be more specialized

• whether inference costs are too high

• and where infrastructure becomes the bottleneck

Achieving economical impact within months not years. While having the flexibility to move up the chain when demand for it grows.

More importantly:
Even if you pre-train it right, but do not fine-tune a model well and fail to deliver on no applications using it, you have failed — with little to no economical impact with the millions invested in.

The reality is, all ideas will carry risks, but starting from applications allows you to test economically multiple ideas at scale, quickly - at the price of a single pre-train model, or a major AI datacenter project.

Its also worth acknowledging: when it comes to new technology where one do not fully understand. Policy has a tendency on focusing on the familiar, infrastructure.

What about AI control and abuse?

For control and AI abuse prevention, these can be safeguarded through policies, that prosecute the misuse of AI. Essentially treating it like a dual use tool.

A mental model i have advised policy maker, is to write policies from the lens where there is access to large pools of outsourced low-cost labor. And apply it to both humans and AI usage.

For example:

• Policies against the usage of large number of fake/paid personas, used for manipulation of public opinion and elections. Done through the usage of large pool of human labor and/or AI agents, pretending to be domestic residents.

• Policies against the usage of false images and videos, for slandering and blackmailing. created through professional photoshop / video editing / AI image or video generation.

• Policies against strengthening cross national prosecution against scams. Done through Human / AI call centers.

The unfortunate reality, as the AI model sizes required to achieve some of these abuse behaviour is shrinking to one that is runnable on “off-the-shelf” hardware like high-end laptops. Because these use cases do not need “large 500 Billion parameter models”

Centralizing into a sovereign AI model and infrastructure. Will not make these tools disappear.

Common push back: But how are we gonna compete with the bigger models?

Here is an industry secret: for specialized production application you do not have to.

You only need to get it to be reliable and good enough for the end-customer use case. Another factor is, when you run things commercially at scale, the bigger the model gets, the more expensive it becomes.

This is reflected in our traffic data usage as well

Because individual users, tend to use AI models across a large spectrum of use cases, individuals tend to use more versatile models (they can pick any of the models on our platform at the same price), where larger models dominate their usage.

However for commercial scale, where the smaller models means “more tokens and requests” for the same dollar. We see heavy reliance of 12B to 32B models in the use cases tuned for them.

You do not need an expensive Einstein model to do everyday college task at scale. Especially when you can get more consistent and even better results, with efficient trained specialized model, for your domain.

We expect this trend of larger off the shelf models being used for exploration, and experimentation to maintain. As it balances the need for fast experimentation and go-to-market. And efficient cost management of AI at scale

One more thing: Pre-trained models are becoming a commodity

Two years ago, a strong open-source model was released every few months.

Today: releases happen monthly, sometimes weekly with the gap closing. All while running on smaller hardware, and outperforming the best close-source models one-to-two years ago.

While major closed source commercial models are still in the lead.

• The gap has been shrinking rapidly in the open model space

• While being smaller and cheaper to run. Today the best open models that can run on a $10k desktop PC can give similar or better results to the larger GPT4 models which required $300k servers in data centers to run.

The pace is so fast that several times many new pre-trained models are obsolete within days. Benefit from this trend by building on pre-trained models, instead of against it.

Few industries exists, where the best commercial solution, which costed 100’s of millions, is made freely available consistently in under 2 years.

Clarifying question: If some of these models can be deployed on cheaper hardware, why is there still strong demand for larger servers?

There are two major factors for AI, the model size, and compute.

Larger model sizes, pushes the minimum hardware requirements for memory chips in the AI system. For example, a high-end Apple PC with 256 GB vram (or 256 Billion Bytes), can run a 120 Billion parameter AI model. But not the larger 1.7 Trillion parameter model, GPT 4 alleged size. Which would requires higher end servers with 2 TB vram (or 2048 Trillion Bytes)

Another major factor is compute. While you can run on a mac studio a 120B model for one user at a time. will struggle for anything more then that. It lacks the compute compared to a high end AI GPU server, which can run for over a 100 users in parallel the same model, at faster speeds.

At scale, when serving thousands or millions of users, these higher end machines are essential. But for business who do not need such volume, open source models help them right size to their needs.

Answer to “guess the model”

About the author

Eugene Cheah is the founder of Featherless AI and co-lead of the RWKV open-source AI project, the first AI project under the Linux Foundation. He works on AI inference, efficient architectures, and building practical AI systems used in production globally.

With our background in Open Source AI - We are interested in collaborating in scaling open source AI models to meet your nations or Soverign AI needs, with your local industries. Feel free to reach out to eugene @ featherless.ai

This is follow-up footnotes - for the previous viral article, originally posted on latent.space. That didn’t make it into the original article (too much math).

This is meant to be a “living post” for any future amendments as well.

It is recommended reading this first for context:

Latent Space

$2 H100s: How the GPU Bubble Burst

Don’t miss Eugene’s responses on the HN and Reddit and Twitter discussions…

Read more

2 years ago · 55 likes · 11 comments · Eugene Cheah

My spreadsheets for calculations can be found here

• new H100 ROI spreadsheet (Aug 2024)

• H100 Infiniband Cluster math spreadsheet (Aug 2024)

Not all H100 clusters are made equal, broadly speaking they are segmented into the following bands, decided by their, InfiniBand.

• <= 2 nodes: Direct Infiniband interconnect (pretty much only unis)

• <= 64 nodes: 1 layer of Infiniband switches (common small clusters)

• <= 2048 nodes: 2 layers of Infiniband switches (most common size range)

• <= 65,536 nodes: 3 layers of Infiniband switches (handful, big players only)

H100 Training clusters are extremely network-hungry, due to the nature of the training process, and are typically deployed with 3.2Tbps Infiniband interconnect. Built using 8 x 400 Gbps connections.

The problem: as the cluster size increases, the more complicated this network becomes to support the levels of interconnectivity required.

This as a result has a direct impact on the unit economics.

PS: 100k H100’s is the approximate cluster size for x.ai cluster

Where each tier of the cluster node sizes (<=64, <= 2048, >2048), has a bump up in the H100 node CAPEX economics; Ranging approximately between $47k to $52k per H100. And why I used the $50k average previously.

Additionally, due to the sheer energy scale of these GPUs. Where each H100 server, consumes as much energy as 4 US households. The numbers projected here are conservative.

For small clusters (<=64), these can possibly be retrofitted in existing data centers.

For larger clusters, the energy consumption starts to rival that of Small or even medium-sized cities. Dedicated cooling and power lines, or even generators will be needed. Dramatically, increasing the setup unit costs involved.

Combined with the high failure rate of H100 (they have <99.9% uptime), Nvidia themselves project a 1:1 ratio for the cluster facility CAPEX & operation costs. So we are being very conservative by using only 50k per H100 node instead.

I will not be surprised if the real number is easily 1.2x higher. Especially for larger clusters, due to how price inelastic H100 servers are when buying from nvidia.

Shoutout: Semianalysis related write ups

These folks are among the best in the industry, for these kind of analysis. And is worth reading, if these kinda of infrastructure deep dives is your thing.

Answers the question on: How much will the setup cost be (minus facility), when you do custom networking solutions instead of following nvidia reference

SemiAnalysis

AI Neocloud Playbook and Anatomy

The rise of the AI Neoclouds has captivated the attention of the entire computing industry. Everyone is using them for access to GPU compute, from enterprises to startups. Even Microsoft is spending ~$200 million a month on GPU compute through AI Neoclouds despite having their own datacenter construction and operation teams. Nvidia has heralded the rapid growth of several AI Neoclouds through direct investments, large allocations of their GPUs, and accolades in various speeches and events…

Read more

2 years ago · 103 likes · 9 comments · Dylan Patel and Daniel Nishball

H100 networking, and failure rates, in even more details

SemiAnalysis

100,000 H100 Clusters: Power, Network Topology, Ethernet vs InfiniBand, Reliability, Failures, Checkpointing

There is a camp that feels AI capabilities have stagnated ever since GPT-4’s release. This is generally true, but only because no one has been able to massively increase the amount of compute dedicated to a single model. Every model that has been released is roughly GPT-4 level (~2e25 FLOP of training compute). This is because the training compute dedicated to these models have also been roughly the same level. In the case of Google’s Gemini Ultra, Nvidia Nemotron 340B, and Meta LLAMA 3 405B, the FLOPS dedicated were of similar magnitude or even higher when compared to GPT-4, but an inferior architecture was utilized, resulting in these models falling short of unlocking new capabilities…

Read more

2 years ago · 170 likes · 22 comments · Dylan Patel and Daniel Nishball

Amendment: in electricity cost calculation

Shout out to and @youjiaccheng for spotting the minor math error, which has been fixed

Featherless.AI plug …

What we do …

At Featherless.AI - We currently host the world’s largest collection of OpenSource AI models, instantly accessible, serverlessly, with unlimited requests from $10 a month, at a fixed price.

We have indexed and made over 2,000 models ready for inference today. This is 10x the catalog of openrouter.ai, the largest model provider aggregator, and is the world’s largest collection of Open Weights models available serverlessly for instant inference. Without the need for any expensive dedicated GPUs

And our platform makes this possible, as it’s able to dynamically hot-swap between models in seconds.

It’s designed to be easy to use, with full OpenAI API compatibility, so you can just plug our platform in as a replacement to your existing AI API for your AI agents. Running in the background

And we do all of this; As we believe that AI should be easily accessible to everyone, regardless of language or social status.

Additional Sources:
- ChatGPT launch date: Wikipedia
- H100 launch date: Tech Power Up Database
- The A100 SXM had 624 bf16 TFlops, the H100 SXM was 1,979 bf16 TFlops
- Microsoft & AWS allocated over $40 billion in AI infra alone: Wall Street Journal
- “600 Billion Dollars “ is about: Sequoia’s AI article
- Nvidia investor slides for Oct 2014: page 14 has the pitch for “data centers”
- Semi Analysis: deepdive for H100 clusters, w/ 5 year lifespan approx for components

A wave of AI foundation model companies, with half a billion dollars raised - reaching their end

Throughout 2023, we see headline after headline. Of various all-star teams, pulling researchers from the top of FAANG, Ivy leagues, Open AI. To create their Avengers of AI.

Since 2024, we have seen the fall, one after another, after asking 100’s of millions after 100’s of millions. Reach their end, as they run out of money

Everyone was trying to follow the same blueprint, of openAI oversimplified

• Step 1: Get the best researchers, Spend $XX million a month, researching and building the best model out there

• Step 2: Keep researching, get results, scale up, Ask for more money for each milestone

• Step 3: ???

• Step 4: Profit; and make a billion-dollar business

The AI market is no longer the same as it was pre-OpenAI - the formula was never reliably repeatable in the first place.

AI does not remove the fundamental principle of businesses needing to make money

But, but … we funded the team that had all the brightest AI researchers, from Google, openAI, Ivy leagues …

That's exactly part of the problem, of how VC’s oversimplified the whole thing. As much as I love their work, what do you think researchers will do with funding…. Research!

Do you not know what the life cycle for a researcher is?

• Step 1: Propose a research idea, get funding grant approval

• Step 2: Get results, Write papers, and Scale up. Ask for more money for each milestone

• Step 3: Repeat steps 1 & 2, with more ideas, or bigger ideas

• Step 4: Accumulate enough paper citations, get university tenure, retire

Noticed how in the researcher's life cycle, at no point was it about monetizing, making profits, or making a billion-dollar business. If anything, research by its nature is very capital-inefficient.

While they are undoubtedly great brilliant minds in research (and I love them for their contributions). They are not the right people to turn research into commercial products. And inversely, ask for more and more money.

While leaving the commercialization process to “someone else”.

VC’s correlated PhDs & papers to commercial success. Over-indexing to OpenAI's outward appearance of success. When literally, some of the most profitable, and capital-efficient teams in AI have the inverse correlation

Wrong Conclusion: OSS / Foundation AI is too expensive?

The continuous repeat of the extremely large funding cycle of 2023, with larger and larger funding cycles, from foundation model companies. Have lead to extremely wrong conclusions from the VC markets

• That foundation model companies require half a billion dollars and more to succeed

• Open-source AI companies can't succeed.

This could not be further from the truth. GPT-4 is estimated to have a training cost of $100 Million in GPU time. And with advancements in AI techniques and technology, this cost has been going down (not up).

Add in labor, and various costs - Any company that raised beyond $250 Million and failed to catch up with GPT-4 is just being “capital inefficient”. A number that is well below many foundation model 2023 capital raise.

So why do they keep failing? It’s back to the basics of “lack of focus” and “financial discipline”. And distracted by research

This can be best reflected by, a failure of the following basic test ….

Litmus Test: “Time to pricing, and payment”

In the same naive mistake, that countless Open Source Companies have encountered, foundation model companies went on the same blind trend of “figuring out how to make money too late”.

And I say this, not only as an observer but as a passionate User as well …

I love you all, but ….

In Rancher's case: I have given talks, workshops, and evangelized ranchers into other startups. But have contributed $0.

• I use their product daily (self-hosted), and would pay to have a good managed SaaS solution, which never existed.

• The only current way to pay them is to be a giant enterprise with a crazy expensive custom enterprise contract, which I am not.

In Stability AI case: I love you, folks. As a GPU donor recipient for the RWKV OSS projects (thanks!). I was literally trying to help pay you, in some shape and form, to help improve your numbers for your investors. But gosh do you make it impossible! I had to fight your system to give you money.

• The paid image generation platform (dreamstudio), is nowhere to be found on your home page and is constantly on page 3 of Google for text-to-image. Refuses a subscription, and is only credits-based. I have to bookmark it, so I can use it every now and then when I need an AI image generated. To effectively give it a few dollars every now and then.

• The model membership, for $20/month, requires me to fill up a form that requires manual review - only to have the reviewer inform me that I do not qualify. As I was not having enough users & making enough money. In both cases I know this, I just want to give you $20/month

For Inflection AI case: How does anyone even pay you, there is no enterprise sales form, nor cloud sales process on your page.

There are already enough problems getting pricing right, for any SaaS/platform growth …

But to make it impossible for customers who want to give you money, to give you any. Is in my opinion the complete opposite of “financial discipline”, which is so self-damaging, it’s investor harm.

Unless your plan was a pure acquisition exit. If you want to make a stable company, constantly releasing wonderful open source code or AI models to the public. You need to have an income stream. Employees need to be paid, Servers / GPUs need to be paid. Investors need to be paid an ROI.

As a business, your number one primary goal is to make Money. We can discuss common good, altruism, etc - AFTER - we make sure your business, is stable enough to stay around and pay the bills while making money in excess.

Even charities and non-profits need a plan on how to get donation money. So does your business, noble goals, and idealistic open source visions aside.

This is a problem not just for AI, but any large-scale open-source project in general. A problem we have seen repeated many times. With idealistic views.

We can do better ….

On the other spectrum, is established companies with Open Source/Core models: MongoDB and Elastic. WIth Open Source/Core products that I have gotten either my startup or other startups I advise to be paying for.

The time to pricing, and payment is relatively straightforward within a few clicks.

It's so straightforward, that this revenue funnel, from one-click users to enterprises - makes them a revenue of $1.5 Billion / $1.1 Billion dollars per year respectively, with an R&D budget of over $300M per year for both.

Enough revenue, or R&D budget, to be training GPT-4 multiple times a year. Enough R&D spend, to outspend the 3 other companies I highlighted, that failed this basic litmus test.

This is despite, having everything about their Open stack being potentially an investor nightmare

• Their software stack has been cloned by every major cloud provider and competitor

• Their open source / open core offering is complete enough, that yes - I can actually self-host everything if I want to - and I do so for my own personal cluster usage

But why do I and several others, then decide to pay them, instead of their cloned offerings - to a tune of billions of dollars

• Production Reliability: Scaling up AI and Databases safely and reliably is incredibly hard, especially in production where you cannot afford any downtime.

• Convenience in scaling up and down: Related to production reliability

• Convenience from the Source: Ease of upgrade and access to updates, before any other providers

• Reasonable Pricing: Without crazy per-seat pricing, it's not much cheaper to get the same performance/service offering on your own, once you factor in upgrade processes required, hardware redundancy, etc. Both Elastic, and MongoDB inversely, get to leverage their economics at scale, and benefits as part of their margins.

• Multi-cloud offering: allow me to easily move between AWS, GCP, and Azure if needed

• Enterprise compliance and security (ISO, and SOC certification processes)

Essentially make it more attractive for me to use your solution and hardware. Then my own cloud hardware, and we the user will pay.

Wrong Conclusion: Is the pricing page all I need?

It is a start and is infinitely better than being user-hostile in making money.

It may be sufficient to get to the thousands, or maybe millions in revenue. But if you want to reach billions, it's all about the details …

Moving from research to production, and sales is a muscle!

A muscle that rarely any one company gets right the first time round, a muscle that needs to be constantly trained and iterated.

Notice the list of reasons previously, on why I and many others choose to pay for the commercial offering. Despite even having the expertise to self-host on our own?

Without exercising the muscles from converting a research product into production and sales, it is easy to make the following mistakes

• Poor production reliability: Rate limits, Downtime, all constantly puts off users

• Missing features that the customer wants: from paid offerings, which the open source offerings provide

• Poor pricing: Users are not idiots, we will shop around, and get pricing drastically wrong compared to your peers. And we will switch. Inversely, price too low, and the company gets burnt.

• Missing conveniences: A better UX/DX, is a very large reason why your commercial offering will be preferred over DIY.

• Bad sales processes: Your enterprise sales team, needs time to find Product Market Fit, needs time to scale.

While I can sing the praises of Mongo, elastic, or even supabase (Postgres). I can also point out various open-source technology projects, which failed on some version of these points badly enough. That their users were pushed into self-hosting or other solutions.

There is no one-size-fits-all solution to these issues, there is only going to market, and iterating with the users. Getting this right for B2B / developers and enterprises is complicated enough entire investors and courses is specializing in this segment (eg. heavybit)

We know all of these things, We have learned it through the cycles of various deep tech startups, and B2B SaaS. Sure investors may have been blinded by greed to repeat OpenAI, through oversimplified pattern matching ( Big name + AI = Profit ?? ).

But why do founders, even business founders, avoid building those muscles - when they know better ???

The problem of incentives from VC’s …

Foundation model / AI companies by their very nature are deep tech. It requires a fairly deep capital of 50M and up.

On the flip side, consumers are very demanding, and besides notable exceptions, are mostly only willing to consider paying once a model reaches GPT 3.5 class and up approximately.

1) VC’s Overfitting to the Product Market Fit revenue rocket

One of the notable issues, that founders constantly discuss - is that once you have revenue - VC’s tend to only focus on the revenue and Product Market Fit curve. Which boils down to: Is it a rocket or not?

This creates a weird situation, where strategically speaking we founders are aware of the uphill sales cycle for anything below a GPT 3.5 class model.

As such, It makes more sense to avoid any revenue until you have an AI model approaching the GPT 3.5 class, after which you can start building a platform. This process is something even several VCs will advise founders on, to maintain focus …

Knowing the tall task involved, many opted to focus on benchmark and open source adoption first before any commercialization … but that has its own weaknesses …

2) Worry about the revenue later instead? You are going to get sniped.

This is what happened to Mistral, as they scaled their model up to Mixtral …

Only to find competing providers, to not only enter the market first to the public before them. But to do so at a lower price point. With their model.

All of which was to be partially expected - because, while Mistral was busy entirely just creating the model - they never built that muscle for bringing their very own model into production and optimizing the user convenience to the sales process. They were isolated from the users.

Inversely, together AI has already been, running the Mistral 7B model. Iterating with the users, on every factor, from convenience to price. It may not have been a major success, but they have gotten the iteration process going. As such, when the mixtral model came out, they were more prepared than the mistral team to publish their API for the model.

Together AI has been exercising the production shipping, and commercial muscle. Mistral didn’t

3) Making it worse: build it close source, behind the scenes? Scale up when we are ready

Basically what Inflection, and a few others did.

• If they avoided revenue, they were forced to be working on the model, without user feedback …

• If they accepted revenue, they will face an uphill task of making consumers care … because they were neither GPT 3.5 class nor open source which they can tinker with … Additionally investors would be doubting them in the early stages due to 1)

• This is also why - nearly every AI startup is a “beta” in the early stages, to have some form of customer users and feedback loop. Without committing to the revenue hockey stick.

Basically, you escalate the stakes. You either clearly beat and provide a strong reason to move from the existing close source model with a Unique Selling Point. With very limited user feedback, till you get to that point.

And if you fail, you will have ended up creating a model, which no one wanted (as it was not adapted to user feedback). A model and platform that no one cared about (as it is not open source), with little to no revenue nor user adoption.

Sounds hard right? Well there is an answer … An alternative …

How Should it be done then? How does one build a foundation model company?

You need a founding CEO

• who has both enough domain expertise and respect in AI to lead an AI team

• with business sensibility,

  • to actually steer the company to profit

  • and not burn more than 100~200 M to make GPT4 class AI

Why not a business CEO, with a AI researcher CTO?
You will need them both to be very very aligned. Hard decisions will need to be made that involves a quarter to half of the company capital. A decision which can only be made with informed context of both the AI training process and business goals. Decisions which either side may fear making or misjudge, due to lack of context of the other side.

With a founding CEO / CTO / Team, that has some mix of the following

• dataset and AI model training and scaling experience

• who has experience building and scaling resource-hungry infrastructure platforms (for inference)

• sales and marketing muscles with either (or both)

  • a DevRel team to help promote the platform adoption

  • or an Enterprise sales and partnership team to close the deals

What if there are talents lacking in specific segments?
That is fine, its hard enough to find the combination of talent required for building and scaling AI models (There are less then 1000 individuals who has experience training a 7B and above model from scratch). Let alone everything else. Discuss with the founding team, a recruitment plan for the required gaps in the team.

With a plan to scale in steps, both the model and platform

• Stage 1: Something to prove the concept / technical approach

• Stage 2: Initial platform and model, which can start closing some customers (even if it is ineffective, unscalable, or slow)

  • This can ideally be within a specific market niche or use case

  • Helps ensure all initial scaling issues on the platform are tackled, and sorted out

• Stage 3: Scaled up, highly performant model and platform, tuned heavily to the user needs (where the hockey stick begins)

  • Ramping up on sales and customer adoption

  • Maybe split into smaller sub-stages (eg. GPT 3.5 class, GPT 4 class)

• All in under 2 years, because AI moves fast

• With Investors, who are willing to commit to such a plan, step by step.

Can’t we skip stage 1 & 2, and go straight to 3?
Unless you are able to find the required talents who have done it all before. Which you do not, as they are all concentrated within OpenAI and Anthropic (not even google counts). Across all the various key disciplines. Then maybe you can remove stage 1 & 2?

And even then, that would be a high risk move to an already high risk endeavor. Its better to just sort out the bumps, across a few months, and the customer feedback loop. To get this right.

For context, openAI had the time between 2018-2020 for stage 1 (GPT-1 & 2). Followed by 2 years+ of stage 2, with early user traction and iteration (GPT-3) between 2020-2022 And only stage 3, towards the end of 2022, early 2023.

But yea, many VC’s loved to ignore the first few stages, and only focus on replicating the last stretch. Without realizing how hard and essential it is to make sure to sort out the platform at scale. After all this is not just a “SaaS template” + SuperBase combo. The success and failure is in the details.

And dun forget to make a profit?

By selling the AI product, at a price point that - actually makes money this time!

OpenAI is reported to be making losses of near $500M a year, or over a million dollar a day. According to various reports, including buisnessinsiders.com

The last part of making profits, making money, over the cost of goods - is important because

• You can cover all bills, to be default alive

• You can funnel the revenue to future models, and customer growth, To grow future revenue

• And recursively scale up the business, and AI model in the process.

Because why not? The market is hungry, over a 100 billion dollars a year hungry, if you get the pricing and scaling right even with existing AI models (GPT 3.5 and GPT 4 class)

Sounds like a saner business plan - better than trying to raise 7 Trillion dollars right?

A self-fullfilling loop that keep scaling past even AGI?

Well, do we have something for you …

Guess what! - that's basically what, me and my team are doing here are Recursal AI are doing.

• We are built using a new AI architecture: That can make a profit, as it’s 100x++ cheaper on inference compared to existing AI architecture (transformers).

• We plan to make not just revenue, but profits: Its part of our founding story, we grew from the ashes of un-stable AI companies. And never want that to happen again

• We do not plan to raise more than $250M: Because, we have always been capital-efficient, and plan to stay that way.

• We just finished Stage 1 (while spending <$2.5M):

  • We proved our architecture, can be scaled like transformers and even outperform LLaMA 2 7B

  • We achieved SOTA status, as a 7B multi-lingual model

  • Multiple orders of magnitude more capital efficient then any LLM group

• We are now at Stage 2:

  • We have our initial commercial cloud platform, and strategic partnerships (more details to come)

  • We have initial customers, giving us a tight feedback loop, and pays us

  • We intend to find PMF with our Unique Selling Point at this stage, across either:

    • Affordable, High throughput support

    • SOTA Multi-lingual support

    • Support for enterprise private cloud / on-premise deployment support

• We are scaling to GPT 3.5 for Stage 3A, Followed by GPT 4 for Stage 3B

• All while building the open-source AI model, open source under the Linux Foundation

• And creating that looping wheel, of profits, powering model improvements, in a stable recursive manner.

  • That makes both investors and employees happily rewarded.

  • A loop I wish to see repeated across more competing open source models.

If that makes sense to you, feel free to reach out

• For investors: founders@recursal.ai

• For those who want to join us: hr@recursal.ai

Recursively: Making AI, for everyone, in every language

~ Until next time 🖖 live long and prosper

While GPT 3+ or ChatGPT, prompt engineering is easier to understand intuitively. With many guides and examples available on the web, and social media. Such as

• https://help.openai.com/en/articles/6654000-best-practices-for-prompt-engineering-with-openai-api

• https://github.com/f/awesome-chatgpt-prompts

Embeddings, requires programming, and are less understood, due to various counter intuitive behaviour on how it works. But, is an extremely powerful tool for search, or to be used together with existing text-based models, for various other possible use cases.

Embeddings is arguably an equally powerful tool, within the AI toolkit, for instruction models. Due to its ability to handle searches across different words and sentences, or even entire languages. Focusing on searching for the relevant document, for any query.

For example, it can be used to power searching and answer from an English-based documentation. In English ...

Or Japanese ...

Or any other languages that the AI model supports. Embeddings can also be used for other tasks such as question-answering, text classification, and generating text.

Note that this article focuses on the search aspect, the answering process is in a follow-up article.

What is an embedding vector?

To generate an embedding vector, one will use an embedding AI model, which converts any text (a large document, a sentence, or even a word), into an "N dimension array", called a vector.

For example a sentence like How do I write a UI test script with Uilicious?

Can be converted into the following array (called a vector) via OpenAI text-embedding-ada-002 model: [0.010046141, -0.009800113, 0.014761676, -0.022538893, ... an a 1000+ numbers]

This vector represents the AI model's summarized understanding of the text. This can be thought of as an "N-word summary" written in a language that only the AI can understand.

Where related documents will have a close distance from each other, based on the AI's understanding of the document (and not its text). This is a major leap beyond the simple keyword searches of classic search engines, as it can handle variations in sentence structure and language (provided the AI model is trained to understand said language).

Take the following as a hypothetical example, oversimplified inaccurately into 2 dimension space so that it is easier to understand:

Which can be visually presented as the following in 2D space.

For example, D1,2,3 are all documents related to how to use Uilicious in various ways and are grouped together in one cluster

D4 and D5, being simply links and having no inherent value beyond that, are grouped separately in another cluster.

Furthermore, D1 & D2 are further grouped together, as they are about Uilicious testing commands, using our very own JavaScript-based testing language.

While D3 is grouped separately, as it is regarding using the web driver protocol on our infrastructure directly, which is intended for a different use case and audiences.

Similarly, for Q1 and Q2, despite the drastic differences in sentence structure and language, because it is essentially the same question, the two questions are grouped together. Additionally, while the question could technically be interpreted both ways (using Uilicious test script, or webdriver protocol), because the question implies the usage of Uilicious test scripting over webdriver, its location is "closer" to D1 & D2, and further away from D3.

As such, despite having huge overlaps in keywords, these nuances in groupings are captured by the AI encoded within the embeddings.

In reality, however, instead of an oversimplified 2 dimension array which is easy for humans to understand, an embedding can easily be a 1,000+ dimension array. This array is unique to the specific AI model used, and cannot be mixed with another AI model's embeddings.

Math notes: N-dimension math is not compatible with 2/3D math

While the oversimplified 2 dimension examples, are good for understanding the high-level concept of grouping relatively to one question (or one point of view), it does not accurately represent an N-sphere, which is the mathematical construct used to describe a space of N dimensions.

Due to complicated N dimension math, you can have situations, where A can be close to B, B can be close to C, but A and C can be considered far from each other. Which is an extremely counter-intuitive gotcha.

This is known as the Curse of Dimensionality

Such distances are only useful when used relative to the same point, and the formulas used. Which can be calculated either using

• Euclidean distance: also known as Pythagorean Theorem on steroids, it is the most common distance metric used and is the straight line distance between two points in N-sphere.

• Cosine Similarity: it is a measure of angular distance between two points in an N-sphere, and is useful for measuring the similarity of documents or other vectors.

• Manhattan or Hamming Distance: these two metrics are used to measure the differences between two vectors, and are useful for measuring the "edit distance" between two strings.

While the effectiveness of each formula has their respective pros and cons, for different use cases. For textual search, it is generally accepted that Euclidean distance "works better" in most cases, and "good enough" for cases where other methods edge out.

All of which, is used in effect to reduce N-Dimensions, to a single dimension (distance), relative to a single point. This as a result, would mean groupings can / may change drastically depending on the question asked.

This "relativity" nature of distances, makes classic database search indexes ineffective.

If that makes no sense, this is how 4 dimension space is visualized properly with math N-Spheres.

Now imagine 1,000 dimensions? Yea it makes no sense.

So without derailing this topic with a PHD paper, I will summarise this as just trusting the math professors.

All we need to understand is that in general, the closer the distance is between two vector embedding points, the increased likelihood they are relevant to each other.

From a practical implementation standpoint. Start with using Euclidean distance first. Before considering using the other formulas which are fine-tuned for better results through trial and error for your use case (not recommended).

Searching embeddings with a vector database

So given that we can convert various documents into embeddings, we can now store in within a database and perform a search with it.

However, unlike an SQL database searching with text, both the search and the data being searched are the vector embedding itself. This means that traditional database search indexes are ineffective when it comes to searching for embeddings.

We can have all your documents embeddings, precomputed, and stored into a vector search database. Which can then be used to provide a list of matches, ranked by the closest distance.

This can be done using existing vector databases such as

• REDIS: a popular open-source database that can be used to store vector embeddings and search them efficiently.

• Annoy: a library created by Spotify which uses an optimized algorithm to search embeddings quickly.

• FAISS: a library created by Facebook which provides efficient search algorithms for large datasets.

One major thing to note, vector search "database tech" is relatively new. Where a large majority of the vector search databases were designed for use cases found in companies like Facebook, Spotify or Google, with record sets in the size of millions or billions. And may not be optimised for small datasets.

This is going to be a constantly changing field in the next few years, here is a github 'awesome-list' to help track and find future vector search databases

As such, in general, we found for small datasets of (<10,000 ~ 100,000 embeddings), keeping the embedding dataset in memory and brute-forcing euclidean-distance-squared is "good enough" for many use cases, and will sometime outperform formal database solutions (which will have disk/network overheads) with something like the following.

The obvious downside to this approach is that the entire dataset must be small enough to fit into memory without the overhead.

Regardless if you are using local in-memory embedding search, or a formal vector search database.

That's it!

Embedding search is just a sort and rank algorithm that works flexibly with various languages, and scenarios. The question for you as the reader is how you can use it. It can be used as it is as a possible google search replacement, or together with other tools, from chat, to games. The possibilities are endless and open to exploration.

~ Until next time 🖖 live long and prosper
Eugene Cheah @ tech-talk-cto.com

All Images used, with their proper attribution

• Man in magnifying glass by Marten Newhall on Unsplash

• Zach Galifianakis Math meme

• File drawers by Jan Antonin Kolar on Unsplash

• Dream big wallpaper by Randy Tarampi on Unsplash

If we don't sort out the legal issues surrounding AI-generated content properly, it could have serious implications for the future of thousands of jobs.

As such, I believe, It's better to address these issues now rather than later, as the potential damage grows with the increased use of AI across all industries.

Throughout this article, you will see the term "works" frequently used to refer to any content to which copyright can be assigned, including images, artworks, and written text. To avoid confusion, the user of the AI model will be referred to as the "human operator", this is separate from the “creator” of the AI model itself.

For the purposes of this article, we will primarily be discussing the legal issues surrounding AI art generators, as they are currently at the forefront of this debate. However, the proposed solution discussed here is applicable to all AI models, including text-based ones.

A crash course on the AI copyright debate, and a disclaimer

To oversimplify an extremely complicated topic that could fill multiple legal essays on its own, the current legal status of AI-generated content is undecided, with lawyers and members of the public advocating for either side.

One of the high-profile cases (which is not finalized) is how the "US Copyright Office" has been going back and forth on the copyright status of an AI-generated comic (link here)

At its core, the AI copyright debate revolves around three major questions:

• Is it permissible (possibly under the concept of fair use) to use copyrighted materials to train an AI?

• Can AI-generated works be copyrighted?

• If so, can an AI model be co-assigned copyright of a generated work (e.g. an artwork) along with its "human operator"?

In general, these are the major arguments for AI.

• Proposition 1: AI learns and creates (words, music, art, etc) like a human

• Proposition 2: AI is just a tool

With the following 3 major opposition

• Opposition 1: Only "real" humans are eligible for copyright

• Opposition 2: Fair use is not automatic and is evaluated on a case-by-case basis

• Opposition 3: Lack of consent, from data sources used in the dataset

There are many additional arguments for and against AI that have been made by both sides, but they are not included here because they are either not as significant as the key points or they are based on ethical considerations rather than existing legal codes.

For example, one notable argument that was left out: is the "for progress" proposition. On how AI should be allowed to learn from copyright materials to encourage progress in society, and that resisting such progress is harmful to society. This is an argument based on ethical and societal considerations and not an existing legal code.

As I am skimming the details of the debate, if you want a more in-depth read, you can consider theverge article - The scary truth about AI copyright is nobody knows what will happen next

Finally, to state the obvious, I am not your lawyer, and the following is not legal advice.

Also, copyright laws differ drastically from country to country (Japan, for example, does not fully recognize fair use).

Now with all of that, out of the way ...

A Proposal to Classify AI-Generated Content and AI Models as Derivative Works with Attribution*

*with various limitations depending on the process used in each step

More specifically, the following sequence is proposed:

• Dataset as a collection of work, from its individual data sources*

• AI Model as derivative work, from the dataset, together with the AI model creator

  • If an AI model is used to train another AI model (instead of a dataset), it is classified as a derivative of the base model, its training data, and its trainer model.

• AI Output as derivative work, from the AI model, together with the "human operator" who provided a "prompt/instruction"*

This proposal allows us to divide the copyright problem into distinct steps and rely on existing legal frameworks and case laws, particularly the legal framework revolving around "Collaborative authorship" of open-source code. It also helps to answer the three questions distinctively:

In this framework, it would also allow answering the 3 questions distinctively.

• Is it permissible, under fair use, to use copyrighted materials to train an AI?

  • No, by default, unless it allows derivative work*

• Can generated works, be copyrighted?

  • Yes*

• If so, can an AI model be co-assigned copyright of a generated work (ie. artworks), together with its human author (user of the AI)?

  • Kind of. It would be attributed to*

This proposal avoids being stuck in the debate of whether an AI performs the creation of work "like a living being."

It also helps to create a clear path for all parties involved in a fair way:

• For artist

  • It gives them control over their own unique brand name, and art style, allowing them to commercialize it where possible

  • It does not stop AI from being used to generate artwork in known older art styles

• For AI Model Creators

  • It provides a legal framework for them to create and commercialize AI models safely

• For Human Operators

  • It allows them to generate AI artwork

  • It allows them to potentially commercialize such artworks for their own personal or commercial uses.

Understanding permissive license (usually with attribution clause)

One major factor that is often overlooked in the debate is how copyright is not binary, and how permissive licenses lie between full restrictive copyright and no copyright.

This proposal intends to take into account how permissive licenses are treated in the process and intends to use such licenses as a means to navigate the legal debate.

Because permissive licenses have been a hallmark in the open-source copyright movement for software coding (e.g. MIT license), this will help lay the groundwork for the proposed approach to existing case laws. (The list of open-source licenses can be found here).

Outside of the software industry, the Creative Commons is its most prominent variant, with over 2 billion works tracked in circulation.

Broadly speaking, permissive licenses are used within three major categories:

1. Content that cannot be used: Share Alike, and No-Derivatives

• No Derivatives is as obvious as it gets, No-Derivatives

  • Creative Commons BY-ND and BY-NC-ND

• Share Alike, or CopyLeft licenses, require the existing license to be copied over into the license of the derivative work. While technically this is possible within smaller focused datasets, for practical purposes this should be avoided due to the logical challenge when merging multiple "CopyLeft" licenses.

  • Creative Commons BY-SA and BY-NC-SA , for software code, an example would be the GPL/AGPL/LGPL licenses.

It is worth reminding the tech bros that no consent (to make derivatives) means no consent.

2. Content that can be used: Typically with attribution

• Derivative and distribution are allowed with attribution

  • Creative Commons BY, for software, an example would be the MIT license

• Derivative and distribution are allowed even without attribution

  • Works in Public Domain, or licensed under Creative Commons Zero

3. Content that can be used: For non-commercial use cases

• Distribution and derivative with attribution, limited to non-commercial use.

  • Creative Commons BY-NC

Placing an early prediction, there will be a derivative of the Creative Commons license which specifically prohibits the usage with a dataset or AI training.

Maybe BY-NADD (No AI nor dataset derivatives)

Important Note: Limitations MUST be carried forward

Permissive licenses are typically written in a way that enforces derivative works, and any collections including their work, or its distribution, to have the same limitation (attribution, non-commercial requirements).

Measures must be taken to ensure the same limitations apply to the dataset, AI model, and their generated content.

Failure to do so would break the current proposal, due to potential loopholes that may allow commercial companies to fund the creation of public datasets under "fair use" only to eventually use them commercially.

For example:

• LAION.AI - was commissioned to build and open-source a "Non-Commercial" dataset

• Stability.ai - built an AI model, based on the LAION.AI dataset, without the "Non-Commercial" restriction, and released it to the public

• Dreambooth.ai, and countless other commercial AI art generators - used the model to build their paid commercial AI art generator.

The end result is, despite starting out in a way that "does not create a competing product," due to its open-source nature, it ends up as a tool, that can be used in commercial settings to replace specific living artists identified by their name, who had their images scrapped within the dataset without consent.

How does it work in practice? - when making AI models

By classifying, downstream datasets, and AI models, as derivative works of their upstream content, copyright can be respected across a chain.

This is similar to how open-source projects work in the software industry, allowing existing laws to work on each layer.

• Dataset layer: Will be based on existing web-scrapping laws

• AI Model layer: Because the final AI Model can be treated as "Code" (it literally can be exported as code), existing open-source laws can handle it. A clear legal path will clear the way for commercial adoption

• AI Output layer: If we treat the AI model as code, with licensing restrictions that force attribution to its output, it falls into place with several existing code licensing models.

In all cases, attribution needs to be maintained up the chain.

For example, all AI models must attribute their datasets, which must in turn provide attribution for their sources.

Once this is established, there will be three major types of AI models:

• Commercial use-friendly AI models, which can be used in commercial settings

  • Creative Commons `BY` or something similar

• Non-Commercial use AI models, which inherited the various "No commercial use" restrictions from the original data

  • Creative Commons `BY-NC` or something similar

• Privately licensed AI models, which will typically be based on a "Commercial use friendly AI model" and finetuned for specific licensed use cases

For right holders (ie. artists), who do not want any of their works to be used to train an AI model, or to be used by others as derivative work. They decide and stay in control, via the licensing terms.

For rights holders, who do allow derivative works, they get to decide if it's restricted to non-commercial uses (non-competing).

Finally, right holders, especially those with very unique styles, can sub-license their data, or AI model, and set their own terms of use. Typically as a way to commercialize their unique style.

How does it work in practice? - when using AI models

Using the AI model is more nuanced. Depending on if the AI is adding IP value, or not, there are two major categories that need to be outlined.

AI as an Intellectual Property generator

The general rule of thumb is that the output would be the derivative of all three participants: the AI model, the human operator (via a prompt typically), and the input material (if any, for example, editing or redrawing an existing image).

This is because the final output would literally be a derivative of all three inputs.

If any of the three enforces a copyright clause (ie. attribution, no commercial use), so will the derivative output. If the AI model follows the Creative Commons model, and the "Human Operator" has full rights to the input, the "Human Operator" would have full rights to the output, with the attribution clause limitation (and the no-commercial clause for no-commercial use AI).

If it's just the "Human Operator" and the AI, the above structure still applies.

AI as a reduction / transformation tool

This is currently limited to the AI text model (as far as I know), where the AI is used in a way that does not add IP value. For example in "Short Summarization, Labeling, or Factual Extraction".

In this case, if the output has no additional content added and is simply the reduction of the input, or transformation (eg. translation), per the user instructions, and was done so in a way that is clearly mapped to the instruction - the AI has arguably not "added any IP or context".

As such, in this case, the AI model should be treated as if it was a tool/extension of the human operator. And the output copyright should be as if the "Human Operator" performed the said instructions manually. (Need legal feedback on this segment)

Will this stop AI from taking artist jobs?

Quite frankly, No, the bar has been raised for being an artist.

If the artist's primary job was commissioned to render artwork in a commonly known style, that is out of copyright (eg. Vincent van Gogh style) or is very common in the public domain.

However, if the artist were to make or remix a newer style, that is unique and distinctive to them.

Then Yes.

This prevents their new style or likeness, from being copied. And allows them to commercialize and be rewarded for their style on their terms. With time, like all artworks - it enters the public domain.

Ultimately, allowing them to make a name and career for themselves, for their unique style.

Into this, strange new AI world

Alright, what's next?

As I am also neither a professional artist, lawyer, nor politician. All of this may just end up being a programmer screaming into the crowd.

However, if this approach makes sense to you, especially if you are an artist, do feedback, forward it to as many legal friends, and start organizing a movement towards this.

For any lawyers who wish to refine this to a legal paper, please do so, reach out to me, and provide credit.

The only paper I can find exploring this angle is "AI Derivatives: the Application to the Derivative Work Right to Literary and Artistic Productions of AI Machines"

At the end of the day, I do wish that we all steer in a direction where, both AI and artists, can work together hand in hand. Either with what's proposed, or any other alternative proposal that respects both sides.

~ Until next time 🖖 live long and prosper 🚀
Eugene Cheah @ tech-talk-cto.com

For more on how AI works by dreaming,

Tech Talk CTO

Introducing AI / ChatGPT, and how it dreams in 2022

This article focuses more of the current state of AI, on 5th Dec 2022. As a means of briefing others outside this industry, what has been going on recently, and what are its limitations. With a focus on recent developments in chatGPT, over long term speculations…

Read more

3 years ago · 1 like · PicoCreator

All Images used, with their proper attribution

• Hand Stacked Coins With Justice Scale by focusonmore.com under CC BY 2.0

• Race Car Photo by Abed Ismail via unsplash.com

• Goals and Plans Photo by Content Pixie on Unsplash

• Creative Commons Licensing chart by kathleen_morris

• Stop Hand Photo by Nadine Shaabana on Unsplash

• Pole Vault Photo by Frans Vledder on Unsplash

• Sunrise Photo by Nicole Avagliano on Unsplash

Let’s ignore the public hype that is chatGPT for a moment, its ethics or its implications to society, and let’s focus on actual field notes in building an AI based on its research and the tech, for your startup or your portfolio.

TLDR 1: Dataset is no longer a hard requirement

TLDR 2: It has very broad use cases

TLDR 3: Cost and Limitations

If you were on holiday, and have no idea what are ChatGPT or LLM (large language models) - you can find the briefing about the technology here :

So what do I mean in longer form?
Grab a cup of coffee and settle in for a bit…

Part 1: Data is no longer a Hard Requirement

Crash Course On Traditional Text-Based AI Data Requirements

Note: This article is discussing only text-based models and does not apply to image models in many cases (e.g., computer vision).

In the past, building an AI model typically involved gathering a large enough dataset for a specific use case. Different AI model training techniques and tools existed to optimize the training process or its computational costs. However, the size and quality of the dataset was still one of the predominant factors in training an AI.

This would result in a graph something like the following:

All numbers are large approximations and will change drastically depending on the use case, AI model, and data quality. The scales on the axes are intentionally vague and inaccurate because they are subjective. For technical numbers, read the papers in the citations for specific use cases.

The general idea remains, however, where small-scale datasets produce effectively random data before slowly becoming better and finally being somewhat useful at approximately the ~500GB part.

This led to the dataset arms race, occurring in nearly every field of specialisation related to AI (especially anything computer vision-related). Which played out across multiple startups from 2009 onwards until today, with a few notable startups being acquired in part for their valuable datasets (and usually talent).

Because the arms race was constantly escalating, it became increasingly difficult for new startups to compete with their AI models (small datasets) against incumbents (larger datasets).

For example, at uilicious.com (low-code UI test automation), we used our limited dataset to train an AI model (called TAMI v0.1). We found that it was spewing garbage half the time, leading us to set the model aside and grow the company without AI while building up our dataset.

In a very overgeneralised and non-technical fashion, AI trained in this way will be referred to as "Specialised Models" from here onwards.

The Start of LLMs or Large Language Models

In the quest for a truly universal or general-purpose AI, especially in the field of human speech interactions (as humans make for the most random of all variables), efforts arose to train a new type of text-based AI for extremely large datasets of public data (think Wikipedia, Quora, StackOverflow, and half of the internet text).

Because these new models pushed the boundaries of dataset sizes and model size (think brain size), the methods used to build them diverged from specialised models (which tend to have more emphasis on accuracy and efficiency).

Text AI models trained in this way are now collectively called "Large Language Models" (LLMs).

The downside of this approach was huge, preventing its early application in many use cases:

• Extremely large datasets need to be built and maintained (GPT-3 used 45 TB)

• Training costs millions of dollars (GPT-3 cost over $4M to train, GPT-3.5/chatGPT is presumably higher)

• Limited use for specialised use cases due to the lack of private training data

• Lower accuracy than specialised models in specialised use cases

• Expensive to run (more on this in limitations later)

LLM was dominated by Google (who already had the data and had the motive to do so for their AI assistant) and OpenAI initially. Later, Microsoft, Amazon, Apple, Salesforce, Meta, and a few others joined the race.

However, due to the sheer size and cost to train such a model, it's generally reserved for major tech companies with deep pockets.

Though the first few generations of LLMs may have produced disappointing results, as they lost out to nearly every specialised model in every task, things changed as years passed and they scaled up in both dataset size and model size.

Their benefits became more visible:

• Facts and accuracy aside, they became really good at talking to humans

• In many (but not all) cases: they are really good at learning new specialised knowledge when given the datasets in an appropriate format
  (you can't just dump it in)

This made drastic changes to the curves:

It also sparked a new wave of AI-backed business models in 2019. Startups could now train new AI models on top of existing LLMs with datasets within their reach, from chatbots to presentation slide generators, code copilots, copywriting and even D&D gamemasters.

No longer was these AI models in the exclusive domain of large tech companies. For a small fee for training and running the AI on their infrastructure, OpenAI and Google started opening up their models for others to build upon as a means to profit from the LLMs.

This was beneficial to startups, too, as they no longer needed millions of dollars in R&D investment required to build these large models internally, allowing them to reach the market faster with their proof-of-concepts.

Data was still, in a way, king. It may have moved the curve to be more accessible, but it still required a team to build up large datasets and tune the model. As a result, while many of the 2019 startups found it significantly easier to build their prototypes, many found it difficult to cross the "useful" threshold, as they needed to scale up their dataset with diminishing returns.

This is consistent with early internal trials of uilicious.com TAMI AI models - basically it was between interesting and "meh".

But that's when things start to really change in 2022....

How Large Language Models are Few-Shot Learners

The release of GPT3.5 (or chatGPT as it's known on the internet) last month (December 2022) had huge implications for the way AI models are trained and fine-tuned.

It has unlocked the possibility of creating usable AI with extremely small datasets - something that most startups have access to or can create by hand. This is a fundamental shift in how we think about AI training.

At uilicious.com, we were stunned when we discovered that a small sample dataset smaller than 1GB, of our larger ~100GB dataset. When converted and optimised with the new training techniques, had exceeded the "useful threshold" - the point at which the AI could be utilised by our users and outperform everything we had previously.

While subsequent experiments with larger datasets showed diminishing returns. The key takeaway, was how little data was needed to "make a useful product". It took us less time to build a proof-of-concept AI, fine-tuned for our proprietary use case, than it took to write this article.

Using GPT3.5 as a basic building block, it is now possible to build useful AI applications for various use cases without needing a specialised team or individual.

Depending on the use case, the dataset can be as small as a single sentence or paragraph, or up to 100MB or 1GB - a size that is achievable for many startups.

Alternatively, if you can get chatGPT to act in a way that you find useful and of value to your startup. You can build it as a dedicated AI service.

The drastic reduction in required dataset made it possible for us to build a "useful" AI model with only a small fraction of our full dataset - something that had previously been "useless" or "impossible" at our scale.

In many ways data is no longer king (thought still important), finding and building useful application is the true kingmakers with this new AI.

Screenshot is a demo of our AI in action, generating full UI testing scripts, from a prompt, as an example integration.

Something which is now doable in a week, with the right team, and the right data.

Do reach out to us if you are interested in the future of AI and UI testing.

Part 2: The Hunt for Use Cases (and AI Interns) has Begun

The intern framework for finding use cases

GPT-3.5 and all other large language models have limits on their accuracy and reliability. This is partly due to their nature of dreaming out answers.

While in theory (yet to be proven), given a large enough specialised dataset, its accuracy can be drastically improved for specialised use cases. However, if you have such a large dataset, chances are you could already build "specialised models".

Ultimately still, this limit on accuracy and reliability is only a critical issue in sensitive industries (e.g. medicine). For most cases, it is merely a distraction in finding use cases. Once the AI crosses the “good enough” threshold.

A healthier and more realistic approach to the problem is to imagine GPT-3.5 as a remote and paid “smart college intern on demand”

Because realistically, the AI has all the same limitations in such use cases, in addition to being remote and online:

• Never trust interns to be 100% right all the time (let alone diagnose medicine)

• Create a workflow that allows the intern to safely provide useful value, with the appropriate supervision, guidance, and planning

• For one-day or one-month interns, you would create a formal training plan to quickly get them productive in one hour or one week of training, respectively- You should be paying your interns; they are not free (both for humans and AI)

The only real benefit the AI model has over real-life interns is:

• They are not affected by school seasons

• You can scale both up and down at any second

• The training is something you can do once per batch and scale up to as many instances

The downside the AI has over the humans is that they cannot get you coffee in person.

Once you frame it in these terms, it becomes significantly easier to figure out how to integrate the AI into existing business processes or products.

Need to generate article ideas? Get a one-day intern.

Need to convert some proprietary Java to JavaScript code? Get a one-month intern with some basic training on how to do the process.

And so on and so forth - where at no point in the process should the human be removed from the loop in supervising and iterating with the intern.

On a slightly more technical level:

- A one-day intern can be quickly built using prompt engineering; anything you manage to get chatGPT to do fits into this category. The downside is there is a practical limit (about 2000 words) of what you can fit into the training data. The plus side is you can experiment and set this up in seconds, and it's very easy to test with chatGPT.

- A one-month intern (or anything in between) is where you start having a formal training dataset, which they can memorise and learn. They will be able to handle more situations according to the training materials. The downside is you actually need to prepare that training material; and its a more technically involving process.

Part 3: Cost and Business Limitations

Cost per Prompt Will Make or Break the Business Model

This is the biggest weakness for this newer approach in AI, which is based on bigger and better models. Unfortunately, there is no free lunch. While it is cheaper and easier to train for specific tasks in terms of dataset size, it is significantly more expensive to run compared to more traditional AI models.

It is not cheap; the cost per prompt and answer range anywhere between one cent and fifty cents, depending on how much data was needed to train or use in the process. This is drastically higher than a typical API server, which can handle a million requests per dollar.

To put it in simpler terms, it costs more in server hardware to handle one AI query for one user within a given second than it does to serve a million user requests for a typical medium-sized Shopify website.

This is not due to OpenAI or Azure soley trying to make a profit; it is down to the pure server hardware infrastructure required to run such large models.

As a result, as powerful as it is, incorporating such Large Language Model AI has a hefty price tag and may not be applicable for all use cases due to this limitation alone.

The end result is that, while many use cases can benefit from the use of such an AI, not all use cases can afford it; and this should be an important consideration for any implementation.

Example A: Support Service Chat

For a support service desk, a typical support staff can handle ten customers per hour, with each customer having an average of fifteen back-and-forth prompts. If it is five cents per prompt - this adds up to $7.50 s an hour if an AI was used to mimic a single support staff.

This is not only cheaper than the typical US call center staff median salary of $15 per hour, it is also much more flexible (no staffing overhead, can scale up and down instantly).

Likewise, it is also possible to take the same "intern" approach, where these support AI serve only as L1 support, allowing the humans to handle the more complicated cases. In this scenario, it makes sense when done and scaled appropriately on a per-prompt (or per-hour) basis.

Example B: A SaaS Service to Draft Emails

The average office worker replies to approximately forty emails per work day, or approximately 880 emails per month. Even at five cents per email, that would be $44 per month on average per user just to handle the email replies.

What makes it worse is that it's reasonable to expect that, with such a service, an office worker would be able to reply to more emails on average. It would not be unreasonable for the average to double to two thousand emails, or one hundred dollars a month, in pure AI costs alone.

In this case, if the SaaS startup were to do a simple, let's say ten-dollar-per-month pricing, they can get into heavy potential losses over time.

This pricing cost and business model runs counter to the typical, per-user pricing model that is common in SaaS. Which is why it is common for such integrations to have a "credits" system as a means to cap the usage, and a means of billing, of such an AI.

It is expected that overtime, with better fine-tuning, competition, and cost optimisation, the price per prompt can go down. Another notable method is to use the original more expensive AI first at launch, while collecting more data, which is then used to train a more specialised and cheaper model. However, all of these methods go into deep technical details, which may be unique for each use case, and generally require significant time and effort.

And even then, while it can be a drastic ten times in savings, it is fundamentally more expensive than traditional SaaS API services.

OpenAI Has an Effective Monopoly (For Now)

Prices May Be Stuck Until Competition Arrives

While there are existing open-source Large Language Models, in very frank terms, they are either comparable to GPT2 or somewhere in between that and GPT3.5.
For some simple cases, once they start building up a reasonable dataset, these smaller (and cheaper) models may be useful to migrate into as a means of cost cutting.

However, for other complex cases, such a move may not be possible due to the complexity of their AI, giving OpenAI an effective monopoly with no incentive for lower pricing.

It is believed, however, that over the course of the next one or two years, the open-source community will catch up and, in doing so, perhaps allow prices to improve due to better alternative infrastructure providers. However, because this is an uncertain future, it is worth highlighting.

OpenAI Is a Potential Unwilling Competitor for Some Models

While not intentionally, it is critical that startups within this space build feature sets that are defensible beyond their prompt-to-text bots.

For example, there were several smaller startups who built GPT3-based bots or SaaS around specific prompt-to-text use cases, like name generators or idea generators with a simple interface.

Literally overnight, with the launch of chatGPT, these small single-use-case text-to-text SaaS were made redundant, as everyday folks can now get the same functionality via chatGPT for free. While it may not have been OpenAI's intention to compete with the very partners that build on them, it may be its inevitable nature to do so, as they keep improving their model and chatGPT.

As such, to ensure this does not repeat itself, it is critical for any business model around this technology to figure out what additional value they provide beyond just prompt-to-text, perhaps better user experience or integrations with existing toolings, etc.

Lastly: It Ain't Perfect

Reminder of the intern model, do not expect to use this for curing cancer tomorrow. So please do not go stuffing AI into every product and startup on Earth when it makes no benefit to the end user to do so.

~ Until next time 🖖 live long and prosper
Eugene Cheah : CTO of uilicious.com

Additional notes that didn't fit in

swyx also does an excellent job in trying to consolidate various information, in this rapidly chaotic and growing space, which is very worth reading

• The Day The AGI Was Born: https://lspace.swyx.io/p/everything-we-know-about-chatgpt

• How Open Source is eating AI: https://lspace.swyx.io/p/open-source-ai

• His live AI notes: https://github.com/sw-yx/ai-notes/blob/main/TEXT.md#chatgpt

Currently BLOOM is the main opensource contender for GPT3 (not 3.5): https://towardsdatascience.com/run-bloom-the-largest-open-access-ai-model-on-your-desktop-computer-f48e1e2a9a32

In the AI / ML industry, the ability for a LLM to quickly learn new concepts and apply them is quantified and measured by tests called "Zero-Shot", "One-Shot" and "Few-Shot" learnings.

Generally the better the AI does in these test, the less data you need to train it for your use case.

In my opinion: This make sense in hindsight - who would have thought then a Neural Network modeled after humans, would act like humans. And benefit from the T-Shape education model. Where a large breadth of generalised knowledge, help improve ones ability to learn specialised knowledge in one are of expertise. (this statement is not backed by any data)

Other sources and citations

• GPT-3 45TB training set: https://www.springboard.com/blog/data-science/machine-learning-gpt-3-open-ai/

• GPT-3 estimated $4 million++ plus training cost: https://lambdalabs.com/blog/demystifying-gpt-3

• Bloom AI, an opensource competitor $1.6 million training cost: https://techcrunch.com/2022/07/12/a-year-in-the-making-bigsciences-ai-language-model-is-finally-available/

• T-Shaped model of education and work

  • https://wordspy.com/index.php?word=t-shaped

  • https://collegeinfogeek.com/become-t-shaped-person/

• Large language models papers, for zero and few shot learnings

  • https://arxiv.org/abs/2205.11916 (zero shot)

  • https://arxiv.org/abs/2005.14165 (few shot)

• Call center salary rates in US : https://www.comparably.com/salaries/salaries-for-call-center-agent

• Average email sent by an office worker a day: https://www.templafy.com/blog/how-many-emails-are-sent-every-day-top-email-statistics-your-business-needs-to-know/

• OpenAI announcement of chatGPT / GPT3.5 : https://openai.com/blog/chatgpt/

• Unsplash photo credits

  • https://unsplash.com/photos/7swaW1bYpWI

  • https://unsplash.com/photos/1iVKwElWrPA

  • https://unsplash.com/photos/SELXIJwN24s

  • https://unsplash.com/photos/3xwrg7Vv6Ts

  • https://unsplash.com/photos/-G3rw6Y02D0

This article focuses more of the current state of AI, on 5th Dec 2022. As a means of briefing others outside this industry, what has been going on recently, and what are its limitations. With a focus on recent developments in chatGPT, over long term speculations.

For those in the tech / AI industry, if you need an article to explain what is going on to your non tech friends, this article is written for that use case.

Recommended music with the article - GITS Stand Alone Complex: Be Human
@youtube-music @spotify

In 2022 artificial intelligence (AI) has made some major strides in its capabilities. Largely in the two following area

• Prompt to text generation

• Prompt to image generation

Collectively, these "prompt-to-output" models are designed to take a prompt (or a question) and generate a finished product in either text (like a clickbait headline, or programming code) or image form (an illustrated art image, or stock images) based on that prompt.

This type of AI is incredibly powerful due to how simple it is to use, and has the potential to revolutionise many different fields, and can literally feel like magic.

An example would be how a single prompt like

Hey, I'm trying to come up with some interesting, fantastical way of decorating a living room for a design project

Can be used to generate the following image

Or write business plans for a website

Or even be used to help generate entire novels

Because its full scope of its capabilities really requires you to try it, to better understand the potential for yourself. I would highly recommend giving these services a try, while its still free at https://chat.openai.com/chat
(it may turn to a paid service in future)

Holy Crap, can it generate anything?

Yes, but that includes dreaming up lies.

In an extremely convincing fashion that even experts can find it hard to tell the difference.

It can also be very stubborn in how it present itself, so as hard as i tried to get it to help me write this article, unfortunately it just couldn't be "me".

In a weird way, you should view these AI as unique assistant who ...

• is a naive smart kid

• who is slightly high or dreaming in the clouds

• have very unique and flawed views

• and is very willing to bluff their way, in providing a response, even if they dun have the answers

If you ever dealt with such an individual, you know that there are random nuggets of pure gold wisdom that they will say from time to time, and bulls#!t you will need to filter, as it dreams up the answer.

The later act of filtering, is potentially the biggest danger of this new technology.

As it can be very very difficult, because of how convincing the AI structure its answer, if you are not an expert in the respective field.

For example, as an experiment, I asked it questions in domains i had zero (or even negative) expertise, like cooking or chemistry. And I realised I have nearly no capability, in validating its bluffs.

This is despite my best attempts to google and fact check it at times, where it may give sometimes answers that is slightly different, but I have honestly no idea if its right or wrong. Unless I try it myself.

Which can potentially be dangerous, seeing as how it can get things wrong in my area of expertise.

How is that possible, wasn't it trained on nearly a terabyte of data?

Yes, but it is not thinking, it is dreaming.

To elaborate, let’s start with an oversimplified explanation on how these AI models fundamentally work.

First the AI is given a large dataset of articles or images from various sources, it uses this data to provide the foundation of its knowledge, loosely connected together. This is then trained together with possible prompts and responses given by the user.

Once this is completed, we have a working neural network or trained model with all this loosely connected data. Frozen in time.

When a prompt or question is given, the AI is not thinking of an answer or deliberating from the trained model. In a way, it is hallucinating the answer using the snapshot of knowledge it has.

Now, lets use a human analogy instead

The human analogy, would be your brain right now, with all your past knowledge, or experiences in school or in life. Snapshot and recorded, in your head. Minus your emotions, and your feelings, or anything hormonal related.

When a prompt is given, imagine it as being told to you, before going to sleep into a dream. For example "go buy milk tomorrow, when you wake up".

For most folks, sleep is when they as an individual have little to no control with how the story goes (ignoring the lucky few lucid dreamers).

In this state, your mind in its dream state, starts to take that last phrase that is stuck in your head, and starts dreaming a story around that phrase.

Where you are an individual inside a dream, and your brain the dream world doing its best to be as convincing as possible that its real and not a dream world, continuously writing the story back and forth with your interactions and actions.

All while using the snapshot of all your past knowledge and experience, in your brain. To make your dreamworld as realistic as possible

Flipping it back around

You, with your prompts is a participant in a dream.

With the AI being the dream world, trying its best to be, as convincingly as possible, a good dream. Where it answer your questions in a very believable manner.

And just like your dream world, it can be at times entirely realistic, and factually accurate, or as imaginary (and perhaps exciting) as it wants to be

As such, this dreamlike state, allows it both to make basic math mistakes with high confidence

To hilariously answer questions as a cat when instructed

To very impressively, even emulating interactions with a virtual machine (rather accurately!)

What does it mean for the future of work?

It's difficult to predict exactly how the advancements in AI will impact the future of work, but there are some potential scenarios to consider.

One thing that seems clear is that the current wave of "prompt-to-output" models is having its "iPhone revolution moment". Just as the iPhone changed how everyday people interact with software and ushered in the era of smartphones, prompt-to-output models have the potential to change how we work and interact with tools.

It has already started fundemnatlly changing industries, today. Especially in SEO copywriting, and stock illustration / photo industries.

The next wave could manifest in waves of AI assistant work tools and practical enhancements to existing tools, ultimately making automation more accessible and lowering the barrier of entry for performing tasks.

Regardless of the forms of changes, this is a significant development and every startup founder (including me) and product owner should be taking note of how to integrate AI into their business. Just as the smartphone/iPhone era made entire industries and tools obsolete and created new domains of app-based tools, the rise of AI has the potential to do the same.

However, it's important to note that not all tools will be impacted by AI. Some industries, may not see much change beyond a new skin or UI. Only time will tell which side your business falls on.

PS: if you are feeling FOMO, and is in the tech industry, chill - the iPhone and app revolution happened in months and years, not days. Do go on with your life, and your holiday plans.

As long as you are not ignoring the potential changes, and making steps towards it - you are good.

If you are outside the tech industry, keep a look out, be ready to learn new tools, across the next few years - that may change your industry. And be prepared to learn and use them to your advantage.

~ Until next time 🖖 live long and prosper