ChatGPT recently became the fastest growing product in the history of technology. The impressive technological demonstration of intelligent language capabilities has captured the imagination of customers, as well as technology experts. The global tech industry has turned its attention swiftly to developing products that use this form of artificial intelligence.
This technology is powered by an LLM, or large language model. LLMs are computer programs that are fed large amounts of text. These programs then identify patterns in this text. When learning from an incredible amount of human-produced text available on the internet, combined with increasingly available computing power and improved algorithms, these programs produce impressive results that could improve our productivity significantly.
ChatGPT was launched in English, and over time we will see the expansion of this technology to other languages. But when can we expect the boosts in productivity promised by large language models to reach Pakistanis in their native languages?
Technology is often built first for the markets it is developed in. This is why new computer technology is often English first, given that America is the home for academia and industry in this field. Increasingly, we will also see hubs in China and India. Given Pakistan’s global position in the technology market, we may not see gains as fast as other countries.
A more worrying concern still, is that even when technology is built elsewhere it will know proportionately less about Pakistan than other parts of the world. Artificial intelligence learns from existing published text. Unfortunately on the internet today, Pakistani languages – and by extension, knowledge about Pakistan – do not rank highly in published content. As an example, Urdu and Punjabi using the Shahmukhi script are the 20th and 22nd largest languages in the world by native speakers. Yet in terms of the size of their respective Wikipedia instances, they rank only 55th and 89th.
To obtain a foothold in the mind of global technology we must make it easier for Pakistanis to use computers in their native languages. This includes Urdu, but also regional languages such as Punjabi, Sindhi, Pashto, Balochi and the dozens of other smaller languages native to this country.
Technology for these languages is dated and not easily accessible to most consumers. Most Urdu publishing uses the InPage word processor, developed in India in the 1990s. We have had few developments, especially ones not reliant on Western technology makers, since that time.
Today, text produced in Pakistan uses pirated versions of the InPage software, losing even the best facilities available in that package. The text we produce is riddled with errors of spelling and spacing. We use only a handful of fonts, with many known issues. New users find it onerous to understand Urdu typing, and are not aided by easily available dictionaries, spelling and grammar correction that is available to them in English. As a result, even the most populist communication needs such as advertising, customer support, and even government communication have resorted to using Roman script.
There is nothing inherently wrong with using the Roman script, but what our need to resort to it represents is what we should be worried about. Young people find it difficult to engage with Urdu or their native languages in the scripts they have been written in historically. Over time they will find it harder still to engage with historical material produced in those languages, and may be dependent on English solely for their academic and professional pursuits. At that moment, we will lose a lot of cultural relevance for future generations, as well as new technology.
How can we address these issues? We must make strategic investments in producing critical infrastructure to allow us to better represent and utilize our native languages in modern technology. This includes investments in dictionaries, fonts, typing, and voice.
Together with researchers at Harvard University and the Technology for People Initiative at LUMS, we developed a best in class dataset of Urdu text. This dataset, called Makhzan, sourced data from publishers that we knew to have a high editorial standard when it came to written Urdu. We fixed tens of thousands of errors in this text, and reformatted it to be best suitable for training AI programs, and auditing the input of these programs. After putting in years of effort, we worked with our team and donors to make Makhzan totally free to use for commercial, academic and personal projects.
I then used Makhzan to build a breakthrough Urdu keyboard for smartphones. This Urdu keyboard, called Matnsaz, uses the easiest keyboard layout ever introduced for the Urdu language, by reducing the number of keys. Matnsaz combines the separate keys for multiple letters sharing the same shape (such as jeem, cheh, heh and kheh) and combines them into one button. It then uses the intelligence learned from our dataset to add dots automatically. The result is a new keyboard that is easy to use for children, the elderly, students and professionals. And despite being easy to learn, it produces better text with less errors. Matnsaz is built for modern devices and uses the latest technologies available on the platforms that Urdu speakers are using.
This work and its design emerged from years of research with our customers. We studied the history of technology built for the Arabic script, interviewed customers of various ages, and tested this new technology with hundreds of people over many years. The result is a demonstration that we do not need to depend on individuals from outside our region to build technology for us.
The growth of broadband internet in Pakistan will enable one of the largest new groups of users to access the internet in the coming years. Building successful technology for this new population in their own languages is an incredible opportunity. Technology is most useful when it understands and solves problems for its users. A technology world envisioned and built in an English-speaking mould cannot benefit Pakistanis the same way as a new design methodology rooted in our own culture.
My choice to build new typing technology, and open-sourced resources for new language tools, was built on the belief that enabling the written word is the foundation of furthering new understanding of our people. If we create new writing tools for the young Urdu speakers, they will be better able to capture the lives they lead in new literature. Our disabled are more likely to find new media with subtitles, scientific and sociological studies will better understand local opinions.
When looking at the global languages of business, media and politics, one can draw the conclusion that holding on to Urdu, and our native scripts, does not open new doors. But I venture that we cannot enable new business, media and politics fully without holding on to Urdu. Often, work done for our culture is considered only with an altruistic lens. But there is a real economic motivation that should encourage local software developers and technology companies to make long-term investments in local languages.
This technology is powered by an LLM, or large language model. LLMs are computer programs that are fed large amounts of text. These programs then identify patterns in this text. When learning from an incredible amount of human-produced text available on the internet, combined with increasingly available computing power and improved algorithms, these programs produce impressive results that could improve our productivity significantly.
ChatGPT was launched in English, and over time we will see the expansion of this technology to other languages. But when can we expect the boosts in productivity promised by large language models to reach Pakistanis in their native languages?
Technology is often built first for the markets it is developed in. This is why new computer technology is often English first, given that America is the home for academia and industry in this field. Increasingly, we will also see hubs in China and India. Given Pakistan’s global position in the technology market, we may not see gains as fast as other countries.
A more worrying concern still, is that even when technology is built elsewhere it will know proportionately less about Pakistan than other parts of the world. Artificial intelligence learns from existing published text. Unfortunately on the internet today, Pakistani languages – and by extension, knowledge about Pakistan – do not rank highly in published content. As an example, Urdu and Punjabi using the Shahmukhi script are the 20th and 22nd largest languages in the world by native speakers. Yet in terms of the size of their respective Wikipedia instances, they rank only 55th and 89th.
To obtain a foothold in the mind of global technology we must make it easier for Pakistanis to use computers in their native languages. This includes Urdu, but also regional languages such as Punjabi, Sindhi, Pashto, Balochi and the dozens of other smaller languages native to this country.
Technology for these languages is dated and not easily accessible to most consumers. Most Urdu publishing uses the InPage word processor, developed in India in the 1990s. We have had few developments, especially ones not reliant on Western technology makers, since that time.
Today, text produced in Pakistan uses pirated versions of the InPage software, losing even the best facilities available in that package. The text we produce is riddled with errors of spelling and spacing. We use only a handful of fonts, with many known issues. New users find it onerous to understand Urdu typing, and are not aided by easily available dictionaries, spelling and grammar correction that is available to them in English. As a result, even the most populist communication needs such as advertising, customer support, and even government communication have resorted to using Roman script.
There is nothing inherently wrong with using the Roman script, but what our need to resort to it represents is what we should be worried about. Young people find it difficult to engage with Urdu or their native languages in the scripts they have been written in historically. Over time they will find it harder still to engage with historical material produced in those languages, and may be dependent on English solely for their academic and professional pursuits. At that moment, we will lose a lot of cultural relevance for future generations, as well as new technology.
How can we address these issues? We must make strategic investments in producing critical infrastructure to allow us to better represent and utilize our native languages in modern technology. This includes investments in dictionaries, fonts, typing, and voice.
Together with researchers at Harvard University and the Technology for People Initiative at LUMS, we developed a best in class dataset of Urdu text. This dataset, called Makhzan, sourced data from publishers that we knew to have a high editorial standard when it came to written Urdu. We fixed tens of thousands of errors in this text, and reformatted it to be best suitable for training AI programs, and auditing the input of these programs. After putting in years of effort, we worked with our team and donors to make Makhzan totally free to use for commercial, academic and personal projects.
I then used Makhzan to build a breakthrough Urdu keyboard for smartphones. This Urdu keyboard, called Matnsaz, uses the easiest keyboard layout ever introduced for the Urdu language, by reducing the number of keys. Matnsaz combines the separate keys for multiple letters sharing the same shape (such as jeem, cheh, heh and kheh) and combines them into one button. It then uses the intelligence learned from our dataset to add dots automatically. The result is a new keyboard that is easy to use for children, the elderly, students and professionals. And despite being easy to learn, it produces better text with less errors. Matnsaz is built for modern devices and uses the latest technologies available on the platforms that Urdu speakers are using.
This work and its design emerged from years of research with our customers. We studied the history of technology built for the Arabic script, interviewed customers of various ages, and tested this new technology with hundreds of people over many years. The result is a demonstration that we do not need to depend on individuals from outside our region to build technology for us.
The growth of broadband internet in Pakistan will enable one of the largest new groups of users to access the internet in the coming years. Building successful technology for this new population in their own languages is an incredible opportunity. Technology is most useful when it understands and solves problems for its users. A technology world envisioned and built in an English-speaking mould cannot benefit Pakistanis the same way as a new design methodology rooted in our own culture.
My choice to build new typing technology, and open-sourced resources for new language tools, was built on the belief that enabling the written word is the foundation of furthering new understanding of our people. If we create new writing tools for the young Urdu speakers, they will be better able to capture the lives they lead in new literature. Our disabled are more likely to find new media with subtitles, scientific and sociological studies will better understand local opinions.
When looking at the global languages of business, media and politics, one can draw the conclusion that holding on to Urdu, and our native scripts, does not open new doors. But I venture that we cannot enable new business, media and politics fully without holding on to Urdu. Often, work done for our culture is considered only with an altruistic lens. But there is a real economic motivation that should encourage local software developers and technology companies to make long-term investments in local languages.