Is Apple feeling the pressure?
According to The Information, Apple is ramping up its efforts to accelerate the development of LLM. Not only has the company significantly increased its research budget-burning through millions of dollars daily-but it has also poached numerous engineers from Google. Apple employees believe their Apple GPT model has surpassed the capabilities of GPT-3.5.
Soon, Siri will be more intuitive. Users can simply instruct it to "create a GIF from the last five photos and send it to my friend," and Siri will seamlessly execute the command without any manual intervention. The core team behind these advancements has been revealed, with many key players hailing from Google.
AI Head Regrets: A Step Behind
Apple had the potential to be OpenAI. Four years ago, Apple's AI chief, John Giannandrea, assembled a team to develop conversational AI, essentially a large language model.
While this move was visionary, it was a tad late. Last fall, OpenAI's release of ChatGPT garnered global attention. Several Apple insiders mentioned that the company wasn't entirely unprepared for the rise of large language models. However, Giannandrea had previously expressed skepticism about the utility of AI-driven chatbots. Now, Apple is determined to develop its own large model, regardless of the cost.
A 16-Person Core Team Racing to Develop LLM
How significant is this investment? Sam Altman once mentioned that OpenAI's GPT-4, which took months to train, cost over $100 million. In comparison, Apple's team, named Foundational Models, consists of around 16 people but has a daily training budget that runs into the millions.
The team, comprised of several former Google engineers (who previously reported to Giannandrea at Google), is led by Ruoming Pang. After 15 years at Google, Pang joined Apple in 2021. Insiders suggest that this team plays a role similar to the AI labs at Google and Meta-researchers develop AI models, while other departments integrate these models into products.
Furthermore, recent research papers and LinkedIn profiles indicate that Apple has at least two other teams working on language or image models.
Siri's Major Upgrade
Apple's team believes their most advanced model, Ajax GPT (or Apple GPT), has already surpassed GPT-3.5. With such a powerful language model, Apple's suite of products is set for a significant upgrade.
For instance, with a simple command, Siri can automatically create a GIF and send it to a contact in the phone. Additionally, Apple plans to launch an app called Shortcuts, allowing users to manually program sequences of app functionalities. These features are expected to debut in next year's iOS update.
However, how LLM will be integrated into products remains undecided. Apple has always championed user privacy, favoring offline, on-device processing over cloud-based solutions. Insiders reveal that Apple GPT boasts over 200 billion parameters. Running such a massive model requires not only immense computational power but also significant storage-challenging demands for an iPhone.
Apple's Transformation: Becoming More "Google-like"
When Giannandrea first joined Apple, his goal was to infuse more AI into Apple's software, like Siri. After being overshadowed by ChatGPT's success, he finally embraced the potential of AI chatbots. Fortunately, one of Giannandrea's wise decisions was to make Apple more "Google-like."
As a result, Apple employees were granted greater freedom and flexibility for research and publishing, leading to the existence of the Foundational Models team. Previously, Apple's restrictions led to talent attrition.
Top Talent: Poached from Google
Apple's team is a powerhouse of talent. The precursor to Foundational Models was led by Dutch computer scientist Arthur Van Hoff.
Van Hoff, an early member of the Sun Microsystems team that created Java in the 1990s, joined Apple in 2019. Initially, he was tasked with developing a new version of Siri (codenamed Blackbird), but Apple abandoned this version.
Later, he shifted his focus to LLM. Initially, the team was small, with notable members being British researchers Tom Gunter and Thomas Nickson from Oxford University, who specialized in NLP.
In 2021, Ruoming Pang joined Apple to assist with LLM training. Unlike other researchers, he was allowed to remain in New York, as Apple hoped to establish a machine learning outpost there.
Servers: Google is the First Choice
Apple's decision to bring on Pang reflects the company's growing realization of the importance of LLM in machine learning.
After OpenAI released GPT-3 in June 2020, Apple's machine learning team clamored for more funds to train their models. To save costs, Apple executives have historically encouraged engineers to use the more affordable Google Cloud services over Amazon's.
Apple Embracing "Open Source"?
Interestingly, under Ruoming Pang's influence, the Foundational Models team discreetly uploaded the machine learning framework used to train Ajax GPT, named AXLearn, to GitHub in July. AXLearn, based on Google's open-source JAX framework and accelerated linear algebra XLA, is optimized for Google's TPU and can be used for rapid machine learning model training.
However, Apple hasn't publicly released the data used to train the models. The reason for Apple's release of AXLearn remains unclear, but it's typically done to allow other engineers to make improvements.
