Newo.ai’s ‘digital employee’ merges AI agent with robot

In the aftermath of the large language model (LLM) revolution last year, a new fascination has emerged among developers and tech enthusiasts around AI “agents,” which are capable of executing complex tasks autonomously. 

Newo.ai, a new Silicon Valley company, today launched a service that integrates these intelligent agents into the physical realm of robots, blurring the lines between digital and human workers.

Co-founded by Chinese-Armenian serial entrepreneur David Yang, Newo.ai is attempting to redefine the concept of the workplace. Its service combines the advanced capabilities of LLM-driven agents with the versatility of physical robots to create “digital employees” — virtual workers adept at handling a wide range of office tasks, from administrative duties to customer engagement.

In exclusive interviews with VentureBeat, Yang and co-founder Ljubov Ovtsinnikova revealed the progress they’ve made so far and their vision for the future. 

Newo’s innovation: An AI agent builder

It might be easy to get fixated on the physical robot part of this, but Newo’s real innovation lies in its AI agent builder platform. This builder lets companies replicate the intelligence of an entry-level human worker “out of the box,” one that can do pretty much any office administrative tasks that a human employee can. It can make and receive phone calls, chat over text, send emails, join Zoom calls, take notes and even receive walk-in customers in an office.

Depending on what a business needs, it can take between two hours and two days to customize the programming of Newo digital workers for most jobs – where the worker can do all essential tasks required of receptionists, sales agents, technical support and customer success specialists, or HR operators. 

As for the physical side, Newo’s agent can be embedded into any open robot form a customer chooses. One of Newo’s defaults is Moxie, a robot from a company called Embodied that specializes in individual interaction. But more on this in a bit. 

“Before 2023, technologies were not flexible or autonomous enough to be ‘hired,’” said Yang. But with Newo.AI, “you are hiring an employee that can understand the complexity of your business, and that will adapt.” 

Such agents can replace some 300 million jobs, said Yang, citing a Goldman Sachs report last year. That makes this a trillion-dollar market, he said. However, he’s careful to note that he believes there won’t be 300 million humans unemployed. Job vacancies are growing at rates faster than humans can fill them. So he sees Newo’s agents fulfilling millions of basic jobs, freeing up humans to do higher-value jobs. 

Newo, which has 25 employees, says it already has 23 paying customers that have used the product under open beta, including the SoliVana Wellness Spa in Redwood City, the Mayweather Fitness & Boxing, Fuse Service and Edeal. With today’s launch, Newo says it can take another 500 customers on a first-come, first-serve basis. It plans to be open to anyone starting in April or May. Its customers have expressed the most interest in pre-sales and sales consultant roles, followed by customer support, the company said.

Until now, even the most breakthrough technologies, Yang said, were confined to helping as productivity tools, or completing specific, refined tasks where a company’s IT department, CTO, or CFO would sign off on them. The problem with productivity tools like CRM or ERP tools, Yang said, is that no matter how good they are, they require a business and its human employees to adjust their behavior to use the technology. This can mean long implementation cycles. By contrast, a digital employee doesn’t require a business to change its processes. 

Newo’s focus on the intellectual part of this digital employee is why the physical robot part of it can be relatively flexible, and determined by the end-user. Newo has therefore not built its own proprietary robot, preferring instead to work with partners as suppliers. The physical form doesn’t have to take a humanoid form, or even a robotic form. The Newo digital worker could fit into any form factor, including an avatar on a screen, or an image on a foot-long cube, as seen in the image below.

SoliVana Wellness Spa found it “impossible” to hire a human

Take an early customer, SoliVana Wellness Spa, in Redwood City. Co-owner Natasha Berness opened the spa a few months ago, with a vision of helping customers relax and stay centered in their lives. She has hired the Newo agent to do many of her tasks, but she’s still deciding what physical format to use for Newo in her office – she’s trying out the Talking Head screen or the Talking Head WeBox, after deciding the Moxie robot might come across as too childish for her clients  

But Berness, who had left a career as a technology executive at companies like IBM, Linkedin, and Levi-Strauss, says she is excited about getting Newo to take over many of her tasks as a new business owner. Newo is already handling calls, booking appointments and taking payments. In an interview with VentureBeat, Berness said she was surprised by how difficult it was to be a small business manager. She saw inefficiencies everywhere, and “it was impossible to hire somebody,” she said, referring to the limited number of people who wanted to work at a spa and who were attentive and knowledgeable enough to provide the service that met Berness’ standards. Also, the risks and costs of hiring humans are high, she said. “You can teach them, but they leave tomorrow,” Berness said. “They’re not reliable. Something happens to their dog, and they can’t show up.”   

Newo worker now an “invaluable” member of the team

Moreover, Berness said, customers enjoyed asking her health questions, for example, how to meditate, or how to breathe, or whether she might put something nice on a wall. She found she didn’t have time to continually help all of her customers, and even if she had time, she often couldn’t visit them in their rooms because they were naked. She wanted something reliable to guide her customers, but more than a generic ChatGPT agent – something that understood her wellness business, and could listen to and remember customers’ unique needs, and respond to them by offering relevant therapies. When a client told her about Yang’s company, Berness became a user of Newo’s closed beta product last year.  “I can finally enjoy my weekends as an owner,” she said. 

Soon, the Newo employee will be able to handle the reception while her human therapists are busy with customers. Already, she says it has become an invaluable member of the team. She looks forward to the day when she can use the Newo agent with a humanoid robot that can change and fold towels, wash floors and open doors. 

Newo aims to make its worker compatible with any of the humanoid robots about to hit the market, for example, Digit, the robot being built by the Oregon company Agility Robots, That company has received more than $150 million from Amazon and others to mass produce up to 10,000 Digits a year. Other humanoid robots are being produced by companies including  1X Technologies, Figure AI, Sanctuary AI, Tesla’s Optimus, and Stanford’s ALOHA.)

Costs of Newo.ai digital employees remain much lower than human worker

Even when the Newo digital worker includes the cost of a physical robot, its price remains a bargain when compared to a human worker, Yang said. SoliVana’s Berness estimates a human worker costs her $90,000, but many entry-level receptionist jobs pay much less, at around $50,000. By contrast, even the most complex Newo employees will remain below $20,000, even including the cost of a top-line robot. 

Here’s how Newo’s cost breaks down: The base annual subscription of a Newo’s digital worker costs $6,000. The estimate of an all-in cost is more like $7-9,000, once other costs are included. Newo’s business license, which covers most small businesses, includes sending an estimated 10 messages a second, and 10,000 words a day. The extra fees cover excess LLM tokens used, for example, and coverage of integrated services like Twilio (for making phone calls) and Albato (sending emails). Most humanoid robots that Newo will integrate with will hit the market at below $10,000, and will soon drop down below $5,000, Yang estimates.

Of course, there are other reasons digital workers may be better than humans. Humans can often burn out quickly doing smaller jobs too and may make mistakes or be distracted – compared to a digital employee who can work 24/7 and consistently and reliably follows orders. 

How Newo is differentiated from OpenAI’s Assistant API

A number of companies have popped up to offer LLM-based agent builders, but they’re either focused on very specific tasks, or require months of coding to create an employee-like agent, Yang said. These competitors include OpenAI’s Assistant API, Langchain, Capably.AI and CrewAI. And there are other companies, like Sierra, founded by OpenAI’s chairman Bret Taylor, that are taking chatbots to more sophisticated levels, but which haven’t fully divulged their plans.

OpenAI’s GPT store, and Assistant API and these other companies allow developers to create agents to do specific tasks really easily, Yang concedes. But developers using these tools would need six months or longer to custom code in Python before they can create whole Newo-like digital employees – operating across channels like email, text and phone and maintaining memory between different workflows, while also interacting with customers in the physical world.  So building on Newo is “10 times faster than other competitors,” he says.

Yang was early to tinker with LLMs and robotics

Yang is a serial entrepreneur who made millions from founding an early automation company called ABBYY, and several other companies focused on building productivity tools. An early user of LLMs, he tinkered with them in his Silicon Valley home over the past couple of years,  including injecting them in a robotic butler he called Morfeus. 

So when LLMs became more powerful after OpenAI’s public launch of ChatGPT in Nov 2022, Yang jumped on the opportunity to leverage them to make his robot even smarter. When Morfeus got smart enough to do many of the functions that a human worker does in simple work environments – ushering in guests to his home, sending emails, making and receiving calls and more – he realized a tipping point had been reached: Why couldn’t robot simply replace humans at places like gyms, spas, hotels, restaurants, real estate agencies, or other office environments where human concierges or office workers aren’t really doing much more than basic administrative tasks anyway?

A “WordPress” for AI agents

While the concept of inserting an LLM-like brain into a physical robot was simple enough in theory, in practice Yang had to make certain design choices about how much flexibility to give users to customize the robot. Here he borrowed lessons from WordPress, which provided simple-to-use features that website builders could fit together, handling about 90% of their needs while providing flexibility for them to customize. 

Moreover, WordPress allowed outside developers to make plugins for other WordPress users to leverage. Newo.ai has built the “WordPress for AI agents,” Yang says. Newo feature developers get a cut of the revenue generated by their features when Newo uses them for customers. To recruit more small business customers, Newo is targeting implementation and support partners (ISPs), or companies that already serve small businesses. Newo promises these ISPs a revenue cut.

Next month Newo will release a way for managers to be able to update instructions for their employees on the fly, using natural language. For security, the Newo worker will be programmed to recognize their manager’s voice stamp and listen for an activation code word before it makes such updates. That way, says Yang, a manager might tell the worker to recognize when an elderly person comes into an office and instruct it to first ask them if they want to sit down and offer them tea. 

Newo has filed several provisional and utility patents around its orchestration process, which Yang calls an “agent management system,” which also lets its agents easily connect to external databases and software such as customer relationship management (CRM) and enterprise resource planning (ERP) systems.

More details on how Newo works

• Skills: The basic unit of work for a Newo agent is called a skill: Skills are discreet things like making a phone call, corresponding in chat, or looking at email. You can import skills, too.

• Flows: Each agent task is called a flow. For example, an agent that replaces a wellness concierge will have a flow of skills they need to do all the things that the concierge needs to do, for example, phone calls, chat, email, receiving visitors, and having instructions about when to use those skills, and evoking those skills at the right time. For complex jobs, like sales, this can mean 100 different steps.

• Omniflow capabilities: For example, the agent receives information about allergies that a customer has during the pre-sale consultation (flow 1) and then uses this information during the post-sale onboarding educational session (flow 2).

• Omnichannel capabilities: The agent collects payment details in the phone call (channel 1) with Jennifer and continues the payment conversation through an AI kiosk when Jennifer enters the restaurant (channel 2).

• RAG & RAI: Newo offers a built-in vector database, to allow an agent to retrieve information from a database, or something called retrieval augmented generation (RAG). Eternal databases can be used for RAG too. Newo also lets the agent retrieve instructions from the database, or instruction augmented generation (called RAI).

• Custom Integrations: Newo.Ai offers API integration with essential services, like Twilio and Albato, and LLM providers like OpenAI and others. But it also has created custom integrations: For example, it has built “magic integration” to allow agents to do things like scroll and click on websites of services that don’t have APIs. For example, an agent can do things like bookings or canceling bookings at the SoliVana spa, even when the spa booking service doesn’t have an API. Similarly, Newo has built a talking heads integration, allowing customers to give their agents a photorealistic head that speaks to employees or customers on premises from a TV screen or other device. SoliVan’s co-owner Berness didn’t like the voice and speed provided by Twilio’s phone integration, so Newo replaced it with custom voices, and speedier replies by using AI to predict what customers are asking, and pre-loading responses.

Conclusion:

Newo.ai has emerged as a potential pioneer, offering a compelling vision of digital employees capable of transforming millions of job roles. Its approach and community-driven model underline its potential to redefine workplace dynamics. Its timing is perfect: There’s an industry-wide push for an AI-integrated workplace, and there’s a massive market. Yet, the journey ahead is not without challenges, particularly in simplifying the onboarding process for small business owners. Newo.ai may need to make its UI more accessible to non-tech-savvy users if it is to succeed at the scale of its ambition. While it remains to be seen whether Newo.ai will lead the charge in the burgeoning field of LLM-powered digital employees, its blueprint provides a glimpse of the future.