Transcript
Mark:
Welcome to PeopleTech, the podcast of the HCM Technology Report. I’m Mark Feffer.
My guest today is Roberto Masiero who runs the innovation lab at ADP. Over the summer they launched Roll, a payroll app designed for small business. The product uses a conversational interface to let business owners run their payroll by saying something like, “Run my payroll.” There’s a lot going on under this particular hood, and we’re going to talk about really smart apps, building products around data, and what comes after Roll on this edition of PeopleTech.
Hi, Roberto. Welcome. Recently you added AI to Roll, your app, and you said that adding AI to Roll was the first step in leveraging generative AI to develop or enhance new tools built all around your workforce data. I’m wondering why you started with Roll.
Roberto:
Roll is one of the newest products at ADP, and another big reason that we… So it was a good platform to start with, and another big factor on that decision is that Roll was already designed from the beginning with AI and ML as a core component of the systems design. Roll is a conversational UI, and I think either by luck or vision we decided when we designed Roll to use a simple conversational UX that makes the system very intuitive for our users. As we all know, back in December of last year, here comes OpenAI with ChatGPT and, as the name say, it uses a chat, so chat or a conversational UI is what made this new wave of AI tools and advancements with the large language model ubiquitous. Everybody now has access.
Our system being chat based, so you run your payroll, you do your hires, your terminations, your promotions, everything just chatting with Roll, it was a obvious choice to ingrain the GenAI capabilities in the same UX. So nowadays, instead of simply doing the transactional part of the system like the hire, the payroll, you can also augment Roll with GenAI. If you ask a question like, “How much should I spend in marketing at my small business?” We will both gather that information from our own data, our own knowledge, but also reach out to the LLM to get a more public sort of information and bring that into the chat with our user.
Mark:
Now, ADP also said that Roll’s new features, these AI features, are a part of a broader generative AI strategy for the company. Can you tell me about that strategy?
Roberto:
Some of it I can. I think, like every company these days, GenAI came as a transformative technology that can make our products better, so every single area of ADP and not only on the client-facing but also on the back end on our service side, so all different products and all different sort of layers of those products are being rethinked and we’re trying to strategize on what are the best places to leverage this new technology. For instance, on our service side, when a client calls we are already using GenAI to classify those calls, to create the summaries of those calls, to help our service agents augment their knowledge with an LLM sitting by their side, but the idea is we’re also doing that on our reporting tool using AI to instead of having to sort of design a report, simply say what you want and the report will be generated for you.
You will see more and more of that both on our insights when you run payroll. We are starting to design GenAI that will look at your payroll history and not only give you the typical insights, “Oh, your overtime was a little higher this month or whatever.” It will actually understand the changes and basically with the generative capability give you a description of why this paycheck is maybe different than the previous one with all the rationale behind it. You will start to see that permeate all of our products with this new sort of intelligence embedded in the flow of work.
Mark:
All of these things going on embedded in the flow of work and the other things that you mentioned, are these the kind of features that users will really notice or are they more the kind of things that are going to show up through better performance and accuracy?
Roberto:
I think some things will be more noticeable. Like I said, when you are designing a report you’ll be able to just describe it, so that changes dramatically how the user experiences the system. Some of them will be just organic, as you said. Some of them will be more about a better insight on your pay statement or a better and faster contact with our service agent because he or she is being augmented with an LLM behind them. I think the idea is you will see different facets. I don’t think you can say some of them will not be features, some of them will clearly be features. Some of them will be just a better or a more performant presentation of the existing features we have.
Mark:
ADP has a massive product line, and I’m wondering how do you coordinate and prioritize your work with AI? I mean, do different groups approach it in their own way? Is there an overarching master plan? How are you tackling it?
Roberto:
That’s a great question. What we did at ADP is that we created sort of a center of excellence, a group that is managing all of the ADP initiatives to make sure that, first of all, we have some governance but also we have reuse, collaboration, exchange of ideas. But independently, every product is looking at its own domain because no one knows better that product than the team that is designing it, building it, ideating on it. We don’t believe that we can centralize innovation. That innovation should come from each one of the products, understanding where there is a need and where there is a opportunity to fulfill with this new technology, with GenAI, but we also believe that we need to have this central organization that can disperse that knowledge, that can facilitate one team hearing about what another team have done and then maybe saying, “Oh my God, I could do this on my own product or I should look at this particular area or pain that I have that could be resolved with an LLM.”
It’s a hybrid sort of approach where we do have a center of excellence, a governance body that includes privacy, legal, as well as obviously the technology side, the architecture, the data scientists, but we put it all together, security, privacy, legal aspects, all of it with the technology to make sure that if we go with one solution, that solution will preserve sort of the authority about the data within ADP, within our clients, and not make that data public because we deal with sensitive data. A lot of that is part of this governance or this central organism to make sure that the groups don’t stray away from those rules, those design assumptions.
Mark:
What do you think comes first? Are you thinking about AI or the product’s capability as a first step? In other words, are you looking for places to use AI or are you seeing a need and AI happens to be the solution, or is it both?
Roberto:
It’s both. I think it’s actually both. We do have some groups at ADP, including mine, that are running what we call more like horizon three or more outlandish, let’s say, ideas about what GenAI can do and change even the nature of our business per se, how we can leverage the data that we have at ADP in addition to what the LLM’s providing. You just saw probably announcement about a week ago that now OpenAI allows you to sort of fine-tune their own model, so you can put a layer on top of the major LLM that they have with your own knowledge and capabilities.
This field is changing so fast that I think it’s both. Every week there is something different that we can see a problem that we had being fulfilled by that new technology, but at the same time I don’t think we can be focused on just resolving for the reality we have. We need to have groups looking at what can be a new reality, what can be a product that we never thought of that can be created just by using this technology, so it is both.
Mark:
How do you leverage AI from one product to another? I mean, can the capabilities you’ve developed for Roll be incorporated into solutions for enterprise companies, for example?
Roberto:
Yes. Yes. I think, as I said before, this central group is the one that distributes this information. Definitely. I mean, at the end of the day, an employee is an employee. In a small company, in a large company, the needs are kind of the same, you want to be paid correctly, on time, you want to understand your benefits, you want to understand your career path. Obviously in a small business some of this are in lower scale, in a large business you might have more career latitude, other places to go in the company, but the needs are the same. I think the toolsets that we build, the features that we put in place for the employees, the managers, the big payroll administrators, all can kind of feed from one another, all can sort of build upon what was already built. What we did in Roll, we’re definitely sharing with the rest of the company and I think it’s going to make the whole of ADP portfolio better.
Mark:
When I was involved in product development back in the day, we used to say that the simpler a feature was to use, the more powerful and sophisticated the technology had to be behind it. Is that still true? Is that just sort of a given in any kind of technology development or-
Roberto:
I think you still hold it true in there. It is the case. I mean, for me it’s like where do you put the cognitive burden, right? I mean, if you want to make something that it’s intuitive, that really reduces the cognitive burden off the user using the system, that will move into the design, the architecture, the technology that you use in the system itself. What we did with Roll to try to simplify everything to a simple chat, which a lot of people at the beginning was like, “Okay, this is not even possible. How can I run payroll, do complex things like a hire just by chatting with the computer?”
It was a lot of engineering, a lot of ML, back in the days not even LLMs but using different models and algorithms for in our back end to be able to provide that simplicity to the user. I think it still holds true. It is an effort, but it’s definitely something that pays off when you see clients come in, run their payroll in less than 30 seconds just by telling the computer what they need and we provide the summary of the payroll, they just say, “Yeah, this is okay. Go.” They don’t worry. I think that is the type of user experience that we want, the simplicity of something that is familiar to me, I chat with, I send messages through my family, to friends and all that, and why not? Why just have that same simplicity on enterprise software as well?
Mark:
When ChatGPT first came out, there was a lot of breathless coverage, everybody was excited, everybody was seeing potential, and more recently there’s been a lot of questions bubbling up about how does it handle diversity or how does it keep things in the proper context or what have you. Are there issues or concerns you have about generative AI that people aren’t really talking about, something that you’ve got your eye on or a little worried about?
Roberto:
I think for us, the big thing with this large language models is to make sure that they don’t hallucinate. I don’t think they have the same issue that we saw with some other experiments where there was sort of improper behavior from the model. One thing that I think this company’s understood is that they needed to address that day one or it will create an issue for their product. I think that the companies address that using what we call the steerability of the model. You can really steer these models to sort of do and say what you want.
The main issue that we face is how to keep the model creative in terms of coming up with interesting responses, good rationale on the questions you ask, but not hallucinate because payroll is an exact science. I cannot come up with things, I cannot say things that are not truth, but I think with a lot of effort, with a lot of understanding what was the best provider for us to use, what was the best model for us to use for certain use case because we have… You probably know this, but you have GPT-4 and then you have GPT-3.5, 3.5 Turbo. The lower ones are faster.
The 4 is slower per se but it’s more precise on the steerability of the model, so we ended up deploying a combination of them depending on the use case. If I really need to steer the model, make sure that it doesn’t hallucinate, we use 4. If I need something quick and fast, we use 3.5. Also if we want to be more creative, 4 with what we call… We can deal with the temperature of the model as well, say, “How much do you want to go into different areas?” So a lot of engineering behind the scenes. That for me is the main issue. How do we make sure that the model stay without hallucinating? Not much I think what we used to be concerned about, it becoming adversarial or improper.
Mark:
I love the idea of AI hallucinating.
Roberto:
I know.
Mark:
Just in case people who are listening don’t really understand what the term means, could you tell me about it?
Roberto:
Well, AI, like us, you can say is creative if I can speak in layman’s terms here. It actually can be very persuasive, telling you a story, giving you an answer that has no grounding in reality, it’s not truth, but it will write to you as if it was absolutely the truth. It is like it’s straying out of the facts of what it was trained on and it’s giving it creative abilities. That’s why it’s so good if you ask it to write a poem that never exists or come up with a story. Those are wonderful things if you’re in the creative space, not so much if you are in a more financial mathematical space. You want one plus one to be always two. Those are sort of the things that… Because it will probably tell you a great story why one plus one is three and a very believable one, even mention sources and things, but that is hallucination and that’s what we need to sort of make sure that the models don’t stray into that.
Mark:
Roberto, thank you very much for talking with me today. It’s been fun, and I hope we’ll talk again.
Roberto:
Yes. Thank you for the opportunity and looking forward for our next chat.
Mark:
My guest today has been Roberto Masiero from the innovation lab at ADP, and this has been PeopleTech, the podcast of the HCM Technology Report. We’re a publication of RecruitingDaily. We’re also a part of Evergreen Podcasts. To see all of their programs, visit www.evergreenpodcasts.com. To keep up with HR Technology, visit the HCM Technology Report every day. We’re the most trusted source of news in the HR tech industry. Find us at www.hcmtechnologyreport.com. I’m Mark Feffer.
Image: ADP
