All about Function Calling in LLMS

Introduction

The biggest problem with LLMs for the longest time was not being able to access real-time data and get hallucinated responses.

To fix this, OpenAI added a new feature called function calling to their chat completion API. This feature made the language models way better by giving them access to work with other external tools and APIs more effectively.

In this Article, We’ll explore the nuances of Function Calling, what it is, how it works, usages, challenges, and many more!

So, without delaying further, let's START!

What’s Function Calling?

Before we move further, let’s understand what function calling actually is!

In simple words, function calling is a feature that allows Large Language Models (LLMs) to interact with external functions, APIs, or tools by generating appropriate function calls based on user inputs.

This capability helps LLMs to perform tasks beyond text generation, such as fetching real-time data, executing computations, and integrating with various services.

"By facilitating the use of structured enterprise data with AI, function calling isn't just an innovation – it's a game-changer."

In essence, function calling transforms LLMs into powerful tools capable of handling a wide range of tasks by leveraging external resources and real-time data.

🎥 If You prefer video more than article, you can also check the following video:

How it works?

Contrary to what its name suggests, LLMs never actually call our functions. In fact, for Function Calling to work there doesn't even have to be any real functions!

Essentially, all it does is attempt to generate the parameters for hypothetical or potential functions based on the JSON schema you provide.

Didn’t get that?

No worries, Let’s understand this in-depth:

1. User Query:
First, the user makes a request to the LLM in the natural language. (e.g., "What’s the weather in New York?" or "Create a new user account.").

2. LLM Processing:

• After receiving the request, Our LLM parses that request and breaks it into smaller chunks based on the semantics of the input, key phrases, and overall meaning. This helps to understand what the user is asking for.

• Next, Our LLM determines what the user wants to achieve. This could be specific actions such as finding the weather of some place, creating a user, calculating a sum, etc.

• It also understands the context of the request to ensure that it’s calling the correct function.

3. Function Matching & Selection:

• The identified user intent is then matched against a set of predefined functions that the LLM can perform.

• These functions represent various tasks the model can facilitate, such as:

  • Answering a question
  • Fetching live data
  • Performing an action (e.g., booking a ticket, updating a database)

• Based on the closest match between the user's intent and the available functions, the system selects the most appropriate function to execute.

4. Parameter Extraction & JSON Generation:

• The selected function often requires specific information or parameters to perform its task effectively.

• The system will then extract those parameters from the user’s input and generate a JSON object to execute functions effectively.

• The JSON object contains the function name and required parameters extracted from the user’s input.

5. Executing the Function:

• Next, Using the JSON object created in the last step, it calls the selected function with the provided parameter.

• Then, the selected function executes based on the provided parameters and returns a structured response

6. Response Generation & Output:

Now, Our LLM generates a response, based on the result obtained from executing the function.

This multi-stage approach allows the LLM to break down complex tasks into manageable units, making it more efficient and accurate in understanding and responding to user input.

Each stage plays a crucial role in ensuring the chatbot operates effectively and provides valuable interactions with users.

Why to use Function Calling?

You might be thinking why do we need function calling? Our LLMs are very capable on their own!

Yes, They are indeed capable! But in some scenarios, they struggle to give proper responses:

Access Real-time data

One of the biggest issues with LLMs was being unable to access the real-time data. For that LLMs sometimes hallucinate and cannot answer questions that require real-time information.

For example, If we ask the LLM “What’s the weather in Kolkata?”, It won’t be able to generate a proper response due to lack of information.

But with Function calling, it can use the weather API to get the data of Kolkata and generate an accurate response with that.

Perform complex tasks beyond text generation

LLMs are great at generating coherent and contextually relevant text, but some tasks require more than just text generation.

For example, If we ask the LLM to book a train ticket. Without Function Calling, the LLM can only simulate the conversation but cannot complete the actual booking process.

However, with Function Calling, the LLM can interact with the Train's booking system to find available slots and book a train ticket for us.

Since LLMs cannot learn, once trained, in the current technology framework, function calling plays an important role in removing limitations of LLMs like lack of mathematical calculation capabilities or awareness of time or access to the latest information.

Challenges of Function Calling

Every feature has its pros and cons, function calling is no different.

While function calling enhances the capabilities of LLMs, it has some limitations too. Here are the main challenges:

Knowing When to Call the API

One of the primary challenges of using function calling is to understand the right time to call an external API. LLMs need to understand when it is really required to invoke a function to call external API.

For example, If we ask the LLM “What’s the capital of India?”, the model should directly answer this as this information is already stored in its memory.

Whereas, if we ask “What’s the temperature of Kolkata?”, the model needs to understand that in order to answer this question it is required to call a function.

Handling multiple functions

Sometimes, LLMs need to manage multiple functions which is a complex task to manage. The model needs to coordinate several tasks at once and ensure each step is completed correctly.

Using Context Effectively

Another crucial challenge is to remember the context of the request effectively. The model has to understand what has been discussed previously and apply that information to the current request.

For example, If we ask “What’s the weather in Mumbai?” and then later ask “And what about tomorrow?", the model should understand that "tomorrow" refers to the weather in Mumbai and not ask you to specify the location again.

By addressing these issues, we can make LLMs even more powerful and useful.

Function Calling in Action

Enough talking about Definitions, It’s time to build.

Now, we’ll build a Customer Support tool that answers queries about the product. To build this we’ll use:

• Node.js for the backend

• Nebius AI for inference models

Step 1: Set Up Your Node.js Project

To start with, Create a New Node.js Project and install the required dependencies

npm i dotenv openai zod

Next, Create a .env file in the root folder and add the Nebius API Keys:

NEBIUS_API_KEY=your_nebius_api_key

💡 You can get your API Keys Here.

Initialize TypeScript by running the following command in your terminal:

tsc --init

And, Go to the tsconfig.ts file and change the following paths to set up source and output directories:

"rootDir": "./src", 
"outDir": "./dist",

Now, In our root file, we’ll import the necessary libraries:

//src/indeex.ts
import OpenAI from 'openai';
import { z } from 'zod';
import { zodResponseFormat } from 'openai/helpers/zod';
import * as dotenv from 'dotenv';

dotenv.config();

Next up, we’ll initialize our OpenAI Client with Nebius inference models.

const client = new OpenAI({
  baseURL: "https://api.studio.nebius.ai/v1/",
  apiKey: process.env.NEBIUS_API_KEY,
});

Step 2: Define the Function for Product Queries

Now, We’ll use function calling to fetch product details dynamically. Let’s define the tool first

const tools = [
  {
    type: "function" as const,
    function: {
      name: "get_product_information",
      description: "Get information about product plans including features, pricing, and support details",
      parameters: zodResponseFormat(ProductQuery, "product_query").json_schema.schema,
    },
  }
];

Now we’ll create the function that will fetch product details. For this example, I’ve created dummy data.

const ProductQuery = z.object({
  plan: z.enum(["basic", "premium", "enterprise"]),
  infoType: z.enum(["features", "pricing", "support"]),
});

 const productInfo = {
    basic: {
      features: ["5 users", "Basic reporting", "Email support"],
      pricing: "$10/month",
      support: "Email support with 24h response time"
    },
    premium: {
      features: ["Unlimited users", "Advanced reporting", "Priority support"],
      pricing: "$50/month",
      support: "Priority email and chat support with 4h response time"
    },
    enterprise: {
      features: ["Custom solutions", "Dedicated account manager", "24/7 support"],
      pricing: "Contact sales",
      support: "24/7 phone and email support with 1h response time"
    }
  };

//Product Information Function with Dummy data
async function getProductInformation(args: z.infer<typeof ProductQuery>) {
  return {
    plan: args.plan,
    infoType: args.infoType,
    information: productInfo[args.plan][args.infoType]
  };
}

Now, we will process user queries and let our LLM call the function dynamically.

async function handleCustomerQuery(userQuery: string) {
  const response = await client.chat.completions.create({
    model: "meta-llama/Meta-Llama-3.1-8B-Instruct-fast",
    messages: [{
      role: "system",
      content: "You are a helpful customer support agent. Use the available function to fetch accurate product information."
    }, {
      role: "user",
      content: userQuery
    }],
    tools: tools,
    tool_choice: "auto",
  });

  const toolCall = response.choices[0].message.tool_calls?.[0];
  if (toolCall && toolCall.function) {
    const functionArgs = JSON.parse(toolCall.function.arguments);
    const result = await getProductInformation(functionArgs);

    // Get AI to format the response nicely
    const finalResponse = await client.chat.completions.create({
      model: "meta-llama/Meta-Llama-3.1-8B-Instruct-fast",
      messages: [
        {
          role: "system",
          content: "You are a helpful customer support agent. Format the information in a friendly, clear way."
        },
        {
          role: "user",
          content: userQuery
        },
        {
          role: "assistant",
          tool_calls: [toolCall]
        },
        {
          role: "tool",
          tool_call_id: toolCall.id,
          content: JSON.stringify(result)
        }
      ]
    });

    return finalResponse.choices[0].message.content;
  }

  return response.choices[0].message.content;
}

Here we are using the Meta-Llama-3 model, But You can choose from a wide range of models provided by Nebius.

Finally, we’ll create an example function with a query:

async function main() {
  const customerQuery = "What are the features of your premium plan?";
  const response = await handleCustomerQuery(customerQuery);
  console.log("Customer: " + customerQuery);
  console.log("AI Agent: " + response);
}

main();

Now, Let’s run this in the terminal, run the following command:

node dist/index.js

And we’ll get the following response

Conclusion

That’s it! We’ve just built an AI-powered customer support assistant using Nebius AI’s inference models and function calling. 🎉

💡 What’s next?

• Connect this to a real database for dynamic product data.

• Extend functionality to support multiple queries in a single conversation.

• Add a front-end interface for a full-fledged AI chatbot!

📌 Complete Source Code: GitHub Repository

Found this helpful? Drop a comment below and share your thoughts! 🚀

🔗 Stay Connected: Twitter | LinkedIn | YouTube | GitHub

Thanks for reading till the end!