API call from Lambda and save response in S3

Today I am going to explain how to call an external API from AWS Lambda and save the response in S3 bucket in a text file. I will use AWS cloud watch event to call the Lambda function after every 5 minutes. Nodejs will be used to write down Lambda function

Lets first create a Lambda function. Login into AWS management console and goto Lambda. Follow the below steps,

• Click on Create function
• Name function CallExternalAPI
• Select runtime Node.js. When I was writing this article Node.js 12.X runtime was available.
• In Permission, create a new role with Lambda basic permission.
• Click create and Lambda function will be created.

Node fetch

There are many way to call external APIs from AWS Lambda functions. After little search I find out a very simple and easy way to call API using node-fetch. Node fetch is a light weight module to call API using window.fetch in Nodejs. I suggest you must explore this npm package. Follow the below steps and create a node project on your local machine.

• Open an empty folder and run command in command prompt/terminal,
  npm init -y
  This will create package.json file in the folder.
• Create an index.js file.
• Install the node-fetch npm package by running the below command
  npm i --save node-fetch

In the next step we are going to upload the nojde project in AWS. For this follow the below step.

• Select index.js and node_modules folder and add into a zip folder.
• On AWS Lambda screen click on Action and upload zip file.

We will use AWS-SDK to save the data in S3 bucket. To call an external API as I mentioned before I am going to use node-fetch npm package. Add/copy the below code in index.js file and save.

const AWS = require('aws-sdk');
const fetch = require('node-fetch');
const s3 = new AWS.S3();
exports.handler = async (event) => {
const res = await fetch('https://jsonplaceholder.typicode.com/users');
const json = await res.json();
console.log(json);
};

In the above code first two line I added will import AWS SDK and node-fetch package. In the next time creating an S3 object. So that we can access S3 bucket. In the event handler I am calling the JSONPlaceholder public api for testing. Then display the result in log. As you can see that with node-fetch how easy it is to call API. Now to test the API lets create a test event and remove the default test cases and make it empty (see the image below). Next save the Lambda function and click on Test. You will see the successful test result in the execution result window. See the image below.

Now we are successfully able to call the API. Next step is to save the result in a S3 bucket. I created a S3 bucket with name geeksexternalapicall. Now let’s modify the code. You can see the below code, after calling the API, I added a variable and set bucket name, key will the name of the file and value or body will the result fetch from API and we are going to save the API result in text format. That is why I put content type as ‘text/plain’. And then following line will upload the data in S3 bucket s3.upload(params).promise();.

const AWS = require('aws-sdk');
const fetch = require('node-fetch');

const s3 = new AWS.S3();

exports.handler = async (event) => {
  const res = await fetch('https://jsonplaceholder.typicode.com/users');
  const json = await res.json();
  var params = {
      Bucket: 'geeksexternalapicall',
         Key: 'test.txt',
         Body: JSON.stringify(json),
         ContentType: 'text/plain',
   };
   
   var s3Response = await s3.upload(params).promise();
};

Save the function and click on Test button. What you see in the execution result is Access Denied, as you see in the image. Why is that. Because we have to give access to Lambda function write file in S3 bucket.

Let’s give access to Lambda function to read and write on S3 bucket. From AWS management console goto IAM and goto roles. For the Lambda function the role gets created is CallExternalAPI-role-qofvxofb, select the role and click on Add Inline Policy. Select the following actions from the inline policy,

• Service – Select S3
• Actions – Read – select GetObject
• Actions – Write – select PutObject
• Actions – Permission – select PutObjectAcl
• Resources – I want to access only the bucket I created for this demo instead of giving access to all the buckets in S3. Click on ARN and give bucket name and for objects we can give access to any.
• Click on Review Policy and give any name and create policy. I name policy name callexternalapipolicy.

Once the policy is created you can see the Policy JSON as below,

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:PutObjectAcl"
            ],
            "Resource": "arn:aws:s3:::geeksexternalapicall/*"
        }
    ]
}

Now we attached policy with our Lambda function which gives access to write on the S3 bucket we created. Come back to Lambda management console and click on Test button and you will see the function is successfully executed and a text file gets created in S3 bucket. You can goto the bucket and view the file.

Now my next requirement is to make a CRON job and call this Lambda function after a specific interval. The reason is to get updated result or data from the API. In the demo I am calling a sample API so it always give same result but you can check from the last modified date and time of the file. To make a CRON job, we will create Cloud Watch Events. Click on CloudWatch from AWS management console and under events click on rules and create a new rule. Select schedule from event source and for the demo purpose I set to 5 minutes. Select Lambda function in Target and click on configure function. Give any name and create function.

There you go. Now if you refresh the bucket you will see the file modify date is updated. I hope you like the post. I try to make it simple so that any one can easily understand. Please share you thoughts, if you like it or how can I make it better.