Objective:

Amazon Athena is an interactive query service that makes it easy to analyze data in Amazon S3 using standard SQL. In this use case we will use Athena to analyze our S3 access logs through some simple steps from AWS web console.

  • Create two S3 buckets.
  • One that has to be monitored.
  • Another one that will be used to store the access logs of the first bucket
  • Store all the user access logs of the first bucket in the second bucket.
  • For safe delete operations ,apply MFA .
  • Create multiple IAM users.
  • Preform some actions through different users in the first bucket to generate some logs.
  • Integrate the second bucket with Athena .
  • Perform logs analysis.

Steps to implement the use case:

  1. Create different S3 buckets. one that has to be monitored and other that will store the access logs of the first bucket.
  • Go to management console And search for service S3:

  • Click on create bucket : Here user has to provide name of bucket and other require information and after this click on create bucket.

Now created buckets are visible in S3.

2. Create multiple users to perform operations on S3 buckets: 

  1. Need to enable server access login for first bucket : AWS S3 > Select Bucket > properties > server access login > enable

  1. Enable versioning for S3 :You can use versioning to preserve, retrieve, and restore every version of every object stored in your Amazon S3 bucket. 
  • Click on s3 and choose bucket Click on Properties > versioning 
  • By default bucket versioning status is disabled.

  1. Implement MFA for delete operations: To Implement MFA for delete operation versioning of S3 bucket is must. 
  • MFA should assign for root/IAM users

 Aws console   >    IAM    >   User (click-on) > security credential   > assign MFA device > Virtual MFA device >  download Google authentication to your computer/Android  > scan the Qr code > enter both codes > Assign  

  1. Perform different operations on S3 buckets: 
  • S3 operations: Perform various actions on S3 bucket like upload download, delete etc. through different users.
  • Add file remove file on management console : AWS services > S3 > buckets > click on bucket and perform actions

  • Add file remove file on management console : AWS services > S3 > buckets > click on bucket and perform actions

  1. Integrate Athena with S3 for logging
  • Search for Athena in services 
  • To create a database
    1. Open the Athena console. If this is your first time visiting the Athena console, you’ll go to a Getting Started page. Choose Get Started to open the Query Editor. If it isn’t your first time, the Athena Query Editor opens. Choose the link to set up a query result location in Amazon S3.
  • In the Settings dialog box, enter the path to the bucket that you created in Amazon S3 for your query results. Prefix the path with s3:// and add a forward slash to the end of the path.
  • Click Save.
  • In the Athena Query Editor, you see a query pane. You can type queries and statements here.
  • Create a database named as “mydatabase” using the CREATE DATABASE statement.
  • Choose Run Query or press Ctrl+ENTER.
  • Confirm that the catalog display refreshes and mydatabase appears in the Database list in the navigation pane on the left.
  1. Create Athena table:
  • Select database as “mydatabase“. 
  • Choose the plus (+) sign in the Query Editor to create a tab with a new query. You can open up to ten query tabs at a time.
  • In the query pane, enter the CREATE TABLE statement.
  • At the end of the query replace “myregion” with the AWS Region that you are currently working.
CREATE EXTERNAL TABLE IF NOT EXISTS cloudfront_logs (
  `Date` DATE,
  Time STRING,
  Location STRING,
  Bytes INT,
  RequestIP STRING,
  Method STRING,
  Host STRING,
  Uri STRING,
  Status INT,
  Referrer STRING,
  os STRING,
  Browser STRING,
  BrowserVersion STRING
  ) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.RegexSerDe'
  WITH SERDEPROPERTIES (
  "input.regex" = "^(?!#)([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+([^ ]+)\\s+[^\(]+[\(]([^\;]+).*\%20([^\/]+)[\/](.*)$"
  ) LOCATION 's3://athena-examples-myregion/cloudfront/plaintext/';

All the user access logs can be verified at the second bucket:

  1. Resource clean up: follow below steps to clean up all the resources created for this use case.
  • Go to service : S3 > Buckets > select bucket and click on Empty

  • For logs bucket we have activated MFA for delete operation: Therefore to delete this bucket need to deactivate MFA and versioning and then user can empty the bucket and delete it.
  • Now delete S3 bucket manually through console.

Kindly comment here or mail us at support@data-stats.com and edugenixcloud@gmail.com in case of any doubts. We will reach you as soon as possible.


Categories: AWS

0 Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

Insert math as
Block
Inline
Additional settings
Formula color
Text color
#333333
Type math using LaTeX
Preview
\({}\)
Nothing to preview
Insert