{"id":569,"date":"2025-12-18T12:25:43","date_gmt":"2025-12-18T12:25:43","guid":{"rendered":"https:\/\/guillaumesblog.net\/?p=569"},"modified":"2025-12-18T12:52:01","modified_gmt":"2025-12-18T12:52:01","slug":"analysing-vm-logs-at-scale-with-ai-big-data-toolset-like-aidp","status":"publish","type":"post","link":"https:\/\/guillaumesblog.net\/index.php\/analysing-vm-logs-at-scale-with-ai-big-data-toolset-like-aidp\/","title":{"rendered":"Analysing VM logs at scale with AI-big data toolset like AIDP"},"content":{"rendered":"\n

One way to get your logs back into OCI is to use the out of the box Custom Log Monitoring agent. The other way is to install Fluentd manually, and that is what we are going to do so we can keep control of the output my way and get it onto Object Storage for further analysis.<\/p>\n\n\n\n\n\n\n\n

\"\"<\/figure>\n\n\n\n

First is to follow the instructions located here<\/a> and install fluentd, then activate and enable the service with systemctl. Second step is to navigate to \/etc\/systemd\/system\/td-agent.service and edit the file, just change fluentd user to root user like so below, the service now run as root. (which is not great for security but let’s leave that aside for a moment.)<\/p>\n\n\n\n

[Service]\nUser=root\nGroup=root<\/code><\/pre>\n\n\n\n
and restart the systemctl to reload the configuration with systemctl daemon-reload<\/p>\n\n\n\n
Third, navigate to the configuration file and enter this configuration, we are monitoring the httpd access log in our case, check the <source> paragraph below;  We are sending the logs to OCI object storage, the @type s3 works perfectly in this situation as OCI offers s3 compatible APIs. The <buffer> configuration allows you to change multiple output parameters and in my config the agent outputs .json files onto my ‘compute_log_bucket’ bucket storage. Change aws_key_id and aws_sec_key with the corresponding entries from OCI, instructions are here.<\/a><\/p>\n\n\n\n
<source>\n  @type tail\n  format apache2\n  path \/var\/log\/httpd\/access_log\n  pos_file \/var\/log\/fluent\/httpd.access_log.pos\n  tag s3.apache.access\n<\/source>\n\n<match s3.*.*>\n  @type s3\n\n  aws_key_id key\n  aws_sec_key secret\n  s3_bucket compute_log_bucket\n  s3_region uk-london-1\n  s3_endpoint https:\/\/namespace.compat.objectstorage.uk-london-1.oci.customer-oci.com\n  path logs5\/\n  store_as json\n\n  <format>\n    @type json\n  <\/format>\n\n  <buffer>\n    @type file\n    path \/var\/log\/fluent\/s3\n    timekey 1h\n    timekey_wait 5m\n    chunk_limit_size 128m\n    timekey_use_utc true\n  <\/buffer>\n\n  time_slice_format %Y%m%d%H%M\n<\/match>\n<\/code><\/pre>\n\n\n\n
Lastly enable auth principal<\/a> with a dynamic group and the corresponding policy so your source VMs can write onto an Object storage bucket. That’s what the result would look like:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
And that’s great, because now you’ve got tons of log data dumped into cheap storage. The idea then is to have this raw data available to perform any type of analysis and unlock the power of a Lakehouse platform. In this case I will use the Oracle AIDP (AI Data platform), which is an all in one big data and AI toolbox.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
On this toolbox you can organise your data sources, assign ownerships, spin up Spark clusters, layout governance, all in the click of a button and it will abstract the complex set-ups in the back-end to get it running, also call in AI models directly.<\/p>\n\n\n\n
Let’s spin up a Spark cluster and analyse the logs coming from my VMs, and again these are webserver access log, so why not have a look at where the requests come from. We’ve got to imagine that at scale we would monitor hundred or thousands of machines and join together a lot of data to perform our analysis, hence the requirement for Spark. Also, this model could serve as a basis for a near<\/em> real-time monitoring cockpit, so we could ban troublesome clients.<\/p>\n\n\n\n
The code is here<\/a> if you are curious… but in a nutshell – We put together a dataframe and aggregate the number of visits by hosts<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Let’s now have a look at where the mystery visitors come from<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
And we have got as top three, drumroll please:<\/p>\n\n\n\n