discuss.newrelic.com Open in urlscan Pro
162.247.242.36  Public Scan

Form analysis
1 forms found in the DOM

POST /login

<form id="hidden-login-form" method="post" action="/login" style="display: none;" __bizdiag="695534669" __biza="WJ__">
  <input name="username" type="text" id="signin_username">
  <input name="password" type="password" id="signin_password">
  <input name="redirect" type="hidden">
  <input type="submit" id="signin-button" value="Log In">
</form>

Text Content

Products
Pricing
Solutions
Help Center
About

Learn

 * Docs
 * Build on New Relic
 * Explore open source projects
 * Training

Get Help

 * Community forum
 * 
 * Global technical support
 * Expert services

New Relic One

 * Platform Overview
 * Telemetry Data Platform
 * Full Stack Observability
 * Applied Intelligence

By Topic

 * DevOps
 * Cloud Adoption
 * Native
 * Digital Customer Experience

By Industry

 * E-Commerce and Retail
 * Media
 * Public Sector

By Technology

 * Amazon Web Services
 * Pivotal Cloud Foundry
 * Microsoft Azure
 * Google Cloud Platform
 * Kubernetes
 * 

Our Customers

Over 17,000 customers love New Relic, from Fortune 500 enterprises to small
businesses around the globe.

Our Blog

The latest news, tips, and insights from the world of New Relic and digital
intelligence.

Our Company

 * About Us
 * Leadership
 * Meetups and Events
 * Resources
 * Social Impact

 * Investor Relations
 * Newsroom
 * Partner Program
 * Contact Us


TIMESTAMP OF LOGS ARE WRONG WHEN LOGS ARE SEND FROM AWS KINESIS DATA FIREHOSE

Support: Telemetry Data Platform (TDP)Log Management
aws logs
Log InSign Up
 * 
 * 

We've updated this site, please refresh, or you may experience unexpected
behavior.



WELCOME TO THE NEW RELIC COMMUNITY FORUM!

This is a public forum for users to discuss questions around software analytics,
performance monitoring, and using New Relic products.

To participate, just Log In with your New Relic account. Make sure to also
review our Community Guidelines, Terms of Service, and Privacy Policy.

X





TIMESTAMP OF LOGS ARE WRONG WHEN LOGS ARE SEND FROM AWS KINESIS DATA FIREHOSE

Support: Telemetry Data Platform (TDP)Log Management
aws logs

You have selected 0 posts.

select all

cancel selecting

Sep 3
3 / 4
Oct 5

2d ago

PlannitIT
3
Sep 3


Hello,

We use AWS WAF and send our logs to AWS Kinesis Data Firehose to finally publish
them in New Relic Logs. This functionality works very well, on the other hand
all our timestamps are wrong. It seems changed by the date and time on
ingestion, the real date in log seems lost or ratified by NR Logs. As you
probably know, the Kinesis delivery streams run every X seconds or Y buffer
size. So we have a log bundle in GZIP filand posted on the New Relic endpoint
(https://aws-api.newrelic.com/firehose/v1 2). The date and time found in this
bundle of logs seem to have changed to the date and time of ingestion.

Here is a summary of the timestamps of my example:

 * WAF Sample in AWS Console : Fri Sep 03 2021 16:06:35 GMT-0000
 * Kinesis Delivery streams (Backup settings that send logs to S3 Bucket) : Fri
   Sep 03 2021 16:06:35 GMT+0000 (epoch 1630685195053)
 * New Relic Logs (json) : Fri Sep 03 2021 16:07:29 GMT+0000 (epoch
   1630685249525)
 * New Relic Logs (table) : September 3, 2021 12:07:29.525

I attach an example of full logs of WAF, Kinesis and New Relic logs.
logs.txt (11.5 KB)


waf_sample1818×771 87.2 KB



FYI, I am also with AWS support to find the root cause, but for the moment it
seems really New Relic the cause, because the backup logs that I find in S3 are
good and have the right timestamp.

Regards,

1 Reply


 * Wrong timestamp and incorrect log order

 * CREATED
   
   Sep 3

 * LAST REPLY
   
   2d
 * 3
   
   REPLIES

 * 377
   
   VIEWS

 * 2
   
   USERS

 * 4
   
   LIKES

 * 1
   
   LINK

 * 3
   

PlannitIT
Sep 3


Also, I see the timestamps spike every 60 secondes.



NRLogs_timestamps_spike1641×257 13.8 KB



1 Reply



1 month later
dzevenbergenNR Support
PlannitIT
2d


Hey @PlannitIT,

This is currently the expected functionality of our Kinesis Firehose log
ingestion.

Our engineers are working on a feature that would allow you to use your own
timestamp, but in the product’s current state, you should expect the timestamp
to be the ingestion timestamp.

If you’d like to make this a feature request, you can use #featureidea in your
post.

Thanks for reaching out!

1 Reply
3


PlannitIT
dzevenbergen
2d


Hi @dzevenbergen,

Thank you for your reply

This is normal behavior, but it’s funny that the timestamps are overwritten by
NR’s. That said, if I understand correctly, there is no way to have our AWS WAF
logs with real timestamps?

I believe that it is important to keep control over the timestamps sent in our
logs, it is even essential #featureidea ! Without that, the logs become
unusable.

1






THIS TOPIC WILL AUTOMATICALLY CLOSE IN A YEAR.


Reply


SUGGESTED TOPICS

Topic Replies Views Activity How to separate staging logs from production logs
(Heroku)?
Log Management
logs
1 241 27d Delete the log records PII
Log Management
2 266 18d Logging directly to New Relic instead of local log
Log Management
nodeagent logs
6 193 29d Why does Logs UI not show some neighboring log entires unless you zoom
in?
Log Management
welcome logs
2 159 29d Log Forwarding From Heroku
Log Management
logs heroku
1 127 23d


WANT TO READ MORE? BROWSE OTHER TOPICS IN LOG MANAGEMENT OR VIEW LATEST TOPICS.


[EN.SHARE.TOPIC]



Share



COMPANY

 * Careers and Culture
 * Partner Program
 * Investor Relations
 * NewRelic.org
 * Suppliers Portal

CONNECT

 * Contact Us
 * Request Demo
 * Events

INTERNATIONAL

 * newrelic.jp (日本語)
 * newrelic.fr (Français)
 * newrelic.de (Deutsch)
 * newrelic.co.kr (한국어)

 * 
 * 
 * 
 * 
 * 

©2008-20 New Relic, Inc. All rights reserved
  Terms of Service   Community Terms of Service   DMCA Policy   Privacy Policy
  Cookie Policy   UK Slavery Act of 2015

Invalid date Invalid date