ably.com
Open in
urlscan Pro
2606:4700:10::6814:59c3
Public Scan
URL:
https://ably.com/blog/redis-keys-do-not-expire-atomically
Submission: On November 16 via api from BY — Scanned from DE
Submission: On November 16 via api from BY — Scanned from DE
Form analysis 3 forms found in the DOM
GET /search
<form class="flex items-start" action="/search" method="get">
<div class="relative w-full"><svg class="text-cool-black absolute top-12 left-16" style="width: 1.5rem; height: 1.5rem;">
<use xlink:href="#sprite-icon-gui-search"></use>
</svg><input type="search" name="q" class="ui-input pl-48 h-48" placeholder="Search" autocomplete="off" data-id="meganav-search-input">
<div class="absolute w-full mt-8 z-10 hidden shadow-container rounded-lg bg-white border border-mid-grey" data-id="meganav-search-autocomplete-container">
<ol class="m-16" data-id="meganav-search-autocomplete-list"></ol>
</div>
</div><button type="submit" class="ui-btn-secondary flex-shrink-0 ml-8 sm:ml-16 md:ml-24 xl:ml-32">Search</button>
</form>GET /search
<form class="mb-16" action="/search" method="get">
<div class="relative w-full"><svg class="text-cool-black absolute top-12 left-16 hover:text-gui-hover" style="width:1.5rem;height:1.5rem">
<use xlink:href="#sprite-icon-gui-search"></use>
</svg><button type="button" class="absolute top-12 right-16 p-0 focus:outline-gui-focus m-0 md:hidden invisible" data-id="meganav-search-input-clear"><svg class="text-dark-grey " style="width:1.5rem;height:1.5rem">
<use xlink:href="#sprite-icon-gui-cross-circled-fill"></use>
</svg></button><input type="search" name="q" class="ui-input px-48 h-48" style="max-width:none" placeholder="Search" autocomplete="off" data-id="meganav-mobile-search-input">
<div class="absolute w-full mt-8 z-10 hidden shadow-container rounded-lg bg-white border border-mid-grey" data-id="meganav-search-autocomplete-container">
<ol class="m-16" data-id="meganav-search-autocomplete-list"></ol>
</div>
</div>
</form><form class="col-span-full sm:col-span-4 relative z-10">
<div class="font-sans font-medium uppercase tracking-widen-0.1 text-overline2 text-white mb-8">Enter your email</div><input class="ui-input" type="email" name="EMAIL" value=""><button type="submit" class="ui-btn-invert mt-24"
disabled="">Submit</button>
</form>Text Content
* ProductsProducts
THE ABLY PLATFORM
Easily power any realtime experience in your application. No complex
infrastructure to manage or provision. Just a simple API that handles
everything realtime, and lets you focus on your code.
Explore how it works
PRODUCTS
* Pub/Sub Channels
Build infinitely scalable realtime applications.
* Spaces (Beta)
Create multi-user collaborative environments.
* LiveSync (Pre release)
Keep clients in sync with any relational database.
TECHNOLOGY
* Predictable performance
A low-latency and high-throughput global network.
* Guaranteed ordering & delivery
Data is delivered - in order - even after disconnections.
* Fault tolerant infrastructure
Redundancy is built in at global and regional levels.
* High scalability & availability
Built for scale with legitimate 99.999% uptime SLAs.
* Global edge network
An edge network of 15 core routing datacenters and 205+ PoPs.
Explore Four Pillars of Dependability
* SolutionsSolutions
SOLUTIONS
* Live Chat
Deliver highly reliable chat experiences at scale.
* Multiplayer Collaboration
Bring collaborative multiplayer experiences to your users.
* Data Broadcast
Broadcast realtime event data to millions of devices around the globe.
* Data Synchronization
Keep your frontend and backend in realtime sync, at global scale.
* Notifications
Deliver cross-platform push notifications with a simple unified API.
* Asset Tracking (Beta)
Track assets in realtime with a solution optimised for last mile
logistics.
INDUSTRY
* EdTech
Deliver interactive learning experiences.
* FinTech
Deliver personalised financial data in realtime.
* Automotive, Logistics, & Mobility
Power diagnostics, order tracking and more.
* B2B Platforms
Empower your customers with realtime solutions.
* Healthcare (HIPAA)
Provide trustworthy, HIPAA-compliant realtime apps.
* eCommerce & Retail
Enrich customer experiences with realtime updates.
* Sports, Media & Audience Engagement
Deliver engaging global realtime experiences.
* Gaming
Power ultra fast and reliable gaming experiences.
* IoT & Connected Devices
Monitor and control global IoT deployments in realtime.
* CompanyCompany
WHY COMPANIES CHOOSE ABLY
* Customers
Ably supports customers across multiple industries.
* Case studies
Discover how customers are benefiting from Ably.
* Compare our tech
Choose the right realtime service.
* Partners
Ably collaborates and integrates with AWS.
* Resources
Learn more about realtime with our handy resources.
* About Ably
Find out more about Ably’s mission.
* Careers
Discover our open roles and core Ably values.
* Events
Join Ably at upcoming events.
BLOG
* No more refreshing: Achieving low latency data with Ably and Confluent
Cloud
Nov 15, 2023
* How to create an Avatar Stack using Ably and Flowbite React
Nov 14, 2023
* Announcing the winners of the Ably Realtime Hackathon
Nov 13, 2023
More from our Blog
* DevelopersDevelopers
EXPLORE
* Documentation
Technical guides to help you build with Ably.
* Quickstart guides
Documentation to help you get started quickly.
* Integrations
Find out more about Ably integrations.
* Live examples
Discover our features and their use cases.
* SDKs
Download an SDK to help you build realtime apps faster.
* Tutorials & Demos
Get stuck in with our hands-on resources.
* Chat apps reference guide
Learn how to build chat apps with Ably.
* Multiplayer reference guide
Learn how to build collaborative features with Ably.
QUICK LINKS
* Discord
* GitHub
* Changelog
* Status
* Support & FAQs
* Pricing
* Contact us
* Login
*
Search
Popular pages
* How does Ably work?
* Quickstart guide
* Publish/Subscribe Messaging
* Platform
Support
* Sign up free
* Login
* Popular pages
* How does Ably work?
* Quickstart guide
* Publish/Subscribe Messaging
* Platform
* Products
Back
THE ABLY PLATFORM
Easily power any realtime experience in your application. No complex
infrastructure to manage or provision. Just a simple API that handles
everything realtime, and lets you focus on your code.
Explore how it works
PRODUCTS
* Pub/Sub Channels
Build infinitely scalable realtime applications.
* Spaces (Beta)
Create multi-user collaborative environments.
* LiveSync (Pre release)
Keep clients in sync with any relational database.
TECHNOLOGY
* Predictable performance
A low-latency and high-throughput global network.
* Guaranteed ordering & delivery
Data is delivered - in order - even after disconnections.
* Fault tolerant infrastructure
Redundancy is built in at global and regional levels.
* High scalability & availability
Built for scale with legitimate 99.999% uptime SLAs.
* Global edge network
An edge network of 15 core routing datacenters and 205+ PoPs.
Explore Four Pillars of Dependability
* Solutions
Back
SOLUTIONS
* Live Chat
Deliver highly reliable chat experiences at scale.
* Multiplayer Collaboration
Bring collaborative multiplayer experiences to your users.
* Data Broadcast
Broadcast realtime event data to millions of devices around the globe.
* Data Synchronization
Keep your frontend and backend in realtime sync, at global scale.
* Notifications
Deliver cross-platform push notifications with a simple unified API.
* Asset Tracking (Beta)
Track assets in realtime with a solution optimised for last mile
logistics.
INDUSTRY
* EdTech
Deliver interactive learning experiences.
* FinTech
Deliver personalised financial data in realtime.
* Automotive, Logistics, & Mobility
Power diagnostics, order tracking and more.
* B2B Platforms
Empower your customers with realtime solutions.
* Healthcare (HIPAA)
Provide trustworthy, HIPAA-compliant realtime apps.
* eCommerce & Retail
Enrich customer experiences with realtime updates.
* Sports, Media & Audience Engagement
Deliver engaging global realtime experiences.
* Gaming
Power ultra fast and reliable gaming experiences.
* IoT & Connected Devices
Monitor and control global IoT deployments in realtime.
* Company
Back
--------------------------------------------------------------------------------
WHY COMPANIES CHOOSE ABLY
* Customers
Ably supports customers across multiple industries.
* Case studies
Discover how customers are benefiting from Ably.
* Compare our tech
Choose the right realtime service.
* Partners
Ably collaborates and integrates with AWS.
* Resources
Learn more about realtime with our handy resources.
* About Ably
Find out more about Ably’s mission.
* Careers
Discover our open roles and core Ably values.
* Events
Join Ably at upcoming events.
BLOG
* No more refreshing: Achieving low latency data with Ably and Confluent
Cloud
Nov 15, 2023
* How to create an Avatar Stack using Ably and Flowbite React
Nov 14, 2023
* Announcing the winners of the Ably Realtime Hackathon
Nov 13, 2023
More from our Blog
* Developers
Back
--------------------------------------------------------------------------------
EXPLORE
* Documentation
Technical guides to help you build with Ably.
* Quickstart guides
Documentation to help you get started quickly.
* Integrations
Find out more about Ably integrations.
* Live examples
Discover our features and their use cases.
* SDKs
Download an SDK to help you build realtime apps faster.
* Tutorials & Demos
Get stuck in with our hands-on resources.
* Chat apps reference guide
Learn how to build chat apps with Ably.
* Multiplayer reference guide
Learn how to build collaborative features with Ably.
QUICK LINKS
* Discord
* GitHub
* Changelog
* Status
* Support & FAQs
* Pricing
--------------------------------------------------------------------------------
Contact usSign up free
On this page
Back to all blog posts Back
Ably engineering
Ably engineering
4 min read•Last updatedUpdated Jul 19, 2022
REDIS SCRIPTS DO NOT EXPIRE KEYS ATOMICALLY
Written by
Andrew Dunstall
This short post by a member of Ably's engineering team describes how we resolved
a problem that is typical of the challenges we face each week. We thrive on
solving hard distributed system problems that are mostly platform agnostic and
theoretical in nature, and this is the first post in a long-term series of
articles about things we've learned recently.
HOW WE USE REDIS AT ABLY
Ably is a platform for pub/sub messaging. Publishes are made on named channels,
and clients subscribed to a given channel have all messages on that channel
delivered to them. We use Redis, a distributed in-memory database for key-based
storage, to store various entities such as authentication tokens and ephemeral
channel state. It’s a good fit for temporary storage of messages while we
process them.
We have billions of active Redis keys at any given time, which are sharded
across numerous Redis instances. The sharding strategy places related keys in
the same shard so that we can perform operations that update related keys
atomically. We use Lua Redis scripts extensively to query and update keys and
rely on the atomicity of script execution to preserve the integrity of values of
related keys. That is, either all commands in the script run, or none at all
run, and no other commands execute at the same time.
We also use expiring keys extensively; the nature of the Ably service is that
much of the state of a channel is ephemeral and only retained for a limited
period of time (typically 2 minutes). We set keys to have a TTL so they
auto-expire.
THE ISSUE
The integrity of a set of related keys requires that either all keys exist, or
none exist. We had assumed that the atomic nature of script execution would also
apply to expire operations invoked by a script, but it isn't in fact true that
naively expiring multiple keys in the same script will preserve that integrity.
While expire operations execute atomically within the same script (with no
opportunity for intervening operations to occur), nonetheless the timestamps
associated with each expire operation are not necessarily the same.
Running TIME shows two different values:
-- time.lua
local a = redis.call('time')
local b = redis.call('time')
return {a, b}
$ ./redis-cli --eval /app/time.lua
1) 1) "1638280442"
2) "996960"
2) 1) "1638280442"
2) "996966"
Checking the actual expiry time:
-- expire_check.lua
redis.call('set', 'foo', '1')
redis.call('expire', 'foo', 1)
-- slow calls...
redis.call('set', 'bar', '2')
redis.call('expire', 'bar', 1)
local fooExpiry = redis.call('PEXPIRETIME', 'foo')
local barExpiry = redis.call('PEXPIRETIME', 'bar')
return {fooExpiry, barExpiry}
$ ./redis-cli --eval /app/expire_check.lua
1) (integer) 1638280843717
2) (integer) 1638280843730
The expire might not be pin-point accurate, and it could be between zero to 1
milliseconds out.
The implication is that there could be times at which some keys have expired,
but other related keys have not and this could lead to an inconsistent state.
OUR SOLUTION
The solution is to use EXPIREAT to set an absolute expiry time for all related
keys, rather than rely on a relative expiry time through the TTL.
The Redis documentation is not clear if multiple key expiry is guaranteed to
occur at the same time if keys have the same EXPIREAT setting. To be cautious,
we reordered key expiry to ensure that, regardless, we avoid inconsistency.
-- expire_new.lua
-- Unix time
local now = redis.call('time')[1]
local expiry = now + 1
redis.call('set', 'foo', '1')
redis.call('expireat', 'foo', expiry)
-- slow calls...
redis.call('set', 'bar', '2')
redis.call('expireat', 'bar', expiry)
local fooExpiry = redis.call('PEXPIRETIME', 'foo')
local barExpiry = redis.call('PEXPIRETIME', 'bar')
return {now, fooExpiry, barExpiry}
$ ./redis-cli --eval /app/expire_new.lua
2) (integer) 1638281266000
3) (integer) 1638281266000
This is typical of one of the many engineering problems we troubleshoot and
solve each week here at Ably.
Fancy working with us in the realtime sphere? Our engineers have a range of
broad technology skills across infrastructure, security, distributed systems,
and beyond.
You can find us on Twitter or LinkedIn, and apply to join us in one of our open
roles.
| Discuss this post on Hacker News |
--------------------------------------------------------------------------------
LATEST FROM ABLY ENGINEERING
* Stretching a point: the economics of elastic infrastructure ?
* A multiplayer game room SDK with Ably and Kotlin coroutines ?
* Save your engineers' sleep: best practices for on-call processes
* Squid game: how we load-tested Ably’s Control API
* How to connect to Ably directly (and why you probably shouldn't) – Part 1
* Migrating from Node Redis to Ioredis: a slightly bumpy but faster road
* No, we don't use Kubernetes
On this page
1. How we use Redis at Ably
2. The issue
3. Our solution
1. Latest from Ably Engineering
BUILD DEPENDABLE REALTIME APPS
Ably offers versatile, easy-to-use APIs to develop powerful realtime apps.
Guaranteed to deliver at scale.
Start building
RECOMMENDED ARTICLES
Ably engineeringCollaborative experiences
17 min read
CURSOR EVERYWHERE: AN EXPERIMENT ON SHARED CURSORS FOR EVERY WEBSITE
Aug 18, 2023
Ably engineering
30 min read
GROUP CHAT APP WITH OPENAI'S GPT
May 24, 2023
Ably engineering
14 min read
CRDTS ARE SIMPLER AND MORE COMMON THAN YOU THINK
Feb 14, 2023
JOIN THE ABLY NEWSLETTER TODAY
1000s of industry pioneers trust Ably for monthly insights on the realtime data
economy.
Enter your email
Submit
How we use cookies to improve your experience.
Accept and close
THE ABLY PLATFORM
Easily power any realtime experience in your application via a simple API that
handles everything realtime.
* Pub/sub messaging
* Push notifications
* Third-party integrations
* Multiple protocol messaging
ABLY IS FOR
* Ably Asset Tracking
* Extend Kafka to the edge
* EdTech
* Automotive, Logistics, & Mobility
* B2B Platforms
* Healthcare
* eCommerce & Retail
* Sports & Media
* Gaming
* IoT & Connected Devices
DEVELOPERS
* Start in 5 minutes
* Documentation
* Tutorials
* Changelog
* Support & FAQs
* SDKs
* System status
WHY ABLY
* Customers
* Case Studies
* Four Pillars of Dependability
* Compare our tech
* Multi protocol support
* Third-party integrations
ABOUT
* About Ably
* Pricing
* Blog
* Careers
* Open protocol policy
* Press & Media
* Contact us
--------------------------------------------------------------------------------
We're hiring!Learn more at Glassdoor
We're hiring!Learn more at Glassdoor
--------------------------------------------------------------------------------
CookiesLegalsData ProtectionPrivacy
SOC 2 Type 2
Certified
HIPAA
Compliant
EU GDPR
Certified
256-bit AES
Encryption
sprite-discord sprite-facebook sprite-github sprite-glassdoor sprite-google
sprite-icon-display-48hrs sprite-icon-display-about-ably-col
sprite-icon-display-api-keys sprite-icon-display-api
sprite-icon-display-asset-tracking-col sprite-icon-display-browser
sprite-icon-display-calendar sprite-icon-display-call-mobile
sprite-icon-display-careers-col sprite-icon-display-case-studies-col
sprite-icon-display-chat-col sprite-icon-display-chat-stack-col
sprite-icon-display-chat-stack sprite-icon-display-cloud-servers
sprite-icon-display-compare-tech-col sprite-icon-display-customers-col
sprite-icon-display-data-broadcast-col
sprite-icon-display-data-synchronization-col sprite-icon-display-docs-col
sprite-icon-display-documentation sprite-icon-display-events-col
sprite-icon-display-examples-col sprite-icon-display-gdpr
sprite-icon-display-general-comms sprite-icon-display-hipaa
sprite-icon-display-integrations-col sprite-icon-display-it-support-access
sprite-icon-display-it-support-helpdesk
sprite-icon-display-kafka-at-the-edge-col sprite-icon-display-laptop
sprite-icon-display-lightbulb-col sprite-icon-display-live-chat
sprite-icon-display-map-pin sprite-icon-display-message
sprite-icon-display-padlock-closed sprite-icon-display-platform
sprite-icon-display-play sprite-icon-display-privacy-shield-framework
sprite-icon-display-push-notifications-col
sprite-icon-display-quickstart-guides-col sprite-icon-display-resources-col
sprite-icon-display-sdks-col sprite-icon-display-servers
sprite-icon-display-shopping-cart sprite-icon-display-sla
sprite-icon-display-soc2-type2 sprite-icon-display-tech-account-comms
sprite-icon-display-tutorials-demos-col sprite-icon-display-virtual-events-col
sprite-icon-display-virtual-events sprite-icon-gui-ably-badge
sprite-icon-gui-arrow-bidirectional-horizontal
sprite-icon-gui-arrow-bidirectional-vertical sprite-icon-gui-arrow-down
sprite-icon-gui-arrow-left sprite-icon-gui-arrow-right sprite-icon-gui-arrow-up
sprite-icon-gui-burger-menu sprite-icon-gui-check-circled-fill-black
sprite-icon-gui-check-circled-fill sprite-icon-gui-check-circled
sprite-icon-gui-checklist-checked sprite-icon-gui-clock sprite-icon-gui-close
sprite-icon-gui-copy sprite-icon-gui-cross-circled-fill
sprite-icon-gui-cross-circled sprite-icon-gui-dash-circled
sprite-icon-gui-disclosure-arrow sprite-icon-gui-document-generic
sprite-icon-gui-enlarge sprite-icon-gui-external-link
sprite-icon-gui-filter-flow-step-1 sprite-icon-gui-filter-flow-step-2
sprite-icon-gui-filter-flow-step-3 sprite-icon-gui-history sprite-icon-gui-info
sprite-icon-gui-link-arrow sprite-icon-gui-link sprite-icon-gui-live-chat
sprite-icon-gui-minus sprite-icon-gui-plus sprite-icon-gui-quote-marks-solid
sprite-icon-gui-refresh sprite-icon-gui-resources sprite-icon-gui-search
sprite-icon-gui-tick sprite-icon-gui-warning
sprite-icon-live-updates-results-metrics-col sprite-icon-multi-user-spaces-col
sprite-icon-social-x sprite-icon-tech-apachekafka sprite-linkedin sprite-quote
sprite-stackoverflow sprite-twitter sprite-youtube