ably.com
Open in
urlscan Pro
2606:4700:10::6814:59c3
Public Scan
URL:
https://ably.com/blog/redis-keys-do-not-expire-atomically
Submission: On November 16 via api from BY — Scanned from DE
Submission: On November 16 via api from BY — Scanned from DE
Form analysis
3 forms found in the DOMGET /search
<form class="flex items-start" action="/search" method="get">
<div class="relative w-full"><svg class="text-cool-black absolute top-12 left-16" style="width: 1.5rem; height: 1.5rem;">
<use xlink:href="#sprite-icon-gui-search"></use>
</svg><input type="search" name="q" class="ui-input pl-48 h-48" placeholder="Search" autocomplete="off" data-id="meganav-search-input">
<div class="absolute w-full mt-8 z-10 hidden shadow-container rounded-lg bg-white border border-mid-grey" data-id="meganav-search-autocomplete-container">
<ol class="m-16" data-id="meganav-search-autocomplete-list"></ol>
</div>
</div><button type="submit" class="ui-btn-secondary flex-shrink-0 ml-8 sm:ml-16 md:ml-24 xl:ml-32">Search</button>
</form>
GET /search
<form class="mb-16" action="/search" method="get">
<div class="relative w-full"><svg class="text-cool-black absolute top-12 left-16 hover:text-gui-hover" style="width:1.5rem;height:1.5rem">
<use xlink:href="#sprite-icon-gui-search"></use>
</svg><button type="button" class="absolute top-12 right-16 p-0 focus:outline-gui-focus m-0 md:hidden invisible" data-id="meganav-search-input-clear"><svg class="text-dark-grey " style="width:1.5rem;height:1.5rem">
<use xlink:href="#sprite-icon-gui-cross-circled-fill"></use>
</svg></button><input type="search" name="q" class="ui-input px-48 h-48" style="max-width:none" placeholder="Search" autocomplete="off" data-id="meganav-mobile-search-input">
<div class="absolute w-full mt-8 z-10 hidden shadow-container rounded-lg bg-white border border-mid-grey" data-id="meganav-search-autocomplete-container">
<ol class="m-16" data-id="meganav-search-autocomplete-list"></ol>
</div>
</div>
</form>
<form class="col-span-full sm:col-span-4 relative z-10">
<div class="font-sans font-medium uppercase tracking-widen-0.1 text-overline2 text-white mb-8">Enter your email</div><input class="ui-input" type="email" name="EMAIL" value=""><button type="submit" class="ui-btn-invert mt-24"
disabled="">Submit</button>
</form>
Text Content
* ProductsProducts THE ABLY PLATFORM Easily power any realtime experience in your application. No complex infrastructure to manage or provision. Just a simple API that handles everything realtime, and lets you focus on your code. Explore how it works PRODUCTS * Pub/Sub Channels Build infinitely scalable realtime applications. * Spaces (Beta) Create multi-user collaborative environments. * LiveSync (Pre release) Keep clients in sync with any relational database. TECHNOLOGY * Predictable performance A low-latency and high-throughput global network. * Guaranteed ordering & delivery Data is delivered - in order - even after disconnections. * Fault tolerant infrastructure Redundancy is built in at global and regional levels. * High scalability & availability Built for scale with legitimate 99.999% uptime SLAs. * Global edge network An edge network of 15 core routing datacenters and 205+ PoPs. Explore Four Pillars of Dependability * SolutionsSolutions SOLUTIONS * Live Chat Deliver highly reliable chat experiences at scale. * Multiplayer Collaboration Bring collaborative multiplayer experiences to your users. * Data Broadcast Broadcast realtime event data to millions of devices around the globe. * Data Synchronization Keep your frontend and backend in realtime sync, at global scale. * Notifications Deliver cross-platform push notifications with a simple unified API. * Asset Tracking (Beta) Track assets in realtime with a solution optimised for last mile logistics. INDUSTRY * EdTech Deliver interactive learning experiences. * FinTech Deliver personalised financial data in realtime. * Automotive, Logistics, & Mobility Power diagnostics, order tracking and more. * B2B Platforms Empower your customers with realtime solutions. * Healthcare (HIPAA) Provide trustworthy, HIPAA-compliant realtime apps. * eCommerce & Retail Enrich customer experiences with realtime updates. * Sports, Media & Audience Engagement Deliver engaging global realtime experiences. * Gaming Power ultra fast and reliable gaming experiences. * IoT & Connected Devices Monitor and control global IoT deployments in realtime. * CompanyCompany WHY COMPANIES CHOOSE ABLY * Customers Ably supports customers across multiple industries. * Case studies Discover how customers are benefiting from Ably. * Compare our tech Choose the right realtime service. * Partners Ably collaborates and integrates with AWS. * Resources Learn more about realtime with our handy resources. * About Ably Find out more about Ably’s mission. * Careers Discover our open roles and core Ably values. * Events Join Ably at upcoming events. BLOG * No more refreshing: Achieving low latency data with Ably and Confluent Cloud Nov 15, 2023 * How to create an Avatar Stack using Ably and Flowbite React Nov 14, 2023 * Announcing the winners of the Ably Realtime Hackathon Nov 13, 2023 More from our Blog * DevelopersDevelopers EXPLORE * Documentation Technical guides to help you build with Ably. * Quickstart guides Documentation to help you get started quickly. * Integrations Find out more about Ably integrations. * Live examples Discover our features and their use cases. * SDKs Download an SDK to help you build realtime apps faster. * Tutorials & Demos Get stuck in with our hands-on resources. * Chat apps reference guide Learn how to build chat apps with Ably. * Multiplayer reference guide Learn how to build collaborative features with Ably. QUICK LINKS * Discord * GitHub * Changelog * Status * Support & FAQs * Pricing * Contact us * Login * Search Popular pages * How does Ably work? * Quickstart guide * Publish/Subscribe Messaging * Platform Support * Sign up free * Login * Popular pages * How does Ably work? * Quickstart guide * Publish/Subscribe Messaging * Platform * Products Back THE ABLY PLATFORM Easily power any realtime experience in your application. No complex infrastructure to manage or provision. Just a simple API that handles everything realtime, and lets you focus on your code. Explore how it works PRODUCTS * Pub/Sub Channels Build infinitely scalable realtime applications. * Spaces (Beta) Create multi-user collaborative environments. * LiveSync (Pre release) Keep clients in sync with any relational database. TECHNOLOGY * Predictable performance A low-latency and high-throughput global network. * Guaranteed ordering & delivery Data is delivered - in order - even after disconnections. * Fault tolerant infrastructure Redundancy is built in at global and regional levels. * High scalability & availability Built for scale with legitimate 99.999% uptime SLAs. * Global edge network An edge network of 15 core routing datacenters and 205+ PoPs. Explore Four Pillars of Dependability * Solutions Back SOLUTIONS * Live Chat Deliver highly reliable chat experiences at scale. * Multiplayer Collaboration Bring collaborative multiplayer experiences to your users. * Data Broadcast Broadcast realtime event data to millions of devices around the globe. * Data Synchronization Keep your frontend and backend in realtime sync, at global scale. * Notifications Deliver cross-platform push notifications with a simple unified API. * Asset Tracking (Beta) Track assets in realtime with a solution optimised for last mile logistics. INDUSTRY * EdTech Deliver interactive learning experiences. * FinTech Deliver personalised financial data in realtime. * Automotive, Logistics, & Mobility Power diagnostics, order tracking and more. * B2B Platforms Empower your customers with realtime solutions. * Healthcare (HIPAA) Provide trustworthy, HIPAA-compliant realtime apps. * eCommerce & Retail Enrich customer experiences with realtime updates. * Sports, Media & Audience Engagement Deliver engaging global realtime experiences. * Gaming Power ultra fast and reliable gaming experiences. * IoT & Connected Devices Monitor and control global IoT deployments in realtime. * Company Back -------------------------------------------------------------------------------- WHY COMPANIES CHOOSE ABLY * Customers Ably supports customers across multiple industries. * Case studies Discover how customers are benefiting from Ably. * Compare our tech Choose the right realtime service. * Partners Ably collaborates and integrates with AWS. * Resources Learn more about realtime with our handy resources. * About Ably Find out more about Ably’s mission. * Careers Discover our open roles and core Ably values. * Events Join Ably at upcoming events. BLOG * No more refreshing: Achieving low latency data with Ably and Confluent Cloud Nov 15, 2023 * How to create an Avatar Stack using Ably and Flowbite React Nov 14, 2023 * Announcing the winners of the Ably Realtime Hackathon Nov 13, 2023 More from our Blog * Developers Back -------------------------------------------------------------------------------- EXPLORE * Documentation Technical guides to help you build with Ably. * Quickstart guides Documentation to help you get started quickly. * Integrations Find out more about Ably integrations. * Live examples Discover our features and their use cases. * SDKs Download an SDK to help you build realtime apps faster. * Tutorials & Demos Get stuck in with our hands-on resources. * Chat apps reference guide Learn how to build chat apps with Ably. * Multiplayer reference guide Learn how to build collaborative features with Ably. QUICK LINKS * Discord * GitHub * Changelog * Status * Support & FAQs * Pricing -------------------------------------------------------------------------------- Contact usSign up free On this page Back to all blog posts Back Ably engineering Ably engineering 4 min read•Last updatedUpdated Jul 19, 2022 REDIS SCRIPTS DO NOT EXPIRE KEYS ATOMICALLY Written by Andrew Dunstall This short post by a member of Ably's engineering team describes how we resolved a problem that is typical of the challenges we face each week. We thrive on solving hard distributed system problems that are mostly platform agnostic and theoretical in nature, and this is the first post in a long-term series of articles about things we've learned recently. HOW WE USE REDIS AT ABLY Ably is a platform for pub/sub messaging. Publishes are made on named channels, and clients subscribed to a given channel have all messages on that channel delivered to them. We use Redis, a distributed in-memory database for key-based storage, to store various entities such as authentication tokens and ephemeral channel state. It’s a good fit for temporary storage of messages while we process them. We have billions of active Redis keys at any given time, which are sharded across numerous Redis instances. The sharding strategy places related keys in the same shard so that we can perform operations that update related keys atomically. We use Lua Redis scripts extensively to query and update keys and rely on the atomicity of script execution to preserve the integrity of values of related keys. That is, either all commands in the script run, or none at all run, and no other commands execute at the same time. We also use expiring keys extensively; the nature of the Ably service is that much of the state of a channel is ephemeral and only retained for a limited period of time (typically 2 minutes). We set keys to have a TTL so they auto-expire. THE ISSUE The integrity of a set of related keys requires that either all keys exist, or none exist. We had assumed that the atomic nature of script execution would also apply to expire operations invoked by a script, but it isn't in fact true that naively expiring multiple keys in the same script will preserve that integrity. While expire operations execute atomically within the same script (with no opportunity for intervening operations to occur), nonetheless the timestamps associated with each expire operation are not necessarily the same. Running TIME shows two different values: -- time.lua local a = redis.call('time') local b = redis.call('time') return {a, b} $ ./redis-cli --eval /app/time.lua 1) 1) "1638280442" 2) "996960" 2) 1) "1638280442" 2) "996966" Checking the actual expiry time: -- expire_check.lua redis.call('set', 'foo', '1') redis.call('expire', 'foo', 1) -- slow calls... redis.call('set', 'bar', '2') redis.call('expire', 'bar', 1) local fooExpiry = redis.call('PEXPIRETIME', 'foo') local barExpiry = redis.call('PEXPIRETIME', 'bar') return {fooExpiry, barExpiry} $ ./redis-cli --eval /app/expire_check.lua 1) (integer) 1638280843717 2) (integer) 1638280843730 The expire might not be pin-point accurate, and it could be between zero to 1 milliseconds out. The implication is that there could be times at which some keys have expired, but other related keys have not and this could lead to an inconsistent state. OUR SOLUTION The solution is to use EXPIREAT to set an absolute expiry time for all related keys, rather than rely on a relative expiry time through the TTL. The Redis documentation is not clear if multiple key expiry is guaranteed to occur at the same time if keys have the same EXPIREAT setting. To be cautious, we reordered key expiry to ensure that, regardless, we avoid inconsistency. -- expire_new.lua -- Unix time local now = redis.call('time')[1] local expiry = now + 1 redis.call('set', 'foo', '1') redis.call('expireat', 'foo', expiry) -- slow calls... redis.call('set', 'bar', '2') redis.call('expireat', 'bar', expiry) local fooExpiry = redis.call('PEXPIRETIME', 'foo') local barExpiry = redis.call('PEXPIRETIME', 'bar') return {now, fooExpiry, barExpiry} $ ./redis-cli --eval /app/expire_new.lua 2) (integer) 1638281266000 3) (integer) 1638281266000 This is typical of one of the many engineering problems we troubleshoot and solve each week here at Ably. Fancy working with us in the realtime sphere? Our engineers have a range of broad technology skills across infrastructure, security, distributed systems, and beyond. You can find us on Twitter or LinkedIn, and apply to join us in one of our open roles. | Discuss this post on Hacker News | -------------------------------------------------------------------------------- LATEST FROM ABLY ENGINEERING * Stretching a point: the economics of elastic infrastructure ? * A multiplayer game room SDK with Ably and Kotlin coroutines ? * Save your engineers' sleep: best practices for on-call processes * Squid game: how we load-tested Ably’s Control API * How to connect to Ably directly (and why you probably shouldn't) – Part 1 * Migrating from Node Redis to Ioredis: a slightly bumpy but faster road * No, we don't use Kubernetes On this page 1. How we use Redis at Ably 2. The issue 3. Our solution 1. Latest from Ably Engineering BUILD DEPENDABLE REALTIME APPS Ably offers versatile, easy-to-use APIs to develop powerful realtime apps. Guaranteed to deliver at scale. Start building RECOMMENDED ARTICLES Ably engineeringCollaborative experiences 17 min read CURSOR EVERYWHERE: AN EXPERIMENT ON SHARED CURSORS FOR EVERY WEBSITE Aug 18, 2023 Ably engineering 30 min read GROUP CHAT APP WITH OPENAI'S GPT May 24, 2023 Ably engineering 14 min read CRDTS ARE SIMPLER AND MORE COMMON THAN YOU THINK Feb 14, 2023 JOIN THE ABLY NEWSLETTER TODAY 1000s of industry pioneers trust Ably for monthly insights on the realtime data economy. Enter your email Submit How we use cookies to improve your experience. Accept and close THE ABLY PLATFORM Easily power any realtime experience in your application via a simple API that handles everything realtime. * Pub/sub messaging * Push notifications * Third-party integrations * Multiple protocol messaging ABLY IS FOR * Ably Asset Tracking * Extend Kafka to the edge * EdTech * Automotive, Logistics, & Mobility * B2B Platforms * Healthcare * eCommerce & Retail * Sports & Media * Gaming * IoT & Connected Devices DEVELOPERS * Start in 5 minutes * Documentation * Tutorials * Changelog * Support & FAQs * SDKs * System status WHY ABLY * Customers * Case Studies * Four Pillars of Dependability * Compare our tech * Multi protocol support * Third-party integrations ABOUT * About Ably * Pricing * Blog * Careers * Open protocol policy * Press & Media * Contact us -------------------------------------------------------------------------------- We're hiring!Learn more at Glassdoor We're hiring!Learn more at Glassdoor -------------------------------------------------------------------------------- CookiesLegalsData ProtectionPrivacy SOC 2 Type 2 Certified HIPAA Compliant EU GDPR Certified 256-bit AES Encryption sprite-discord sprite-facebook sprite-github sprite-glassdoor sprite-google sprite-icon-display-48hrs sprite-icon-display-about-ably-col sprite-icon-display-api-keys sprite-icon-display-api sprite-icon-display-asset-tracking-col sprite-icon-display-browser sprite-icon-display-calendar sprite-icon-display-call-mobile sprite-icon-display-careers-col sprite-icon-display-case-studies-col sprite-icon-display-chat-col sprite-icon-display-chat-stack-col sprite-icon-display-chat-stack sprite-icon-display-cloud-servers sprite-icon-display-compare-tech-col sprite-icon-display-customers-col sprite-icon-display-data-broadcast-col sprite-icon-display-data-synchronization-col sprite-icon-display-docs-col sprite-icon-display-documentation sprite-icon-display-events-col sprite-icon-display-examples-col sprite-icon-display-gdpr sprite-icon-display-general-comms sprite-icon-display-hipaa sprite-icon-display-integrations-col sprite-icon-display-it-support-access sprite-icon-display-it-support-helpdesk sprite-icon-display-kafka-at-the-edge-col sprite-icon-display-laptop sprite-icon-display-lightbulb-col sprite-icon-display-live-chat sprite-icon-display-map-pin sprite-icon-display-message sprite-icon-display-padlock-closed sprite-icon-display-platform sprite-icon-display-play sprite-icon-display-privacy-shield-framework sprite-icon-display-push-notifications-col sprite-icon-display-quickstart-guides-col sprite-icon-display-resources-col sprite-icon-display-sdks-col sprite-icon-display-servers sprite-icon-display-shopping-cart sprite-icon-display-sla sprite-icon-display-soc2-type2 sprite-icon-display-tech-account-comms sprite-icon-display-tutorials-demos-col sprite-icon-display-virtual-events-col sprite-icon-display-virtual-events sprite-icon-gui-ably-badge sprite-icon-gui-arrow-bidirectional-horizontal sprite-icon-gui-arrow-bidirectional-vertical sprite-icon-gui-arrow-down sprite-icon-gui-arrow-left sprite-icon-gui-arrow-right sprite-icon-gui-arrow-up sprite-icon-gui-burger-menu sprite-icon-gui-check-circled-fill-black sprite-icon-gui-check-circled-fill sprite-icon-gui-check-circled sprite-icon-gui-checklist-checked sprite-icon-gui-clock sprite-icon-gui-close sprite-icon-gui-copy sprite-icon-gui-cross-circled-fill sprite-icon-gui-cross-circled sprite-icon-gui-dash-circled sprite-icon-gui-disclosure-arrow sprite-icon-gui-document-generic sprite-icon-gui-enlarge sprite-icon-gui-external-link sprite-icon-gui-filter-flow-step-1 sprite-icon-gui-filter-flow-step-2 sprite-icon-gui-filter-flow-step-3 sprite-icon-gui-history sprite-icon-gui-info sprite-icon-gui-link-arrow sprite-icon-gui-link sprite-icon-gui-live-chat sprite-icon-gui-minus sprite-icon-gui-plus sprite-icon-gui-quote-marks-solid sprite-icon-gui-refresh sprite-icon-gui-resources sprite-icon-gui-search sprite-icon-gui-tick sprite-icon-gui-warning sprite-icon-live-updates-results-metrics-col sprite-icon-multi-user-spaces-col sprite-icon-social-x sprite-icon-tech-apachekafka sprite-linkedin sprite-quote sprite-stackoverflow sprite-twitter sprite-youtube