urlscan.io logo urlscan.io
SecurityTrails logo

community.spiceworks.com Open in urlscan Pro
45.60.13.212  Public Scan

Back to summary
Submitted URL: https://freshlistsinfo.com/campaigns/vg103t50otbf4/track-url/jf0625od3y1c1/68462fe491eaa3b5ef09d5b15239d46d58fcb53d
Effective URL: https://community.spiceworks.com/topic/2288745-uh-oh-trouble-in-o-z?source=start&pos=12?source=navbar-global?source=start&pos=34&...
Submission: On August 28 via api from US — Scanned from NL

Form analysis 1 forms found in the DOM

<form>
  <i class="1693259188294 mag-glass"></i>
  <input class="1693259188294 search-input" autocomplete="off" placeholder="Search Spiceworks">
  <i class="clean-icon"></i>
  <div class="1693259188294 trending-topics"></div>
  <div class="1693259188294 search-box-results"></div>
</form>

Text Content

Home
 * News & Insights
   * News & Insights Home
   * Artificial Intelligence
   * Innovation
   * IT Careers & Skills
   * Cloud
   * Cyber Security
   * Future of Work
   * All Categories
   * Marketing
   * HR
   * Finance
 * Community
   * Ask question
   * Community Home
   * Spiceworks Originals
   * Cloud
   * Collaboration
   * Networking
   * Water Cooler
   * Windows
   * All forums
   * How-Tos
   * Scripts
   * Vendors
   * Meetups
 * Reviews
 * Online Events


Login Join
Login Join


 * Home
 * Spiceworks
 * Features


UH-OH! -- TROUBLE IN O-Z

Posted by sladeashford on Sep 17th, 2020 at 10:51 AM
Features IT Jobs / Careers

This is in regard to a Microsoft Exchange DAG (Database Availability Group) for
a very large food franchise that's based in a few countries around the world.
What I thought would be a ridiculously easy change ended up bringing the entire
O-Z database to a crashing halt.

At the time I had done a 2007 to 2013 exchange as well as a fresh installation
of exchange 2013 so I thought I was this super hot exchange guru.

The Drive the database was sitting on was running extremely low on hard drive
space, but because it was a physical server with partitions, increasing space
was not as straight forward as it is now with a Virtual Environment. I arranged
new drives and installed them into the server, set up a logical disk for the
RAID configuration, and continued to bring the new partition online. I gave it a
drive letter and then moved the database, which all went super smoothly.

When I mounted the database it started giving issues which I was able to get
past eventually but there was no mail flow to anyone that had a mailbox residing
in that database.

Due to the times this franchise is open it was extremely difficult to arrange a
good time that suited them as there were large amounts of data that had to be
copied so a long weekend would have been perfect. Unfortunately I was told to
start Saturday afternoon after 14:00.

Sunday night at around 18:00, after spending the entire Saturday night and
Sunday power googling with no resolution, I gave a friend a call, someone I know
is an exchange guru, not just a self-proclaimed one like I was. Well after
hearing what happened he spent a few good hours on it but wasn't able to find a
solution. Bear in mind business starts at 07:00 on Monday morning.

He eventually figured out what the problem was early on Monday morning but the
steps we needed to take to fix the problem were RIDICULOUS. If you have your
server configured with C:\, E:\, F:\ and G:\ and put the A-G database on E and
O-Z on F:\ then all servers brought into the DAG need to have the exact same
configuration with the database copy residing on the same drive as the other
server. I obviously moved the drive to H:\ and the other servers were like Hell
no, Ours is on F:\ and that's how the fight started.

Business only commenced Wednesday afternoon. All stores that began with an O all
the way to Z had no mail.

Well, as you can imagine, the client was spitting flames and I am sure someone
blew a blood vessel or suddenly developed a case of Chronic High Blood Pressure.
The client and I had an extremely good relationship as I had been given the task
of Server and network escalation for the onsite guys and if a "priority 1,
severity 1" came in for them I had to drop everything I was doing to assist
them.

However, from the time I called the client on Sunday afternoon to tell him
what's happened and that I needed to get someone with far more knowledge and
experience to assist me till the day email started to flow again, he pretty much
spoke 60 decibels louder towards me. I could see the anger and frustration just
consume his entire demeanor.

Thankfully, once mail was flowing again our relationship turned to normal again.
He always phoned me on my cell for issues or if he needed it done quickly rather
than going through the switchboard, helpdesk, and account manager. Strangely
enough, they still trusted me in the environment which I was really grateful
for, but it did change my cowboy mentality regarding making changes to an
environment I 'think' I know.

My direct boss and the account manager that I had to give feedback to constantly
were probably the best at holding in their emotions. Once I called them on
Sunday, just before the client, I could hear that initial like "THIS IS UTTER
DEVASTATION," but they did everything they could to assist me and talk to the
client on my behalf. I am extremely short tempered and I will have my say if you
come gunning at me. If you shout at me in a work environment I lose respect
instantly. If something happens, call me into an office so we can talk humanely
to each other. If I do the same thing again, well then, I expect that person to
be a little ill-tempered and I will accept the consequences. I don't think I
would have been able to keep my temper under control if they hadn't been as
supportive as they were.

Trying to troubleshoot or make progress on the problem at hand but your phone is
constantly ringing to give someone else an update or asking how long it's going
to take, all while knowing I completely messed up (keeping it clean language
wise), with no sleep I was right on the edge at times. I was typing powershell
commands with the downward force of a sledge hammer, like trying to push the
keys out the other side of the keyboard.

It taught me a lot about keeping cool under pressure, always ensuring your
knowledge on the latest tech and environments is up to date and rather take a
little longer to assess, understand and plot the best strategy to achieve the
task before naively and arrogantly making changes.


Got a good story about a time you messed up at work? Your fellow SpiceHeads want
to hear it! Send us an email (news@spiceworks.com) with your Uh-Oh! story!
You'll earn 250 spice points when your feature gets published! Read about other
IT pros' awkward moments in the Uh-Oh Features group.

Spice (76) Reply (13)
flagReport
sladeashford
This person is a verified professional.
Verify your account to enable IT peers to see that you are a professional.
poblano

24X7 CYBERSECURITY MIT SOC UND INCIDENT RESPONSE

Sep 21 @ 4:00 PM Webinar Webinar : 24x7 CyberSecurity mit SOC und Incident
Response Event Details Opens a new window View all events



13 REPLIES

 * thomasstenhaug
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   poblano
   Sep 17th, 2020 at 12:41 PM
   
   shiiiiieeett well its these kind of situations that really stick with you.
   Glad you got it to work. And still managed to get a good relationship with
   the client.
   
   Edit: also first, that never happends almost.
   
   Spice (7) flagReport
   Was this post helpful? thumb_up thumb_down
 * DailyLlama
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   ghost chili
   Sep 17th, 2020 at 1:18 PM
   
   Ugh, what a mess! Good that you were able to figure it out and fix it in the
   end.
   
   Spice (2) flagReport
   Was this post helpful? thumb_up thumb_down
 * john.chuma
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   cayenne
   Sep 17th, 2020 at 2:41 PM
   
   That brings me back to the says of when I was at a MSP - some of the Exchange
   mailboxes were corrupted (large mailboxes) and were pegging the CPU on the
   Exchange server/s. We initially thought it was a bad server, but through
   working with Microsoft support around the clock of nearly a week, we figured
   it out. The client moved to O365 shortly after that.
   
   Spice (2) flagReport
   Was this post helpful? thumb_up thumb_down
 * bengarbutt
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   serrano
   Sep 17th, 2020 at 3:23 PM
   
   I'm glad that my Surname starts with a G after reading this..... 
   
   Spice (3) flagReport
   Was this post helpful? thumb_up thumb_down
 * Jason1121
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   thai pepper
   Sep 17th, 2020 at 5:59 PM
   
   Great story.  Been there a few times in the past with Exchange. Drive space
   was usually the cause.
   
   It's situations like you describe that taught me to give Exchange 4x the room
   I think it might need.  Like you said though - with virtualization it's a
   whole lot easier to expand drive space.
   
   Exchange being down for a long time is the reason I put Mail Enable in front
   of the Exchange server.  Users could then use the webmail interface for
   email.  Not as good as Outlook but it works in a pinch.
   
   Spice (4) flagReport
   Was this post helpful? thumb_up thumb_down
 * deanmoncaster
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   datil
   Sep 17th, 2020 at 6:09 PM
   
   "how can this happen".
   
   it is a computer, stupid things happen all the time for no apparent reason!
   
   Spice (6) flagReport
   Was this post helpful? thumb_up thumb_down
 * DailyLlama
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   ghost chili
   Sep 17th, 2020 at 6:33 PM
   
   > deanmoncaster wrote:
   > 
   > "how can this happen".
   > 
   > it is a computer, stupid things happen all the time for no apparent reason!
   
   "no apparent reason" usually means users!
   
    * local_offer Tagged Items
    * deanmoncaster
   
   Spice (5) flagReport
   Was this post helpful? thumb_up thumb_down
 * Stay_Dandy
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   thai pepper
   Sep 17th, 2020 at 8:06 PM
   
   I had an issue just this morning where I had to just stay cool. Especially
   when they went behind my back to my boss to try to get me in trouble. It can
   be so hard sometimes. Good job getting it resolved but I feel your
   frustration.
   
   Spice (2) flagReport
   Was this post helpful? thumb_up thumb_down
 * David Whitehead
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   chipotle
   Sep 17th, 2020 at 8:21 PM
   
   Aside from the fact that you were able to get things going again, I
   especially liked the " If you shout at me in a work environment I lose
   respect instantly.". I'm the same way.
   
   Spice (4) flagReport
   Was this post helpful? thumb_up thumb_down
 * Manwiththeforce
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   jalapeno
   Sep 17th, 2020 at 10:49 PM
   
   I remember a situation with Exchange 2013 I ran into - I was upgrading
   Exchange 2003 all the way to 2013 (including the hops in the middle). I spent
   countless hours before beginning just the first migration, researching,
   planning, guide creation off of other guide creations, more researching, more
   planning, revising my guide. Went through all the migrations up to 2013
   flawlessly. I thought I had this exchange thing down. Waited a week for it to
   settle, researched some more, revised my guides some more, had a different
   guide up from someone more knowledgeable than me, went to migrate to Exchange
   2013 and ran into issue after issue after issue. I spent days on this, got my
   boss involved, he spent days, got Microsoft involved.....it was a soul
   crushing learning event.
   
   Spice (5) flagReport
   Was this post helpful? thumb_up thumb_down
 * 
 * John5152
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   habanero
   Sep 18th, 2020 at 11:05 AM
   
   And this is why Office365 is the best thing since sliced bread - no more
   Exchange to screw up - I mean manage!
   
   
   Spice (3) flagReport
   Was this post helpful? thumb_up thumb_down
 * StapleBench
   This person is a verified professional.
   Verify your account to enable IT peers to see that you are a professional.
   poblano
   Sep 22nd, 2020 at 6:50 AM
   
   I was just hired as IT support right out of high school for the world's
   largest car dealership at the time. It was around Y2K and I was tasked to
   look at the corporate internal email system for the top management. Internal
   email for like 12 people.... It was an aging physical Compaq server running
   on NT4 with MS Email for workgroups... There was a qualification test that
   Compaq had that certified the hardware RAID for Y2K that I stared at about 10
   pm. Sometime around 1 am Monday morning I got a BSOD 0x0000007e. FML!..
   Somehow figured out where the tape backups were, loaded drivers, and restored
   that server before 6 am before the big boss got in and saved the day. Nobody
   knew how bad it could have been. I didn't admit how bad it could have been.
   All they knew was there I was, waiting to make sure email was working early
   in the morning.  
   
   Spice (4) flagReport
   Was this post helpful? thumb_up thumb_down
 * burncreek
   sonora
   Sep 29th, 2020 at 10:36 AM
   
   > David Whitehead wrote:
   > 
   > Aside from the fact that you were able to get things going again, I
   > especially liked the " If you shout at me in a work environment I lose
   > respect instantly.". I'm the same way.
   
   Same here. I’m pretty short tempered as well and will also damn right have my
   say if you come gunning at me, as sladeashford so accurately puts it. This
   has proved to be a problem in Sweden, where I live. Here, the majority of
   people are utterly afraid of getting into any kind of conflicts, be it small
   or even smaller. Many take such as personal insults, whereas for me it’s more
   of a way of clearing the air right then and there.
   I’m not saying one way is better than the other, only different. We’re all in
   the same boat and hell, I’ll buy you a beer later to make up, okay? :-)
   
   
   flagReport
   Was this post helpful? thumb_up thumb_down

lock

This topic has been locked by an administrator and is no longer open for
commenting.

To continue this discussion, please ask a new question.




READ THESE NEXT...


 * SNAP! -- BLUE SUPERMOON, LOCH NESS DRONES, 50TB MAGNETIC TAPE, COFFEE
   CONCRETE
   
   Spiceworks Originals
   
   Your daily dose of tech news, in brief. Welcome to the Snap! Flashback:
   August 28, 2009: The End of AppleTalk (Read more HERE.) Bonus Flashback:
   August 28, 1993: Galileo spacecraft flies by asteroid Ida (Read more HERE.)
   You need to he...

 * 


 * BITLOCKER WEAKNESS - PERHAPS?
   
   Security
   
   Generally, we think that bitlocker is secure but as with many secure systems
   there will be a weakness.
   This https://hackaday.com/2023/08/25/bypassing-bitlocker-with-a-logic-analzyer/ article
   describes how it is possible to get hold of the bitlocker key st...


 * NERD JOURNEY # 229 - A DEPTH AND A BREADTH WITH CHRIS WILLIAMS
   
   Best Practices & General IT
   
   Many times we need someone to recognize our talent and encourage us to pursue
   a specific role or direction.  Chris Williams was a psychology student and a
   gamer, and when a computer science professor learned that he had networked
   all the computers in his ...


 * SPARK! PRO SERIES - 28TH AUGUST 2023
   
   Water Cooler
   
   Today in History: 1988 - NATO Air-show accident An air show involving
   military jets at the Ramstein Air Base in Germany turns tragic on August 28,
   1988 when three jets collide in mid-air and fall into the crowd. Sixty-nine
   of the 100,000 spectators...


 * WHAT ARE YOUR FAVORITE AT-WORK SNACKS?
   
   Spiceworks
   
   Today is "Crackers Over the Keyboard" day, a somewhat bizarre holiday that
   few know about who aren't actively searching for obscure holidays online...
   but such is my lot. The original intention of the holiday I think was a small
   rebellion... knowing wh...

 * About
 * Contact
 * Support
 * Press / Media
 * Careers
 * SpiceWorld
 * Blog
 * * 
   * 
   * 
   * 

 * Sitemap
 * Privacy Policy
 * Terms of Use
 * Guidelines
 * Accessibility Statement
 * Do Not Sell My Personal Information
 * © Copyright 2006 - 2023 Spiceworks Inc.






WE CARE ABOUT YOUR PRIVACY

If you consent, we and our partners can store and access personal information on
your device to provide a more personalised browsing experience. This is
accomplished through processing personal data collected from browsing data
stored in cookies. You can provide/withdraw consent and object to processing
based on a legitimate interest at any time by clicking on the ‘Manage
Preferences’ button.Our Privacy Policy


WE AND OUR PARTNERS PROCESS DATA TO:

Store and/or access information on a device. Personalised ads and content, ad
and content measurement, audience insights and product development. Our Partners

Reject All I Accept
More Options