list back up
jwest at classiccmp.org
Sun Nov 2 16:39:22 CST 2014
The classiccmp server and it's backups were on a temporary san that hit a
bug and rebooted itself. Needless to say, dropping iscsi to a hypervisor is
not a good thing. The classiccmp vm thus rebooted itself, ran fsck, and (as
best as I can tell) the only thing corrupted was the cctalk and cctech
config.pck files (which contain the list settings and membership). The only
backup had the same corruption on it, because it ran the backup before I
noticed the problem. Anyways.
Knowing the recovery wouldn't be quick/easy anyways, I took the opportunity
to just build a new classiccmp server VM from scratch. The latest OS,
patches, dependencies, libraries, and ports. This new VM is on the new
production san so there's a pretty fair amount of additional disk space
thrown at classiccmp (and thus bitsavers). It's always amazing when I
revisit this just how many packages/services are on that machine that a
reload from scratch requires. It's been a busy few days/nights.
Things you should know:
1> The membership list for each list was recovered thanks to the hard
work of DBoone. He processed the last few days of maillogs to get the
subscriber lists back. THANK YOU D.
2> Given how we had to do #1, it is possible that a few folks that had
subscriptions weren't automatically re-subscribed (someone who set their
membership on 'no email' could be one example). Those folks will notice and
3> Given the very bizarre way we had the cctalk and cctech lists "joined
at the hip", I do not recall the magic we did to get that working properly.
I am aware that posts from one probably aren't being redirected to the other
like it used to (if at all). I'm working on refiguring that out, and any
suggestions (offlist) would be most appreciated.
4> Given that I had to recreate the lists configurations from scratch,
I'm pretty certain that I have some settings different than used to be. If
you find anything different that matters, let me know.
5> The "new" classiccmp server is currently hosting the mailing lists,
the classiccmp.org website, plus all websites that are underneath
classiccmp.org (ex. www.classiccmp.org/hp , www.classiccmp.org/cpmarchives ,
etc.). Thus, you may notice a different ip address and hostname on the
server. The new name/IP will stick till any future rebuilds.
6> The "old" classiccmp server is still up, and is hosting all the
websites that have their own domain name. Time permitting, I'll be migrating
those one at a time to the new server. During that time period - however
long it takes - ftp and rsyncd will not be available to prevent loosing
changes when cutover occurs.
7> I do not recall who all the folks were that were helping review/gate
posts between cctalk/cctech. If you were one of those folks, please let me
know asap off-list so I can get you re-added.
8> For people who have websites hosted on the classiccmp server, be
aware the new server is running mysql56, php56, and apache24 (with mod_itk).
Before your website is moved, let me know if you're aware of any breakage
that using those new versions may cause. I'd also prefer to transform any
myisam databases to innodb unless you're aware of an issue that may cause.
9> Please do not post to the list or email me directly about "hey, the
archives are gone!". They have been preserved, and time permitting they will
get imported back into the "new" mailing list (along with the missing 199x
bits that were recently mentioned). Rumor has it that someone has
volunteered to improve the archive search mechanism in the not too distant
10> Many people have asked me over the past year or so to host their
classiccmp-related website, and I pushed them off due to the lack of disk
space. That is no longer an issue so if you're one of the people wanting
that (and still do), now is the time to get back in touch with me. As
always, we'll host any classiccmp-related website free of charge.
11> Within the next 30 days (at most) we'll be adding the classiccmp server
to our CDN service. The advantage: I would expect to be able to remove
any/all bandwidth limits previously in place and folks will be able to
download files from the bitsavers master very quickly and your
classiccmp-hosted websites will be a lot more snappy. The downside is,
you'll have to either be ok with it taking a while for new/changed content
on your site to be visible world-wide, or you'll need to take a look at the
api (or via a CDN website login) to invalidate the cache when you change
something. In most cases, this can be easily automated.
12> During the mailing list outage, all websites and bitsavers were still
up. It was just the mailing lists that were affected.
I'm sure there's more that I've forgotten. But.. Enjoy!
More information about the cctech