Wednesday, 2009-02-25

*** johndoigiii has quit IRC00:16
*** johndoigiii_ has joined #cc00:28
*** nathany has quit IRC00:31
nkinkadepaulproteus: Any advice about alpine? Perhaps a base config file that would be a good starting point?01:03
*** stevel has quit IRC01:07
*** stevel has joined #cc01:08
paulproteusnkinkade, If you're to use it against Gmail for IMAP, I'd google alpine+gmail.01:09
paulproteusOtherwise for now I'll remain silent so I can finish some more skin work and run off.01:09
nkinkadepaulproteus: No, I plan to use it with dovecot off my own server.01:09
nkinkadeBut no need to stop what you're doing now for this.01:09
nkinkadeI'll figure something out for now.01:10
*** tianshiz has joined #CC01:13
*** tianshiz has left #CC01:13
*** rohitj has joined #cc01:19
paulproteusBovinity, My request for your CSS help is in http://code.creativecommons.org/issues/issue139 .01:27
Bovinityah yes, that needs some love01:28
Bovinitymarked todo tomorrow01:28
paulproteusSweet.01:32
*** johndoigiii_ has quit IRC01:35
*** johndoigiii has joined #cc01:39
*** Bovinity has quit IRC01:55
*** jgay has quit IRC02:00
*** stevel has quit IRC02:17
*** mlinksva has quit IRC03:06
*** nkinkade has quit IRC05:08
*** rohitj has quit IRC06:16
*** isforinsects has quit IRC07:58
*** sama has joined #cc08:36
*** isforinsects has joined #cc13:46
*** balor has joined #cc13:52
*** UncleCJ has joined #cc14:06
*** johndoigiii has joined #cc14:14
*** UncleCJ2_ has joined #cc14:28
*** Phew^ has joined #cc14:38
*** Phew^ has left #cc14:38
*** balor has quit IRC15:22
*** jgay has joined #cc15:27
*** nathany has joined #cc15:34
*** nkinkade has joined #cc15:35
nkinkadenathany: Any thoughts on Giorgos' email?15:53
nathanynkinkade: many :)15:53
nathanyi'm going to chat with mike and will reply after the staff call15:54
nkinkadeMy personal feeling is that I'm generally averse to handing out a handful of shell accounts just to skin and configure a wiki.15:54
nathanyyeah, i concur15:54
nkinkadeSo shall I hold off on replying for now?15:54
nathanyyes, please15:54
nkinkadeI have a couple alternative ideas.15:54
nkinkadeI'll email them to you.15:54
nathanyi'll get a reply out this morning after i confirm with mike -- just want to double check on the funding we received for that (if there are requirements/strings)15:55
nkinkadeOnes you've surely already thought of, but I'll send them just to make sure we're on the same page.15:55
nathanysure, that'd be great15:55
*** rohitj has joined #cc15:55
*** sama has quit IRC16:42
*** TRD has joined #cc16:49
nathanyhey johndoigiii -- i'm going to update your key16:53
johndoigiiiokay great thx16:53
nathanydo you want me to keep both in the system? (not sure if you'll use the old one for your desktop)16:53
nathanyjohndoigiii: i kept both; done16:55
johndoigiiisorry bout that, no I will not need the other one16:56
paulproteusnkinkade, I'll grant user:mt admin privileges on cz.16:57
paulproteusre: Michelle's email and Pootle16:57
nkinkadepaulproteus: Okay, perfect.  Thanks for letting me know.16:58
paulproteus...if I can figure out how...16:59
paulproteus...do you know how I should do this?16:59
paulproteusNever mind, found it.17:00
paulproteusnkinkade, Done.17:01
nkinkadepaulproteus: As an addendum to the emails about CZ, I noticed another language admin the other day who was missing commit rights, when I thought they should have had them.17:12
nkinkadeFor Vietnamese.17:12
nathanynkinkade: i just replied to giorgos17:12
nkinkadeBut, then again, I do recall not checking the "Commit" right for many admins before.17:12
nkinkadenathany: That seems right to me.  Thanks.17:13
*** TRD is now known as TRD|Away17:15
nkinkadeI just answered 3 consecutive info@ emails where people distinctly used the word registration in connection with the license chooser, and who were wondering where there registration code was.17:20
nkinkadeWe have really got to change the messaging relating to that tool.17:20
*** stevel has joined #cc17:26
nathanyor make it harder to send info@ emails ;)17:30
nathany(just kidding!)17:30
*** balor has joined #cc17:38
*** Bovinity has joined #cc17:43
paulproteusnkinkade, BTW dealing with the XHTML attachment stuff is really easy in alpine.17:44
nkinkadepaulproteus: Probably because Alpine is natively able to display HTML inline?17:45
nkinkade*is not17:45
paulproteusIt is, but its handling of attachments is superb.17:46
paulproteusYou just sort of zoom in and get the ability to save the attachments.17:46
paulproteusIt is also displaying them inline.17:46
paulproteusBut if you ask for attachment view you get them.17:46
nkinkadeNice.17:46
nkinkadeBy the way, I tried sup-mail earlier this morning after having to install 4 src packages from Debian as dependencies.17:47
paulproteusWhat do you make of it?17:48
nkinkadeThere were some errors at a few points.17:48
nkinkadeI'll need to read more about it.17:48
paulproteusOkay.17:48
nkinkadeNo fatal errors, but didn't give me confidence.17:48
* paulproteus nods17:48
*** everton137 has quit IRC17:52
*** TRD|Away has quit IRC18:06
nkinkadenathany: There is definitely a bug in CiviCRM re: that email this morning.18:20
nkinkadeI'm about to have it tracked down, fixed and reported, but it's taking up my time now rather than the wiki stuff.18:20
nkinkadeJust wanted to let you know.18:20
nathanynkinkade: thanks18:21
nkinkadeIt's not likely to show up for many people, but a bug in the donation software seems more important than the wikis for the moment.18:21
nathanyabsolutely18:22
jgayanybody here going to the CC Boston Salon tonight?19:01
*** mattl has joined #cc19:07
mattlisforinsects: coming out tonight?19:07
paulproteusnkinkade, Er, I thought Czech was cz, not cs.19:23
* paulproteus checks Pootle19:23
paulproteusI just bounced you a message to justify the above "Er, I thought".19:23
paulproteusNope, it's totally cs. Weird, but okay.19:24
nkinkadepaulproteus: Is there something I need to do?19:25
mattlcs is when .cz and .sk was .cs19:25
paulproteusNot really.19:25
paulproteusmattl, Point taken.19:25
paulproteusnkinkade, What do you know about the addition of the lang hy to i18n?19:33
paulproteuspaulproteus@a7:/var/lib/pootle/checkouts/cc_org/cc-i18n-trunk/cc_i18ntools$ ls -l /var/lib/pootle/checkouts/cc_org/cc-i18n-trunk/i18n/hy/19:33
paulproteustotal 019:33
paulproteuspaulproteus@a7:/var/lib/pootle/checkouts/cc_org/cc-i18n-trunk/cc_i18ntools$ ls -l /var/lib/pootle/checkouts/cc_org/cc-i18n-trunk/po/hy/19:33
paulproteustotal 10019:33
paulproteus-rw-rw-r-- 1 pootle pootle 90771 2009-02-24 17:41 cc_org.po19:33
paulproteus-rw-rw-r-- 1 pootle pootle    54 2009-02-24 17:41 pootle-cc_org-hy.prefs19:33
paulproteus...which is to say, there is a PO-style file but no CC-style file.19:33
paulproteusAnd tragically, the PO-style file isn't a Subversion checkout.19:33
paulproteusDid you create the hy language recently, and if so, following what directions? I should ensure all the docs create everything correctly.19:34
*** rohitj has quit IRC19:34
paulproteusIt seems to be Armenian. Looking at http://translate.creativecommons.org/hy/cc_org/translate.html?untranslated=1&editing=1 in Pootle, it's a total mess; it appears to be some combination of CC-style and PO-style.19:37
Bovinitynkinkade: creativecommons.org has fallen over20:00
johndoigiiiyeah getting 50320:00
Bovinityvarnish, specifically20:00
Bovinityjohndoigiii: hi john. Alex @ cc sf here.20:01
johndoigiiiBovinity: hey Alex, I was just looking up bovinity in the contact info on teamspace20:01
Bovinityhehe20:02
paulproteuspaulproteus@a5:~$ curl http://creativecommons.org20:02
paulproteuscurl: (6) Couldn't resolve host 'creativecommons.org'20:02
paulproteuswell that's odd.20:02
* paulproteus adds 4.2.2.4 to resolv.conf on a520:02
paulproteusBovinity, johndoigiii: fixed?20:03
Bovinityaye20:03
paulproteus"yay"20:03
paulproteusemailing webmaster@ now20:03
johndoigiiiyup20:03
johndoigiiipaulproteus: you have  sec to answer a quick question about wpLicense?20:05
paulproteusjohndoigiii, Sure.20:05
johndoigiiipaulproteus: I'm just toying around with it, ensuring it still works correctly in wp 2.7 and noticed that it is still using jswidget .92, is there any particular reason for this?20:05
paulproteusNo reason at all.20:06
paulproteusWe should bump it.20:06
johndoigiiiohh okay, I wasn't sure20:06
johndoigiiiwell bumping up breaks the form processsing20:06
paulproteusWell in that case we ought to bump it and test it and then release it!20:07
johndoigiiiokay, how many projects rely on jswidget?20:07
paulproteusWe haven't been reading the logs, but at least WpLicense and the Six Apart thing they+we announced in 2007.20:08
paulproteusTypePad.20:08
paulproteusBut everyone links to the version they last tested with, so API changes are okay.20:08
nkinkadepaulproteus: Bovinity: nathany: I went to get a bit to eat for lunch, just got back with it, and am still eating.  I'll look into a5, but it's not like I'm the only one who could possible resolve the issue.  If a5 goes down, the first person to notice should do something about it right away, not just post a message to me on #cc and forget about it.20:09
paulproteusnkinkade, Yeah, and I did; see your email.20:09
paulproteusTurns out IMHO that the problem would not have been solved by just restarting Apache or Varnish.20:11
paulproteusSo a careful eye was needed, at least.20:11
nkinkadeIt's still down as far as I can tell.20:11
paulproteuser, and I get the same error from curl20:12
paulproteusweird20:12
nathanypaulproteus: what's 4.2.2.4?20:12
nkinkadeprintk: 2847 messages suppressed.20:12
nkinkadenf_conntrack: table full, dropping packet.20:12
paulproteusA Verizon DNS server that everyone knows the IP of.20:12
nathanynkinkade, paulproteus: seems to be up for me although super slow20:12
paulproteusFor small values of everyone.20:12
nathanynkinkade: should i have gotten an email re: a5?20:12
paulproteus(now works for me again too)20:12
paulproteusYeah, I think the DNS requests from a5 are timing out because its network queue is so long.20:13
nkinkadedmesg is full of "possible SYN flooding on port 80. Sending cookies."20:13
paulproteusA local caching DNS server *on* a5 would fix this.20:13
nkinkadeAnd also the messages I posted a minute ago.20:13
paulproteusnkinkade, Ya20:13
nathanyright, i saw that, it just implied that i would have seen it previously20:13
paulproteusI don't recall how to look into the length of the rxqueue and txqueue in Linux.20:14
paulproteusBut I do believe that the curl error is directly related to the Varnish error.20:14
paulproteus(and both now work fine)20:14
paulproteus(I'm running curl on a5, to be clear)20:14
paulproteusBut then again, why would Varnish rely on DNS when the default backend is named by IP?20:15
nkinkadeMaybe a transient DNS issue caused a major backlog of connections.20:15
paulproteus503 again20:16
paulproteusnkinkade, How can I ask Varnish why it reports 503? (Should it just be restarted?) Also feel free to tell me you're looking into it and I shouldn't bother you with questions.20:17
paulproteuspaulproteus@a5:~$ curl 127.0.0.1:808020:17
paulproteuscurl: (7) couldn't connect to host20:17
paulproteus(no running apache...)20:17
paulproteus(a running apache exists now)20:18
paulproteus(And the site works fine)20:18
paulproteusBut I should do things other than track a5's status....20:18
nkinkadepaulproteus: Apache had died?20:18
paulproteusPerhaps the autorestart scripts just kicked it?20:18
paulproteusWhen I wrote the curl error, "ps auxw | grep apache" listed no apache2 processes.20:19
nkinkadeIt seems that Apache is gone again.20:20
paulproteusYou can see in 'ps' that the processes all have a start time of 14:18 (CST), AKA 2 minutes ago.20:20
nkinkadeSYN_RECV: 25220:20
nkinkadeESTABLISHED: 1057820:20
nkinkadeTIME_WAIT: 1459520:20
nkinkadeFIN_WAIT1: 69820:20
nkinkadeFIN_WAIT2: 165820:20
nkinkadeTotal unique connections: 1808820:20
nkinkadeThose were the stats just reported from little script.20:21
nkinkade18K unique connections?20:21
paulproteusStill, why would Apache die?20:21
paulproteusNote this:20:21
paulproteusa5:~# tail -f /var/log/apache2/error.log20:21
paulproteus[Wed Feb 25 14:20:03 2009] [notice] caught SIGTERM, shutting down20:21
paulproteusI think the autorestart scripts may be killing it?20:21
paulproteusThey see slow Apache and think "restart it" rather than "wait for it", yes?20:21
nkinkadeI just disabled the script.20:22
paulproteusno running apache20:22
paulproteusi'll restart it20:22
paulproteusnow there's a running apache20:22
paulproteusAnd I'll keep "tail -f"ing the Apache2 error log.20:23
paulproteusLoad average is 2.90 and falling20:24
paulproteusPerhaps due to the downtime, though20:24
nkinkadeeth0 is pumping about about 1.2M/sec20:25
paulproteusOf what? Static files, mostly?20:25
nkinkadeJust using iptraf, so I can't tell.20:25
paulproteusThis is why I liked my old Apache performance analysis patch.20:25
paulproteusYou could tell where the time went.20:25
nkinkadeIt's not terrible, but that's pretty busy.20:25
paulproteusIf it's static files, my Athlon in 2001 could do that without breaking a sweat.20:25
paulproteusAnd load average is < 1.20:26
nkinkadeNow it's down to it's normal 500K/sec20:26
paulproteusI guess I'll go back to other things? I think the restart script hurt us here rather than helped us, but maybe with a longer timeout period it would have been okay.20:27
nkinkadeI have this sense that when any sort of problem arises that it causes a negative feedback cycle of backed up connections.20:27
paulproteusI have this sense that we need more data.20:27
nkinkadeI'm talking generals, not specifics, with that comment.20:27
paulproteusAnd also it'd be great if we could temporarily disable just the dynamic parts of the site.20:27
paulproteusYeah, I know, but I believe that a few specifics could fix the whole general problem.20:27
paulproteusIf we could just switch off the dynamic parts of the site to weather load storms we'd be totally fine.20:28
Bovinitypaulproteus: if one sets "Watch this page" on teamspace, does it email in the event of changes, or just highlights on the history page?20:28
paulproteusBovinity, Just highlights, unless you configure a user pref iirc.20:28
Bovinitypaulproteus: aha, indeed20:29
paulproteus(Whether or not the server *honors* that user pref is a LocalSettings.php option.)20:33
paulproteusnkinkade, If nathany thinks it's a good use of time, maybe you and I can/should sit down sometime and think about how we can better-handle load storms?20:33
paulproteus"sit down" used lightly.20:33
paulproteusnkinkade, Also, you might like Splunk.20:34
nkinkadepaulproteus: Sure.  Something needs to be done.20:34
paulproteusI haven't used it but it looks very nice (for automated log analysis).20:34
nkinkadeThe kernel logs on a5 are packed to the gills with possible syn flood warnings and then nf_conntrack: table full, dropping packet errors.20:35
paulproteusWhy are we even doing connection tracking in the kernel?20:35
paulproteusWe're running a stateful firewall because...?20:35
paulproteusYou can increase table size by adding line in /etc/sysctl.conf:20:36
paulproteusnet.ipv4.netfilter.ip_conntrack_max=<size>  ex. 16384 <http://forum.openwrt.org/viewtopic.php?id=17558>20:36
paulproteusBut why does a5 need a stateful firewall in the first place?20:36
paulproteusShorewall, I suppose.20:37
paulproteusI'm still of the opinion that's it's more trouble than it's worth.20:37
paulproteusBut they're your servers, not mine, to make that decision on, so you might find some solace in increasing the conntrack table size.20:37
paulproteusI think that the firewall buys us nothing.20:38
paulproteusI'd be curious to hear your thoughts.20:39
paulproteusWe both like making broad generalizations. Anyway, we can have this conversation another time.20:39
paulproteusnkinkade, More pressing to me is the discussion above about hy and Pootle.20:40
paulproteusMaybe you can comment on that when you have a moment.20:41
paulproteus(or feel free to tell me I should email you instead)20:41
*** rejon1 has joined #cc20:42
nkinkadepaulproteus: Yes, I added Armenian yesterday.  You were copied on the email in which I stated that I had added it.20:43
nkinkadeI didn't do anything special.  I just went to Admin -> Languages and added the language.20:43
nkinkadeNothing more, nothing less.20:43
* paulproteus nods20:43
paulproteusThat way is totally broken now that we have the svn stuff set up behind Pootle's back.20:43
nkinkadesvn has always behind Pootle in our setup.20:44
nkinkadeAt least as long as I've been at CC.20:44
paulproteusHmm, okay.20:44
rejon1crap20:44
nkinkade(Pootle was installed not long after I arrived.20:44
paulproteusWell, now that we have *my* svn stuff set up behind Pootle's back, that's totally broken.20:44
nkinkadeI'll just let you handle Pootle stuff in the future.20:45
paulproteusThat sounds like new breakage on my part. Let me see what the docs say you should have done to see if I can say you should have followed them. (-;20:45
paulproteusWell, if the docs are up-to-date, there's no reason we need to have a bus factor of one.20:45
paulproteusAnd if the docs aren't, then we have a bus factor of ca. 0.9.20:45
*** nathany has quit IRC20:51
*** nathany has joined #cc20:51
*** rejon1 has left #cc20:53
*** tvol has joined #CC20:55
paulproteusnkinkade, tty at 2 about VNC, and after that I'll work out the Pootle documentation stuff20:55
nkinkadepaulproteus: Yeah, let's look at VNC at 2.20:56
*** mlinksva has joined #cc21:00
*** rohitj has joined #cc21:12
*** Aggrav8d has joined #cc21:16
*** Aggrav8d has left #cc21:16
*** [mharrison] has quit IRC21:17
nkinkadepaulproteus: I'm giving a shot to your idea of just ditching netfilter/shorewall.  On a5 I stopped shorewall and there are now no rules, but I'd like to remove the dozens of modules that it installed.21:19
nkinkadeCan you think of any reason that I shouldn't remove them all?21:19
nkinkadeI can unload them in the same order it loaded them.21:19
nkinkadeI just wanted to be sure that nothing else needed those modules for any reason.21:20
*** balor has quit IRC21:21
Bovinitynathany: http://blag.xkcd.com/2009/02/25/kindle-2/21:30
nkinkadeRe: the modules ... apparently they weren't needed by anything because I removed them all and things seem okay.21:43
nathanyphew21:57
nathany;)21:57
nathanyBovinity: so i "found" the CD Maxi single of "Love, etc"... the mixes aren't bad21:58
Bovinity"found" you say21:58
nathanyit was just lying there!21:58
paulproteusnathany, Howdy22:02
paulproteusEr, I meant that for nkinkade, but you too!22:02
nathanygee thanks ;)22:02
paulproteusnkinkade, If rmmod works, then the modules weren't in use.22:03
Bovinityhttp://www.prefixmag.com/media/pet-shop-boys/love-etc-video/26367/22:03
paulproteusIt's always safe to try it.22:03
paulproteusnkinkade, Let's talk VNC right now; I'll move to a conf. room.22:03
nathanyBovinity: http://www.youtube.com/watch?v=b7TSDUHhPIw22:04
*** balor has joined #cc22:04
nkinkadepaulproteus: Shall we do a conference call?!22:04
nkinkadepaulproteus: What I had done with the modules was this:22:05
nkinkade# tac shorewall/modules | sed -e '/^#/d' | cut -d' ' -f 2 | xargs modprobe -q -r22:05
nkinkadeThere were dozens of modules in that file, and I wasn't 100% sure they were all *just* for shorewall/iptables.22:06
nkinkadeBut apparently they were, though there are still some iptables related modules installed.22:06
*** stevel has quit IRC22:06
nkinkadeIt seems that that might be normal, though, as I see them on rose.makesad.us too.22:06
paulproteusCool.22:08
paulproteusnkinkade, Let's try Ekiga; you start it up and I'll start up mine.22:08
paulproteusYou could read the list and stuff first...22:08
paulproteusI would do:22:08
paulproteus# tac shorewall/modules | sed -e '/^#/d' | cut -d' ' -f 2 | xargs echo modprobe -q -r22:08
paulproteusfor a dry run first, but you can read the modules list anyway so that's not necessary probably.22:08
nkinkadepaulproteus: I had already reviewed the list manually with a xargs echo.22:08
*** stevel has joined #cc22:08
paulproteusCool22:08
nkinkadeThere were just *so many* modules that it seemed ab-diculous.22:08
nkinkadeCalling sip:paulproteus22:09
paulproteusnkinkade, Okay22:09
nkinkadeSeems not to have worked.22:10
paulproteusI'll hang up on the echo test.22:10
paulproteusI can call 500@ just fine.22:10
paulproteusWant to join me in some ekiga.net conference?22:10
nkinkadeI don't get a ringtone for you.22:10
nkinkadeLet's try that.22:10
paulproteusFirst I'll try ringing you.22:10
nkinkadeOne sec.22:10
nkinkadeIt's using my skrs. instead of the headset!22:11
nkinkadepaulproteus: Try that call again.22:11
paulproteusOkay.22:12
*** tvol has quit IRC22:17
*** rohitj has quit IRC22:29
*** balor has quit IRC22:32
*** stevel has quit IRC22:33
*** Bovinity has quit IRC22:36
*** Bovinity has joined #cc22:36
*** tvol has joined #CC22:38
*** tvol_ has joined #CC22:47
*** stevel has joined #cc22:51
*** tvol has quit IRC22:54
Bovinitypaulproteus: nathany: http://learn.creativecommons.org/community/For_Learners?useskin=ahab22:56
nathanyBovinity: i think my preference would be for two separate lines of links (page views v. toolbox)22:58
nathanyperhaps we'd also add code to make the toolbox disappear if you're not logged in?22:58
Bovinity+1 for the latter22:58
*** stevel has quit IRC22:59
*** stevel has joined #cc23:00
*** stevel has quit IRC23:07
*** stevel has joined #cc23:08
paulproteusIn conclusion: VNC with "tight" encoding is way faster than the fruity DimDim thing.23:12
nkinkadeI agree.  And also in conclusion a 5 to 10 second delay in speaking is not conducive to effective communication.23:15
BovinityDIM fail$23:15
*** rohitj has joined #cc23:16
*** paulproteus has left #cc23:18
*** paulproteus has joined #cc23:19
paulproteusFor some unknown reason I left the #cc channel.23:20
paulproteusAnyway, time to check on those Pootle docs.23:20
johndoigiiihey guys, quick q... if you see a bug and am able to fix it and commit, there should be no need to issue a bug in roundup right?23:23
paulproteusjohndoigiii, Agreed, though if it's related to something one of us is working on it's good to let us know (unless you know we'll find it via a commit log email).23:23
johndoigiiioo ok, its just something small on CCnet23:24
paulproteusCool.23:24
johndoigiiijust wanted to make sure i wasn't breaking any process you guys might have established23:24
paulproteusMake sure you comply with what appears to be the branching process that that project uses.23:24
johndoigiiiwill do23:25
paulproteusnkinkade, So I think the web interface to create a language in Pootle should not really be used. When adding languages to cc_org, can I have you just "svn copy" en_US's i18n/ and po/ files and then ssh to the pootle user and svn update and restart Pootle?23:25
paulproteushttp://wiki.creativecommons.org/Translating_with_Pootle/Administration#Add_an_entirely_new_language is I think the process that fits in best with my new svn setup.23:25
nkinkadepaulproteus: That sounds good, and I've actually done that in the past. :-)23:26
paulproteusAh, cool. (-:23:26
nkinkadeBut I never was sure whether that was necessary or if Pootle would do it all correctly.23:26
paulproteusI think that you not doing it that way is what broke the expectations of the post-commit hook, which broke commit.23:26
paulproteusBy "I think that" I mean "".23:27
* paulproteus nods23:27
paulproteusnkinkade, Do you want to re-do that language, or should I?23:31
nkinkadepaulproteus: Since you are already in there, maybe you could go ahead? :-)23:34
paulproteusI'm not, but I can, sure. (-:23:34
paulproteusI'll handle it and test it and close that email exchange from earlier today.23:34
paulproteusAnd in the future, please do follow that. (-:23:35
Bovinitynkinkade: do you recall ohw you got the back plate out of the office freezer?23:36
nkinkadeBovinity: Did it freeze up again?23:37
Bovinitynkinkade: yup23:37
nkinkadeThere are 4 or 6 little screws with a hex-head.23:38
Bovinitymain fridge is nice and warm.23:38
nkinkadeI don't think it was necessary to remove more than those.23:38
nkinkadeHowever, it still didn't want to come off until some of the ice had melted already.23:38
Bovinityand the cover for two of the screws doesnt want to come out either23:42
nathanyBovinity: http://store.muledesign.com/3289059765_088439b51f_o.php23:54

Generated by irclog2html.py 2.6 by Marius Gedminas - find it at mg.pov.lt!