Friday, 2010-06-04

*** mralex has quit IRC00:00
*** JoiIto has joined #cc00:23
*** sudiptamondal has joined #cc00:35
*** JoiIto has quit IRC00:36
*** sudiptamondal1 has joined #cc00:39
*** luisv has quit IRC00:40
*** sudiptamondal has quit IRC00:40
*** akila87 has joined #cc00:43
*** JoiIto has joined #cc00:44
*** JoiIto has quit IRC00:48
*** MarkDude has quit IRC01:04
*** igorlukanin has joined #cc01:20
*** igorlukanin1 has joined #cc01:21
*** igorlukanin1 has quit IRC01:36
*** nkinkade has quit IRC01:52
*** JoiIto has joined #cc01:59
*** oshani has quit IRC02:17
*** JoiIto has quit IRC02:29
*** sudiptamondal1 has quit IRC02:35
*** Kaetemi has quit IRC03:23
*** Kaetemi has joined #cc03:23
*** JoiIto has joined #cc04:13
*** JoiIto has left #cc04:17
*** akozak has quit IRC05:36
*** Kaetemi has quit IRC06:04
*** sudiptamondal has joined #cc06:33
*** wormsxulla has quit IRC07:27
*** michi_ has joined #cc07:28
*** michi_ has quit IRC07:34
*** sama_ has joined #cc07:41
*** wormsxulla has joined #cc07:44
*** sudiptamondal has quit IRC08:35
*** sudiptamondal has joined #cc08:56
*** erlehmann has joined #cc08:59
*** anttil has quit IRC09:01
*** akila87 has left #cc09:09
*** anttil has joined #cc09:13
*** wormsxulla has quit IRC09:51
*** wormsxulla has joined #cc10:07
*** anttil has quit IRC10:09
*** akila87 has joined #cc10:10
*** bassel has joined #cc10:23
*** bassel has quit IRC10:42
*** sudiptamondal has quit IRC11:21
*** anttil has joined #cc11:31
*** user01 has joined #cc12:02
*** user01 has quit IRC12:04
*** user01 has joined #cc12:05
*** user01 has quit IRC12:11
*** Roux_taff has joined #cc12:57
*** Roux_taff has left #cc12:57
*** tvol has joined #cc13:18
*** everton137 has joined #cc13:25
*** oshani has joined #cc13:25
*** sama_ has quit IRC13:35
*** sama_ has joined #cc13:35
*** sama_ has quit IRC13:37
*** sama_ has joined #cc13:38
*** everton137 has quit IRC13:41
*** sama has joined #cc13:58
*** sama_ has quit IRC14:01
greg-gJED3: question about the humanizer, I thought I had the morepermissions stuff correct on my test page, but the humanizer is giving "Permissions beyond the scope of this public license are available at ."14:03
*** akila87 has quit IRC14:08
*** akila87 has joined #cc14:10
*** Odemia has quit IRC14:12
*** Odemia has joined #cc14:13
*** Oxygeek has joined #cc14:17
*** nkinkade has joined #cc14:19
*** Oxygeek has left #cc14:20
*** sama_ has joined #cc14:40
*** sama has quit IRC14:43
*** akila87 has left #cc14:52
*** mattl_ is now known as mattl14:56
*** mattl has joined #cc14:56
*** Kaetemi has joined #cc14:56
*** sudiptamondal has joined #cc15:06
*** oshani has quit IRC15:11
*** oshani has joined #cc15:12
*** jgay has joined #cc15:15
*** sama_ has quit IRC15:16
*** sama_ has joined #cc15:16
*** sama_ has quit IRC15:28
*** oshani has quit IRC15:33
*** akila87 has joined #cc15:34
*** sudiptamondal1 has joined #cc15:34
*** oshani has joined #cc15:34
*** sudiptamondal has quit IRC15:36
*** sudiptamondal1 has quit IRC15:39
*** sudiptamondal has joined #cc15:41
*** oshani has quit IRC15:46
JED3nkinkade: ping15:55
nkinkadeJED3: Here.15:56
JED3could i solicit your help for just a sec?15:56
*** erlehmann has quit IRC15:57
JED3the scraper went down from what looks like when the logs were rotated, creating new empty log files with invalid permissions15:57
JED3nkinkade: do you know much about the SuexecUserGroup configuration for mod_fcgid?16:00
nkinkadeJED3: No, I know little to nothing about the SuExec stuff.16:01
nkinkadeSo does the scraper need to be restarted after the logs are rotated?16:01
*** JoiIto has joined #cc16:01
JED3nkinkade: when the logs are rotated they are set with permissions that the scraper cannot write to them16:02
nkinkadeJED3: Logrotate can handle that and set the new log with certain perms.16:03
JED3the current SuexecUserGroup is set to "john john"16:03
nkinkadeCheck the "create" directive of logrotate.16:03
nkinkadecreate mode owner group16:03
nkinkadeJED3: What do the perms need to be, and owner/group?16:04
JED3nkinkade: does it make more sense to fix this in the logrotate or changed the suexecusergroup to something like "www-data www-data"16:04
nkinkadeJED3: If you are happy and comfortable with the current user/group of the log files, from a security standpoint, then I'd say we just use logrotate to fix it.16:05
JED3nkinkade: i'd prefer that the logs not be owned by 'john'16:06
nkinkadeJED3: Should root own them?16:06
JED3same for the scraper's fcgi process16:06
nkinkadeJED3: What should the fcgi process run as?16:06
JED3i'm going to change the suexecusergroup then16:06
JED3www-data users?16:06
nkinkadeSeems to me it's better john, than root.16:06
nkinkadeOh, www-data could be good, I think.16:07
nkinkadewww-data doesn't own much.16:07
JED3oh but www-data wont work will it? does the user need to be a sudoer?16:07
nkinkadeJED3: I don't think so.16:07
nkinkadeSuExec will just run the process as the owner of the script, right?16:08
JED3nkinkade: not sure, i dont know much about suexec16:08
nkinkadeJED3: I'm pretty sure that's how it works.  If www-data owns the script, then the resulting process will be run as user www-data.  Though I could be wrong.  Try it.16:09
*** nyergler has joined #cc16:09
*** oshani has joined #cc16:22
JED3nyergler: ping16:22
*** akila87 has left #cc16:22
*** akozak has joined #cc16:26
*** luisv has joined #cc16:28
nyerglerJED3, pong16:29
JED3nyergler: hi, i'm trying to set where i want the python-egg-cache to be using buildout, but the config doesn't seem to be taking16:30
JED3nyergler: is the configuration option "environment-variables" or "environment" ?16:31
nyerglerenvironment, i think16:31
nyerglerJED3, how are you trying to set it?16:32
JED3neither have worked for me...16:32
JED3environment =16:32
JED3    PYTHON_EGG_CACHE ${buildout:directory}/var/.python-eggs16:32
JED3nyergler: ^^16:33
nyerglerJED3, in the zc.recipe.egg section?16:34
JED3yes, tried it there and in the main buildout part16:34
nyerglerJED3, looks like you have the format wrong16:35
nyerglerJED3, looks like you need to use the "custom" entry point for the recipe, too16:36
nyerglerJED3, possibly unrelated, did you see my email re: the scraper?16:36
JED3nyergler: directly related to my questioning :)16:37
JED3the logs were rotated with different permissions, decided to change the suexecusergroup of the scraper fcgid to www-data, can't run the scraper as it tries to create a python-egg cache in the /var/www/ directory which is the home of the newly appointed user, trying to change that now.16:39
JED3^^ which is why i'm asking buildout questions16:39
nyerglerJED3, ah16:40
nyerglerJED3, I really hate egg caches16:41
nyerglerJED3, why not just set unzip=true in buildout, blow away your eggs, re-run buildout so that eggs are unzipped, and ignore the cache?16:41
nyergler(of course I'd suggest backing up your current deployment just in case something goes horribly, unexpectedly awry).16:42
nyerglerJED3, but regardless, it looks like you have to specify environment as a buildout section, not as inline variable assignments16:43
JED3nyergler: trying unzip now16:43
JED3nyergler: yeah but when i used the egg:custom recipe didn't seem to like how i was declaring the egg dependencies16:43
*** mralex has joined #cc16:51
*** bassel has joined #cc17:00
nyerglerJED3, did unzip work?17:07
JED3nyergler: yes that did work, but now i'm chasing down why fcgid is sending out 503's17:07
JED3i can run the server as www-data (same user thats declared in the suexecusergroup config) in the foreground just fine17:09
*** sudiptamondal has quit IRC17:18
*** oshani has quit IRC17:29
*** sudiptamondal has joined #cc17:37
*** sudiptamondal has quit IRC17:49
*** jgay has quit IRC17:54
*** jgay has joined #cc17:54
*** jgay has joined #cc17:55
*** jgay has joined #cc17:57
*** jgay has quit IRC17:58
nyerglerJED3, if you haven't already moved on, might try turning up the log level to get more details about the problem17:58
*** sama_ has joined #cc17:58
*** jgay has joined #cc17:58
*** jgay has joined #cc17:58
nkinkadeJED3: mralex:  This page on the CC Wiki is hanging, and eventually returns a 503 from Varnish:17:59
nkinkadeThe very odd thing is that it's not even getting far enough for Apache to log the request.18:00
JED3nyergler: any idea what this is "mod_fcgid: can't apply process slot for /var/www/"18:00
nkinkadeDoes anything immediately occur to either of you?18:00
mralexno idea18:00
nyerglerJED3, googling seems to imply that it's a permissions issue18:02
*** jgay has quit IRC18:03
*** luisv has quit IRC18:05
*** luisv has joined #cc18:06
*** Kaetemi has quit IRC18:08
*** Kaetemi has joined #cc18:08
JED3nyergler: nkinkade figured it out18:17
nkinkadeWhat was it?18:17
JED3suexec won't run anything thats group writable18:17
nkinkadeThat's a good one to tuck into the back of my head.18:18
JED3:/ good to know in the future before i go down that rabbit hole again18:18
nyerglerdidn't we just talk about that a week ago?18:18
nyergleror maybe that was with webber18:18
JED3yeah i'd never have dealt with that before18:18
JED3nkinkade: i still need to change the logrotate, when do the logs normally rotate?18:19
nkinkadeJED3: I'm not sure of the exact time.18:20
nkinkadeI'm happy to do it, or you can look in /etc/logrotate.d18:20
JED3was going to pack up and head into the office as my battery is dying18:20
nkinkadeJED3: So I should just have logrotate set the logfile to owner www-data with mode 644?18:21
nkinkadeOr 640, or evern 600?18:21
JED3nkinkade: i just editted it to "john everette"18:21
nkinkadeJED3:  everett needs to have write permissions.18:21
JED3did you change it to 660 or was it already like that?18:22
nkinkadeNot only to the log file, but also for the directory.18:22
JED3nkinkade: everett has write18:22
nkinkadeCool.  And those perms are already set in the logrotate conf file, right?18:23
JED3does www-data need write, is that what you meant?18:23
JED3create 660 john everett18:23
*** oshani has joined #cc18:23
JED3heading into the office now, back in a bit18:24
nkinkadeBut JED3: should the script have create 660 www-data everett?18:25
JED3nkinkade: no, i ended up reverting the suexec user back to john while i was trying to figure out what was going on18:26
*** Imz has joined #cc18:26
nkinkadeAh, okay, we'll now you know where to change it if you decide to use www-data at some point.18:26
JED3i suppose i could change it to www-data if it makes the most sense for us to do it that way18:26
JED3 yeah18:27
nkinkadeI don't think it really matters.  john is fine.18:27
*** JED3 has quit IRC18:32
*** TDJACR has quit IRC18:42
*** TDJACR has joined #cc18:42
*** Imz has left #cc18:43
*** TDJACR has quit IRC18:46
*** TDJACR has joined #cc18:47
nyerglernkinkade, are you looking at the wiki?18:59
nkinkadenyergler: It seems to be up to me.18:59
nkinkadeIs it down for you?18:59
nkinkadeThat page loads fine for me.18:59
nyergleri get:19:00
nyerglerFatal error: Allowed memory size of 52428800 bytes exhausted (tried to allocate 74 bytes) in /var/www/  on line 509619:00
nyergler(for main page)19:00
nyerglerFatal error: Allowed memory size of 52428800 bytes exhausted (tried to allocate 1966080 bytes) in /var/www/ on line 463719:00
nyerglerfor Grants19:00
nyergleralso, nagios sent an email with "String not found"19:00
nkinkadeInteresting. I'm not seeing that for some reason, even upon refresh.19:00
nyerglernkinkade, seems to be fine now19:02
nkinkadenyergler: One thing is that had APC disabled for testing.  Perhaps that was causing an issue?19:05
nkinkadeBut strangely it wasn't causing an issue for me here??19:06
nkinkadenyergler: For what it's worth, I think I may have found the problem, and it was Jesse W. that inadvertently pointed it out.  Special:NewFiles, hangs, to the point where Apache never even logs the requests.19:09
nkinkadeWhile I was busy trying to figure out why, the machine locked up and I had to reboot.19:09
nkinkadeI wasn't sure if it was my testing that did it, or that in combo. with some crawlers.19:09
nkinkadeSadly wiki-staging, doesn't exhibit the problem, so I can't do testing on the testing site.19:10
*** JED3 has joined #cc19:12
JED3nyergler: have a sec?19:22
nyerglerJED3, sure19:22
greg-gJED3: btw, did you get a chance to take a look at the Humanizer's output for More Permissions when using ?19:23
greg-gshould I just open a ticket instead of bugging you again? ;)19:23
JED3greg-g: no, actually you raised an interesting question for nry and I19:27
greg-guh oh19:27
greg-gmy bad19:27
JED3we didn't think about the case where people may be using mailto links for cc+19:28
greg-gahh, yeah19:29
*** luisv has quit IRC19:33
*** luisv has joined #cc19:35
*** luisv has quit IRC19:38
*** luisv has joined #cc19:45
*** luisv has quit IRC19:50
*** michi_ has joined #cc19:57
nkinkademralex: Do you know if there is any reason we should be using ImageMagick to convert images on the wiki as opposed to PHPs internal tools?20:19
mralexnkinkade: do we have a version of php installed that has support for its built in image handling features? (libgd, if i recall)20:21
nkinkademralex: We do:
mralexnkinkade: then i see no reason not to use it — assuming it's just a configuration option away?20:23
nkinkademralex: It is, and the default is to use gd, but for some reason we had our install using ImageMagick.20:23
nkinkadeTurns out it was ImageMagick dieing causing Special:NewFiles to hang.20:24
nkinkadeI think ImageMagick was perhaps hittings a shell memory limit.20:24
mralexi imagine there was a reason for it, maybe we didn't have GD installed.20:24
nkinkadeYup.  I re-enabled ImageMagick and bumped $wgMaxShellMemory to a much higher value and it worked.20:26
nkinkadeBut since we don't need to do that tweaking, I just set the wiki to use GD.20:27
*** wormsxulla has quit IRC20:32
mralexnkinkade: you know, the only reason i can think of why we weren't using GD was php's own memory limits...20:40
greg-gJED3: awesome, thanks for updating the humanizer20:41
nkinkademralex: Hmm.  I guess let's just see how things go without ImageMagick.20:42
JED3greg-g: thats actually the code for the deeds :)20:44
greg-goh right, because your humanizer just uses that? cool20:44
greg-gthankya much20:47
JED3thank you20:47
greg-gany time20:48
*** wormsxulla has joined #cc20:49
*** oshani has quit IRC20:57
*** oshani has joined #cc20:58
*** michi_ has quit IRC21:19
*** tantek has joined #cc21:21
*** sama_ has quit IRC21:55
*** oshani has quit IRC21:55
mralexnkinkade: um, it appears the source= querystring isn't overriding the default contribution source for OneClick22:11
mralexwhich i thought i had overridden temporarly in OneClickConfig.php, but apparently that file isn't used any more. :/22:12
nkinkademralex: Hmm.  You're right.  Let me check it.22:12
mralexi see it's defined in OneClick.class22:12
nkinkademralex: The only file used now is OneClick.class.php (and a small wrapper/router OneClick.php)22:13
nkinkademralex: Do you want me to fix it, or are you in there doing it?22:14
nkinkadeI see the issues.22:14
nkinkadeIf you're not already doing it, then I'll fix it now.22:14
mralexi'm peeking at the code, but you know where to go, so i'll let you handle it :)22:15
nkinkademralex: in __construct I was referencing $this->_request_data['source'], which didn't exist.22:16
nkinkadeShould have been $this->_request['source']22:16
nkinkadeI apparently change the name, but failed to update that one.22:16
mralexah, i see22:16
nkinkadeI'll go in and manually change the source for any contributions since ... ???22:16
nkinkadeWhat date?22:16
nkinkademralex: ^^22:17
mralexsince june 1 that haven't already been modified22:17
*** oshani has joined #cc23:00
JED3nyergler: ping23:02
nyerglerJED3, pong23:02
JED3nyergler: going to ask about how to handle restricting openid's in ccnet.php, but i'd rather do it next week23:04
nyerglerJED3, ok23:04
*** akozak has quit IRC23:13
*** tvol has quit IRC23:14
nyerglernkinkade, we were thinking hardware RAID for that server, right?23:28
nkinkadenyergler: We certainly can.23:28
nkinkadeI don't think it's critical, but it's always nice.23:29
nyerglerjust doing some poking at that23:29
nkinkadeI think it doesn't cost anything else from Dell on those servers to get hw RAID, or at least not much more.23:29
nyerglernkinkade, well it seems like there's some incremental cost on the one i was looking at, and they also want you to get all the drives in the configuration, too23:30
nkinkadenyergler: I think I had a decent box spec'ed a PowerEdge R410???23:30
paulproteusHardware RAID is usually worse than software RAID.23:31
nkinkadepaulproteus: In what sense?23:31
paulproteusMost "hardware raid" is what Linux calls "fakeraid" which iirc relies on a Windows driver to do the RAID.23:33
paulproteusAnd then there's the whole write-back cache thing that took down LiveJournal.23:33
paulproteusFor 36 hours or something.23:33
paulproteus"But unknown to us, the raid cards didn't disable the write caching on the drives themselves.... which is frickin' useless!"23:34
paulproteusRead that paragraph for more.23:34
paulproteusThe more incoherent caches you have, the more bad news you'll have in case of emergency.23:34
paulproteusLinux software RAID is smart about these things. I'd actually trust it.23:34
paulproteusHumorously mentions that they like "Battery-backed write-back caches."23:35
paulproteusSure, in 2004 they did.23:35
nyerglerpaulproteus, interesting; the adaptec story isn't that concerning to me... i'd be clear enough to get something reported to work with debian/ubuntu for hardware raid23:36
nyergler(having been around the host raid block myself a few times)23:36
* paulproteus nods and shrugs.23:36
nyerglerbut i am interested to hear that you are comfortable with software RAID23:36
paulproteusI wouldn't bother with hardware RAID. As far as I can tell, it buys you possible LiveJournal style insanity at the benefit of... nothing.23:36
nyergleri think the benefit is supposed to be that it offloads parity calculation from the CPU23:37
paulproteusYeah, Linux md has been through years and years of testing.23:37
paulproteusBoo frickin' hoo.23:37
paulproteusYou're going to use RAID1, right?23:37
* paulproteus rolls his eyes.23:37
paulproteusI mean, *fine*...23:37
paulproteusI would look seriously at RAID1.23:37
nyerglerfor simplicity reasons?23:38
paulproteusSimplicity and performance.23:38
nkinkadepaulproteus:  How then would you handle 4 disks, for example?23:38
paulproteusTwo RAID1s?23:38
nkinkadeRight with a 50% loss of actual capacity.23:38
nyergler"disks are cheap", paulproteus says23:38
paulproteusWith blindingly fast 2TB disks costing $200-400, is that really a problem?23:38
nyerglernkinkade, raid 1 would actually make our life slightly simpler23:39
paulproteusLast I saw, the CC servers used, like, 100GB of storage each or something.23:39
nkinkadepaulproteus:  It would just mean a more expensive server with an enclosure for at least 8 disks.23:39
nyerglerwrt to the Amz move23:39
paulproteusHuh? re: 8 disks23:39
paulproteusYou need to store 8TB?23:39
paulproteusPlus the in-case-of-emergency scenario is so easy with RAID1.23:39
nkinkadeWhy does RAID1 make life simpler over RAID5?23:39
paulproteus"Oh, the first disk doesn't boot? Okay, fall over to the second." Every BIOS has that built-in. Plus resilver is faster.23:39
nkinkadeIt's all handled by either the software driver or the hardware.23:39
nyerglernkinkade, i guess i wasn't convinced you could start an array from a disk with data on it (non-destructively)23:40
nyerglerif that's possible, nm my comment23:40
paulproteus(Honestly, eight disks?)23:40
nkinkadenyergler: Perhaps you can't.23:40
nkinkadepaulproteus: No, but maybe 3.23:41
nkinkadeIt would be 6 with a RAID1 setup.23:41
paulproteusHow many TB of storage do you need?23:41
nyerglernkinkade, what's our current backup usage look like?23:41
paulproteusDisk speed is usually the bottleneck on a server. That, and not enough RAM (and then swapping, God help you).23:41
nkinkade4T may not be enough in the long run.23:41
nkinkadenyergler: Order what makes most sense to you.23:41
nyerglernkinkade, gee, thanks :)23:41
paulproteusRAID5 with writes is tragic compared to RAID1.23:41
nyerglerpaulproteus, that is true (although "tragic" is probably a little strong ;) )23:42
paulproteusI don't mean to push you guys around, just wanted to share my thoughts. (-:23:42
nyerglerbtw, i learned about RAID 6 today23:42
nyerglerwho knew23:42
paulproteusOh God.23:42
nkinkadeThis is just a log storage and backup machine.23:42
nyerglerpaulproteus, i appreciate the input23:42
nkinkadeSpeed doesn't really matter.23:42
paulproteusnkinkade: Oh, that makes a huge difference! (-:23:43
paulproteusYeah, in that case, whatever. I almost wouldn't bother with RAID. RAID 5, by all means.23:43
nkinkadeReliability does matter, but I'm not hearing anyone say, "Oh, never, *ever* use RAID5 for backups, because your data will be lost surely."23:43
paulproteusThere was that bad summer I had when a RAID1 disk failed, and within 36h the other RAID1 disk failed.23:43
nyerglerRAID 6! ;)23:44
nkinkade4TB is enough for the near future.  But if we get a machine with only a 4 drive enclosure and 2 RAID1 pairs, then we have no room to grow.23:44
paulproteusI like 4-drive 1U machines.23:44
paulproteusA lot.23:45
paulproteusOr 12-drive 2U machines.23:45
paulproteusMan, I really want one of those.23:45
nkinkadepaulproteus: That is what I had in mind.23:45
paulproteusI have a 4-drive 1U. That's nice, but 12 disks... man.23:45
nkinkadeA 1U machine with 4 possible drives.23:45
* paulproteus nods.23:45
nyerglerpaulproteus, do you build your own machines or buy assembled w/o OS?23:45
nkinkadeMy main reason for thinking RAID5, was to maximize the space and storage capacity of that 1U machine.23:45
paulproteusYeah, RAID 5 for backup storage makes good sense. Don't let me stop you; performance doesn't matter.23:46
paulproteus(in the way it would for many other uses)23:46
paulproteus(i.e., your disks/filesystem will never be the bottleneck)23:46
nkinkadeWrite-back caches and battery backed RAID cards don't seem like an issue for us.23:46
paulproteusnyergler: I build my own, but if I were you I'd try to find a SuperMicro-based assembler that would assemble+warranty a whole machine.23:47
nkinkadeIt was simply a matter of making the most of space (physical space, not just disc capacity).23:47
paulproteusThe crazy awesome computer store on 4th st & Howard (iirc)...23:47
paulproteus...Central Computer, right...23:47 one of those.23:47
paulproteusBuilding servers is something I think I'm done with.23:47
nyerglerpaulproteus, yeah, i'm done with it, too23:47
nkinkadenyergler: Seriously, if you think RAID1 is the way to go, then it's all the same to me.  I think it'll mean paying more for a 2U server to be sure we have some growing room, but the budget is your concern, not mine.23:48
nyerglernkinkade, right23:49
nyerglerok, i'm not doing anything this afternoon, will return to this monday23:49
paulproteusNow that I understand, I'd go software RAID5. Sorry to push so hard then immediately about-face when I learned context. (-:23:49
paulproteusI guess hardware RAID5 is good because you can boot off a degraded array.23:50
paulproteus(i.e., you can put / on RAID5 that way)23:50
paulproteusIf you go that route, majorly majorly do your homework.23:51
paulproteusI feel like there's so many possible screw-ups with hardware RAID that I don't want to touch it.23:51
paulproteusBut there are surely people who know this stuff and could happily help.23:51
paulproteusSome of them might even hang out on #cernio (a Bay Area tech cooperative)23:51
paulproteusesp. since you're CC and all that (-:23:51
nyerglerok cool23:52
paulproteusP.S. I love that I can buy a toaster oven on Newegg.23:52
nyerglerwell i'm out for the day... ttyl  guys :)23:52
*** nyergler has quit IRC23:52
paulproteusI'll go back to my regular plan of listening to Rilo Kiley and cleaning house.23:52
paulproteusnkinkade: Sorry to barge in and sound like a know-it-all when I clearly didn't (-:23:52
*** JoiIto has quit IRC23:58

Generated by 2.6 by Marius Gedminas - find it at!