| *** johndoigiii has quit IRC | 00:01 | |
| *** jgay has quit IRC | 01:10 | |
| *** parker-fcnyu has joined #cc | 02:13 | |
| *** dini has joined #cc | 02:24 | |
| *** Danny_B has quit IRC | 03:00 | |
| *** Danny_B has joined #cc | 03:00 | |
| *** stevel has quit IRC | 03:13 | |
| *** parker-fcnyu has quit IRC | 03:14 | |
| *** dini has quit IRC | 03:25 | |
| *** Danny_B has quit IRC | 03:36 | |
| *** stevel has joined #cc | 03:38 | |
| *** Danny_B has joined #cc | 03:53 | |
| *** K`Tetch has quit IRC | 04:29 | |
| *** johndoigiii has joined #cc | 04:31 | |
| *** cchelpbot has joined #cc | 05:14 | |
| *** stevel has quit IRC | 05:28 | |
| *** stevel has joined #cc | 05:32 | |
| *** parker-fcnyu has joined #cc | 06:17 | |
| *** johndoigiii has quit IRC | 07:01 | |
| *** Roderick_ has quit IRC | 07:42 | |
| *** stevel has quit IRC | 08:06 | |
| *** Roderick_ has joined #cc | 09:21 | |
| *** haoyu_ has joined #cc | 10:19 | |
| *** haoyu has quit IRC | 10:20 | |
| *** haoyu_ is now known as haoyu | 10:25 | |
| *** stevel has joined #cc | 11:39 | |
| *** Orango has quit IRC | 11:44 | |
| *** haoyu has quit IRC | 12:15 | |
| *** [mharrison] has quit IRC | 12:15 | |
| *** paulproteus_ has joined #cc | 12:28 | |
| *** paulproteus has quit IRC | 12:28 | |
| *** haoyu has joined #cc | 12:29 | |
| *** kreynen has joined #cc | 13:01 | |
| *** kreynen has quit IRC | 13:37 | |
| *** parker-fcnyu has quit IRC | 13:44 | |
| *** michaelkrnac has joined #cc | 13:57 | |
| *** kreynen has joined #cc | 14:09 | |
| *** jgay has joined #cc | 14:34 | |
| *** parker-fcnyu has joined #cc | 14:47 | |
| *** nathany has joined #cc | 14:49 | |
| *** nathany has quit IRC | 14:50 | |
| *** nathany_ has joined #cc | 14:50 | |
| *** Peter__ has joined #CC | 14:54 | |
| *** Peter__ is now known as Danny | 14:54 | |
| Danny | What's needed to put a wiki under a creative commons license? Do all the contributors need to give permission? | 14:55 |
|---|---|---|
| nathany_ | Danny: typically, yes, unless some assignment of copyright has occured | 14:58 |
| nathany_ | many systems allow you to include language on the editing screen that states that by submitting to the wiki, the contributor agrees to have their content licensed | 14:59 |
| Danny | nathany_, wow, that's a pain. probably makes sense to start a new wiki starting with creative commons and phace out the old one | 14:59 |
| Danny | phase* | 14:59 |
| nathany_ | Danny: I mean, this isn't legal advice, so a lawyer might say differently | 14:59 |
| nathany_ | but yeah, you're right | 14:59 |
| Danny | damn | 14:59 |
| Danny | oh well | 14:59 |
| Danny | thanks! | 14:59 |
| nathany_ | does the existing one have a license? | 14:59 |
| Danny | none | 15:00 |
| *** nathany_ is now known as nathany | 15:00 | |
| nathany | ah | 15:00 |
| nathany | darn | 15:00 |
| Danny | yeah, oh well | 15:02 |
| Danny | thanks for the help | 15:02 |
| Danny | it might be better to start fresh anyways | 15:02 |
| *** michaelkrnac has quit IRC | 15:09 | |
| *** Danny has quit IRC | 15:12 | |
| *** lolo has joined #cc | 15:12 | |
| *** parker-fcnyu has quit IRC | 15:14 | |
| *** nkinkade has joined #cc | 15:15 | |
| *** parker-fcnyu has joined #cc | 15:28 | |
| *** K`Tetch has joined #cc | 15:32 | |
| *** tanjir_ has joined #cc | 15:34 | |
| *** tanjir has quit IRC | 15:34 | |
| *** [mharrison] has joined #cc | 15:36 | |
| *** johndoigiii has joined #cc | 16:07 | |
| *** parker-fcnyu has quit IRC | 16:22 | |
| *** Bovinity has joined #cc | 16:27 | |
| *** K`Tetch has quit IRC | 16:30 | |
| *** K`Tetch has joined #cc | 16:30 | |
| nkinkade | nathany: There was a slight misconfiguration of Nagios from a few days back that was causing us not to get SMS alert (only email alerts). I'm going to manually trigger a failure condition for forum.creativecommons.org, so you can ignore an SMS that comes in shortly for that. | 16:33 |
| nathany | nkinkade: got it | 16:34 |
| nkinkade | cc.engine was just down on a5 and we didn't get alerts. I noticed that paster was up to 2.3G of resident memory, so it seems that the watchdog script wasn't working right. I'm checking that too. | 16:35 |
| *** paulproteus_ is now known as paulproteus | 16:47 | |
| *** radiance29 has joined #cc | 17:11 | |
| *** radiance29 has left #cc | 17:11 | |
| *** lolo has left #cc | 17:35 | |
| *** kreynen_ has joined #cc | 17:39 | |
| *** kreynen has quit IRC | 17:55 | |
| *** johndoigiii is now known as johndoigiii-lunc | 18:39 | |
| *** johndoigiii-lunc is now known as johndoig-lunch | 18:40 | |
| nathany | nkinkade: i assume you didn't figure out the watchdog situation? | 18:44 |
| nathany | i just killed paster to get it responsive again | 18:44 |
| nathany | :( | 18:44 |
| nkinkade | nathany: I haven't checked it yet. | 18:45 |
| nathany | ok | 18:45 |
| nkinkade | I suspect it has to do with the way ps is reporting the size of the process. | 18:45 |
| nkinkade | Perhaps above a certain size it reports at N.NG instead of in KB?? | 18:45 |
| nathany | nkinkade: yes | 18:47 |
| nathany | it does | 18:47 |
| nkinkade | The watchdog currently kills paster when it get's to 1G. Maybe we should back that down to 800M or 900M. | 18:48 |
| nkinkade | nathany: I just set it to 750000KB, which is what it used to be before you made those changes to cc.engine. Let's see where that gets us. | 18:49 |
| nathany | ok | 18:49 |
| *** balleyne has joined #cc | 19:15 | |
| nathany | nkinkade: ping | 19:28 |
| nkinkade | nathany: Yeah. | 19:28 |
| nkinkade | a5 is having issues. | 19:28 |
| nathany | right | 19:28 |
| paulproteus | ps_mem.py is way more reliable for measuring RAM. How are you measuring? | 19:28 |
| paulproteus | Need a hand? | 19:28 |
| paulproteus | (more reliable than e.g. ps) | 19:28 |
| nathany | any idea what it is? | 19:28 |
| nathany | except that top, etc don't report things as being pegged | 19:29 |
| nkinkade | Not yet. Memory looks good and CPU looks good. | 19:29 |
| nkinkade | There are around 4000 network connections in the TIME_WAIT state. | 19:29 |
| nkinkade | That seems high. | 19:29 |
| paulproteus | That's free of cost, though, which is nice at least. | 19:29 |
| nathany | except that things queue up behind them, rgiht? | 19:29 |
| nkinkade | But overall thoughput is about normal at around 600K/s | 19:29 |
| nathany | paulproteus: suggestions? | 19:30 |
| paulproteus | What's the service time of an average HTTP request now? What indicates to you there is a problem at all? | 19:30 |
| nathany | (or nkinkade) | 19:31 |
| nathany | paulproteus: it's not responding :) | 19:31 |
| paulproteus | TCP queue, it looks like? | 19:31 |
| paulproteus | Right, I see that now. | 19:31 |
| paulproteus | Apache on 8080 is immediate | 19:31 |
| paulproteus | Varnish is not | 19:31 |
| nathany | weird | 19:31 |
| paulproteus | The varnish processes are spending a lot of time writing to disk | 19:31 |
| paulproteus | You can see that in iotop | 19:31 |
| paulproteus | Let me see where their disk i/o is going | 19:32 |
| paulproteus | It could be that Apache is fast because there's no kernel queue on port 8080 | 19:32 |
| nathany | (presumably to the cahce) | 19:32 |
| paulproteus | Maybe Varnish is handling all the connections it can; can we ask Varnish to increase that #? | 19:32 |
| paulproteus | Let me read the varnish conf docs | 19:32 |
| nathany | it seems to be abetting | 19:34 |
| nathany | (er, nevermind) | 19:34 |
| nkinkade | My browser seems to be hanging on getting data from google-analytics.com | 19:34 |
| paulproteus | That's a separate issue, since "time curl http://a5.creativecommons.org/" takes 4eva | 19:35 |
| nathany | right | 19:35 |
| paulproteus | (whereas :8080 in that curl returns instantaneously) | 19:35 |
| nathany | yea | 19:35 |
| paulproteus | Wow, so Varnish has about 1000 sockets open. | 19:35 |
| paulproteus | It's not hurting at all. | 19:36 |
| paulproteus | It's just got a bunch of connections doing nothing. | 19:36 |
| nkinkade | I restarted Varnish a minute ago, and that didn't seem to do anything. | 19:36 |
| paulproteus | I really want to see if we can just ask Varnish to accept more simultaneous connections. | 19:36 |
| paulproteus | The other thing I'm interested in doing is just proxying all HTTP straight to Apache "for now" | 19:37 |
| nkinkade | paulproteus: Give that a shot. I suspect that Apache will then become overwhelmed. | 19:38 |
| nkinkade | But it's worth a try. | 19:38 |
| nkinkade | I guess I could put in a Varnish rule that just passes everything to Apache. | 19:38 |
| paulproteus | But I don't think that'd help; Varnish would still queue the connections. | 19:40 |
| paulproteus | I used a "simple TCP proxy" and now Apache receives the connections (indirectly) | 19:40 |
| nkinkade | For me, requests to :8080 are hanging too. | 19:40 |
| paulproteus | Yeah, I bet this just overloaded Apache in an instant. (-: | 19:41 |
| paulproteus | Reverting | 19:41 |
| paulproteus | (varnish restarted; creativecommons.org loads quickly) | 19:41 |
| paulproteus | (...and now not quickly) | 19:41 |
| paulproteus | :8080 is quick now htough | 19:41 |
| paulproteus | * -p listen_depth=4096 (default 1024) | 19:43 |
| paulproteus | That's what we ought to change | 19:43 |
| paulproteus | at http://varnish.projects.linpro.no/wiki/Performance | 19:43 |
| paulproteus | one sec, let me see how to | 19:43 |
| nathany | i saw that... looks like a CLI option? | 19:43 |
| paulproteus | Ya, added and restarted | 19:44 |
| paulproteus | Responsive now | 19:44 |
| paulproteus | Still responsive | 19:44 |
| paulproteus | (kinda?) | 19:45 |
| paulproteus | Not really anymore... | 19:45 |
| nkinkade | It's quick again. | 19:46 |
| nkinkade | I might have slowed because I restarted Apache. | 19:46 |
| paulproteus | Okay, yeah | 19:46 |
| paulproteus | Why did you restart Apache? | 19:46 |
| nkinkade | What we were seeing looked vaguely like a problem Varnish used to have with KeepAlives in Apache, so I had disabled them to see. | 19:46 |
| paulproteus | Oh yeah | 19:46 |
| paulproteus | No longer fast for me, fwiw | 19:46 |
| paulproteus | :8080 is fast still | 19:47 |
| paulproteus | We need more sockets in Varnish, I wonder? | 19:47 |
| paulproteus | let me read more | 19:47 |
| nkinkade | No here either. | 19:47 |
| nathany | it seems like it's slow, but struggling along better than it was previously | 19:48 |
| paulproteus | 3s response time now, which is bearable | 19:48 |
| paulproteus | (as measured with time curl http://a5.creativecommons.org/ ) | 19:49 |
| paulproteus | time curl http://creativecommons.org/ # still waiting, >15s, wonder why | 19:49 |
| paulproteus | The TCP sockets still aren't being picked up by Varnish quickly enough. | 19:50 |
| nathany | is it getting busy waiting for connections to return from zope/apache? | 19:50 |
| nathany | (no idea, speculation) | 19:50 |
| paulproteus | increasing thread pool count | 19:51 |
| paulproteus | So one question is, are our connections mostly coming from one source? If so, can we attempt to degrade service for that source? | 19:53 |
| * paulproteus checks | 19:53 | |
| paulproteus | Basically, no | 19:55 |
| nathany | paulproteus: is iptraf not helpful here? | 19:55 |
| nathany | it seems to suggest lots of traffic from one ip | 19:55 |
| paulproteus | I can't listen if you are, with it | 19:56 |
| nathany | paulproteus: i'm out | 19:56 |
| paulproteus | I was using netstat -a -n # list of ips, then uniq -c to generate the histogram | 19:56 |
| nkinkade | paulproteus: I checked that already. | 19:56 |
| nkinkade | It's seems like a heterogeneous pool of IPs. | 19:56 |
| paulproteus | The only really fast growing byte count is my iptraf session over SSH | 19:57 |
| nathany | ah | 19:57 |
| paulproteus | Which is consistent with poor service for everyone | 19:57 |
| nathany | nevermind :) | 19:57 |
| nathany | so is this just a spike in service that we haven't seen before? | 19:57 |
| nkinkade | Interestingly varnishhist seems to be reporting mostly normal graphs. | 19:57 |
| paulproteus | nkinkade, The problem is what varnish doesn't see | 19:58 |
| nkinkade | That makes sense. | 19:58 |
| paulproteus | It's letting connections sit un-handled in the kernel queue | 19:58 |
| nkinkade | But the numbers from varnishstat don't look out of place either. | 19:58 |
| nkinkade | paulproteus: You can run /home/nkinkade/bin/conn_stat.sh for some stats based on netstat. | 19:59 |
| nathany | nkinkade: they wouldn't; it appears the problem is how quickly varnish is picking up connections (if i understand correctly) | 19:59 |
| nathany | (which i'm not sure i do) | 19:59 |
| paulproteus | nathany, That's what I'm saying, at least; if it's correct is a separate issue (-; | 19:59 |
| paulproteus | But I think it is | 19:59 |
| nkinkade | Things look busy, but they don't seem that much busier than what I've seen in the past. | 19:59 |
| paulproteus | I just enabled syncookies. | 20:01 |
| nathany | wow | 20:01 |
| paulproteus | Suddenly it's all fast. | 20:01 |
| nathany | fast now | 20:01 |
| nathany | wtf? | 20:01 |
| paulproteus | I have no idea if these are related. | 20:01 |
| paulproteus | I should dump the traffic and see if it's really a SYN flood | 20:02 |
| nathany | paulproteus: sounds good | 20:02 |
| paulproteus | (still fast) | 20:03 |
| nkinkade | Nice. | 20:03 |
| nkinkade | If that's the case, could it be that netstat wouldn't even list all the SYN requests? | 20:03 |
| paulproteus | I think that's right, yeah | 20:03 |
| paulproteus | It could be that bogus connections were preventing the kernel from handling the real ones | 20:03 |
| nkinkade | Ah. | 20:03 |
| paulproteus | So they were in an even earlier queue | 20:03 |
| nathany | hrm | 20:04 |
| nkinkade | Still quick for me. | 20:04 |
| paulproteus | I have 50M of packets; I guess that's good enough for me. | 20:05 |
| paulproteus | ca. 640K packets | 20:06 |
| paulproteus | stopped dumping | 20:06 |
| paulproteus | That was fun! | 20:07 |
| paulproteus | It's hard to say which Varnish options I added were/are necessary. | 20:07 |
| nkinkade | paulproteus: Which did you add? | 20:07 |
| paulproteus | Now that we know that syncookies fixed it, I wonder if increasing the size of a kernel queue could have also fixed it. | 20:08 |
| paulproteus | But then again an attacker (if it was an attack) could just double/etc the number of SYN packets he sends. | 20:08 |
| paulproteus | -p listen_depth=4096 \ | 20:08 |
| paulproteus | -p thread_pools=4 \ | 20:08 |
| paulproteus | -p thread_pool_max=4000 \ | 20:08 |
| paulproteus | to /etc/default/varnish | 20:08 |
| nkinkade | If the queue you are talking about is before the queue that netstat works with, I wonder if concurrent connection limiting would have done anything. | 20:10 |
| nkinkade | paulproteus: http://varnish.projects.linpro.no/wiki/Performance | 20:11 |
| nkinkade | Is that what you saw earlier? | 20:11 |
| paulproteus | Yes, that's what I read. | 20:11 |
| paulproteus | What are you saying about concurrent connection limiting? | 20:12 |
| paulproteus | Through all of this, it's interesting that port 22 and 8080 were super responsive. | 20:14 |
| nkinkade | One of the things I had been looking forward to with the upgrade to Lenny was the ability to limit concurrent connections on a per IP basis ... just to avoid one address slamming the machine. | 20:15 |
| nkinkade | Like you were able to do with netcat one time. | 20:15 |
| paulproteus | How can you do that limit? Is it a netfilter feature? | 20:15 |
| * paulproteus nods | 20:15 | |
| nkinkade | paulproteus: Yeah, it's a netfilter feature ... rather it's some Netfilter module. | 20:15 |
| nkinkade | ip_connlimit, or something like that. | 20:16 |
| nkinkade | I can't remember now. | 20:16 |
| nkinkade | But if turning syncookies on fixed this, then I imagine that it was at a level that the kernel could identify the IP address and perhaps could have taken action, like just dropping it. | 20:16 |
| nkinkade | I'm not sure about that, though. | 20:17 |
| nkinkade | And it certainly does nothing much for any distributed flood of traffic. | 20:17 |
| paulproteus | Yeah | 20:17 |
| nkinkade | But I was also confused as to why SSH and everything else was so responsive during that whole ordeal. | 20:17 |
| paulproteus | We'll see when we look at that dump (/root/2009-04-21-packets-for-a-minute-or-two.tcpdump) I guess. | 20:17 |
| paulproteus | Okay, I'm going to go have lunch, and then head to the office. | 20:23 |
| *** parker-fcnyu has joined #cc | 20:30 | |
| *** johndoig-lunch is now known as johndoigiii | 20:35 | |
| *** michaelkrnac has joined #cc | 20:39 | |
| *** michaelkrnac has quit IRC | 20:51 | |
| *** Orango has joined #cc | 21:00 | |
| *** Orango has joined #cc | 21:01 | |
| *** Orango has quit IRC | 21:10 | |
| *** Orango has joined #cc | 21:13 | |
| *** parker-fcnyu has quit IRC | 21:24 | |
| *** kreynen_ has quit IRC | 21:30 | |
| *** kreynen has joined #cc | 21:34 | |
| *** [mharrison] has quit IRC | 21:47 | |
| *** tanjir__ has joined #cc | 21:48 | |
| *** balleyne has quit IRC | 21:49 | |
| nathany | nkinkade: you around? | 21:50 |
| johndoigiii | nathany: http://dpaste.com/36462/ are the wrapping span's necessary? | 21:50 |
| nkinkade | nathany: Yeah. | 21:51 |
| nathany | johndoigiii: i wouldn't think so; couldn't you just put style="display:none" on the span with the tal:omit-tag=""? | 21:51 |
| nathany | (we can probably remove the tal:omit-tag anyway since it's disregarded) | 21:51 |
| nathany | nkinkade: i'm looking @ the planet bug (finally/again) | 21:51 |
| nkinkade | Ah. Thanks. | 21:51 |
| nkinkade | I feel like I should have been investigating that myself a bit. | 21:52 |
| nathany | question re: the plugin... why do we look for an id that matches "license_url" on the parent? | 21:52 |
| nathany | (it's ok, pretty low priority) | 21:52 |
| nkinkade | My assessment that it was rdfadict was largely based on the word RDFa showing up somewhere in the backtrace. | 21:52 |
| nathany | (my question above has zero to do with the bug, it just seems weird | 21:52 |
| nkinkade | I'll have to look at the plugin again to answer that. | 21:53 |
| nkinkade | It's been a quite some months since I touched it, maybe as much as a year. | 21:53 |
| nathany | no problem | 21:53 |
| nkinkade | nathany: Do you mean on line 107? | 21:55 |
| nathany | yes | 21:55 |
| *** parker-fcnyu has joined #cc | 21:56 | |
| nkinkade | Hmm. I'm not really sure anymore. At this point I can't really say more than my note indicates. | 21:57 |
| nkinkade | I'm not sure how I determined before that THE license URL would always be wrapped in a <span> tag with id=license_url | 21:59 |
| nathany | lol | 22:00 |
| nathany | yeah, me either :) | 22:00 |
| nkinkade | haha. | 22:00 |
| nkinkade | Ah, now I see. See ./software/themes/cc/index.html.tmpl | 22:00 |
| nkinkade | Look at line 109. | 22:01 |
| nathany | oh, the filter gets the rendered page, not the source | 22:01 |
| nkinkade | Yeah. | 22:01 |
| nathany | that makes [more] sense now | 22:01 |
| nkinkade | In this case it does. | 22:01 |
| nathany | nkinkade: anyway, i fixed the other issue but not sure if i should commit to trunk, production, ...? | 22:02 |
| nathany | so i reassigned to you for committing :) | 22:02 |
| nkinkade | There are two ways Planet Venus filters can work. 1) By getting each entry as it's parsed 2) By getting the final rendered page. | 22:02 |
| nathany | got it | 22:02 |
| nkinkade | Where was the other issue, by the way? | 22:02 |
| nkinkade | Are you working on a local checkout? | 22:04 |
| nathany | nkinkade: nope, i fixed it on a8 | 22:04 |
| nathany | it's in the get_license_name filter | 22:04 |
| nkinkade | Fuck. | 22:04 |
| nkinkade | Sorry. | 22:04 |
| nathany | nkinkade: no problem | 22:04 |
| nkinkade | And there I went blaming it on poor rdfadict. :-) | 22:05 |
| nathany | btw, where does the code live that pulls out the license URL? | 22:05 |
| nathany | i assume it's in a different filter/plugin? | 22:05 |
| nathany | we need to do something similar there to fix http://code.creativecommons.org/issues/issue284 | 22:05 |
| nkinkade | The code that pulls out the license URL lives somewhere in the planet code. | 22:05 |
| nkinkade | I had to alter the core code to make the plugin work. | 22:05 |
| nathany | ah | 22:05 |
| nkinkade | I think the changes are visible in svn: | 22:05 |
| nkinkade | http://code.creativecommons.org/viewsvn/planet/ | 22:06 |
| nathany | awesome | 22:06 |
| nkinkade | http://code.creativecommons.org/viewsvn/planet/trunk/software/planet/shell/tmpl.py?r1=8298&r2=10188 | 22:07 |
| nkinkade | Somewhere around there. | 22:07 |
| nkinkade | As far as your change, the best would be to checkout planet, make the change in trunk, then svnmerge it to production, then svn up ... the usual. | 22:08 |
| nkinkade | paulproteus: What's the status with newer versions of svn and svnmerge? | 22:08 |
| nkinkade | Didn't you say that lately svnmerge was basically part of subversion? | 22:08 |
| *** tanjir_ has quit IRC | 22:10 | |
| nathany | nkinkade: it sort of is, but it's implemented in a way that's incompatible with the svnmerge we've been using | 22:10 |
| nathany | you can get it to upgrade but it means upgrading the server + everyone's client | 22:10 |
| *** haoyu has quit IRC | 22:10 | |
| nathany | (although most people's clients are probably already updated) | 22:10 |
| *** haoyu has joined #cc | 22:10 | |
| * paulproteus nods in nkinkade's and nathany's general direction | 22:12 | |
| johndoigiii | nathany, if you add anything to the attributes of the span tag it will it in with the ${i18n_name_of_item} and thus break the markup | 22:13 |
| nathany | johndoigiii: that's incredibly depressing | 22:14 |
| nathany | nkinkade: if I kill the license cache planet will recreate it, right? | 22:46 |
| nathany | (cache/license_mappings/license.maps) | 22:47 |
| nkinkade | nathany: Yeah, it'll get recreated. | 22:52 |
| nkinkade | It should be safe to do something like rm -rf cache/* | 22:52 |
| nathany | nkinkade: thanks, i just assumed that and tried it :) | 22:52 |
| nkinkade | :-) | 22:52 |
| nkinkade | nathany: For those Deeds that you need generated. Is that all license versions, or just 3.0? | 22:55 |
| nathany | nkinkade: all versions | 22:55 |
| nkinkade | nathany: If -v isn't specified for mkdeeds, will it generate them all, or is there some way to tell it to do all without using a comma separated list? | 22:56 |
| nathany | nkinkade: iirc you can omit it and it'll do them all | 22:56 |
| nkinkade | I'll give it a go. | 22:56 |
| nathany | it always starts with a list of every license and then prunes it based on the command line | 22:56 |
| nathany | (iirc) | 22:57 |
| *** kreynen_ has joined #cc | 23:03 | |
| *** nathany has quit IRC | 23:07 | |
| *** kreynen has quit IRC | 23:20 | |
| *** kreynen_ has quit IRC | 23:22 | |
| *** at1z0r has joined #cc | 23:24 | |
| *** at1z0r has left #cc | 23:24 | |
| *** nathany has joined #cc | 23:46 | |
| nkinkade | paulproteus: Do you happen to be logged into a5? | 23:47 |
| nathany | paulproteus: nkinkade: did we disable syncookies? | 23:47 |
| nkinkade | nathany: I haven't touched a5 since the problem earlier. | 23:47 |
| nkinkade | nathany: Looks like paster got up to 2.4G again. | 23:49 |
| nkinkade | I'm restarting it. I'm not sure why the watchdog script suddenly stopped working. | 23:49 |
| paulproteus | nkinkade, I do not | 23:49 |
| paulproteus | 18:49:39 up 8 days, 6:31, 3 users, load average: 49.93, 107.61, 62.99 | 23:49 |
| paulproteus | Sweet | 23:49 |
| nathany | nkinkade: we should figure out what's up with that | 23:50 |
| nathany | sooner rather than later | 23:50 |
| nkinkade | paulproteus: Do you have a second to take a look at the watchdog script? | 23:50 |
| paulproteus | Which app is paster, and can't we just move it to cycling processes via FCGI? Sorry that I don't know much about this deployment so will sound clueless. )-: | 23:50 |
| paulproteus | nkinkade, You got it, tell me where it is | 23:50 |
| nkinkade | I'm just about to head out the door ... a friend is on the way to pick me up in about 5 minutes. | 23:50 |
| nathany | for fuck's sake | 23:50 |
| nathany | (sorry) | 23:50 |
| nathany | nkinkade: where does the watchdog live? | 23:50 |
| nkinkade | a5:/root/bin/check_webservices.sh | 23:51 |
| paulproteus | I think we can handle it, nathany | 23:51 |
| paulproteus | Wow, not a trivial script. (-: | 23:51 |
| nathany | (the FFS wasn't directed at you, nkinkade -- of course it's end of day for you :) ) | 23:51 |
| nkinkade | paulproteus: You should be able to see at the top where I set how much memory is the max for paster. | 23:51 |
| paulproteus | nkinkade, Ya | 23:51 |
| nkinkade | It's nearly the end of the day for SF, too. | 23:52 |
| paulproteus | "Luckily" my day got started late | 23:52 |
| nathany | paulproteus: as discussed before, Zope isn't amenable to fcgi handling or we may have moved to that previously | 23:53 |
| nathany | heh | 23:53 |
| paulproteus | I can chat with NK about it, nathany, if you need to go | 23:53 |
| nathany | i'll hang out a bit | 23:53 |
| nathany | NK has to run, i believe | 23:53 |
| paulproteus | Oh oops, I read the wrong nathan in the 5min message | 23:53 |
| nkinkade | paulproteus: I've got to run, but as long as cc.engine can stay going for about 2 hours, then I can look more into when I get back after dinner. | 23:53 |
| nathany | nkinkade: thanks... paulproteus will send you an email and let you know if we do anything :) | 23:54 |
| paulproteus | ACK | 23:54 |
| nkinkade | The script had been working just fine for months and months. | 23:54 |
| nkinkade | Suddenly today it seems to not be catching paster (cc.engine) before it's memory usage spirals out of control. | 23:54 |
| nathany | paulproteus: yes, complex, but i think the part we need to care about is the first 10-15 lines | 23:54 |
| nathany | (the part that looks @ paster) | 23:54 |
| * paulproteus installs emacs | 23:54 | |
| nkinkade | Unless of course it's because the memory usage grows from acceptable to distastrous in less than 2 minutes. | 23:54 |
| paulproteus | ne'er mind it's already there | 23:55 |
| nathany | i suppose that's possible but seems unlikely | 23:55 |
| nkinkade | Okay. I'll check back on #cc when I get back, and maybe send an email me if you find something. | 23:55 |
| paulproteus | + '[' 7604 106540 -gt 750000 ']' | 23:55 |
| paulproteus | /root/bin/check_webservices.sh: line 22: [: too many arguments | 23:55 |
| nkinkade | Doh! | 23:55 |
| nathany | LOL | 23:56 |
| nkinkade | Okay. I'll check back in a bit later. | 23:56 |
| paulproteus | I'm looking into why that is... | 23:56 |
| * paulproteus is running with bash -x | 23:56 | |
| nkinkade | Thanks paulproteus. sorry about the bum script. | 23:56 |
| paulproteus | Heh, that's life. (-: | 23:56 |
| * paulproteus eats some chocolate chips and carries on | 23:56 | |
| nathany | paulproteus: so it had an incorrect #! or you're still looking @ it? | 23:59 |
| paulproteus | I'm still looking | 23:59 |
Generated by irclog2html.py 2.6 by Marius Gedminas - find it at mg.pov.lt!