| *** tomkap has quit IRC | 00:07 | |
| *** tomkap has joined #cc | 00:08 | |
| *** nevyn has quit IRC | 00:28 | |
| *** tomkap has quit IRC | 00:33 | |
| *** oshani has joined #cc | 00:44 | |
| *** MarkDude has quit IRC | 00:49 | |
| *** akila87 has joined #cc | 01:03 | |
| *** akila871 has joined #cc | 01:27 | |
| *** akila87 has quit IRC | 01:29 | |
| *** Doom2pro has joined #cc | 01:34 | |
| *** Doom2pro has left #cc | 01:34 | |
| *** bassel has quit IRC | 01:47 | |
| *** pmiller has joined #cc | 01:57 | |
| *** MarkDude has joined #cc | 02:13 | |
| *** oshani has quit IRC | 02:13 | |
| *** nkinkade has quit IRC | 02:23 | |
| *** pyrak has quit IRC | 02:38 | |
| *** pyrak has joined #cc | 02:39 | |
| *** pmiller has left #cc | 02:53 | |
| *** mralex has quit IRC | 03:35 | |
| *** akila871 has quit IRC | 03:53 | |
| *** MarkDude has quit IRC | 04:25 | |
| *** akila87 has joined #cc | 04:33 | |
| *** loprox has joined #cc | 04:34 | |
| *** loprox has left #cc | 04:34 | |
| *** mralex has joined #cc | 04:53 | |
| *** nevyn has joined #cc | 04:56 | |
| *** mralex has quit IRC | 04:57 | |
| *** pmiller has joined #cc | 05:20 | |
| *** pmiller has quit IRC | 05:23 | |
| *** pmiller has joined #cc | 05:25 | |
| *** MarkDude has joined #cc | 05:28 | |
| *** MarkDude has quit IRC | 05:31 | |
| *** JoiIto has joined #cc | 06:21 | |
| *** JoiIto has left #cc | 06:21 | |
| *** tomkap has joined #cc | 06:42 | |
| *** pmiller has left #cc | 06:50 | |
| *** bassel has joined #cc | 07:03 | |
| *** akila87 has left #cc | 07:22 | |
| *** thvdburgt has joined #cc | 07:45 | |
| *** thvdburgt has left #cc | 07:45 | |
| *** wormsxulla has quit IRC | 08:18 | |
| *** cchelpbot has joined #cc | 08:23 | |
| *** pmiller has joined #cc | 08:31 | |
| *** wormsxulla has joined #cc | 08:33 | |
| *** nevyn has quit IRC | 08:56 | |
| *** nevyn has joined #cc | 08:59 | |
| *** akila87 has joined #cc | 09:37 | |
| *** bassel has quit IRC | 09:52 | |
| *** phurl has joined #cc | 10:06 | |
| *** pmiller has left #cc | 10:10 | |
| *** phurl has quit IRC | 10:11 | |
| *** erlehmann has joined #cc | 10:20 | |
| *** wormsxulla has quit IRC | 10:34 | |
| *** wormsxulla has joined #cc | 10:46 | |
| *** bassel has joined #cc | 10:56 | |
| *** Sudipta has joined #cc | 11:04 | |
| *** phurl has joined #cc | 11:16 | |
| *** akila87 has quit IRC | 11:38 | |
| *** erlehmann has quit IRC | 11:59 | |
| *** anttil_ has quit IRC | 12:06 | |
| *** anttil has joined #cc | 12:12 | |
| *** BjornW has joined #cc | 12:14 | |
| *** oshani has joined #cc | 12:25 | |
| *** thisisparker has joined #cc | 12:39 | |
| *** bassel_ has joined #cc | 12:45 | |
| *** bassel has quit IRC | 12:49 | |
| *** bassel_ is now known as help | 12:49 | |
| *** help is now known as bassel | 12:49 | |
| *** oshani has quit IRC | 12:56 | |
| *** Sudipta has quit IRC | 13:24 | |
| *** iTune has joined #cc | 13:42 | |
| *** oshani has joined #cc | 14:02 | |
| *** papy|away is now known as papyromancer | 14:10 | |
| *** wormsxulla has quit IRC | 14:19 | |
| *** wormsxulla has joined #cc | 14:36 | |
| *** akila87 has joined #cc | 14:40 | |
| *** davis has joined #cc | 14:47 | |
| davis | hello | 14:47 |
|---|---|---|
| paroneayea | hello | 14:47 |
| * greg-g waves | 14:47 | |
| *** phurl has quit IRC | 14:48 | |
| davis | I am using System V semaphores on a 2.4 kernel. I am wondering if there is a way to free a semaphore which has been created by a nonexistant program. (I.e. the process which did semget() crashed.) I've tried semctl() with a IPC_RMID but it fails. semop() with IPC_NOWAIT does not help either. | 14:50 |
| *** iTune has left #cc | 14:50 | |
| paroneayea | davis: this isn't the C Compiler channel, sorry! | 14:50 |
| paroneayea | it's the Creative Commons channel :) | 14:50 |
| davis | lol | 14:51 |
| davis | i thought it was ##C | 14:51 |
| davis | sorry my bad | 14:51 |
| paroneayea | :) | 14:51 |
| paroneayea | np! | 14:51 |
| davis | best of luck | 14:51 |
| *** davis has left #cc | 14:51 | |
| *** akila871 has joined #cc | 14:57 | |
| *** akila871 has joined #cc | 14:58 | |
| *** akila87 has quit IRC | 14:59 | |
| *** erlehmann has joined #cc | 15:03 | |
| *** nkinkade has joined #cc | 15:20 | |
| *** akila871 has left #cc | 15:34 | |
| *** oshani has quit IRC | 15:58 | |
| *** thisisparker has quit IRC | 16:06 | |
| *** thisisparker has joined #cc | 16:12 | |
| *** haoyu has joined #cc | 16:20 | |
| *** coolingfan has joined #cc | 16:21 | |
| *** coolingfan has left #cc | 16:22 | |
| *** erlehmann has quit IRC | 16:23 | |
| *** akozak has joined #cc | 16:29 | |
| *** oshani has joined #cc | 16:50 | |
| *** JED3 has quit IRC | 16:50 | |
| *** mralex has joined #cc | 16:51 | |
| *** haoyu has quit IRC | 17:23 | |
| *** bassel has quit IRC | 17:23 | |
| *** bassel has joined #cc | 17:24 | |
| *** nyergler has joined #cc | 17:36 | |
| *** nyergler has joined #cc | 17:36 | |
| akozak | interesting: http://weblog.muledesign.com/2010/06/unsuck_it_special_byebye_my.php | 17:39 |
| *** JED3 has joined #cc | 17:41 | |
| JED3 | hey nkinkade, do we have nagios configured to monitor i.cc.org? | 17:45 |
| nkinkade | JED3: Yeah. | 17:45 |
| JED3 | i.dot was just down, but i didnt see any nagios messages | 17:45 |
| nkinkade | JED3: How long was it down for? | 17:46 |
| nkinkade | Do you get emails for all Nagios alerts? | 17:46 |
| JED3 | i'm not sure | 17:46 |
| JED3 | nkinkade: yes | 17:46 |
| JED3 | at least, I assumed that i do | 17:47 |
| nkinkade | JED3: I see that Nagios is still configured to query i.CC.org on a8 ... the only reason it's still working is because Varnish is still forwarding requests to the backup machine. | 17:47 |
| JED3 | ahh, does nagios follow redirects to check status? | 17:48 |
| nyergler | varnish doesn't redirect -- it makes a proxy request on your behalf | 17:49 |
| nyergler | transparent to nagios/users | 17:49 |
| JED3 | at any measure, i'm looking around the logs on i.dot and i'm not seeing any evidence as to why apache when down | 17:50 |
| JED3 | went* | 17:50 |
| nkinkade | JED3: Varnish doesn't actually redirect, but just fetches the content from a remote backend. The client has no idea. | 17:52 |
| nkinkade | Oh, I didn't read nyergler reply. Sorry. | 17:52 |
| nkinkade | JED3: So absolutely Apache went down on backup.creativecommons.org, or you just noticed i.CC.org problems for a minute? | 17:53 |
| JED3 | i noticed it was down, saw a varnish error response, and then promptly restarted apache | 17:54 |
| *** thisisparker has quit IRC | 17:55 | |
| JED3 | it took me about 5 minutes of noticing dead badges on different pages until i ssh'ed in to restart | 17:56 |
| *** nyergler has quit IRC | 18:06 | |
| *** papyromancer has quit IRC | 18:37 | |
| *** papyromancer has joined #cc | 18:37 | |
| nkinkade | Hmm. a5 has over 10,000 unique hosts connected to it at this moment. | 18:37 |
| paroneayea | that's over 9000! | 18:38 |
| paroneayea | (sorry) | 18:38 |
| nkinkade | Maybe I missed the reference. | 18:38 |
| akozak | heh | 18:39 |
| akozak | what you say | 18:39 |
| paroneayea | nkinkade: you missed the reference, but you didn't miss much :) | 18:39 |
| nkinkade | Keep in mind, that is unique *hosts*, not unique connections. There are a lot more connections. | 18:40 |
| nkinkade | a5 is consequently dragging a bit. | 18:40 |
| nkinkade | Still working, but just a bit slowly. | 18:40 |
| mralex | paroneayea: NINE THOOOUUUUSSSAAAAND | 18:45 |
| mralex | JED3: weird, you keep doing something on zupport that nukes the URL path settings of some pages | 18:48 |
| paroneayea | mralex: What nine thousand? | 18:48 |
| paroneayea | ! | 18:48 |
| JED3 | mralex: yes i've had to reset them a couple times now | 18:53 |
| *** BjornW has quit IRC | 19:03 | |
| *** papyromancer is now known as papy|food | 19:16 | |
| paroneayea | nkinkade: man, no kidding, a5 is slow | 19:32 |
| paroneayea | almost a bit nervous to restart the engine with the new translations | 19:33 |
| * paroneayea tests building out on staging first | 19:34 | |
| paroneayea | I'm always nervous to run ./bin/buildout on the live server lest it fail halfway through and make the python environment completely fucked | 19:34 |
| nkinkade | paroneayea: How long does buildout take? | 19:36 |
| nkinkade | Is it at all disk intensive? | 19:36 |
| nkinkade | If so, you should always run with "sudo ionice -c3 <command>" | 19:36 |
| *** balleyne has joined #cc | 19:36 | |
| paroneayea | nkinkade: it's not too disk intensive, but it does take as long as 5-10 minutes | 19:36 |
| paroneayea | because it queries a bunch of servers to check for new packages | 19:37 |
| paroneayea | and sometimes it rebuilds lxml or something. | 19:37 |
| *** phurl has joined #cc | 19:38 | |
| paroneayea | it's faster if you do -N but I need it to fetch new packages | 19:38 |
| paroneayea | on that note, I didn't actually know about ionice, I just knew about nice | 19:39 |
| paroneayea | I like how you can also read it as Ion Ice | 19:39 |
| paroneayea | which sounds very futurist | 19:39 |
| *** balleyne has quit IRC | 19:41 | |
| akozak | or I on Ice | 19:41 |
| mralex | something something Ice something something Neuromancer something something hack the Gibson | 19:41 |
| *** balleyne has joined #cc | 19:41 | |
| akozak | haha | 19:41 |
| paroneayea | mralex: win | 19:41 |
| paroneayea | nkinkade: if the server is under so much strain atm though, maybe I should avoid wiping the deed cache? | 19:44 |
| nkinkade | paroneayea: It appears to be a little better now. | 19:44 |
| nkinkade | paroneayea: All our servers are very sensitive to heavy disk I/O, so any time you do anything that might load it up that will run for more than a minute or two should probably be run using ionice. | 19:46 |
| nkinkade | For example, running the following on any of our busier servers would surely cause all services to fail in a short time: | 19:47 |
| paroneayea | whew | 19:47 |
| paroneayea | engine back up | 19:47 |
| nkinkade | $ dd if=/dev/zero of=big.bin bs=1G count=25 | 19:47 |
| paroneayea | normal restart time | 19:47 |
| paroneayea | always makes me nervous anyway | 19:47 |
| paroneayea | nkinkade: that sure is a big.bin | 19:48 |
| nkinkade | paroneayea: On occasion I have to create one that is 50G, for Varnish. | 19:48 |
| nkinkade | Running *that* command without ionice would be about equivalent to pulling the power cable from the server. | 19:49 |
| paroneayea | ;x | 19:50 |
| akozak | i dont understand why the os doesnt stop that from happening | 19:53 |
| nkinkade | paroneayea: Did you say you were going to hold off on resetting the cache? | 19:55 |
| nkinkade | How about just deleting all Greek Deeds? | 19:55 |
| paroneayea | nkinkade: I could do that | 19:55 |
| nkinkade | find all deed.el an delete them. | 19:55 |
| nkinkade | and of course all files like deed inside of the gr/ dirs | 19:55 |
| nkinkade | akozak: The OS doesn't know which process is most important unless you tell it so. | 19:56 |
| akozak | hm | 19:56 |
| akozak | but couldnt there be some way to avoid one process hogging all the resources | 19:57 |
| paroneayea | nkinkade: done! | 19:57 |
| akozak | unless you allow it | 19:57 |
| paroneayea | akozak: you can set default nice levels iirc | 19:57 |
| nkinkade | akozak: The OS doesn't allow that. | 19:57 |
| paroneayea | oh | 19:57 |
| paroneayea | maybe I'm wrong :) | 19:58 |
| nkinkade | The OS shares the resources among processes, but some processes are very sensitive to delays in I/O operations. | 19:58 |
| nkinkade | If Varnish/Apache can't get requests out the door immediately, then requests start to stack up, the performance goes down and it all snowballs until the machine grinds to a halt., | 19:59 |
| akozak | right, i guess my question is why the os wouldnt try to mitigate that or something. i also don't really know what im talking about. | 19:59 |
| akozak | ah i see | 19:59 |
| akozak | that makes sense | 19:59 |
| nkinkade | Hence the existence of nice, renice and ionice. | 19:59 |
| paroneayea | I'm pretty sure I got the IonIce upgrade to my blaster in Metroid at some point. | 20:01 |
| nkinkade | I'm not expert on all this either, but I mostly understand what is happening a relatively high level. | 20:01 |
| nkinkade | paroneayea: I see you just deleted Greek deeds, no? | 20:01 |
| paroneayea | nkinkade: yeah | 20:03 |
| paroneayea | hence <paroneayea> nkinkade: done! | 20:03 |
| nkinkade | Awesome. Thanks! I look forward to the day when I can log into a convenient web interface and just press a button to launch translations. | 20:03 |
| nkinkade | ;-) | 20:03 |
| paroneayea | nkinkade: me too :) | 20:05 |
| nkinkade | Although, it is pretty easy to just say "Hey, paroneayea, launch Greek." | 20:06 |
| nkinkade | Maybe even easier than logging into a web interface and clicking a button. | 20:06 |
| nkinkade | It it weren't for the lack of instant gratification, I'd just as soon stick with our present mode of operation. | 20:07 |
| nkinkade | :-) | 20:07 |
| paroneayea | cloudsource your tasks with paroneayea-update | 20:07 |
| akozak | now thats job security | 20:07 |
| paroneayea | lol | 20:07 |
| paulproteus | nkinkade: and ionicer :P | 20:08 |
| * paulproteus waves while updating search.agshare.org. | 20:09 | |
| nkinkade | Ooo. Yeah, I forgot paulproteus' nice tool that allow regular users to ionice stuff. | 20:09 |
| nkinkade | (as usually on root can do it) | 20:09 |
| *** JoiIto has joined #cc | 20:09 | |
| nkinkade | *only | 20:10 |
| nkinkade | paulproteus: Have you got ionicer into Debian yet? | 20:10 |
| nkinkade | I say that half jokingly, but actually part serious. | 20:10 |
| nkinkade | The ability for a regular user to run ionice seems nice. | 20:10 |
| paulproteus | nkinkade: I feel the same way about it (-: | 20:10 |
| paulproteus | Like the opposite of "Don't tread on me". | 20:11 |
| *** JoiIto has quit IRC | 20:14 | |
| *** Kaetemi has joined #cc | 20:16 | |
| nkinkade | paulproteus: What's the meaning of this on the discovered VPS? | 20:17 |
| nkinkade | -rw-r--r-- 1 501 staff 273 Sep 3 20:53 /etc/hosts | 20:17 |
| * paulproteus blinks | 20:17 | |
| paulproteus | It doesn't look familiar to me, and looks quite icky. | 20:18 |
| nkinkade | securetty and securetty.dpkg-old have the same issue. | 20:18 |
| nkinkade | I guess I'll just fix it up. | 20:18 |
| paulproteus | Well, let me do a "find -uid 501" first | 20:19 |
| *** JoiIto has joined #cc | 20:19 | |
| paulproteus | It's just the files you pointed to. | 20:19 |
| paulproteus | How very odd. | 20:19 |
| paulproteus | My current hypothesis would be that the machine was pwned by some RedHat-centric attacker (hence user IDs based on 500, not starting at 1000 like Debian). | 20:20 |
| paulproteus | But I don't even see why they'd do that. | 20:20 |
| paulproteus | So one alternative is some weird configuration script for some program. | 20:20 |
| paulproteus | September 3... I don't think I was doing much on that VPS that Friday. | 20:20 |
| nkinkade | paulproteus: Yeah, it's odd. I only knew about it because dpkg complained when I was upgrading some packages a minute ago. | 20:20 |
| paroneayea | (03:19:46 PM) Christopher Webber: are we doing a tech call this week? | 20:21 |
| paroneayea | (03:20:27 PM) Nathan Yergler: Uh, no. I'm on an airplane in chicago right now :) | 20:21 |
| nkinkade | paulproteus: It was probably me on Sept 3, updating /etc/hosts | 20:21 |
| paulproteus | Hmm, okay. | 20:21 |
| *** bassel has quit IRC | 20:41 | |
| nkinkade | paulproteus: Did you change your user passwd on that machine around a week ago? | 20:54 |
| paulproteus | Yeah, I'm pretty sure. | 20:54 |
| nkinkade | Okay. Good ... just making sure. :-) | 20:55 |
| nkinkade | The permissions on those 3 files is a bit disconcerting. | 20:55 |
| nkinkade | It's a mystery to me how they could have got that way. | 20:56 |
| *** oshani has quit IRC | 21:05 | |
| *** papy|food is now known as papyromancer | 21:08 | |
| *** bassel has joined #cc | 21:18 | |
| *** oshani has joined #cc | 21:40 | |
| *** oshani has quit IRC | 21:51 | |
| *** balleyne has quit IRC | 22:03 | |
| *** DNS777 has quit IRC | 23:05 | |
| *** papyromancer is now known as papy|away | 23:08 | |
| *** JED3 has quit IRC | 23:16 | |
| *** phurl has quit IRC | 23:40 | |
| *** akozak has quit IRC | 23:49 | |
| *** pmiller has joined #cc | 23:51 | |
Generated by irclog2html.py 2.6 by Marius Gedminas - find it at mg.pov.lt!