Wednesday, 2009-04-22

paulproteusThere's two paster PIDs00:00
paulproteusSo this line goes horribly wrong:00:00
paulproteusCCENGINE_MEM=$(ps u -d | grep paster | grep -v grep | awk '{print $6}')00:00
* nathany wonders if he changed the way something's deployed00:00
nathanyoh, right00:00
paulproteusI will fix that now.00:00
paulproteusmta, yeah00:01
nathanywhen i was updating the MTA web app00:01
paulproteusShould this cycle the MTA process in the same way?00:01
nathanyright... i thought i was being clever by compositing the WSGI apps00:01
nathanypaulproteus: sure00:01
* paulproteus giggles.00:01
nathanythey run in similar manners00:01
nathanyi doubt the MTA app will have the same issue, but whatever00:01
paulproteusBTW mod_wsgi plus daemon mode seems safe to me, if it can do process cycling (haven't read that far in its docs yet).00:01
paulproteusAll I really want for Christmas is process cycling.00:01
* nathany wonders if there's an FCGI to WSGI container00:02
paulproteusflup_fcgi seems to be that00:02
nathanypaulproteus: i thought we were using mod_wsgi daemon mode previously for... the scraper? whatever it was that we ditched mod_wsgi for00:02
paulproteusThat's possible, I didn't know about the daemon mode vs. not-daemon-mode distinction then.00:02
nathanyi did notice a release announcement saying they'd fixed a memory leak in a recent release00:02
paulproteusI am still hesitant to use mod_wsgi, make no mistake. (-:00:03
nathanyi understand :)00:03
*** johndoigiii has quit IRC00:04
paulproteusWhat the heck kind of stupid init script doesn't leave a PID file.00:05
paulproteusWith all due respect...00:05
paulproteusLooks like /etc/rc2.d/S99cc_engine-run-cc_engine is generated code, generated by buildout?00:06
nathanypaulproteus: correct00:06
paulproteusAnd you just cp that to /etc/init.d/ ?00:06
paulproteus(Not saying that's bad, just trying to understand)00:06
nathanypaulproteus: uh... not sure... i think it's more likely the one in /etc/init.d is automatically created, copied to rc2.d00:07
paulproteusRight, ACK00:07
nathanysince buildout wouldn't know about appropriate run levels00:07
paulproteus"the one in /etc/init.d is automatically created" - by buildout, and that's why you run the buildout as root?00:07
nathanypaulproteus: zdaemon may be able to be coerced to leave a PID file00:07
nathanypaulproteus: that was true hysterically00:07
paulproteuspaster can; I'm reading its docs now00:08
nathanyi'm not 100% certain it's true today00:08
* paulproteus appears to be eating chocolate chips for dinner00:08
paulproteusLayers upon layers, but I think I'm starting to understand00:10
nathanypaulproteus: yes :)00:10
paulproteusSo: zdaemon.conf is what runs bin/paster00:13
nathanypaulproteus: right00:13
paulproteusIs that the canonical place for the "bin/paster serve deploy.ini" string that is run (presumably by the shell)?00:13
paulproteus'Cause I changed it, and when I restart cc.engine I get a warning I don't yet understand: WARNING! zdrun is managing a different program!00:14
paulproteusour program   = ['bin/paster', 'serve', '--pid-file=/var/tmp/', 'deploy.ini']00:14
paulproteusdaemon's args = ['bin/paster', 'serve', 'deploy.ini']00:14
paulproteusI don't know what zdrun is yet; going to Google it now00:14
nathanypaulproteus: zdrun is zdaemon00:14
nathanyit's complaining because you told it to restart but pulled the rug out from under it00:14
nathanypaulproteus: zdaemon/zdrun actually acts as a process manager, but doesn't do memory based cycling00:15
nathanythat's why we can just kill paster and it comes back00:15
paulproteusAh, neat.00:15
nathany"it keeps coming back!"00:15
* paulproteus laughs 00:15
paulproteusI have a PIDfile00:16
paulproteusNow to update the script00:16
* paulproteus does so, slowly and wrongly at first apparently00:17
paulproteusThis way of measuring RAM usage is still wrong, but the script now works the way it did ca. 2d ago00:20
nathanypaulproteus: thanks00:20
paulproteusBut now with the righteousness and sanity of being based on a pid file. (-:00:20
nathanyi think its close enough00:20
nathanymy personal goal is to eliminate the need00:20
* nathany has some ideas but of course they take time and experimentation00:20
paulproteusD'oh, Googling [zdaemon fcgi] gives back our own frickin' commit messages.00:21
paulproteuspython-paste seems to support being used as FCGI via flup:
paulproteusIf you want, I can try to look into feasibility of that for now00:23
paulproteusInstead I could make the NK memory checker script either (a) run for MTA, too, or (b) use ps_mem for accurate counting, or (c) go back to the other things I could do00:24
nathanypaulproteus: let me look @ how we're doing paste for one second...00:24
paulproteusBTW schoen@eff showed me a text adventure game program for the G1...00:24
paulproteusHe's REALLY excited about it. It's called "Twisty," for "Twisty passages."00:24
nathanypaulproteus: cool!00:25
nathanypaulproteus: so i suspect that the paste FCGI server expects to be daemonized as opposed to using a socket00:26
nathanybut i'd be open to you looking at that and playing with it on your local box/staging00:26
nathany(in trunk)00:26
paulproteusit's fine with me if it's daemonized; it would probably launch a UNIX socket file and wait for requests there as a background daemon, and when Apache gets fed up with it'll kill that mofo00:27
paulproteusSamuel L. Jackson style00:27
paulproteusDaemon doesn't mean TCP, necessarily00:27
nathanyright, i know that -- that's what we're doing for paster, iirc00:27
paulproteusLooks to me like Paster launches its own HTTPd on localhost:9080 to me; I'm not sure I understand what you're saying.00:28
nathanypaulproteus: true; for some reason i was thinking that the FCGI process cycling worked by running the app and interacting through stdin/out00:28
paulproteusI'm Googling [zdaemon process cycling] and hit #1 is:00:28
paulproteusnkinkade, nathany: I'll set it to cycle cc.engine when it's memory usage ... nathany, there actually is a zdaemon process watching paster so you may be able ...00:28
paulproteusFrickin' awesome. |-:00:29
nathanypaulproteus: in fairness i think others have less pain associated with zdaemon00:29
nathanywe're abusing Zope a little with cc.engine in the way we're doing things, which i think causes some of these problems00:29
paulproteusIs that because they write code without memory leaks? I don't see why anyone would bother doing that when they could just kill the process every once in a while. (-;00:29
nathany(and we'll stop when frank's work lands)00:29
nathanypaulproteus: yes... and they actually use the persistent object store like sane people00:30
nathanyanyway, we can talk about this in depth tomorrow if you want (it'd be fun, i suspect)00:30
* paulproteus does a double-take on that sentence, but okay00:30
paulproteuslol! re: "it'd be fun"00:30
paulproteusSure, I guess this is stable for now, and we can call it a zdaemon-day.00:30
nathanypaulproteus: we do lots of object creation on the fly00:30
nathanyi can explain why and hang my head in shame tomorrow00:30
paulproteusThat sounds awesome, let's do it.00:31
paulproteus</zdaemon> for today, or should I add one or two enhancements to NK's script (Your choices are MTA support and memory calculation)00:32
paulproteusBTW we should do more IRC conversations in the style of text adventure games.00:32
paulproteusI'll try to play some so I know the genre better.00:32
nathanypaulproteus: i think there's plenty of other stuff to do today; let's talk about zdaemon-ness tomorrow and figure out if there are more things to do00:33
paulproteusI like that plan.00:33
nathanyi don't expect the MTA to have the same issues as cc.engine00:33
paulproteus"famous last words"00:33
* nathany is aware of that and is looking cautiously behind himself00:33
paulproteusBut I'm fine with that being your call and me doing some of the other tasks today promises.00:33
nathanyok, i'm off for the day-- talk to you tomorrow00:34
nathany(oh, and please drop NK an email to let him know what's up, paulproteus )00:34
paulproteusdoing so now00:34
*** nathany has quit IRC00:34
*** Bovinity has quit IRC00:59
*** johndoigiii has joined #cc01:27
*** jgay has quit IRC02:01
*** nkinkade has left #cc02:36
*** stevel has quit IRC03:32
*** stevel has joined #cc04:01
*** johndoigiii has quit IRC04:04
*** johndoigiii has joined #cc05:15
*** tanjir__ is now known as tanjir05:19
*** michaelkrnac has joined #cc05:59
*** stevel has quit IRC06:26
*** tanjir has quit IRC06:50
*** johndoigiii has quit IRC07:03
*** sama has joined #cc07:33
*** linuksamiko has joined #cc09:24
*** linuksamiko has quit IRC09:54
*** stevel has joined #cc10:58
*** balleyne has joined #cc11:01
*** balleyne has quit IRC11:22
*** lolo1 has joined #cc11:27
*** coolvik has joined #cc11:42
*** coolvik has left #cc11:42
*** sama has quit IRC11:43
*** sama has joined #cc11:44
*** haoyu_ has joined #cc11:52
*** sama has quit IRC12:00
*** sama has joined #cc12:01
*** lolo1 has left #cc12:13
*** kreynen has joined #cc13:21
*** Danny_B has quit IRC13:27
*** Danny_B has joined #cc13:28
*** lotia has joined #cc13:28
*** balleyne has joined #cc13:30
*** balleyne has quit IRC13:35
*** Danny_B has quit IRC13:47
*** Danny_B has joined #cc13:48
*** kreynen_ has joined #cc14:15
*** kreynen has quit IRC14:17
*** Danny_B has quit IRC14:21
*** michaelkrnac has quit IRC14:22
*** Danny_B has joined #cc14:22
*** jgay has joined #cc14:24
*** parker-fcnyu has quit IRC14:30
*** Danny_B has quit IRC14:31
*** Danny_B has joined #cc14:31
*** nathany has joined #cc14:39
*** johndoigiii has joined #cc14:57
*** nkinkade has joined #cc15:01
*** parker-fcnyu has joined #cc15:34
*** lotia has quit IRC15:47
*** jgay has quit IRC15:52
*** michaelkrnac has joined #cc15:58
*** jgay has joined #cc16:12
*** Bovinity has joined #cc16:14
nkinkadenathany: What's the jurisdiction code for the Unported licenses?  I tried "Unported" and "unported" but neither worked.16:26
nathanynkinkade: "-"16:26
nkinkadeNo problem, but I'll be asking you that again in a few months. :-)16:27
*** sama has quit IRC16:28
*** parker-fcnyu has quit IRC16:30
*** jgay_ has joined #cc16:32
*** lotia has joined #cc16:32
*** jgay has quit IRC16:36
*** jgay_ has quit IRC16:44
*** jgay has joined #cc16:46
haoyuhi nkinkade17:51
nkinkadehaoyu: Hi.17:51
haoyunkinkade, I have drop you a mail about the database error problem17:51
haoyuI got it again17:51
nkinkadeI saw it a few minutes ago.17:51
nkinkadeCan you cut-n-paste the exact error and email it to me?17:52
haoyunkinkade, the error is as same as the previous error, i think17:52
nkinkadeDo you get it when you browse the page, or just when you use api.php?17:52
haoyuWhen I browse the page17:52
haoyuWhen I use api.php it also return a response saying database error17:53
*** [mharrison] has joined #cc17:55
haoyunkinkade, the wireshark capture file is the session that updating the page at first time, which is no DB locking error but just get no response from server17:57
*** lotia has quit IRC17:57
nathanyjohndoigiii: so where are we with the i18n of the new partner interface stuff?17:58
nathanyyou made a comment yesterday about... something being included?17:58
johndoigiiiworking on it right now and am a bit puzzled17:58
nathanycan i help resolve any puzzles?17:58
haoyunkinkade, after the first time updating, every access to the page will get a DB error.17:59
johndoigiiinathany: right now I am placing these urls in spans like you said, but hiding the content is difficult18:00
johndoigiiithe only method I have found to work was the example I showed yesterday18:00
nathanyjohndoigiii: with the wrapped span?18:00
nathanyjohndoigiii: great, do that18:00
nathanyand then manually massage the i18n string in master/cc_org.po18:00
johndoigiiialright, so in the po file, we won't need the preceding ${url_name} thats produced from these spans?18:01
nathanyjohndoigiii: right18:01
nathanywhy don't you do the nested spans and update master/cc_org.po18:01
nathanycommit those and i'll double check before you sync+cc2po18:01
johndoigiiiawesome, I'll get this merged and commited asap18:02
nathanyjohndoigiii: yeah, don't merge to production yet, let's double check first :) (but i'm 95% sure this is right)18:02
johndoigiiioh I meant i18nmerge18:02
johndoigiiiI should also have that license select for commoner at some point this afternoon18:05
*** everton137 has joined #cc18:13
nkinkadehaoyu: Try those links again.18:38
nkinkadeIt seems there may have been some old locks hanging around.  I'm curious as to why MySQL doesn't flush those locks after some period, or rollback the transaction.18:39
nkinkadeIt's strange that it seems to just leave them hanging for days.18:39
haoyunkinkade, yes they can be accessed now. may I try the api.php now?18:40
nkinkadehaoyu: Yeah, try it again.18:41
nkinkadeThose locks were hanging around since before I told Varnish to leave requests to api.php alone.18:41
haoyuI tried to update this page:
haoyuand seems it hang again18:42
haoyuI guess updating via api.php may cause these locks... don't know why18:44
haoyuNow I'll try to update Romina/Flag via tor18:45
nkinkadehaoyu: Did you say that using api.php works when going directly to port 8080?18:47
haoyuthat's for file uploading18:48
haoyuI can't made file uploading works when using port 80. But wiki page updating usually works well with both 80 or 808018:49
haoyuthe file uploading wouldn't be related to this problem (db error) I think18:50
*** lotia has joined #cc18:53
nkinkadehaoyu: So do you have a workaround?18:56
haoyuThe workaround is I can manually update these four pages - so far, there's only four18:58
*** julieboy401 has joined #cc18:58
*** julieboy401 has left #cc19:01
*** parker-fcnyu has joined #cc19:03
*** michaelkrnac has quit IRC19:23
nkinkadenathany: I'm going to have to do the building of these Deeds on one of the servers because it's taking too long, and eventually I'll have to shutdown my laptop later.  I'm trying to run buildout for cc.engine on a6 and getting this:19:24
nkinkadeVersionConflict: (setuptools 0.6c8 (/usr/lib/python2.5/site-packages), Requirement.parse('setuptools==0.6c9'))19:24
nkinkadeIs there a way to fix the version at 0.6c8?19:24
nkinkadepaulproteus: yo19:24
nkinkadeI guess you would know the answer to that as well.19:24
paulproteusI don't know why nathany depends on that version of setuptools.19:25
paulproteusWe could upgrade the version on a6.19:25
paulproteusThanks for the --no-heading. (-:19:26
* paulproteus does that19:27
paulproteusNow retry?19:27
nkinkadepaulproteus: doing buildout again?  You upgraded setuptools on a6?19:29
paulproteusI just upgraded setuptools on a6, and now it's your move.19:32
nkinkadeError: Couldn't find a distribution for 'ctypes'.19:32
haoyunkinkade, I'll sleep now. If you have no idea about why the DB errors happens, just clear the locks is ok. then I'll manually update these pages and blacklist them in my wiki bot. That would be a workaround.19:33
haoyuthanks nkinkade! :)19:33
paulproteusnkinkade, ctypes is part of Python now.19:33
paulproteusYou should just comment out that requirement.19:34
nkinkadehaoyu: That sounds like a plan.19:34
nkinkadeSo you need me to clear some of the locks again?19:34
paulproteus(nkinkade, or you should run buildout with Python 2.4)19:34
nkinkadeYou can kill individual mysql threads, but last time I just restarted MySQL. :-)19:34
nkinkadepaulproteus: Thanks.  I'll comment it out in the config.19:34
haoyunkinkade, yes I think they locked again19:35
nkinkadehaoyu: I'll just restart MySQL and hope that does it.  Have a good night!19:36
haoyuthank you nkinkade, good night! :)19:36
nkinkadepaulproteus: It seems that there is no config that specifies ctypes.  It must figure out that it needs it by looking at import statements or something.19:38
paulproteusBuh, can I see what you're seeing somehow?19:38
nkinkadeDo you just recommend to run buildout with python 2.4?19:38
paulproteusTell me where to cd / what to run / how to share your screen?19:38
paulproteusYes, python2.4 is a better plan in this case probably (cc.engine?).19:38
nkinkadeYeah, a6.19:38
nkinkadeI've got a screen session.19:38
nkinkadeYeah, cc.engine.19:39
nkinkadeI grepped -r for ctypes and it only found matches in binary files and some python code.19:39
paulproteusYeah, use python2.419:40
paulproteusnathany, SYN - cc.engine still wants python2.4, right?19:40
paulproteusThanks, I see that19:40
nkinkadeI missed that one earlier ... I guess I could comment that reference to ctypes out.19:40
paulproteuscc.engine iirc requires 2.4, so just buildout with that I think.19:40
nkinkadeI thought cc.engine was now 2.5 compatible.19:40
paulproteus"Well clearly it's not"19:41
nkinkadeIt was something nathany did when we upgraded to Lenny.19:41
nkinkadeAt least it's runtime compatible with 2.5.19:41
paulproteusSure you can try commenting it out and rebuildout-ing19:41
nkinkadeTrying that now, and it seems to have got us past that hurdle.19:43
nkinkadeMaybe there will be more waiting.  You can see in your own terminal ... :-)19:43
paulproteusI'm watching, yeah (-:19:43
paulproteusA hard-coded dependency on pytz is just *so* broken.19:43
paulproteusLooks like it's ZODB3 that does that.19:44
paulproteusGetting distribution for 'pytz==2007k'. # this means if some country updates daylight savings, we'll never notice19:44
nkinkadeI haven't the least idea what pytz is, or why doing that would be bad.19:44
nkinkadeOh, python timezone.19:44
paulproteusThe risk of letting me watch things is I'll complain about them. (-:19:44
nkinkadeWell, I wonder if that wouldnt' have happened if I hadn't specified -N19:45
nkinkadeBuilding Deeds takes eons.19:45
nkinkadeMy machine has been regenerating the Unported Deeds for about the past 5 hours.19:46
nkinkadeAnd I've still got 1 other jurisdiction and 7 or 8 more locales to rebuild.19:46
paulproteusI wonder what the bottleneck is.19:46
nkinkadeHence wanting to do it in a detached screen session on a6. :-)19:46
nkinkadepaulproteus: I've wondered myself.19:47
nkinkadenathany explained to me once that ./bin/mkdeeds is really just a modified version of cc.engine itself .. that it sets up a server, then makes a request to the server, then writes the Deed that the cc.engine code automatically generates.19:48
nkinkadeSomething like that.19:48
paulproteusSure, but that doesn't explain why it takes so long!19:48
paulproteusAnyway your buildout finished. (-:19:48
nkinkadeI'm tempted to run it without -N to see if gets some newer version.19:48
nkinkadeDoes -N only affect already installed components or also components to be fetched?19:49
*** Orango has quit IRC19:50
paulproteusI think just already-installed ones.19:50
nkinkadeThat's what I had been thinking, so maybe pytz 2007 actually is hardcoded.19:51
paulproteusI think so too.19:51
nkinkadeI'll worry about later ... much later.19:51
paulproteusI'm going to worry about the NEXT FIVE MINUTES!19:52
nkinkademkdeeds just blew up.19:54
nkinkadeStarting to wonder if I should have builtout with 2.4.19:55
paulproteusBoom boom boom boom, mighty fine mkdeeds.19:55
paulproteusEr, what's cc.engine using Maildir for!?20:08
nkinkadeSo CC needs a RAQ?20:12
nkinkadeI have no idea what cc.engine uses maildir for.  I just re-ran buildout for good measure and it pulled in that updated.20:13
paulproteusnkinkade, Great20:15
nkinkadepaulproteus: johndoigiii: Are we having our tech meeting today?20:15
paulproteusOh I hope so!20:15
nkinkadeIt's supposed to be in 15 minutes.20:15
paulproteusnkinkade, re: the DB nonsense: Maybe if I were you, I'd mysqldump the database, drop the DB entirely, recreate it, and re-import the data from the mysqldump.20:16
paulproteus'cause like what is this trash?20:16
paulproteus(the locking)20:16
nkinkadepaulproteus: Which DB?  monitor?20:16
nkinkadeSeems weird, for sure.20:17
nkinkadeThough I'm not sure how recreating the DB would fix this.20:17
paulproteusMaybe the MySQL database has some fruity invalid state, I dunno.20:17
*** robmyers has joined #cc20:18
nkinkadeI seems as if something about api.php is causing some condition where the connection goes away, but the thread in MySQL keeps on living, with a lock on some row.20:18
nkinkadeBut who knows, maybe the DB is messed up somehow.20:18
paulproteusThe whole thing seems pretty "Should Never Happen" to me, but I haven't dived in deep yet.20:20
paulproteusjohndoigiii, You're WFH today, right? Okay if we do our tech check-in meeting at two?20:25
johndoigiiiyeah, whatever you guys agree on is fine with me20:26
*** kreynen has joined #cc20:51
nkinkadenathany: Whenever you have a second, could you help me debug a mkdeeds errors on a buildout in my homedir on a6?20:58
nathanynkinkade: what's up?20:59
nkinkadeBuilding Deeds it taking much longer than I had anticipated, so I'll need to build them in a detachable screen session on a6.20:59
nkinkadehttperror_seek_wrapper: HTTP Error 500: Internal Server Error20:59
nkinkadeI'm not sure how to find out what the error is.20:59
nkinkadeOne thing of note is that I did buildout with python2.521:00
nathanyweird; this is the production branch?21:00
nkinkadeYou could login to a6 and su nkinkade.21:00
nkinkadeThen screen -x21:00
nathanyCannot open your terminal '/dev/pts/4' - please check.21:01
nathany(is what i get when i try that21:01
nkinkadeAre you me?21:01
paulproteussudo chown nkinkade $(tty)21:01
*** kreynen_ has quit IRC21:01
paulproteusRun my command as nathan before running nkinkade's command as nkinkade.21:01
nathanynkinkade: where does the error get printed out?21:02
nkinkadeOn stderr.21:02
nkinkadeOr perhaps it's stdout21:02
nathanynkinkade: this feels really familiar...21:02
nkinkadeCan you see the error?21:02
paulproteusAlso, like, meeting o'clock?21:03
nkinkadeOr alternatively, you could just run something like: ./bin/mkdeeds -j cz -v 3.0 -o ~/cc_svn/license.rdf/trunk/licenses/21:03
nkinkadeFrom a5:/home/nkinkade/cc_svn/cc.engine/branches/production/21:03
nkinkadeOf course it'll work better if you sudo su nkinkade21:04
nkinkadeI can call the conference line whenever you guys want to meet.21:04
nathanynkinkade: paulproteus johndoigiii i'm still trying to wrap up this sync up meeting with ben21:04
nkinkadeI may have to checkout a bit early today, in about 30 minutes to an hour.21:05
nathanynkinkade: so does it print out the error on stderr and then exit, or does it print it when you kill the process?21:05
nkinkadeMy lease ran out today, so I have to leave.  But I've got to return this cable modem to the provider by 7PM.21:05
paulproteusACK to all; just lemme know when we want to meet.21:05
nkinkadenathany: It print it to the console.21:05
nathanynkinkade: got it21:05
*** lotia has quit IRC21:06
nathanynkinkade: i'm going to try the buildout + run on my laptop21:08
nkinkadenathany: It must be something specific to a6's environment because things work just fine on my laptop.21:10
nathanynkinkade: sigh21:10
nkinkadeIf the Deeds built faster then I'd just do it all on my laptop, but I've got to suspend it once I leave in a while.  I guess I could leave it running on battery power for a few hours while it's in transport, but that's not ideal.21:11
nathanynkinkade: right21:12
paulproteusDoes it bind to a port that's in use by some other service?21:13
paulproteus(Actually I guess it uses wsgi testing, which is in-process no-TCP?)21:13
nathanypaulproteus: it doesn't really bind21:14
paulproteusRight, exactly.21:15
paulproteusjohndoigiii, nkinkade - call the conf line, and we'll join you from a conf room in 1m21:17
nathanynkinkade: how did you build out the license engine w/ 2.5? a clean checkout fails with a ctypes dependency error (as it should)21:19
*** lotia has joined #cc21:19
*** lotia has quit IRC21:20
*** johndoigiii_ has joined #cc21:24
*** johndoigiii__ has joined #cc21:28
*** nkinkade has left #cc21:33
*** johndoigiii_ has quit IRC21:34
*** johndoigiii has quit IRC21:41
*** qwerty has joined #CC21:45
*** qwerty is now known as Guest3854421:46
*** Guest38544 is now known as veran21:46
*** kreynen has quit IRC21:57
*** kreynen has joined #cc22:07
*** lotia has joined #cc22:10
*** veran has quit IRC22:28
*** robmyers has quit IRC22:33
*** tanjir has joined #cc22:58
*** nathany has quit IRC23:09
*** [mharrison] has quit IRC23:11
*** lotia has quit IRC23:15
*** kreynen has quit IRC23:17
*** haoyu_ has joined #cc23:26
*** haoyu has quit IRC23:27

Generated by 2.6 by Marius Gedminas - find it at!