#ceph IRC Log


IRC Log for 2011-02-09

Timestamps are in GMT/BST.

[0:17] <Tv|work> there's one really ugly problem i have with autotest right now, and it's that it doesn't re-download "external tests" defined in tarballs
[0:17] <Tv|work> which makes it really hard to develop tests
[0:17] <Tv|work> you need to ssh ... sudo rm -rf ... all the time
[0:18] <cmccabe> tv: can you explain what an external test is
[0:18] <Tv|work> combine this with >1 test machine and you don't otherwise care what machine ran your test
[0:18] <Tv|work> and that's a recipe for confusion
[0:18] <Tv|work> cmccabe: a way to avoid having to have your test be part of the checked-out source tree on the autotest server machine
[0:19] <Tv|work> basically run_job('http://url-to-a-tarball')
[0:19] <cmccabe> tv: I haven't thought of a really good reason why tests shouldn't be part of the source tree
[0:19] <Tv|work> cmccabe: developing them is painful if you need to ssh to autotest server, shut things down, git pull, start things up...
[0:20] <cmccabe> tv: I guess my concept of a test framework was integrated with git, so that you always tested a particular commit
[0:20] <Tv|work> i think they just started with some funny assumptions, because of their legacy
[0:20] <Tv|work> cmccabe: you can tell a test to use e.g. a specific version of ceph
[0:20] <Tv|work> cmccabe: but don't confuse test version vs ceph version...
[0:21] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[0:21] <cmccabe> tv: yeah, I can see how that would be desirable
[0:22] <cmccabe> tv: the problem with versioning the tests and ceph separately is that you'll end up with a lot of undocumented assumptions about what version of a test works with what version of the code
[0:22] <Tv|work> yeah but that's the lesser of two evils imho
[0:22] <cmccabe> tv: like, oh, don't use test foo with code before a certain commit, because it doesn't have the right hooks.
[0:22] <Tv|work> the other one is "yeah we found a bug in stable ceph, but we can't add a test for it without releasing a new stable ceph"
[0:23] <Tv|work> tests just evolve separately from releases
[0:23] <cmccabe> tv: well, you can always cherry-pick.
[0:23] <Tv|work> you can do things like have a test say "skip me if your ceph doesn't contain commit XXX"
[0:24] <Tv|work> i mean, automated
[0:24] <cmccabe> tv: versioning isn't always that straightforward though
[0:24] <cmccabe> tv: sometimes interfaces change
[0:24] <Tv|work> yeah, life's a bitch
[0:24] <cmccabe> tv: I had to deal with cross-module dependencies before... was part of a group that managed them
[0:25] <Tv|work> this is why you don't want to manage a long-living stable branch
[0:25] <Tv|work> it takes resources
[0:25] <cmccabe> tv: well, customers may disagree.
[0:25] <Tv|work> hey customers are used to give you those resources, if necesary
[0:25] <Tv|work> that's the whole point of it
[0:26] <jantje> we have a test branch for every developement branch
[0:26] <Tv|work> but in some ways making *you* feel the pain for incompatible changes is just fair -- your customers will feel it anyway
[0:27] <cmccabe> tv: yeah, but making test authors feel the pain is what I'd prefer to avoid
[0:27] <cmccabe> tv: remember that the APIs that tests will be accessing are in many cases not public
[0:27] <Tv|work> cmccabe: i wish there was a way around that..
[0:27] <jantje> nite
[0:27] <Tv|work> cmccabe: err, what internal apis should integration tests access?
[0:27] <cmccabe> tv: well there is. One repo for tests+code :) It does involve tradeoffs, though.
[0:27] <cmccabe> jantje: night!
[0:28] <cmccabe> tv: my integration tests often ran "debug" commands... undocumented cephtool commands
[0:28] <Tv|work> cmccabe: don't
[0:28] <cmccabe> tv: the point is that tests often have to wait for some weird condition to be true
[0:28] <Tv|work> cmccabe: *expose* any interface you need. think about what the right interface is.
[0:28] <cmccabe> tv: something that users don't care about, but tests do.
[0:28] <Tv|work> cmccabe: the end product will be much better for that!
[0:28] <cmccabe> tv: creating a public API that we'll support forever is a heavy burden
[0:28] <cmccabe> tv: I'd rather not do it just to write a test
[0:29] <Tv|work> e.g. good access to metrics is desirable to end users' monitoring systems
[0:29] <cmccabe> tv: also I wanted to avoid parsing the output of cephtool as much as possible, since parsing is error-prone and formats can change
[0:29] <Tv|work> cmccabe: so make cephtool output a nicer format!
[0:29] <Tv|work> cmccabe: don't workaround crap by crap
[0:29] <Tv|work> others *will* parse the output
[0:29] <cmccabe> tv: here is the thing. cephtool's output is mainly for human consumption.
[0:29] <Tv|work> think about e.g. nagios & munin checks
[0:29] <cmccabe> tv: as such, it can change.
[0:30] <Tv|work> well then there should be a --batch option or something
[0:30] <Tv|work> --json
[0:30] <cmccabe> tv: even if there was, will the JSON api stay the same forever? Or will you need to start versioning?
[0:30] <Tv|work> there's no "forever"
[0:31] <Tv|work> make it a proper feature and it's more likely to survive than your extra debug interfaces
[0:31] <cmccabe> tv: by separating the repos, you are creating a situation where deprecating an API that a test uses is extremely, extremely hard
[0:32] <cmccabe> tv: that will perhaps lead to better APIs, if a lot of thought up-front makes a good API. And perhaps worse ones, if experimentation and change leads to good APIs.
[0:32] <Tv|work> cmccabe: umm, you're mistaken about what repos are even in the conversation
[0:32] <cmccabe> tv: repos = git repos?
[0:32] <Tv|work> cmccabe: even if i try to put our autotests in ceph.git, that's still not the same as the autotest.git source tree on the server
[0:34] <cmccabe> tv: I do understand that it seems cleaner to put all the autotest-related stuff in a separate repo. Perhaps that is why you created autotest.git?
[0:34] <Tv|work> i did not
[0:34] <cmccabe> tv: but there are tradeoffs, of course.
[0:34] <Tv|work> hence, your mistakes
[0:34] <cmccabe> tv: is autotest.git the autotest upstream?
[0:34] <Tv|work> yes
[0:34] <Tv|work> autotest historically bundles its tests in its own source tree
[0:34] <cmccabe> tv: I see
[0:34] <Tv|work> because it was built to test the linux kernel
[0:35] <cmccabe> tv: right, and the Linux kernel's userland API is incredibly highly stable
[0:35] <cmccabe> tv: it can take months of debate to change anything
[0:35] <cmccabe> tv: well, except for that evil debugfs stuff >:)
[0:38] <cmccabe> tv: well, anyway, having two repos will work. If it's the "standard" way to do things with autotest then that is one reason to do it that way.
[0:38] <cmccabe> tv: we can keep the existing debug APIs for now and add to them.
[0:39] <cmccabe> tv: let's please not start parsing cephtool stdout in our tests, though, because otherwise cephtool will be set in stone forever.
[0:39] <cmccabe> tv: use the debug APIs, that's what they're for.
[0:42] <DeHackEd> maybe, but like a DVD-R you only get to write to it ince
[0:42] <Tv|work> cmccabe: now tell me how do i as a sysadmin wanting to install ceph set up monitoring to check that it's healthy
[0:42] <DeHackEd> whoops, wrong window
[0:43] <cmccabe> tv: right now our main mechanism is ./ceph health
[0:43] <cmccabe> tv: ceph health's output is designed for parsing... the first word is something like error, warn, ok
[0:44] <Tv|work> cmccabe: what i'm saying is, either that is sufficient for tests, or it's insufficient for sysadmins; there's a very narrow boundary there
[0:44] <cmccabe> tv: and then there's some free-form text with an explanation
[0:44] <cmccabe> tv: there are a lot of times when you want to force a weird situation in a test that you would never intentionally create
[0:44] <cmccabe> tv: like deliberately creating lost objects
[0:44] <Tv|work> cmccabe: sure, fault injection is special
[0:44] <Tv|work> cmccabe: but seeing the status of it isn't
[0:46] <cmccabe> tv: certain statuses are not interesting to users
[0:46] <Tv|work> i honestly doubt that
[0:48] <cmccabe> tv: well, continuing with fault injection
[0:49] <cmccabe> tv: the number of faults that have been injected is something sysadmins don't want to know, since they won't be injecting faults.
[0:49] <Tv|work> why would tests ask ceph what the test itself did?
[0:49] <cmccabe> tv: perhaps the test wants to know if that has been accomplished correctly
[0:49] <cmccabe> tv: communication is not a one-way street
[0:51] <cmccabe> tv: I think a lot of this will be more obvious once we've written more tests
[0:52] <cmccabe> tv: not just simple tests like "do an rsync hurrr" but things that actually test the fault-tolerance and corner cases that we claim to support.
[0:53] <sagewk> i think even in those cases we should aim for the mechanism to expose results to the test to also be useful for admins.
[0:53] <Tv|work> that's what i said ages ago, and what you seemed to be debating against
[0:54] <cmccabe> sagewk, tv: the problem I encountered when I was first writing my system tests is that I didn't want to parse the ever-changing output of cephtool
[0:54] <Tv|work> yes; don't
[0:55] <cmccabe> sagewk, tv: and I didn't want to create some kind of permanent API before getting any work done
[0:55] <cmccabe> sagewk, tv: hence the debug API
[0:55] <Tv|work> in my mind, the point of all this is not to test the ceph-as-it-is
[0:55] <Tv|work> but to build the ceph-of-the-future
[0:55] <cmccabe> sagewk, tv: I do like the idea of some kind of API that returns JSON with useful info
[0:55] <Tv|work> you don't need to stop everything else until you've abstracted the hell out of everything
[0:56] <cmccabe> sagewk, tv: well, you kind of do, if you want all APIs to be useful for sysadmins, or else not-existent
[0:56] <Tv|work> but i definitely subscribe to the world view of "you need an interface, you add an interface"
[0:56] <sagewk> i don't see a problem with using the interface as it currently exists, and fixing the test if/when we change/improve the interface.
[0:56] <Tv|work> yeah if it's not too horrible to parse
[0:57] <cmccabe> sagewk: can we have some kind of interface that doesn't require parsing for tests
[0:58] <cmccabe> sagewk: python must support some kind of JSON interface where you just do JSON.from_string()
[0:58] <Tv|work> oh god damn
[0:58] <Tv|work> can we please not waste time on this?
[0:58] <Tv|work> it's all obvious once we hit it
[1:01] <cmccabe> tv: well, let me know when the framework is ready. Then I'll try porting the existing tests and we'll "hit it" :)
[1:02] <sagewk> i suspect this will come down to making 'ceph whatever' output more parseable?
[1:03] <Tv|work> sagewk: yup, and sometimes exposing more of it
[1:03] <sagewk> and hopefully imposing some order on the madness that is the current monitor command set and parsing code
[1:04] <cmccabe> sagewk: right
[1:04] <cmccabe> sagewk: see the use of "./ceph pg debug unfound_objects_exist" in test_lost.sh
[1:04] <cmccabe> sagewk: it's an example of a debug command that was added just for a test
[1:05] <cmccabe> sagewk: which we could/should replace with some call to a superior API if we want
[1:10] <Tv|work> ops wants metrics
[1:10] <Tv|work> one good metric is how many objects are in what states
[1:10] <Tv|work> sounds like a match
[1:10] <sagewk> yeah, just a matter of expsoing it in an easy to parse way
[1:11] <Tv|work> the details of that can evolve, that's not the relevant part
[1:11] <Tv|work> but just putting in that feature improves ceph a lot
[1:12] <cmccabe> yeah, more visibility would be nice
[1:16] <cmccabe> sagewk: I wonder if we could write a python library to hydrate our serialized structs?
[1:17] <cmccabe> sagewk: then it would just be a matter of calling osd_stat_t::encode and sending that buffer
[1:17] <cmccabe> sagewk: it's not as ambitious as it might seem, since we could start simple with just handling structs with ints and strings, and branch out from there over time.
[1:20] <sagewk> if there's a clear use? we're adding the work of maintaining independent implementations of encode/decode
[1:21] <cmccabe> yeah it's just an idea... hmm
[1:21] <Tv|work> i'd rather build tests on top of things like "ceph metrics osd.objects.lost" than fiddling with internal protocol packets directly
[1:22] <Tv|work> think of the sysadmins
[1:22] <cmccabe> we could always take the sysfs approach
[1:22] <cmccabe> one key, one value
[1:22] <cmccabe> so ceph metrics osd.objects.lost returns just one value, and ceph metrics osd.objects.unfound returns another, etc.
[2:00] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[2:07] <Tv|work> some gitbuilder downtime, i'm fiddling with the vm
[2:11] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[2:32] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[2:39] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) has joined #ceph
[2:40] * Tv|work (~Tv|work@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[2:42] * greglap (~Adium@ip-66-33-206-8.dreamhost.com) Quit ()
[2:59] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving.)
[3:14] * cmccabe (~cmccabe@c-24-23-253-6.hsd1.ca.comcast.net) has left #ceph
[3:34] * WesleyS (~WesleyS@ Quit (Quit: WesleyS)
[4:14] * bcherian (~bencheria@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[4:57] * votz (~votz@dhcp0020.grt.resnet.group.upenn.edu) Quit (Quit: Leaving)
[5:44] * greglap (~Adium@cpe-76-90-239-202.socal.res.rr.com) has joined #ceph
[5:53] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[6:14] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) has joined #ceph
[7:41] * bcherian (~bencheria@cpe-76-173-232-163.socal.res.rr.com) has joined #ceph
[8:10] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[8:43] * uwe (~uwe@ has joined #ceph
[8:53] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) Quit (Quit: Yoric)
[9:30] * allsystemsarego (~allsystem@ has joined #ceph
[10:25] * Meths_ (rift@ has joined #ceph
[10:32] * Meths (rift@ Quit (Ping timeout: 480 seconds)
[10:34] * Yoric (~David@ has joined #ceph
[11:08] * cclien_ (~cclien@ec2-175-41-146-71.ap-southeast-1.compute.amazonaws.com) has joined #ceph
[11:08] * cclien (~cclien@ec2-175-41-146-71.ap-southeast-1.compute.amazonaws.com) Quit (Read error: Connection reset by peer)
[11:22] * verwilst (~verwilst@router.begen1.office.netnoc.eu) has joined #ceph
[12:18] * tjikkun (~tjikkun@195-240-122-237.ip.telfort.nl) Quit (Ping timeout: 480 seconds)
[12:19] * asachs (~asachs@andre.jnb1.gp-online.net) has joined #ceph
[12:26] * tjikkun (~tjikkun@2001:7b8:356:0:204:bff:fe80:8080) has joined #ceph
[12:27] <asachs> Hi All, I am looking to Ceph for a file archiving cluster and wanted to know more about the stability and where to find a roadmap for the project ?
[12:28] <DeHackEd> http://tracker.newdream.net/projects/ceph/roadmap <- roadmap
[12:29] <DeHackEd> as for stability, check the bug list. it's getting better, but still not quite there I'd say. Try it, but not in a situation where its complete failure would cost you.
[12:31] * Yoric_ (~David@ has joined #ceph
[12:31] * Yoric (~David@ Quit (Read error: Connection reset by peer)
[12:31] * Yoric_ is now known as Yoric
[12:40] <asachs> DeHackEd: thanks for the heads up
[12:46] * asachs (~asachs@andre.jnb1.gp-online.net) Quit (Quit: asachs)
[14:09] * Meths_ is now known as Meths
[15:35] * Yoric (~David@ Quit (Quit: Yoric)
[15:35] * Yoric (~David@ has joined #ceph
[16:41] * ghaskins (~ghaskins@66-189-113-47.dhcp.oxfr.ma.charter.com) Quit (Quit: Leaving)
[17:14] * greglap (~Adium@cpe-76-90-239-202.socal.res.rr.com) Quit (Quit: Leaving.)
[17:34] * bcherian (~bencheria@cpe-76-173-232-163.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[17:37] * verwilst (~verwilst@router.begen1.office.netnoc.eu) Quit (Quit: Ex-Chat)
[17:49] * bchrisman (~Adium@c-98-207-207-62.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:53] * greglap (~Adium@ has joined #ceph
[18:21] * bcherian (~bencheria@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:22] * Tv|work (~Tv|work@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:36] * bchrisman (~Adium@70-35-37-146.static.wiline.com) has joined #ceph
[18:37] * Yoric (~David@ Quit (Quit: Yoric)
[18:37] * uwe (~uwe@ Quit (Quit: sleep)
[18:43] * greglap (~Adium@ Quit (Quit: Leaving.)
[18:46] * joshd (~joshd@ip-66-33-206-8.dreamhost.com) has joined #ceph
[18:49] * cmccabe (~cmccabe@ has joined #ceph
[18:55] <cmccabe> tv: do you remember the syntax for removing a branch on the server?
[18:55] <cmccabe> tv: it was something with a push
[18:57] <Tv|work> so where's the logic that says "2011-02-09 01:21:21.463582 406bdf20" in front of what looks like logs sent to stderr?
[18:57] <Tv|work> cmccabe: git push theremote :thebranch
[18:57] <cmccabe> tv: debug.h
[18:58] <cmccabe> tv: specifically, _dout_begin_line
[18:58] <Tv|work> cmccabe: ohh so the bit of hex is a thread id?
[18:58] <Tv|work> cmccabe: no wonder it looks different on i386..
[18:58] <cmccabe> tv: pthread_self
[18:59] <cmccabe> tv: I have considered the idea of not outputting the date when logging to stderr. It wouldn't be too hard to implement if the will is there
[18:59] <cmccabe> tv: also we probably shouldn't log the date to syslog since syslog takes care of that itself
[18:59] <Tv|work> i'm ok with, just need to fix a regexp to accept the i386 variant..
[18:59] <Tv|work> no dates to syslog, yes on that one
[19:01] <cmccabe> tv: I'll open an issue
[19:03] <prometheanfire> seems to handle suspend/resume just fine
[19:05] <cmccabe> prometheanfire: suspend/resume?
[19:10] <gregaf> bchrisman: did you get a chance to test out the filesystem timeout stuff on a drive pull?
[19:10] <gregaf> http://tracker.newdream.net/issues/735
[19:11] <bchrisman> hey… didn't notice that's been fixed.
[19:11] <bchrisman> I'll test it at some point in the future.. I'm working on trying to figure out what NFS is doing that's trashing ceph.. :)
[19:11] <cmccabe> bchrisman: yeah. Now when the drive is pulled, the btrfs ioctl should time out, and
[19:11] <cmccabe> bchrisman: ceph will log an error and quit
[19:12] <bchrisman> cmccabe: the osd does I presume?
[19:12] <cmccabe> bchrisman: well, cosd will
[19:12] <bchrisman> :)
[19:12] <cmccabe> bchrisman: hopefully leading to a quicker healing process
[19:13] <prometheanfire> cmccabe: running it on my laptop :D
[19:13] <prometheanfire> was testing kvm
[19:14] <cmccabe> prometheanfire: great, we'd love to see the timeout in action. I did a few simulations but didn't pull any drives
[19:14] <gregaf> he meant it survived his laptop going through a suspend-resume cycle...
[19:14] <prometheanfire> no, this is a single node laptop
[19:14] <prometheanfire> gregaf: exactly
[19:14] <cmccabe> gregaf: oh I see.
[19:15] <prometheanfire> no reason it should not survive, but still
[19:15] <Tv|work> the pstree from Inception: DEEPER! http://pastebin.com/raw.php?i=xpdFUH4L
[19:16] <cmccabe> tv: we need to go deeper! oh wait, maybe that's deep enough :)
[19:20] <bchrisman> cmccabe: jyotinder sits next to me up here… says he knew you at lab126… you're the same guy?
[19:20] * WesleyS (~WesleyS@ has joined #ceph
[19:20] <cmccabe> bchrisman: seriously? Jyotindra?
[19:20] <cmccabe> bchrisman: yeah, I knew him back at the Lab... :)
[19:21] <bchrisman> ahh cool yeah..
[19:23] <cmccabe> bchrisman: small world. I was just in Cupertino the other day to eat at the curry house
[19:23] <bchrisman> ahh he was wondering whether you were still up in the bay area or moved down to socal.. :)
[19:24] <cmccabe> bchrisman: nah, NorCal ftw. I am in the city now though
[19:24] <bchrisman> cmccabe: yeah… hard to shake a stick in this valley without hitting former storage industry coworkers.. :)
[19:26] <cmccabe> bchrisman: yup
[19:26] <sagewk> skype at 10:30!
[19:27] <Tv|work> whee gitbuilder-i386 has no errors anymore.. http://ceph.newdream.net/gitbuilder-i386/#origin/master
[19:28] * fzylogic (~fzylogic@ Quit (Quit: fzylogic)
[19:29] <gregaf> signed/unsigned warnings…?
[19:30] <gregaf> I didn't think we had any of those...
[19:30] <cmccabe> gregaf: using different compiler options or archs will often expose warnings
[19:30] <cmccabe> gregaf: gcc's warning messages come out of its optimization phase, so that if you turn off optimizations, some warnings go away :P
[19:31] <gregaf> cmccabe: skype!
[19:33] * Yoric (~David@dau94-10-88-189-211-192.fbx.proxad.net) has joined #ceph
[19:38] * uwe (~uwe@ip-94-79-145-210.unitymediagroup.de) has joined #ceph
[19:44] * bencherian (~bencheria@ip-66-33-206-8.dreamhost.com) has joined #ceph
[19:48] * uwe (~uwe@ip-94-79-145-210.unitymediagroup.de) Quit (Quit: sleep)
[19:51] * bcherian (~bencheria@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[19:56] <Tv|work> smells like the underlying i386 thing is PAGE_SIZE being defined differently
[19:56] <Tv|work> or at least there first few i looked at all involved PAGE_SIZE
[20:00] <cmccabe> tv: PAGE_SIZE should be 4k on both i386 and x86_64
[20:00] <Tv|work> yeah but the warnings were just a MAX of that and something that didn't seem arch-related..
[20:00] <cmccabe> tv: we should be using the posix function that gets pagesize
[20:00] <cmccabe> tv: PAGE_SIZE is deprecated if I remember correctly
[20:41] * uwe (~uwe@ip-94-79-145-210.unitymediagroup.de) has joined #ceph
[20:46] <cmccabe> does anyone have the link to the i386 warnings?
[20:46] <cmccabe> never mind, found it.
[21:06] <Tv|work> yeah i don't know how to edit the ceph.newdream.net sidebar
[21:06] <Tv|work> just -i386 to the existing link
[21:15] <Ormod> cmccabe: btw if we do merge python-rados into the ceph repo, what's the commit policy over there?
[21:16] <Tv|work> honestly, as a Python programmer, I'm not sure why you'd want to merge them
[21:16] <Tv|work> if you have ceph's development headers properly installed, pypi makes installing the python wrapper a single command
[21:16] <Tv|work> that's the way it usually goes
[21:17] <Ormod> Tv|work: python-rados is actually ctypes based, not a python c extension
[21:17] <cmccabe> ormod, tv: yeah, I see that you used ctypes
[21:17] <Tv|work> Ormod: even better, it's actually readable ;)
[21:18] <Tv|work> Ormod: so you don't need anything but ceph's libraries found when the python script is run
[21:18] <Ormod> Tv|work: yup
[21:18] <cmccabe> ormod, tv: the reason why I wanted to merge is so that we could use python bindings in some of our ceph scripts
[21:18] <Tv|work> cmccabe: we use a lot of 3rd party software, you know..
[21:18] <cmccabe> ormod, tv: for example we've considered rewriting mkfs in python (yes, I know mkfs doesn't need this particular library binding)
[21:19] <Tv|work> ohh to provide actual ceph things
[21:19] <Tv|work> yeah that'd be nice
[21:19] <cmccabe> ormod, tv: I also was toying with the idea of writing gceph in python
[21:19] <Tv|work> *that* makes sense to me
[21:19] <Tv|work> gceph?
[21:19] <cmccabe> ormod, tv: the cephtool gui
[21:19] <Ormod> cmccabe: yeah well I'm fine with merging them
[21:20] <cmccabe> ormod: as far as commit policy, we sort of have a core team with push access, and review other patches on the mailing list, usually fairly quickly
[21:20] <cmccabe> cool. I'll check out how hard it is to use automake with this stuff (cringe)
[21:21] <Ormod> mmh, well I'd prefer push access but I can see why you'd want to limit that.
[21:22] <bchrisman> my kernel client's connection to mds is failing (message on mailing list).. is there a different level of mds debugging I should set?
[21:22] <bchrisman> Right now I have it at 0.. but when I upped it to 20.. looked like a lot of ops were coming through.. and I have a whole lot of file operations going on… so I was concerned about logspam.
[21:24] <Ormod> cmccabe: one way to get both might be to use submodules
[21:24] <Ormod> (git submodules that is)
[21:24] <cmccabe> bchrisman: which message on the mailing list are you referring to
[21:25] <cmccabe> bchrisman: is this "Fixing NFS" or a different one
[21:25] <bchrisman> Yeah NFS
[21:25] <bchrisman> that error… I'm trying to trace down why it's losing its connection to the mds..
[21:26] <cmccabe> bchrisman: I see. So you turned up the MDS debugging.
[21:26] <bchrisman> cmccabe: well.. I did.. but looked at the output… and turned it back down
[21:27] <bchrisman> cmccabe: not sure what the levels are reporting.. I can update the wiki if there's an explanation somewhere
[21:27] <cmccabe> bchrisman: is memory tight at all?
[21:28] <cmccabe> bchrisman: also, are all the cmds processes running?
[21:28] <cmccabe> bchrisman: I mean did any of them crash
[21:28] <bchrisman> cmccabe: yeah they are right now… but I'm guessing one crashes during the test (I'm rerunning)
[21:29] <cmccabe> bchrisman: It would be good if you could run the kernel client on a different machine than the servers... there's a well-known deadlock situation if you don't do that
[21:29] <cmccabe> bchrisman: it's more or less the same reason you should never NFS-mount from localhost.
[21:31] <bchrisman> cmccabe: yeah… definitely can be some issues there…. might be a memory issue yes… didn't see an OOM message… but could cause other issue
[21:32] <cmccabe> bchrisman: it's just something to keep in mind. It could also just be a cmds crash or something.
[21:34] <cmccabe> ok, I'm headed to lunch. Will be back in a half hour folks
[21:37] <gregaf> Ormod: cmccabe: Tv|work: I thought we were talking about setting up gitosis to provide push access to libraries which we'd keep in local repos
[21:39] <gregaf> bchrisman: I think you'll pretty much need mds_debug set to 20
[21:39] <gregaf> you could try 10
[21:39] <gregaf> in combination with ms_debug 1 that would let us at least see which op is failing and get some idea of where
[22:11] <Ormod> gregaf: btw about http://tracker.newdream.net/issues/644 , how much better was it with the patch?
[22:11] <gregaf> not a lot :(
[22:12] <gregaf> basically, mknod is just an expensive op and most initial rsyncs involve a lot of mknods
[22:12] <gregaf> (expensive because it requires going over-the-wire to the MDS; it's something I hope to see improved on eventually but isn't a priority right now)
[22:13] <Tv|work> gregaf: i'm sorry i can't parse that sentence ;)
[22:13] <Tv|work> gregaf: the earlier one that mentioned me
[22:13] <Tv|work> sage wanted gitosis, but mostly to make adding repos easier
[22:14] <gregaf> weren't we talking about setting up a repository for the python/php/etc bindings people set up?
[22:14] <gregaf> I thought gitosis meant we could give people selective push access
[22:14] <Tv|work> yeah, it has per-repo acls
[22:14] <gregaf> so we could give them push access to the stuff they contribute and keep the core code requiring patches
[22:14] <Tv|work> but now we've been talking about merging the python stuff
[22:15] <gregaf> oh, dunno about that
[22:15] <sagewk> that issue aside, does itm ake more sense to have the language bindings in ceph.git or in a separate repo?
[22:15] <Tv|work> and frankly in the world of github, i'm not sure we need to provide hosting...
[22:15] <Ormod> gregaf: ok. I'm basically looking at ways of replicating to a distant data center which is why I'm interested
[22:15] <gregaf> Ormod: if you're bandwidth-limited you're unlikely to have an issue
[22:16] <Tv|work> sagewk: cmccabe had a good point earlier in that if we merge the python thing, then we could write ceph utilities in python
[22:16] <Ormod> gregaf: in this case I wouldn't be
[22:17] <Ormod> gregaf: Tv|work I'm fine with keeping the stuff on github/providing commit rights to people if that's the way you want to go
[22:17] <sagewk> it would also make it easier to keep it in sync with the api (not that there should be much drift there :)
[22:17] <Tv|work> sagewk: yeah but you don't want to host every language wrapper out there..
[22:17] <sagewk> fwiw i'm all for having supported python bindings
[22:17] <sagewk> yeah
[22:18] <Tv|work> sagewk: but cmccabe point about using python bindings for ceph core features was a good one
[22:18] <Tv|work> that is, pick one, bless it special, others live elsewhere
[22:18] * bcherian (~bencheria@ip-66-33-206-8.dreamhost.com) has joined #ceph
[22:18] <Tv|work> personally, i'd much rather see 300 line python scripts than 300 line shell scripts
[22:19] <cmccabe> tv: yeah, completely
[22:19] <gregaf> yes please
[22:20] <Tv|work> so it sounds like we want to get the python ctypes rados lib merged right now, any other python wrappers in the same fashion belong there, other language wrappers get encouraged to use github?
[22:21] <cmccabe> tv: yeah I like that idea
[22:21] <cmccabe> tv: we should probably also link to any other wrapper somewhere on the wiki/site
[22:21] <Tv|work> cmccabe: oh yeah, definitely
[22:22] <Tv|work> encourage them to be part of the ecosystem and all that
[22:23] <Tv|work> ohh autotest
[22:23] <Tv|work> you are so quaint
[22:23] <Tv|work> it's like slackware back in the day
[22:23] <Tv|work> it sticks together mostly because of all the special cases littered all around the source
[22:24] <sagewk> works for me.
[22:24] <sagewk> i would rule out pulling other language bindings in later, but c/c++/python is a solid starting point.
[22:24] <sagewk> s/would/wouldn't/ :)
[22:24] <cmccabe> cool
[22:24] <Tv|work> oh it's more, i've seen projects turn into clearinghouses for 10+ language bindings
[22:25] <Tv|work> with moderation, it's doable
[22:25] <cmccabe> tv: isn't slackware still going?
[22:25] * bencherian (~bencheria@ip-66-33-206-8.dreamhost.com) Quit (Ping timeout: 480 seconds)
[22:25] <Tv|work> but not "yea welcome put your code there"
[22:25] <Tv|work> cmccabe: no clue, I jumped ship when we were still using 3.5" floppies..
[22:25] <cmccabe> tv: I think their package managment system is sitll tar.gz too :|
[22:26] <Tv|work> i actually used SLS originally
[22:26] <cmccabe> redhat originally for me
[22:26] <Tv|work> C-64 -> CP/M -> MS-DOS -> OS/2 -> SLS -> Slackware -> Red Hat --around rh4.2--> Debian -> Ubuntu
[22:34] * bcherian (~bencheria@ip-66-33-206-8.dreamhost.com) Quit (Quit: Leaving)
[22:35] <Ormod> Anyway, feel free to merge python-rados or submodule it or whatnot, if possible I'd like to keep push access for it but I can live without if push comes to shove.
[22:36] <Tv|work> Ormod: If nothing else we can give you a repo that's a clone of ceph.git that you can push to.
[22:38] <Ormod> Tv|work: btw, noticed through linkedin that we have mutual acquaintances. Jaakko Niemi and Matti Halme
[22:38] <Tv|work> Ormod: oh huh
[22:38] <Tv|work> I worked with Matti and have known Jaakko for a long time
[22:38] <Ormod> Yeah both worked for F-Secure previously
[22:38] <Tv|work> oh yeah
[22:39] <Ormod> which is how I know them
[22:39] <Tv|work> Jaakko might still be there
[22:39] <Ormod> Nope, he left before christmas
[22:39] <Tv|work> haven't caught up with him since i moved to LA
[22:40] <Ormod> But yeah, small world. :)
[22:40] <Tv|work> yup
[22:40] <Tv|work> Ormod: so do you work for F-Sec, or just know them?
[22:40] <Ormod> Tv|work: I'm a lead architect at f-s
[22:41] <Tv|work> Ormod: do they have R&D outside of .fi these days?
[22:41] <Tv|work> or are you really up north and up late?-)
[22:41] <Ormod> Tv|work: malaysia and france
[22:41] <Ormod> Tv|work: but I'm just up late
[22:41] <Tv|work> ahh I think that's new since my days of consulting for F-sec
[22:44] <Ormod> well the french thing came through an acquisition, malaysian branch got set up 4-5 years ago
[22:44] <Tv|work> yeah my interaction with them was more late 90s
[22:44] <Ormod> We used to have development stuff in san jose but I think that it's more of a sales office these days
[22:44] <Tv|work> ..i think.. my consulting life is a blur ;)
[22:45] <Ormod> but yeah, time to head for bed. g'night. or day probably over there
[22:58] <cmccabe> tv: I thought we killed the configure.ac warnings, but now I see them again?
[22:59] <cmccabe> ormod: good night!
[22:59] <Tv|work> cmccabe: i haven't done anything about them yet
[23:01] * mnigh (~mnigh@75-128-161-124.static.stls.mo.charter.com) has joined #ceph
[23:05] <bchrisman> when ceph -s only returns lines like: '2011-02-09 22:05:08.162337 7f84a9bbc710 -- :/9318 >> pipe(0x7f849c000b00 sd=4 pgs=0 cs=0 l=0).fault first fault', I guess that means it can't talk to the mon?
[23:07] <bchrisman> wondering if this is a fixed or known assertion: http://pastebin.com/EA0YqG75
[23:07] * allsystemsarego (~allsystem@ Quit (Quit: Leaving)
[23:07] <cmccabe> bchrisman: "fault first fault" shows up a lot when there is a communication problem. And yes, it is almost always either a crashed or unresponsive daemon
[23:08] <bchrisman> yeah.. looks like my mon is down… seeing that assert failure in mon log..
[23:10] <cmccabe> can you run "cconf mon data"
[23:11] <bchrisman> returns 'data'
[23:11] <cmccabe> is that the path to a writable directory?
[23:11] <cmccabe> a directory where you can create new files
[23:11] <bchrisman> from root?
[23:11] <bchrisman> yeah
[23:11] <cmccabe> yeah...
[23:11] <bchrisman> oh
[23:12] <bchrisman> root fs full
[23:12] <bchrisman> okay… feh…
[23:12] <bchrisman> too much logging.. 2.4GB in ceph logs.. heh
[23:13] <cmccabe> bchrisman: what branch are you using?
[23:13] <bchrisman> a 0.24 later rc
[23:13] <cmccabe> bchrisman: it looks like the master branch, at least, recently upgraded that error handling path to actually print out the error (in this case, disk full)
[23:13] <bchrisman> ahh ok
[23:14] <bchrisman> cmccabe: cool.. this isn't the issue I'm trying to track down.. and it looks like it's handled better then in the later release.. cool enough
[23:16] <cmccabe> bchrisman: we do generate a lot of logs when logging is turned up. It's probably best to log to a separate partition
[23:17] <cmccabe> bchrisman: the daemons do also handle SIGHUP so logrotate should work. Of course, getting rid of old logs makes debugging harder, so it's a tradeoff.
[23:18] <bchrisman> cmccabe: I'm trying to figure out why the kernel client is getting connection drops to mds… (ie, is the mds dying or whatever)… if I put mds logging up to 20, seems like the output is overkill.
[23:19] <cmccabe> bchrisman: to figure if the mds is dead, check the end of all the mds logs
[23:19] <cmccabe> bchrisman: it should have a stack trace if the mds crashed
[23:19] <cmccabe> bchrisman: also there is the ps-ceph.pl script which will show all the ceph processes running (if you're still on 1 machine)
[23:20] <bchrisman> cmccabe: will do… I don't have to clean up the log files… reimage is a couple minutes… so this will be fin for now..
[23:20] <cmccabe> bchrisman: for multi-node setups, I usually install /etc/init.d/ceph and do "/etc/init.d/ceph status"
[23:21] <bchrisman> ahh okay… yeah.. I use that init script as well...
[23:21] <cmccabe> bchrisman: I think init-ceph assumes that you have one daemon per machine
[23:21] <bchrisman> hmm...
[23:22] <cmccabe> bchrisman: actually, we should fix that. I remember sage recommended running a cosd per disk earlier
[23:22] <bchrisman> cmccabe: yeah.. taht's what I'm running.
[23:22] <cmccabe> bchrisman: so multiple daemons per node is clearly a supported configuration
[23:22] <bchrisman> cmccabe: I've been just using pstree to view visually..
[23:51] * verwilst (~verwilst@dD576FAAE.access.telenet.be) has joined #ceph
[23:59] * uwe (~uwe@ip-94-79-145-210.unitymediagroup.de) Quit (Quit: sleep)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.