#ceph IRC Log


IRC Log for 2011-12-13

Timestamps are in GMT/BST.

[0:01] <todin> cp: you could check out an old version from the git repro, or get an tar.gz http://ceph.newdream.net/download/
[0:07] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Remote host closed the connection)
[0:28] <cp> todin: thanks
[1:02] * aa (~aa@r190-135-31-128.dialup.adsl.anteldata.net.uy) has joined #ceph
[2:28] * Tv (~Tv|work@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[2:40] * adjohn (~adjohn@ Quit (Read error: Connection reset by peer)
[2:40] * adjohn (~adjohn@ has joined #ceph
[2:50] * adjohn (~adjohn@ Quit (Ping timeout: 480 seconds)
[3:04] * bchrisman (~Adium@ Quit (Quit: Leaving.)
[3:15] * joshd (~joshd@aon.hq.newdream.net) Quit (Quit: Leaving.)
[3:16] * cp (~cp@76-220-17-197.lightspeed.sntcca.sbcglobal.net) Quit (Quit: cp)
[3:39] * buck (~buck@bender.soe.ucsc.edu) has joined #ceph
[3:40] <buck> is it common knowledge that you have to use the host name returned by 'hostname' to configure ceph and not a different hostname that resolves to another IP on the same box (think private network) ?
[3:45] * adjohn (~adjohn@70-36-197-80.dsl.dynamic.sonic.net) has joined #ceph
[3:48] * aa (~aa@r190-135-31-128.dialup.adsl.anteldata.net.uy) Quit (Quit: Konversation terminated!)
[3:48] * aa (~aa@r190-135-31-128.dialup.adsl.anteldata.net.uy) has joined #ceph
[4:04] * adjohn (~adjohn@70-36-197-80.dsl.dynamic.sonic.net) Quit (Quit: adjohn)
[4:06] * adjohn (~adjohn@70-36-197-80.dsl.dynamic.sonic.net) has joined #ceph
[4:35] * adjohn (~adjohn@70-36-197-80.dsl.dynamic.sonic.net) Quit (Quit: adjohn)
[4:55] * adjohn (~adjohn@70-36-197-80.dsl.dynamic.sonic.net) has joined #ceph
[5:10] * adjohn (~adjohn@70-36-197-80.dsl.dynamic.sonic.net) Quit (Quit: adjohn)
[5:23] * cp (~cp@c-98-234-218-251.hsd1.ca.comcast.net) has joined #ceph
[5:25] * cp (~cp@c-98-234-218-251.hsd1.ca.comcast.net) Quit ()
[6:13] * sjustlaptop (~sam@96-41-121-194.dhcp.mtpk.ca.charter.com) has joined #ceph
[6:40] * sjustlaptop (~sam@96-41-121-194.dhcp.mtpk.ca.charter.com) Quit (Ping timeout: 480 seconds)
[6:42] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[7:20] * votz (~votz@pool-108-52-122-97.phlapa.fios.verizon.net) Quit (Quit: Leaving)
[7:20] * votz (~votz@pool-108-52-122-97.phlapa.fios.verizon.net) has joined #ceph
[7:43] * The_Bishop (~bishop@port-92-206-76-165.dynamic.qsc.de) has joined #ceph
[8:31] * lxo (~aoliva@lxo.user.oftc.net) Quit (Quit: later)
[8:59] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) has joined #ceph
[9:08] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) has left #ceph
[9:13] * Lo-lan-do (~roland@mirenboite.placard.fr.eu.org) has joined #ceph
[10:10] * Lo-lan-do (~roland@mirenboite.placard.fr.eu.org) has left #ceph
[10:19] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) has joined #ceph
[10:21] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) Quit ()
[10:22] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) has joined #ceph
[10:46] * Sargun (~sargun@208-106-98-2.static.sonic.net) has joined #ceph
[10:46] <Sargun> Hey
[10:53] <fghaas> Sargun: lemme guess, rack awareness? :)
[10:58] * andresambrois (~aa@r186-52-143-67.dialup.adsl.anteldata.net.uy) has joined #ceph
[11:00] <Sargun> fghaas: I know you guys have CRUSH maps.
[11:01] <Sargun> fghaas: I currently have glusterfs in production for basic shared storage
[11:01] <Sargun> I'm wondering if Ceph might be a choice for backing my vm stores
[11:01] <Sargun> This CRUSH stuff, and RBD seems kinda neat
[11:01] <fghaas> seen this I suppose? http://berrange.com/posts/2011/10/12/setting-up-a-ceph-cluster-and-exporting-a-rbd-volume-to-a-kvm-guest/
[11:02] * aa (~aa@r190-135-31-128.dialup.adsl.anteldata.net.uy) Quit (Ping timeout: 480 seconds)
[11:03] <Sargun> yep
[11:59] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Read error: Connection reset by peer)
[11:59] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[11:59] * fronlius_ (~fronlius@testing78.jimdo-server.com) has joined #ceph
[12:10] * bchrisman1 (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[12:12] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[13:40] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[13:59] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[14:06] * fronlius (~fronlius@testing78.jimdo-server.com) has joined #ceph
[14:06] * fronlius_ (~fronlius@testing78.jimdo-server.com) Quit (Read error: Connection reset by peer)
[14:09] * fronlius_ (~fronlius@testing78.jimdo-server.com) has joined #ceph
[14:09] * fronlius (~fronlius@testing78.jimdo-server.com) Quit (Read error: Connection reset by peer)
[14:09] * fronlius_ is now known as fronlius
[14:31] * `gregorg` (~Greg@ has joined #ceph
[14:31] * gregorg_taf (~Greg@ Quit (Read error: Connection reset by peer)
[15:52] * MikeP (~Talan@ Quit (Read error: Connection reset by peer)
[15:52] * MikeP (~Talan@ has joined #ceph
[16:43] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[16:55] <fghaas> tarballs in http://ceph.newdream.net/download/ -- do you build those with git archive, or with make dist?
[17:01] * adjohn (~adjohn@70-36-197-80.dsl.dynamic.sonic.net) has joined #ceph
[17:17] * andresambrois (~aa@r186-52-143-67.dialup.adsl.anteldata.net.uy) Quit (Remote host closed the connection)
[17:41] * elder (~elder@c-71-193-71-178.hsd1.mn.comcast.net) has joined #ceph
[17:42] * adjohn (~adjohn@70-36-197-80.dsl.dynamic.sonic.net) Quit (Quit: adjohn)
[17:43] * lxo (~aoliva@lxo.user.oftc.net) Quit (Remote host closed the connection)
[17:44] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[17:44] * elder (~elder@c-71-193-71-178.hsd1.mn.comcast.net) Quit (Read error: Operation timed out)
[17:47] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) Quit (Ping timeout: 480 seconds)
[17:47] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[17:54] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) has joined #ceph
[17:58] * elder (~elder@c-71-193-71-178.hsd1.mn.comcast.net) has joined #ceph
[17:58] * bchrisman1 (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[18:09] * elder (~elder@c-71-193-71-178.hsd1.mn.comcast.net) Quit (Quit: Leaving)
[18:12] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) Quit (Ping timeout: 480 seconds)
[18:25] * Tv (~Tv|work@aon.hq.newdream.net) has joined #ceph
[18:41] * joshd (~joshd@aon.hq.newdream.net) has joined #ceph
[18:54] <sagewk> fghaas: make dist
[18:56] * bchrisman (~Adium@ has joined #ceph
[19:02] * fronlius (~fronlius@testing78.jimdo-server.com) Quit (Quit: fronlius)
[19:08] * adjohn (~adjohn@ has joined #ceph
[19:48] * fronlius (~fronlius@d219230.adsl.hansenet.de) has joined #ceph
[20:32] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) has joined #ceph
[21:13] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Quit: Konversation terminated!)
[21:13] * eugene (~eugenes@173-166-24-193-newengland.hfc.comcastbusiness.net) has joined #ceph
[21:13] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[21:21] * adjohn (~adjohn@ Quit (Remote host closed the connection)
[21:21] * adjohn (~adjohn@ has joined #ceph
[21:28] * adjohn (~adjohn@ Quit (Quit: adjohn)
[21:42] * verwilst (~verwilst@dD5769656.access.telenet.be) has joined #ceph
[21:50] * The_Bishop (~bishop@port-92-206-76-165.dynamic.qsc.de) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[21:50] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[21:55] <yehudasa_> Tv: I'm still trying to figure out how s3tests.py conf supposed to work
[21:56] <Tv> yehudasa_: can you ask a question?
[21:56] <yehudasa_> Tv: working on it
[21:58] <yehudasa_> Tv: we use s3tests_conf to hold the configuration. We first create an empty object and then we're supposed to fill it in with stuff from the actual config
[21:58] <yehudasa_> Tv: then we do a setdefault
[21:59] <yehudasa_> Tv: so that everything that was not set is getting the default config
[21:59] <yehudasa_> Tv: I'm having trouble finding where we do the second stage.. where we fill in stuff from the actual config
[22:00] <Tv> yehudasa_: are you talking about functional tests or benchmarks?
[22:01] <yehudasa_> Tv: teuthology task
[22:01] <yehudasa_> s3tests.py
[22:01] <Tv> alright trying to understand what you mean by second stage
[22:03] <yehudasa_> Tv: how is the configuration applied into s3tests_conf
[22:04] <Tv> yehudasa_: in the configure function
[22:04] <Tv> lambda: configure(ctx=ctx, config=dict(
[22:04] <Tv> clients=config,
[22:04] <Tv> s3tests_conf=s3tests_conf,
[22:04] <Tv> )),
[22:04] <Tv> for client, properties in config['clients'].iteritems():
[22:04] <Tv> and assign things to s3tests_conf
[22:05] <Tv> and then write it out
[22:05] <yehudasa_> Tv: but that's being called after users were already created
[22:06] <yehudasa_> Tv: so user creation just uses the defaults
[22:06] <Tv> for *what*
[22:07] <Tv> hostname of the rgw doesn't matter for radosgw_admin
[22:07] <yehudasa_> Tv: def create_users(ctx, config)
[22:07] <Tv> default for what
[22:07] <yehudasa_> Tv: user info defaults
[22:07] <Tv> for section, user in [('s3 main', 'foo'), ('s3 alt', 'bar')]:
[22:07] <Tv> _config_user(s3tests_conf, section, '{user}.{client}'.format(user=user, client=client))
[22:08] <Tv> we can do this off irc as soon as i can complete booking these flights, hold on
[22:14] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Remote host closed the connection)
[22:18] <fghaas> Tv or anyone else who knows -- the tarballs in http://ceph.newdream.net/download, do you build those with git archive, or with make dist?
[22:20] <gregaf> sage said make dist
[22:23] * adjohn (~adjohn@ has joined #ceph
[22:23] <fghaas> gregaf: thanks
[22:26] <fghaas> and the general expectation is, of course, that "rpmbuild -tb" on the tarball works, correct?
[22:27] <fghaas> (I ask because a guy in #opensuse-project on freenode ran into problems building that way)
[22:28] <Tv> fghaas: yes on -tb
[22:28] <fghaas> thanks
[22:28] <Tv> fghaas: ever since c16241fd41d0ac3f993cbd6d608fb835f5f77ac8
[22:29] <Tv> fghaas: i tested that on centos 6
[22:29] <Tv> fghaas: oh wait i forget what -ta vs -tb does.. i tested -ta
[22:29] <fghaas> only difference is -ta also builds the srpm
[22:30] <Tv> well then it should work, barring individual issues with e.g. libraries etc
[22:30] <fghaas> right, so evidently c16241fd41d0ac3f993cbd6d608fb835f5f77ac8 made it into 0.35, and that guy is reporting issues on 0.39 ... odd
[22:36] <ajm> http://pastebin.com/CvdmU7P3
[22:36] <ajm> anyone seen anything like this? mds terminates when I access specific directories
[22:37] * cp (~cp@76-220-17-197.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[22:38] <Tv> ajm: that's an awfully short traceback :(
[22:38] <gregaf> if you can turn up mds logging and trigger it again we might be able to work something out
[22:39] <ajm> k
[22:43] <ajm> -rw-r--r-- 1 adam adam 332M Dec 13 16:43 ceph.mds15crash.log
[22:44] <ajm> that one is bigger :)
[22:47] <gregaf> ajm: can you post it somewhere for us?
[22:48] <ajm> bzipping
[22:48] <ajm> http://adam.gs/ceph.mds15crash.log.bz2
[22:52] * The_Bishop (~bishop@port-92-206-76-165.dynamic.qsc.de) has joined #ceph
[23:02] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[23:06] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) Quit (Ping timeout: 480 seconds)
[23:07] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[23:14] <gregaf> ajm: all right, I've grabbed it — it looks like that directory is pretty large?
[23:16] <ajm> in terms of the size of the contents or the # of files?
[23:16] <ajm> it has a few hundred files and might be 10s of gbs
[23:16] <ajm> but I have much larger directories with more files in them elsewhere
[23:16] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Remote host closed the connection)
[23:17] <gregaf> number of files
[23:17] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[23:18] <gregaf> not that it should be a problem, it's just that there's some sort of encoding/decoding problem so I want to know the size of the on-disk directory object is plausible at ~3MB :)
[23:18] <ajm> yeah it sounds plausible
[23:22] <gregaf> well crap; looks like it's corrupted on disk
[23:23] <gregaf> have you had any OSD failures?
[23:23] <ajm> i had one or two issues with osd'es yeah
[23:23] <ajm> but all replicated around the failures
[23:24] <gregaf> yeah; I'm wondering if the primary OSD got some bad stuff on disk
[23:24] <gregaf> ( and the replicas could still be good in that case, depending on what happened)
[23:25] <ajm> how would I check if they differ
[23:25] <gregaf> grab a copy of the object from each OSD and run diff on them ;)
[23:25] <ajm> hrm, which object though :)
[23:25] <gregaf> 10000ad8995.00000000 is the object
[23:26] <gregaf> it's being read from osd.10 right now
[23:26] <gregaf> I think 1.3e55821a is the PG that it's in, so you can check out who the other replicas are
[23:27] <gregaf> (sorry about the spelunking)
[23:27] <ajm> np
[23:28] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) has joined #ceph
[23:28] <ajm> 1.3e5 2063 0 0 0 9589 8852147 216 216 active+clean 67999'38880 109024'109236 [8,7] [8,7] 67999'38880 2011-12-13 00:18:21.823350
[23:28] <ajm> I see 1.35e on 8,7 not 10 ?
[23:30] <gregaf> umm
[23:30] <gregaf> are there are any others that start with 1.3e5?
[23:31] <gregaf> and is your cluster currently stable or are PGs moving around?
[23:31] <ajm> no and its stable at the moment
[23:31] <ajm> 2011-12-13 17:31:27.935059 mon.0 -> 'HEALTH_OK' (0)
[23:32] <gregaf> what's ceph -s?
[23:32] <ajm> 2011-12-13 17:32:35.926965 pg v4014369: 3406 pgs: 3406 active+clean; 22750 GB data, 45914 GB used, 46282 GB / 92197 GB avail
[23:32] <ajm> 2011-12-13 17:32:35.935837 mds e23607: 1/1/1 up {0=15=up:active}, 1 up:standby, 3 up:oneshot-replay(laggy or crashed)
[23:32] <ajm> 2011-12-13 17:32:35.935877 osd e110732: 11 osds: 11 up, 11 in
[23:32] <ajm> 2011-12-13 17:32:35.935978 log 2011-12-13 17:30:45.904926 osd.9 341 : [INF] 0.5c scrub ok
[23:32] <ajm> 2011-12-13 17:32:35.936060 mon e1: 3 mons at {13=,14=,15=}
[23:35] <gregaf> can you paste your pg dump and osd dump?
[23:35] <ajm> sure
[23:37] * verwilst (~verwilst@dD5769656.access.telenet.be) Quit (Quit: Ex-Chat)
[23:37] <ajm> gregaf: http://adam.gs/dump.txt
[23:40] <gregaf> all right, I need to talk to some other people to get into this, but it looks like it might be an OSD issue so it'll be soon :)
[23:42] <ajm> ok
[23:42] <ajm> if you can narrow down I actually do have some older copies of the raw osd data
[23:42] <ajm> so i can possibly find the bad/missing objects
[23:43] * fghaas (~florian@85-127-155-32.dynamic.xdsl-line.inode.at) has left #ceph
[23:44] <gregaf> ajm: actually it's PG 1.21a, sorry for the confusion
[23:44] <gregaf> that maps to OSDs 10 and 6
[23:48] <gregaf> ajm: if they are different and you have any OSD logging enabled we'll be interested in seeing those, too
[23:48] <gregaf> and you should make a copy of the primary's object and then replace it with the replica's and see if that makes it work
[23:48] <ajm> md5 sums differ
[23:49] <ajm> is that sufficent that they are definitely different or is there some per-mds data
[23:49] <ajm> err, per osd data
[23:50] <sjust> ajm: an md5 of the content of the file should match
[23:50] <ajm> ok, definitely doesn't then
[23:50] <ajm> i'll try replacing the primary with the secondary a bit later
[23:50] <ajm> i'd rather not crash the MDS right now
[23:51] <ajm> any idea how this occured? I have at least a few places where this happened :(
[23:52] <sjust> ajm: so far, no, but we are looking into it
[23:52] <ajm> k
[23:53] <ajm> unfortunately I don't run the OSD's with any debug logging enabled (its a lot of logging)
[23:53] <sjust> ajm: yeah
[23:56] <gregaf> well with what we have we can't really tell, unfortunately — the proper fix to this of course is more complete checksumming throughout the system

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.