#ceph IRC Log


IRC Log for 2012-01-25

Timestamps are in GMT/BST.

[0:01] * adjohn is now known as Guest395
[0:01] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[0:07] * Guest395 (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Ping timeout: 480 seconds)
[0:15] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Read error: Connection reset by peer)
[0:15] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[0:37] * ceph (~hylick@ Quit (Quit: Leaving.)
[0:43] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Remote host closed the connection)
[0:47] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Remote host closed the connection)
[0:47] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[0:48] * weetabeex (~joao@ Quit (Quit: weetabeex)
[0:55] * mtk (~mtk@ool-44c35967.dyn.optonline.net) Quit (Remote host closed the connection)
[1:47] * yoshi (~yoshi@p9224-ipngn1601marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[1:50] * amichel (~amichel@ has joined #ceph
[1:55] <amichel> I've just got a sort of ignorant question about multi-device btrfs filesystems and how one would lay out a 45 disk server for ceph OSD at that level. Anyone around that might have some knowledge to drop on me in that area?
[2:12] * aa (~aa@r190-135-16-151.dialup.adsl.anteldata.net.uy) has joined #ceph
[2:19] * bchrisman (~Adium@ Quit (Quit: Leaving.)
[2:21] * amichel (~amichel@ has left #ceph
[3:12] * jojy (~jvarghese@ Quit (Quit: jojy)
[3:13] * aa (~aa@r190-135-16-151.dialup.adsl.anteldata.net.uy) Quit (Quit: Konversation terminated!)
[3:13] * aa (~aa@r190-135-16-151.dialup.adsl.anteldata.net.uy) has joined #ceph
[3:23] * amichel (~amichel@ has joined #ceph
[3:30] * joshd (~joshd@aon.hq.newdream.net) Quit (Quit: Leaving.)
[3:45] * andresambrois (~aa@r186-52-178-254.dialup.adsl.anteldata.net.uy) has joined #ceph
[3:52] * aa (~aa@r190-135-16-151.dialup.adsl.anteldata.net.uy) Quit (Ping timeout: 480 seconds)
[3:53] * amichel (~amichel@ Quit (Quit: Bad news, everyone!)
[3:57] * chutzpah (~chutz@ Quit (Quit: Leaving)
[4:07] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Quit: adjohn)
[4:09] * andresambrois (~aa@r186-52-178-254.dialup.adsl.anteldata.net.uy) Quit (Quit: Konversation terminated!)
[4:09] * andresambrois (~aa@r186-52-178-254.dialup.adsl.anteldata.net.uy) has joined #ceph
[4:50] * andresambrois (~aa@r186-52-178-254.dialup.adsl.anteldata.net.uy) Quit (Ping timeout: 480 seconds)
[6:03] * barkycowbell (~barkycowb@ has joined #ceph
[6:04] * barkycowbell (~barkycowb@ has left #ceph
[6:37] * sage (~sage@cpe-76-94-40-34.socal.res.rr.com) has joined #ceph
[6:40] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[6:55] * wido (~wido@rockbox.widodh.nl) Quit (Remote host closed the connection)
[15:58] -coulomb.oftc.net- *** Looking up your hostname...
[15:58] -coulomb.oftc.net- *** Checking Ident
[15:58] -coulomb.oftc.net- *** No Ident response
[15:58] -coulomb.oftc.net- *** Found your hostname
[15:58] * CephLogBot (~PircBot@rockbox.widodh.nl) has joined #ceph
[15:58] * fronlius (~fronlius@f054098205.adsl.alicedsl.de) has joined #ceph
[16:27] * nolan (~nolan@phong.sigbus.net) Quit (Quit: ZNC - http://znc.sourceforge.net)
[16:28] * nolan (~nolan@phong.sigbus.net) has joined #ceph
[16:34] * ceph (~hylick@ has joined #ceph
[16:42] * bchrisman1 (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[16:43] * ghaskins_ (~ghaskins@68-116-192-32.dhcp.oxfr.ma.charter.com) has joined #ceph
[16:43] * jpieper_ (~josh@209-6-86-62.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com) has joined #ceph
[16:46] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (resistance.oftc.net larich.oftc.net)
[16:46] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) Quit (resistance.oftc.net larich.oftc.net)
[16:46] * aneesh (~aneesh@ Quit (resistance.oftc.net larich.oftc.net)
[16:46] * eternaleye___ (~eternaley@ Quit (resistance.oftc.net larich.oftc.net)
[16:46] * ghaskins (~ghaskins@68-116-192-32.dhcp.oxfr.ma.charter.com) Quit (resistance.oftc.net larich.oftc.net)
[16:46] * Meths (rift@ Quit (resistance.oftc.net larich.oftc.net)
[16:46] * cclien (~cclien@ec2-50-112-123-234.us-west-2.compute.amazonaws.com) Quit (resistance.oftc.net larich.oftc.net)
[16:46] * grape (~grape@ Quit (resistance.oftc.net larich.oftc.net)
[16:46] * acaos_ (~zac@209-99-103-42.fwd.datafoundry.com) Quit (resistance.oftc.net larich.oftc.net)
[16:46] * jpieper (~josh@209-6-86-62.c3-0.smr-ubr2.sbo-smr.ma.cable.rcn.com) Quit (resistance.oftc.net larich.oftc.net)
[16:47] * eternaleye___ (~eternaley@ has joined #ceph
[16:49] * elder (~elder@c-71-195-31-37.hsd1.mn.comcast.net) has joined #ceph
[16:51] * cclien (~cclien@ec2-50-112-123-234.us-west-2.compute.amazonaws.com) has joined #ceph
[16:51] * grape (~grape@ has joined #ceph
[16:51] * acaos (~zac@209-99-103-42.fwd.datafoundry.com) has joined #ceph
[16:52] * Meths (rift@ has joined #ceph
[16:55] * aneesh (~aneesh@ has joined #ceph
[17:02] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[17:17] * grape (~grape@ Quit (Remote host closed the connection)
[17:26] <ceph> can anyone point me in the ceph source to where the metadata for files stored in the file system is extracted (and then cached on the MDS cluster)
[17:38] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) has joined #ceph
[17:43] * bchrisman1 (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:52] * weetabeex is now known as jluis
[17:59] * grape (~grape@ has joined #ceph
[18:08] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[18:25] * jojy (~jvarghese@75-54-228-176.lightspeed.sntcca.sbcglobal.net) has joined #ceph
[18:26] * jojy (~jvarghese@75-54-228-176.lightspeed.sntcca.sbcglobal.net) Quit ()
[18:47] * Kioob (~kioob@luuna.daevel.fr) Quit (Quit: Leaving.)
[18:51] * bchrisman (~Adium@ has joined #ceph
[18:55] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[18:57] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) has joined #ceph
[18:58] * joshd (~joshd@aon.hq.newdream.net) has joined #ceph
[18:59] <gregaf> ceph: I'm not quite sure what you're asking…metadata is stored in objects in RADOS, but it's not "extracted" except in that the MDS reads it out of RADOS
[18:59] <gregaf> that happens throughout the MDS system, largely in MDCache.cc
[19:00] <gregaf> Kioob`Taff1: you want to look at each on-disk copy of it or something?
[19:01] <sage> gregaf, etc: will miss the standup this morning. will be on irc later though
[19:01] <gregaf> there's not a great way still but if you've got a log you can find the inode and then search for that to turn up the objects it's stored in...
[19:01] <gregaf> roger that, sage
[19:08] <ceph> gregaf: thank you. i will look in MDCache.cc because that is part of what i am looking for. i am also interested in where/how/when the metadata contents of each directory are stored in objects
[19:09] <gregaf> each directory is an object which stores all the inodes in that directory, along with some extra directory-specific metadata…
[19:09] <gregaf> changes are written to a journal (which is just striped along objects) and flushed out to the permanent directory objects every so often
[19:10] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[19:11] <ceph> if i was to copy some new data into the file system, what (or who, which file) initiates the metadata object creation for each directory copied?
[19:13] <gregaf> I don't remember what the specific functions are, but you would start in Server.cc::handle_client_mkdir :)
[19:13] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Read error: Connection reset by peer)
[19:13] <gregaf> be back in a bit!
[19:13] <ceph> ok. thanks
[19:15] * lx0 (~aoliva@lxo.user.oftc.net) has joined #ceph
[19:20] * chutzpah (~chutz@ has joined #ceph
[19:42] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[20:12] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[20:26] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Remote host closed the connection)
[20:28] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[20:31] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Read error: Connection reset by peer)
[20:32] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[20:54] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Read error: Connection reset by peer)
[20:54] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[21:15] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Quit: adjohn)
[21:24] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) Quit (Quit: Konversation terminated!)
[21:24] * aa (~aa@r200-40-114-26.ae-static.anteldata.net.uy) has joined #ceph
[21:30] * lollercaust (~paper@179.Red-88-11-190.dynamicIP.rima-tde.net) has joined #ceph
[21:36] * s15y1 (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[22:06] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[22:16] <sage> elder: you saw http://tracker.newdream.net/issues/1990 right?
[22:17] <elder> Yes, Josh pointed it out.
[22:17] <elder> I haven't investigated any further than that thoug.
[22:18] <elder> Also note the "doesn't exist any more" which is the problem with the rebasing that I don't like very much :)
[22:20] <sage> elder: hrm yeah
[22:24] * fronlius_ (~fronlius@e176057155.adsl.alicedsl.de) has joined #ceph
[22:28] * The_Bishop (~bishop@cable-89-16-138-109.cust.telecolumbus.net) Quit (Ping timeout: 480 seconds)
[22:30] * fronlius (~fronlius@f054098205.adsl.alicedsl.de) Quit (Ping timeout: 480 seconds)
[22:30] * fronlius_ is now known as fronlius
[22:31] * The_Bishop (~bishop@cable-89-16-138-109.cust.telecolumbus.net) has joined #ceph
[22:33] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) Quit (Quit: Leaving)
[22:39] <lxo> sage, gregaf, been doing some testing with snapshot timestamps *and* mds loss of dir layout info. it seems that the problems are related
[22:40] <sage> lxo: that sounds right
[22:40] <lxo> AFAICT when the mds kicks a dir out of the cache, when it reads things back, it loses layout info, but the snapshot timestamps are right
[22:40] <sage> lxo: i think in both cases the fix is to journal the old_inode content
[22:40] <sage> lxo: er oh, hmm!
[22:41] <lxo> however, when the mds recovers that info from the mds journal, it gets the timestamps wrong, even though it seems to get dir layout info right
[22:41] <sage> lxo: sounds like journaling of layout is correct, writeback is not; journaling of old_inode is wrong, writeback is correct.
[22:41] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) has joined #ceph
[22:42] <lxo> so I'm inclined to believe that we don't need to journal old_inodes, we just have to deal properly with creation of snapshots when replaying the journal, and review how layout info is stored in/recovered from dir objects
[22:42] <lxo> yep, that's a shorter way to put it
[22:43] <lxo> now, as for fixing old_inodes, I still don't see why we have to do that (don't know enough about the mds journal yet), but it appears to me that the journal should already have all the necessary info
[22:43] <sage> lxo: i think we need to journal old_inodes either way..
[22:44] <elder> sage, your last btrfs hack commit added a build warning because you inserted a printf prior to a local variable declaration in a block.
[22:44] <lxo> as in, if the previous state of the snapshotted directory is in the journal, we can use that info when replaying (?) a snapshot-creation op; if it's not in the journal, we can recover it from the osd dir object
[22:44] <elder> Code still works though--it's legal C now, just wasn't before hence the warning.
[22:44] <sage> lxo: sort of.. the way the mds works snapshots may be updated after they are cowed because client writeback is asynchronous
[22:45] <elder> sage, I am going to recommit those patches soon I think so I will fix it for you (unless you object).
[22:45] <sage> elder: k. we can clean it up or ignore.. it was just for the next qa run so i can see wehre einval is coming from
[22:46] <lxo> as for recovering the info, I don't see that journal old_inodes gains us any info; AFAICT we could avoid the misbehavior by refraining from assuming old_inodes timestamps are the same as the updated dir, and instead fetch them from disk even though the updated dir info is in the journal already
[22:47] <lxo> I can see this would change snapshot data, but metadata?!?
[22:48] <elder> sage, already fixed in my local area; I'll fix it if/when I commit these changes.
[22:48] <sage> there's more in old_inodes beyond mtime: recursive accounting stats, for instance. we don't learn that until potentially well after the snapshot is created (when clients finish writeback), so even tho it's in a different time bucket, it can change going forward, and is part of the directory state.
[22:49] <sage> ...and all such state is journaled when it is updated.
[22:49] <lxo> is there any way to force an mds to fully flush the cache and the journal, and then restart/quit? this would be a nice way to reliably test what I'm observing empirically
[22:49] <sage> lxo: i think 'ceph mds stop 0' will make it flush and shut down
[22:50] <sage> lxo: but fwiw i'm 99% sure the lack of old_inode journaling is an oversight/bug. fixing that should make that half of your problem go away
[22:50] <lxo> will that empty the journal too, or should I do that separately? (or is there some other way to ensure the contents of the journal have made to dirs)
[22:51] <lxo> I believe that, but I think it's overkill. the fact that I don't see subdirectories' timestamps become wrong after the snapshot is taken makes me a bit doubtful that this is necessary. I feel I'm still missing something
[22:57] * diegows (~diegows@50-57-106-86.static.cloud-ips.com) Quit (Ping timeout: 480 seconds)
[23:07] * fghaas (~florian@85-127-92-127.dynamic.xdsl-line.inode.at) has left #ceph
[23:16] * adjohn is now known as Guest505
[23:16] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[23:22] * Guest505 (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Ping timeout: 480 seconds)
[23:31] <sage> i accidentally clobbered the -b qa run .. scheduled a new one.
[23:35] * fronlius (~fronlius@e176057155.adsl.alicedsl.de) Quit (Quit: fronlius)

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.