#ceph IRC Log

Index

IRC Log for 2012-03-07

Timestamps are in GMT/BST.

[0:12] * phil__ (~quassel@chello080109010223.16.14.vie.surfer.at) has joined #ceph
[0:16] * phil_ (~quassel@chello080109010223.16.14.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[0:22] * aliguori (~anthony@32.97.110.59) Quit (Remote host closed the connection)
[0:34] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[0:38] * Tv|work (~Tv_@aon.hq.newdream.net) Quit (Ping timeout: 480 seconds)
[1:02] <sagewk> should i make --concise the default for the 'ceph' command?
[1:02] <sagewk> i'm ready to lose the mon <- message, mon -> reply crap
[1:04] * yoshi (~yoshi@p8031-ipngn2701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[1:05] <joshd1> sagewk: sounds good to me
[1:07] <sagewk> yay, wip_omap!
[1:08] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[1:13] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) has joined #ceph
[1:30] <sagewk> anyone want to look over wip-doc? there are improvements to 'ceph health', and then some quick docs about how to use the new osd cluster introspection stuff we added.
[1:32] <joshd1> sagewk: looking
[1:33] <Volley> just wanted to ask: i try to follow http://ceph.newdream.net/docs/latest/ops/install/mkcephfs/ and i wonder if i shouldn't see some processes after /etc/init.d/ceph start, and ceph ... health just sits there doing nothing ...
[1:33] <sagewk> the init script only starts daemons if the host = line in the ceph.conf matches the local host name (`hostname`)
[1:34] <Volley> hmmm
[1:34] <sagewk> ceph health will hang until the ceph-mon processes are up and running
[1:34] <Volley> argh ...
[1:35] <Volley> yeah, hostname had first char in uppercase, while ... well you can guess the rest. thanks for the hint
[1:36] <sagewk> np
[1:37] <Volley> yeah, looks way better now
[1:40] * BManojlovic (~steki@212.200.240.216) Quit (Remote host closed the connection)
[1:42] <sagewk> http://ceph.newdream.net/docs/wip-doc/ops/manage/failures/
[1:59] <Volley> ok, i get HEALTH OK now ... would next step allready be http://ceph.newdream.net/wiki/Mounting_the_file_system ?
[2:01] <Volley> which mentions cauthtool ... which i didn't yet find ...
[2:02] <joshd1> Volley: everything was renamed awhile ago from c* to ceph-*, so it's ceph-authtool now
[2:03] <Volley> aaah - i see ... and i assume the wiki docu is a bit old sometimes and the other docu is ... not exactely done :)
[2:03] <joshd1> exactly
[2:07] <Volley> looks like it worked ... "mount: error writing /etc/mtab: Invalid argument" is to be ignored?
[2:08] <joshd1> haven't seen that one myself, sounds suspicious
[2:08] <joshd1> can you read/write to it fine?
[2:08] <Volley> feels so
[2:10] <joshd1> maybe just a permissions error writing to /etc/mtab being turned into EINVAL by mount?
[2:11] <Volley> but /etc/mtab did get the line ...: 192.168.0.5:6789:/ /mnt/test ceph rw,relatime,name=admin,secret=<hidden> 0 0
[2:12] <joshd1> huh, weird
[2:13] <iggy> /etc/mtab linked to /proc/mounts maybe
[2:14] <Volley> checking ... indeed ...
[2:20] <Volley> ok, i guess i will experiment quite a bit the next days - just one thing: how ... far can i trust ceph allready? how likely is a data loss, and how likely is the need to setup everything fresh?
[2:21] <iggy> i think the FS is still not suggested for general purpose use
[2:21] <iggy> the other bits i think are safe
[2:21] <iggy> double check that though
[2:22] <Volley> i consider testing with a handful of linux pcs ( debian wheezy, kernel 3.2 ) with uncritical and backuped data - but testing without actually using is kinda ... kinda ... how to explain ... :)
[2:23] <iggy> yeah... we've all been there
[2:24] <Volley> and then? :)
[2:25] <iggy> i haven't crossed that bridge yet
[2:25] <iggy> some people are testing with mirrored data
[2:25] <iggy> and some with reproduceable/generated data
[2:25] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) Quit (Ping timeout: 480 seconds)
[2:26] <Volley> ok, so it seems i need to be very very careful...
[2:26] <iggy> basically yeah
[2:28] * bchrisman (~Adium@108.60.121.114) Quit (Quit: Leaving.)
[2:29] <Volley> hmm ... can i expect to notice when things go wrong? i mean - can i consider silent data corruption or loss as unlikely enough?
[2:29] <Volley> for example i'm using btrfs for like a year now, and i did know the risks, i did run into some issues, but i did notice when things went wrong
[2:38] <iggy> that i'm unsure of
[2:41] <Volley> well, i will probably find out :)
[2:41] <Volley> thanks for your help, time for me to go off ... and i have lots to read and learn and test the next days
[2:41] * Volley (~worf@chello080109200187.3.sc-graz.chello.at) Quit (Quit: Konversation terminated!)
[3:28] * phil__ (~quassel@chello080109010223.16.14.vie.surfer.at) Quit (Remote host closed the connection)
[3:54] * dmick1 (~dmick@aon.hq.newdream.net) Quit (Quit: Leaving.)
[4:14] * joshd1 (~joshd@aon.hq.newdream.net) Quit (Quit: Leaving.)
[4:14] * pruby (~tim@leibniz.catalyst.net.nz) Quit (Read error: Operation timed out)
[4:26] * pruby (~tim@leibniz.catalyst.net.nz) has joined #ceph
[4:26] * chutzpah (~chutz@216.174.109.254) Quit (Quit: Leaving)
[5:43] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) has joined #ceph
[6:59] * Tv__ (~tv@cpe-24-24-131-250.socal.res.rr.com) Quit (Ping timeout: 480 seconds)
[7:00] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) Quit (Quit: Leaving.)
[7:56] * tnt_ (~tnt@37.191-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[9:16] * tnt_ (~tnt@37.191-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[9:27] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[9:35] * Volley (~worf@chello080109200187.3.sc-graz.chello.at) has joined #ceph
[10:15] * stxShadow (~jens@p4FFFED42.dip.t-dialin.net) has joined #ceph
[10:15] * yoshi (~yoshi@p8031-ipngn2701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[10:17] <stxShadow> good morning all
[10:17] * yoshi (~yoshi@p8031-ipngn2701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[10:17] * yoshi (~yoshi@p8031-ipngn2701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[10:18] <stxShadow> another day another problem
[10:18] <stxShadow> i've shutted our fist mon server down. after that i'm not able to use "rbd map" anymore
[10:18] <stxShadow> libceph: mon0 10.10.10.4:6789 connection failed
[10:18] <stxShadow> libceph: mon1 10.10.10.4:6789 connection failed
[10:18] <stxShadow> libceph: mon2 10.10.10.4:6789 connection failed
[10:18] <stxShadow> libceph: mon1 10.10.10.4:6789 connection failed
[10:19] <stxShadow> -> so the client tries to ask the other mons .... but with the ip of the primary (which is down at that moment)
[10:24] * Volley (~worf@chello080109200187.3.sc-graz.chello.at) Quit (Quit: Konversation terminated!)
[10:43] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) has joined #ceph
[11:00] * tnt_ (~tnt@212-166-48-236.win.be) has joined #ceph
[11:14] * Volley (~worf@hexen.cgv.tu-graz.ac.at) has joined #ceph
[11:16] * phil_ (~quassel@chello080109010223.16.14.vie.surfer.at) has joined #ceph
[11:36] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[12:20] * joao (~JL@89.181.145.13) has joined #ceph
[12:24] <Volley> maybe a stupid question, but would it make sense to add clients that have a significant amount of unused local diskspace as storage node? Would the client benefit from the local storage node? Like would ceph distribute data frequently used from that client to that local storage node?
[12:28] <NaioN> Volley: no
[12:28] <Volley> so i would just get a bigger pool
[12:28] * morse (~morse@supercomputing.univpm.it) Quit (Ping timeout: 480 seconds)
[12:29] <NaioN> Do you turn the clients off?
[12:29] <stxShadow> that would be an interesting feature
[12:30] <NaioN> stxShadow: yeah but hen you would need to alter the placement algoritm singificantly
[12:30] <Volley> well, that would have been the next question. but with "clients" i just meant "systems that mount the filesystem" - it would be interesting tough if it made sense to add systems that are not up up 24/7 but maybe only during worktime
[12:30] <stxShadow> --> we use the ceph nodes as server but also as client (kvm virtualisation)
[12:30] <stxShadow> if a vm moves to a different node
[12:31] <stxShadow> it would be nice to have the data "local"
[12:31] <stxShadow> NaioN, -> i know .... you a thought
[12:31] <stxShadow> -you+just
[12:31] <NaioN> stxShadow: I agree with that, but then the algoritm wouldn't be pseudo-random anymore
[12:32] <NaioN> and that's also one of the string points of the algoritm
[12:32] <NaioN> s/string/strong
[12:32] <stxShadow> sure .... i only would like to discribe our usage case
[12:33] <stxShadow> if its not possible ..... thats ok
[12:33] <NaioN> Volley: with the current algoritm you don't have enough control to get the situation you want (storage for the local client on the local client)
[12:35] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) has joined #ceph
[12:37] <NaioN> Volley: something like XtreemFS could do that
[12:37] * eightyeight (~atoponce@pthree.org) Quit (Ping timeout: 480 seconds)
[12:37] <NaioN> it provides in caching of metadata and data
[12:38] <Volley> i was just thinking of alternatives - possibly adding some cache layer on top of the ceph mount ...
[12:38] <Volley> just came across fscache ... need to read more first tough
[12:39] <Volley> and about XtreemFS - will read about that too, thanks :)
[12:39] <stxShadow> i'm looking forward for the "rbd caching" feature
[12:40] <NaioN> stxShadow: me too :)
[12:40] <stxShadow> NaioN, -> what is your use case ?
[12:40] <NaioN> mainly rbd's
[12:41] <stxShadow> same here .....
[12:41] <NaioN> formatted with ext3 and rsyncd
[12:41] <NaioN> as backup volumes
[12:41] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[12:41] <NaioN> and in the furure for KVM
[12:41] <NaioN> but for KVM you can already use the rbd caching
[12:42] <stxShadow> you mean: rbd_writeback_window
[12:42] <NaioN> yes
[12:42] <stxShadow> we have lost some data with that already .....
[12:42] <stxShadow> :S
[12:43] <NaioN> well that's the disadvantage of write cache :)
[12:43] <stxShadow> mainly the boot sectors of the images are broken after "killing" kvm
[12:43] <NaioN> I use SSD's in the nodes
[12:44] <stxShadow> the partition table is broken ..... or the filesystem superblocks
[12:44] <NaioN> as journal devices but that way it depens on how fast your network is
[12:44] <stxShadow> it would help if kvm flushes the data in case of SIGTERM
[12:45] <stxShadow> SSD
[12:45] <stxShadow> as jounal device here too
[12:45] <stxShadow> data is on 4 disk raid0
[12:45] <NaioN> I just make osd's per disk
[12:46] <stxShadow> but you are right .... our test systems is equipped with ssds to
[12:46] <stxShadow> no data loss there
[12:48] <Volley> found some helpful information regarding the different goals of ceph and xtreemfs ... i start to realize that ceph might be great for the storage needs in our server room somewhen, eventually for local workstations as clients, but most likely not for everything that involves slower networks in between
[12:58] * Volley (~worf@hexen.cgv.tu-graz.ac.at) Quit (Quit: Leaving)
[13:29] * jluis (~JL@89-181-145-13.net.novis.pt) has joined #ceph
[13:34] * joao (~JL@89.181.145.13) Quit (Ping timeout: 480 seconds)
[13:50] * Volley (~worf@chello080109200187.3.sc-graz.chello.at) has joined #ceph
[14:15] * aliguori (~anthony@cpe-70-123-132-139.austin.res.rr.com) has joined #ceph
[14:27] * Volley (~worf@chello080109200187.3.sc-graz.chello.at) Quit (Quit: Konversation terminated!)
[14:41] * Volley (~worf@hexen.cgv.tu-graz.ac.at) has joined #ceph
[14:43] * morse (~morse@supercomputing.univpm.it) Quit (Ping timeout: 480 seconds)
[14:50] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[14:55] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) has joined #ceph
[14:58] * morse (~morse@supercomputing.univpm.it) Quit (Ping timeout: 480 seconds)
[15:05] <darkfader> who of you ceph people will be coming to those "world hosting days" in ruhr, germany?
[15:10] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[15:19] * morse (~morse@supercomputing.univpm.it) Quit (Ping timeout: 480 seconds)
[15:24] <jluis> darkfader, when will that happen?
[15:27] <nhm> darkfader: I know Sage has been travelling all over, so perhaps he will be going.
[15:27] <jluis> it would be nice if sage showed up in this side of the atlantic
[15:27] <jluis> good morning nhm :)
[15:28] <nhm> jluis: good morning. :)
[15:28] * jluis is now known as joao
[15:29] <nhm> let me just sy that ohai is a great name for a project.
[15:29] <joao> where did that come from? :p
[15:30] <nhm> joao: http://wiki.opscode.com/display/chef/Ohai
[15:30] <joao> oh, so it exists
[15:30] <nhm> joao: I've been talking to TV about what we should do to gather system information during test runs.
[15:31] <joao> I honestly thought you'd came up with the name and were going to put it to use :)
[15:31] <joao> nhm, that seems like a good idea
[15:31] <nhm> joao: hehe, nope. It fits for the project though.
[15:32] <joao> I'm curious about what "kernel data" does Ohai captures
[15:32] <nhm> yeah, I think I'm just going to install it and try it out as the docs are kind of sparse.
[15:33] <nhm> biggest thing I dislike about it is that we'd need ruby and ohai on every node we want the infromation from.
[15:34] <nhm> For us that may not be a big deal, but for customers...
[15:36] <joao> well, we're talking about *testing* anyway
[15:37] <joao> not production environment
[15:37] <nhm> joao: yeah, but users do testing too before stuff goes into production.
[15:38] <joao> my point being, I suppose the testing environment should not be as strict both in software requirements, as well as foreseen overhead
[15:38] <nhm> joao: And it would be nice if our tests could easily run in other environments.
[15:38] <joao> true
[15:39] <joao> well, I have to traceback my steps and focus on what I have to do
[15:39] <nhm> what are you doing?
[15:39] <joao> working on the workload generator
[15:40] <nhm> ah, what kind of workload?
[15:40] <joao> but damn... I started reading code this morning and I'm sooo far away from where I began I'm not even sure it's related to the initial thing anymore
[15:40] <joao> http://tracker.newdream.net/issues/2087
[15:41] <joao> right now, I've just been reading ceph's code related with the object store, and testing; but the goal is to implement a workload generator that mimics ceph's workload
[15:42] <nhm> joao: ah, good!
[15:42] <joao> not sure about much more than what's described on that issue though
[15:42] <joao> the last couple of days have been my first experience with ceph so far
[15:43] <nhm> joao: Don't worry, I'm a n00b too. :)
[15:44] <joao> its amazing how I haven't been able to find a single reference to what "kernel data" ohai captures
[15:45] <nhm> maybe they just mean they sweep through proc.
[15:45] <joao> maybe
[15:45] <joao> in any case, custom plugin support could come in hand
[15:46] <nhm> yeah
[15:46] <joao> I can think of a couple of plugins useful to capture bug's and warning's when using btrfs :p
[15:46] <nhm> And there doesn't seem to be much else out there. I hate the idea of bringing in so many additional dependencies, but it might be worth it.
[15:47] <nhm> facter perhaps.
[15:48] <joao> afaict, Ohai depends on Chef, which is also ruby
[15:48] <nhm> yeah
[15:49] <joao> well, there is no way to avoid ruby in this case then
[15:52] <darkfader> joao: i saw something on the #ceph twitter foo
[15:52] <darkfader> 22 march or so
[15:52] <joao> ceph has a twitter account?
[15:53] <joao> how am I only knowing about this now?
[15:53] <joao> :o
[15:53] <darkfader> i don't know how expensive the conference is, if it's pocket money-prices, then i'd go for a visit
[15:53] <darkfader> joao: because you have a life? :)
[15:53] <joao> eheh
[15:56] <joao> the funniest thing though, is that apparently I was already following ceph
[15:56] <joao> go figure...
[16:03] <nhm> information overload. :)
[16:10] * morse (~morse@supercomputing.univpm.it) has joined #ceph
[16:13] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) Quit (Quit: Leaving.)
[16:16] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) has joined #ceph
[16:19] <joao> btw, does anyone know where config options such as "osd_journal_size" are described?
[16:21] <joao> nevermind, I think I found something :)
[16:28] * Volley (~worf@hexen.cgv.tu-graz.ac.at) Quit (Quit: Leaving)
[16:34] * jluis (~JL@89.181.145.13) has joined #ceph
[16:34] * joao (~JL@89-181-145-13.net.novis.pt) Quit (Read error: Connection reset by peer)
[16:34] <jluis> well, I suppose the vpn is still busted to some extent
[16:34] * jluis is now known as joao
[16:34] <joao> nhm, are you able to access any of the sepia machines from home?
[16:35] * Volley (~worf@chello080109200187.3.sc-graz.chello.at) has joined #ceph
[16:36] <nhm> joao: nope
[16:40] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) Quit (Quit: Leaving.)
[16:43] * joao (~JL@89.181.145.13) Quit (Ping timeout: 480 seconds)
[16:44] * jmlowe (~Adium@129-79-195-139.dhcp-bl.indiana.edu) has joined #ceph
[16:49] * joao (~JL@ace.ops.newdream.net) has joined #ceph
[16:58] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[17:00] * stxShadow (~jens@p4FFFED42.dip.t-dialin.net) Quit (Ping timeout: 480 seconds)
[17:03] * stxShadow (~jens@p4FFFED42.dip.t-dialin.net) has joined #ceph
[17:04] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) has joined #ceph
[17:06] * stxShadow (~jens@p4FFFED42.dip.t-dialin.net) Quit (Remote host closed the connection)
[17:07] * lofejndif (~lsqavnbok@83TAADXJM.tor-irc.dnsbl.oftc.net) has joined #ceph
[17:07] * lofejndif (~lsqavnbok@83TAADXJM.tor-irc.dnsbl.oftc.net) Quit (Remote host closed the connection)
[17:11] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[17:12] * lofejndif (~lsqavnbok@9KCAAEIRR.tor-irc.dnsbl.oftc.net) has joined #ceph
[17:16] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[17:19] * lofejndif (~lsqavnbok@9KCAAEIRR.tor-irc.dnsbl.oftc.net) Quit (Quit: Leaving)
[17:44] * tnt_ (~tnt@212-166-48-236.win.be) Quit (Ping timeout: 480 seconds)
[17:55] * tnt_ (~tnt@37.191-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:04] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) Quit (Quit: Leaving.)
[18:13] * sjusthm (~sam@24-205-39-1.dhcp.gldl.ca.charter.com) has joined #ceph
[18:19] * Volley (~worf@chello080109200187.3.sc-graz.chello.at) Quit (Quit: Konversation terminated!)
[18:20] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) has joined #ceph
[18:27] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[18:35] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[18:43] * Tv|work (~Tv_@aon.hq.newdream.net) has joined #ceph
[18:56] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[18:56] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) Quit (Ping timeout: 480 seconds)
[18:59] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) has joined #ceph
[18:59] * mkampe (~markk@aon.hq.newdream.net) has joined #ceph
[19:08] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) has joined #ceph
[19:10] * joshd (~joshd@aon.hq.newdream.net) has joined #ceph
[19:18] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[19:21] * adjohn is now known as Guest5406
[19:21] * Guest5406 (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Read error: Connection reset by peer)
[19:21] * _adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[19:21] * _adjohn is now known as adjohn
[19:28] * chutzpah (~chutz@216.174.109.254) has joined #ceph
[19:34] * dmick (~dmick@aon.hq.newdream.net) has joined #ceph
[20:01] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) Quit (Quit: Leaving.)
[20:11] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) has joined #ceph
[20:13] * phil_ (~quassel@chello080109010223.16.14.vie.surfer.at) Quit (Remote host closed the connection)
[20:16] * LarsFronius (~LarsFroni@testing78.jimdo-server.com) Quit (Quit: LarsFronius)
[20:19] * verwilst (~verwilst@dD5769628.access.telenet.be) has joined #ceph
[20:26] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) Quit (Remote host closed the connection)
[20:26] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) has joined #ceph
[20:31] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) Quit (Quit: Leaving.)
[20:51] * dwm__ wonders if his reply to Tv|work's braindump yesterday made any sense.
[20:53] <dwm__> Hmm, does anyone know how ready RBD is to consider using in anger?
[20:54] <nhm> dwm__: I'd say if you install it on a drive, and throw it at someone, it should work well.
[20:56] <iggy> heh
[20:56] <gregaf1> dwm_: I'm not sure if they're forward-thinking or crazy, but people are using rbd, see the mailing list and eg Piston Cloud
[20:57] <gregaf1> really, it should be doing well, except I'm not sure what to make of the trouble that stxShadow, filoo, et al are running into since we're not seeing similar reports from anybody else
[20:58] <gregaf1> anyway, lunchtime
[21:04] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[21:09] * Jaykra (~Jamie@64-126-89-248.dyn.everestkc.net) has joined #ceph
[21:11] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) Quit (Quit: adjohn)
[21:17] <dwm__> Okay, sounds like it's worth playing with.
[21:17] <Tv|work> dwm__: yes, but i'm too swamped to respond right now
[21:17] <dwm__> (I'm being asked about scalable VM options, libvirt + RBD looks mighty attractive on paper.)
[21:18] <dwm__> Tv|work: Hopefully it's the good kind of busy. :-)
[21:19] <Tv|work> dwm__: it's the new hardware kind of busy ;)
[21:19] <dwm__> Don't worry about replying, just hope it was helpful..
[21:19] <dwm__> Ooh, shiny.
[21:33] * guido (~guido@mx1.hannover.ccc.de) Quit (Remote host closed the connection)
[21:33] * guido (~guido@mx1.hannover.ccc.de) has joined #ceph
[21:39] <dmick> heh: dwm__: no, actually sort of tarnished, sadly :)
[21:40] <jmlowe> dwm__: Ive been using libvirt+qemu+rbd with some success
[21:53] * BManojlovic (~steki@212.200.240.216) has joined #ceph
[21:57] <nhm> libvirt+rbd is definitely interesting.
[22:00] <jmlowe> one caution, make sure your underlying storage is rock solid, I've had all kinds of problems related to hardware failures not bubbling up and triggering failover
[22:00] <sagewk> sjust1: do you have a minute to peek at wip-2098?
[22:00] <sagewk> want to make sure i'm going in the right direction before adding the guards to every op
[22:01] * adjohn (~adjohn@rackspacesf.static.monkeybrains.net) has joined #ceph
[22:02] <dmick> heh, I just was looking at the gitbuilder failure there and Josh and I kicked off a rebuild to make it work
[22:03] <dmick> (to make it rebuild with your fix to the .h)
[22:12] <dwm__> jmlowe: Which FS is that sitting on top of?
[22:13] <dwm__> Given btrfs's continuing development, I'm thinking that XFS might be the more robust foundation -- at least for now.
[22:17] <jmlowe> I've used both btrfs and ext4
[22:17] <iggy> there's some bug with xfs currently http://tracker.newdream.net/issues/2003
[22:18] <sagewk> there's also currently a problem with transaction atomicity on any non-btrfs fs.. will be fixed in v0.44.
[22:18] <sagewk> http://tracker.newdream.net/issues/2098
[22:18] <jmlowe> I just replaced 5/24 disks with predictive failures, and I've finally given up using raid 0 pairs in favor of raid 1, so I'm hoping for better results
[22:19] <sjusthm> sagewk: looking
[22:19] <dwm__> Hmm, fragmentation. btrfs with autodefrag would hopefully cope better with this over time.
[22:19] <dwm__> Sounds like there's room for experimentation.
[22:19] <jmlowe> I experienced read and write errors without the logical drives being marked as bad or going offline, ceph had no idea there was a problem and consequently kept on going like there wasn't anything wrong
[22:20] <jmlowe> I saw extremely high fragmentation with ext4, %94+
[22:20] <dwm__> jmlowe: Hmm, you'd kinda hope the filesystem would offline itself in those circumstances.
[22:20] <jmlowe> on the other hand I was running 16 vm's with iozone -a in while true loops
[22:21] <jmlowe> I know, if the disk went away the osd would go down and failover and replication in ceph should happen
[22:24] * lofejndif (~lsqavnbok@9YYAAEFXT.tor-irc.dnsbl.oftc.net) has joined #ceph
[22:26] <sagewk> sjusthm: just repushed wip-2098, slight change in direction
[22:26] <sjusthm> ok
[22:34] * nhorman (~nhorman@99-127-245-201.lightspeed.rlghnc.sbcglobal.net) Quit (Quit: Leaving)
[22:34] <sjusthm> sagewk: there is one problem, fsync won't guarantee that the omap portion of a clone is complete
[22:34] <sagewk> sjusthm: hrm.. so we need to check if omap keys exist, and if so, sync leveldb?
[22:35] <sjusthm> sagewk: I think so
[22:36] <sjusthm> I think it'll normally just be a fs sync, though
[22:36] <sjusthm> let me look around a bit
[22:37] <sagewk> sjusthm: hrm, i don't see where the object_map is flushed in sync_entry()..
[22:37] <Tv|work> uhh leveldb syncs after every operation (/batch)
[22:37] <Tv|work> right?
[22:38] <Tv|work> oh wait sorry that was configurable
[22:38] <Tv|work> depending on WriteOptions
[22:38] <sagewk> hopefully it's not syncing every time.. :)
[22:39] <Tv|work> personally i've always used the batch thing, and a sync after a batch isn't too bad
[22:39] * MarkDude (~MT@c-71-198-138-155.hsd1.ca.comcast.net) Quit (Quit: Leaving)
[22:41] <sjusthm> Tv|work: that's configurable
[22:41] <sjusthm> as we have it set up, it applies the write to the fs, but does not force a sync
[22:42] <sjusthm> if we have DBObjectMap make link and clone synchronous, we should be ok
[22:42] <sjusthm> I wasn't sure if it would sync the previous operations, but apparently it will
[22:48] <sjusthm> sagewk: that seems ok
[22:48] <sjusthm> wip-2098, that is
[22:50] <sagewk> sjusthm: k thanks. i'll have you guys take another look after i do some basic testing
[22:50] <sjusthm> ok
[23:03] * verwilst (~verwilst@dD5769628.access.telenet.be) Quit (Quit: Ex-Chat)
[23:05] * jmlowe (~Adium@129-79-195-139.dhcp-bl.indiana.edu) Quit (Quit: Leaving.)
[23:05] * jmlowe (~Adium@140-182-216-129.dhcp-bl.indiana.edu) has joined #ceph
[23:13] * jmlowe (~Adium@140-182-216-129.dhcp-bl.indiana.edu) Quit (Ping timeout: 480 seconds)
[23:14] * The_Bishop (~bishop@178-17-163-220.static-host.net) Quit (Quit: Wer zum Teufel ist dieser Peer? Wenn ich den erwische dann werde ich ihm mal die Verbindung resetten!)
[23:23] * stxShadow (~Jens@ip-88-153-224-220.unitymediagroup.de) has joined #ceph
[23:23] <stxShadow> good evening all
[23:23] <sagewk> sjusthm: wip-2098 passing my basic tests. need to throw something harder at it now.
[23:23] <nhm> good evening stxShadow
[23:26] <stxShadow> i wrote a question regarding a mon problem to this channel this morning ...... anyone here who could answer it ?
[23:28] * fhgokrikek (~lsqavnbok@659AAACG7.tor-irc.dnsbl.oftc.net) has joined #ceph
[23:30] * lofejndif (~lsqavnbok@9YYAAEFXT.tor-irc.dnsbl.oftc.net) Quit (Ping timeout: 480 seconds)
[23:32] <stxShadow> [417877.894082] libceph: mon1 10.10.10.4:6789 connection failed
[23:32] <stxShadow> [417887.905142] libceph: mon0 10.10.10.4:6789 connection failed
[23:32] <stxShadow> [417897.906446] libceph: mon2 10.10.10.4:6789 connection failed
[23:32] <stxShadow> why is libceph always trying the same ip ?
[23:32] <Tv|work> stxShadow: can you pastebin your ceph.conf?
[23:33] <stxShadow> sure
[23:33] <dwm__> Hmm, suggsets that the client thinks all the MONs have the same address.
[23:33] <Tv|work> dwm__: i'm hoping to see a simple typo in the config, to make it be like that ;)
[23:34] <stxShadow> http://pastebin.de/24011
[23:35] <stxShadow> -> this only happens if mon0 is down .... mon0 owns the 10.10.10.4 ip
[23:36] <stxShadow> i've tried to use the "rbd map" command
[23:36] <stxShadow> this ended up with an input / output error -> dmesg shows up the messages above
[23:37] <stxShadow> if i remove the mon.0 entry from ceph.conf
[23:37] <Tv|work> hmm, nothing obviously wrong in ceph.conf
[23:37] <stxShadow> he uses mon.1
[23:37] <stxShadow> and everything works ok
[23:37] <Tv|work> stxShadow: wait you say dmesg.. so this is the kernel?
[23:38] * jmlowe (~Adium@c-71-201-31-207.hsd1.in.comcast.net) has joined #ceph
[23:38] <Tv|work> stxShadow: kernel client, i mean.. can you share the mount command? (edit out secret, if you have it there)
[23:38] <sjusthm> sagewk: running thrashing + rados with snaps tends to trigger those bugs pretty quickly
[23:39] <sjusthm> if any subset of sepia is usable
[23:39] <joshd> stxShadow, Tv: there's a typo in rbd map causing it to pass n copies of the first monitor
[23:39] <Tv|work> ahh
[23:40] <stxShadow> ah .... a known bug ?
[23:41] <stxShadow> Tv|work: -> i'm not using "mount" :) only rbd
[23:41] * fhgokrikek (~lsqavnbok@659AAACG7.tor-irc.dnsbl.oftc.net) Quit (Quit: Leaving)
[23:41] <Tv|work> yeah sorry misread
[23:41] * lofejndif (~lsqavnbok@82VAAB9O9.tor-irc.dnsbl.oftc.net) has joined #ceph
[23:42] <sjusthm> sagewk: wip_dbobjectmap_sync adds a sync call to DBObjectMap
[23:42] <joshd> stxShadow: no, you just made me notice it
[23:43] <stxShadow> joshd: thanks for checking ....
[23:43] <stxShadow> ;)
[23:45] <joshd> stxShadow: thanks for reporting it
[23:47] <stxShadow> i hope we will find solutions for our other problems too ;)
[23:48] <stxShadow> its late here now .... i have to go to bed now .... good night everyone !!
[23:48] * stxShadow (~Jens@ip-88-153-224-220.unitymediagroup.de) has left #ceph
[23:55] <Tv|work> ok so the new sepia switch config has remnants of the old DH networking setup in it
[23:56] <Tv|work> now i need to figure out what parts are safe to chop out :-/
[23:57] <Tv|work> aaand it routes across all networks by default, which i definitely do not want.. time to kill it with fire

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.