#ceph IRC Log

Index

IRC Log for 2012-08-03

Timestamps are in GMT/BST.

[0:00] <nhm> ok
[0:00] <nhm> What kind of throughput do you see with only new writes?
[0:00] <sjust> a few hundred iops
[0:00] <sjust> very slow
[0:01] <nhm> yeah, that sounds about like what I see.
[0:01] <nhm> Need to go back and look at blktrace to see how many seeks we are actually doing.
[0:09] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) has joined #ceph
[0:11] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) Quit (Quit: Leaving)
[0:55] <sjust> nhm: most of our problem results have so far been from rados bench and the workload generator, right?
[0:57] <nhm> sjust: And RGW, both with rest-bench and slow downs with the functional tests.
[0:58] <nhm> sjust: it's totally file creation times
[0:58] <nhm> sjust: it's just that we get hit harder by it because when an OSD stalls it means the whole cluster stalls
[0:58] <nhm> I gotta run to dinner, but will be bbl
[0:58] <sjust> ok, we do need to do some improvement there, but I'm going to slap together a tool that will actually give us random write numbers
[1:01] * s[X]_ (~sX]@ppp59-167-154-113.static.internode.on.net) has joined #ceph
[1:05] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) Quit (Read error: Operation timed out)
[1:13] * SuperSonicSound (~SuperSoni@28IAAGJND.tor-irc.dnsbl.oftc.net) has joined #ceph
[1:16] * BManojlovic (~steki@212.200.241.106) Quit (Quit: Ja odoh a vi sta 'ocete...)
[1:23] * Tv_ (~tv@38.122.20.226) Quit (Quit: Tv_)
[1:30] * tnt (~tnt@167.39-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[1:41] * LarsFronius_ (~LarsFroni@dyndsl-080-228-197-038.ewe-ip-backbone.de) has joined #ceph
[1:47] * LarsFronius (~LarsFroni@dyndsl-031-150-008-017.ewe-ip-backbone.de) Quit (Ping timeout: 480 seconds)
[1:47] * LarsFronius_ is now known as LarsFronius
[1:52] * s[X]_ (~sX]@ppp59-167-154-113.static.internode.on.net) Quit (Remote host closed the connection)
[1:53] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) has joined #ceph
[2:01] * JJ (~JJ@12.248.40.138) has left #ceph
[2:02] * LarsFronius (~LarsFroni@dyndsl-080-228-197-038.ewe-ip-backbone.de) Quit (Quit: LarsFronius)
[2:06] * bchrisman (~Adium@108.60.121.114) Quit (Quit: Leaving.)
[2:06] * LarsFronius (~LarsFroni@dyndsl-080-228-197-038.ewe-ip-backbone.de) has joined #ceph
[2:13] * LarsFronius (~LarsFroni@dyndsl-080-228-197-038.ewe-ip-backbone.de) Quit (Quit: LarsFronius)
[2:22] * Cube (~Adium@12.248.40.138) Quit (Quit: Leaving.)
[2:22] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Ping timeout: 480 seconds)
[2:24] * loicd (~loic@brln-4d0cc4bd.pool.mediaWays.net) Quit (Quit: Leaving.)
[2:34] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[2:38] * lofejndif (~lsqavnbok@82VAAFFQ9.tor-irc.dnsbl.oftc.net) Quit (Remote host closed the connection)
[2:52] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) Quit (Quit: Leseb)
[2:54] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[2:55] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit ()
[2:57] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) Quit (Quit: Leaving.)
[3:28] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) Quit (Read error: Connection reset by peer)
[3:28] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[3:49] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[4:24] * chutzpah (~chutz@100.42.98.5) Quit (Quit: Leaving)
[4:36] * glowell (~glowell@141.142.134.93) Quit (Remote host closed the connection)
[4:38] * deepsa (~deepsa@122.172.7.232) has joined #ceph
[4:40] * joao (~JL@89.181.157.48) Quit (Read error: Operation timed out)
[4:59] * deepsa (~deepsa@122.172.7.232) Quit (Quit: Computer has gone to sleep.)
[5:33] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) has joined #ceph
[5:40] * deepsa (~deepsa@122.172.35.1) has joined #ceph
[5:46] * MarkDude (~MT@c-98-210-253-235.hsd1.ca.comcast.net) has joined #ceph
[6:12] * SuperSonicSound (~SuperSoni@28IAAGJND.tor-irc.dnsbl.oftc.net) Quit (Quit: Leaving)
[6:12] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) has joined #ceph
[6:23] * joao (~JL@89.181.154.115) has joined #ceph
[6:29] * dmick (~dmick@2607:f298:a:607:592b:48c2:f1f3:4291) Quit (Quit: Leaving.)
[7:01] * asadpanda (~asadpanda@2001:470:c09d:0:20c:29ff:fe4e:a66) has joined #ceph
[7:01] * joao (~JL@89.181.154.115) Quit (Read error: Operation timed out)
[7:01] * jluis (~JL@89-181-151-89.net.novis.pt) has joined #ceph
[7:46] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) Quit (Remote host closed the connection)
[7:50] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[7:51] * tnt (~tnt@167.39-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[8:46] * tnt (~tnt@167.39-67-87.adsl-dyn.isp.belgacom.be) Quit (Ping timeout: 480 seconds)
[8:57] * tnt (~tnt@office.intopix.com) has joined #ceph
[9:01] * BManojlovic (~steki@91.195.39.5) has joined #ceph
[9:05] * loicd (~loic@brln-4dbc27d1.pool.mediaWays.net) has joined #ceph
[9:20] * EmilienM (~EmilienM@191.223.101.84.rev.sfr.net) has joined #ceph
[9:36] * Cube (~Adium@cpe-76-95-223-199.socal.res.rr.com) Quit (Quit: Leaving.)
[9:47] * MarkDude (~MT@c-98-210-253-235.hsd1.ca.comcast.net) Quit (Ping timeout: 480 seconds)
[9:50] * Leseb (~Leseb@193.172.124.196) has joined #ceph
[9:59] <exec> someone knows, what is current file_size_limit on cephfs?
[10:02] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has joined #ceph
[10:12] * MarkDude (~MT@c-98-210-253-235.hsd1.ca.comcast.net) has joined #ceph
[10:12] * ninkotech_lite (~dp@ip4-95-82-152-139.cust.nbox.cz) has joined #ceph
[10:17] * fc (~fc@83.167.43.235) Quit (Quit: leaving)
[10:17] * fc (~fc@83.167.43.235) has joined #ceph
[11:08] * yoshi (~yoshi@p22043-ipngn1701marunouchi.tokyo.ocn.ne.jp) Quit (Remote host closed the connection)
[11:12] <loicd> sileht: (07:51:37 PM) joshd: tnt, loicd, sileht: radogw also breaks up large objects, there's some configurable threshold above which they're broken up iirc
[11:13] <loicd> sileht: it's good news isn't it ?
[11:13] <sileht> loicd yes :)
[11:32] <sileht> loicd, http://ceph.com/wiki/Benchmark#Some_results_according_to_the_number_of_PG
[11:33] * LarsFronius (~LarsFroni@dyndsl-080-228-197-038.ewe-ip-backbone.de) has joined #ceph
[11:33] <sileht> loicd, I have added the results of the rados bench in addition of the ressources utilizations
[11:34] <loicd> it's good. For how long did you run each rados bench ?
[11:35] <sileht> 360 seconds for this results
[11:35] <loicd> http://ceph.com/w/images/9/92/Result-avg-cpu-64-write.png there are dots at the bottom of the graph, what are they ?
[11:36] <sileht> hum, strange
[11:36] <loicd> sileht: you should list questions that do not yet have an answer in a "Pending questions paragraph". Such as : "What are the dots at the bottom of http://ceph.com/w/images/9/92/Result-avg-cpu-64-write.png ?"
[11:37] <sileht> it seems a good idea I will add it
[11:40] <loicd> Also "TODO list" with things such as "repeat the benchmark with more than 360 seconds for each osd bench"
[11:41] <loicd> sileht: s/Residual/Resident/
[11:41] <loicd> Also in the todo probably : why does it get stuck when PG >= 50000
[11:43] <loicd> sileht: I suggest you move http://dl.sileht.net/public/enovance/ceph-pgnum/script/ to ceph.com/wiki so that it is self contained and does not depend on your personal server being up. Although I know you take good care of your servers, no offense ;-)
[11:43] <sileht> loicd, the wiki don't allow to upload .sh and .py files
[11:44] <loicd> name them .txt or whatever ;-)
[11:44] * lofejndif (~lsqavnbok@9YYAAIGHK.tor-irc.dnsbl.oftc.net) has joined #ceph
[11:45] <loicd> sileht: the graphs look good. I see you managed to mix lines and dots, it's a lot easier to understand. And the captions are self explanatory.
[12:13] <sileht> loicd, for dot on the bottom: matplot show the value of the xseries on the x axes
[12:15] <loicd> sileht: thanks for the explanation :-)
[12:16] <sileht> loicd, I have added the result of rados bench for 50000 PGs, it seems that the result for 20000 is not valid.
[12:52] * eternaleye_ (~eternaley@tchaikovsky.exherbo.org) has joined #ceph
[12:53] * eternaleye (~eternaley@tchaikovsky.exherbo.org) Quit (Read error: Connection reset by peer)
[12:53] * joao (~JL@89.181.156.247) has joined #ceph
[12:55] * EmilienM (~EmilienM@191.223.101.84.rev.sfr.net) Quit (Remote host closed the connection)
[12:55] * EmilienM (~EmilienM@191.223.101.84.rev.sfr.net) has joined #ceph
[12:57] * jluis (~JL@89-181-151-89.net.novis.pt) Quit (Ping timeout: 480 seconds)
[13:23] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) has joined #ceph
[13:42] * mtk (ENHMqO0Io0@panix2.panix.com) Quit (Remote host closed the connection)
[13:43] * mtk (~mtk@ool-44c35bb4.dyn.optonline.net) has joined #ceph
[13:55] * ninkotech_lite (~dp@ip4-95-82-152-139.cust.nbox.cz) Quit (Ping timeout: 480 seconds)
[14:34] * mgalkiewicz (~mgalkiewi@staticline58611.toya.net.pl) has joined #ceph
[14:35] <mgalkiewicz> hi is anybody there?
[14:35] <mgalkiewicz> joao: are u there?
[15:07] * LarsFronius_ (~LarsFroni@dyndsl-031-150-015-033.ewe-ip-backbone.de) has joined #ceph
[15:13] * LarsFronius (~LarsFroni@dyndsl-080-228-197-038.ewe-ip-backbone.de) Quit (Ping timeout: 480 seconds)
[15:13] * LarsFronius_ is now known as LarsFronius
[15:23] * dspano (~dspano@rrcs-24-103-221-202.nys.biz.rr.com) has joined #ceph
[15:27] * LarsFronius_ (~LarsFroni@dyndsl-031-150-029-005.ewe-ip-backbone.de) has joined #ceph
[15:30] * LarsFronius (~LarsFroni@dyndsl-031-150-015-033.ewe-ip-backbone.de) Quit (Read error: Operation timed out)
[15:30] * LarsFronius_ is now known as LarsFronius
[15:33] * s[X] (~sX]@ppp59-167-157-96.static.internode.on.net) has joined #ceph
[15:43] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) has joined #ceph
[15:51] * LarsFronius (~LarsFroni@dyndsl-031-150-029-005.ewe-ip-backbone.de) Quit (Quit: LarsFronius)
[17:00] * lx0 (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[17:19] * Tv_ (~tv@38.122.20.226) has joined #ceph
[17:31] * bchrisman (~Adium@c-76-103-130-94.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[17:37] * s[X] (~sX]@ppp59-167-157-96.static.internode.on.net) Quit (Remote host closed the connection)
[17:41] * loicd (~loic@brln-4dbc27d1.pool.mediaWays.net) Quit (Quit: Leaving.)
[17:41] * loicd (~loic@brln-4dbc27d1.pool.mediaWays.net) has joined #ceph
[17:44] * nolan (~nolan@phong.sigbus.net) Quit (Remote host closed the connection)
[17:44] * nolan (~nolan@2001:470:1:41:20c:29ff:fe9a:60be) has joined #ceph
[17:55] * andreask (~andreas@chello062178013131.5.11.vie.surfer.at) Quit (Quit: Leaving.)
[17:59] * Leseb (~Leseb@193.172.124.196) Quit (Quit: Leseb)
[18:08] * mgalkiewicz (~mgalkiewi@staticline58611.toya.net.pl) Quit (Remote host closed the connection)
[18:19] * LarsFronius (~LarsFroni@2a02:8108:380:90:e806:c4a5:f4f7:4c25) has joined #ceph
[18:20] * BManojlovic (~steki@91.195.39.5) Quit (Quit: Ja odoh a vi sta 'ocete...)
[18:22] * tnt (~tnt@office.intopix.com) Quit (Ping timeout: 480 seconds)
[18:23] * Leseb (~Leseb@5ED01FAC.cm-7-1a.dynamic.ziggo.nl) has joined #ceph
[18:25] * LarsFronius (~LarsFroni@2a02:8108:380:90:e806:c4a5:f4f7:4c25) Quit (Quit: LarsFronius)
[18:27] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[18:38] * Cube (~Adium@12.248.40.138) has joined #ceph
[18:40] * lofejndif (~lsqavnbok@9YYAAIGHK.tor-irc.dnsbl.oftc.net) Quit (Ping timeout: 480 seconds)
[18:40] * bchrisman (~Adium@108.60.121.114) has joined #ceph
[18:41] * lofejndif (~lsqavnbok@82VAAFF56.tor-irc.dnsbl.oftc.net) has joined #ceph
[18:48] * LarsFronius (~LarsFroni@2a02:8108:380:90:4407:9a34:f248:e41b) has joined #ceph
[18:50] * tnt (~tnt@167.39-67-87.adsl-dyn.isp.belgacom.be) has joined #ceph
[18:50] * LarsFronius (~LarsFroni@2a02:8108:380:90:4407:9a34:f248:e41b) Quit ()
[19:02] * fc (~fc@83.167.43.235) Quit (Quit: leaving)
[19:04] * chutzpah (~chutz@100.42.98.5) has joined #ceph
[19:08] * dmick (~dmick@2607:f298:a:607:89e4:6d41:7bd9:2e0) has joined #ceph
[19:09] * jbd_ (~jbd_@34322hpv162162.ikoula.com) has left #ceph
[19:10] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) has joined #ceph
[19:11] <mgalkiewicz> Hi guys I have 2 osds in my configuration (ceph 0.48) and following problem http://pastie.org/4384864
[19:12] <mgalkiewicz> what is more I cant list rbd volumes https://gist.github.com/3249641
[19:12] <mgalkiewicz> I havent changed keys or anything
[19:13] <mgalkiewicz> ceph status does not report any problems
[19:30] <dmick> certainly looks like secret id 534 has been lost somehow
[19:32] <dmick> have you accessed this cluster successfully in the past (during this run)?
[19:40] * joshd (~joshd@2607:f298:a:607:221:70ff:fe33:3fe3) has joined #ceph
[19:41] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) has joined #ceph
[19:46] <mgalkiewicz> dmick: I am only using rbd volumes
[19:46] <mgalkiewicz> and it seems that all works
[19:48] <mgalkiewicz> could you help me with debugging it futher?
[19:49] <dmick> mgalkiewicz: ok, but, have you successfully accessed rbd volumes on this cluster run, and this started happening on its own? or has this problem existed since you started this cluster?
[19:50] <mgalkiewicz> the problem started happening on its own
[19:52] <mgalkiewicz> what is more ceph components (mon, osd, mds) started using a lot of memory
[19:52] <mgalkiewicz> more than 100% more
[19:52] <mgalkiewicz> nothing particular in logs except what I have pasted
[19:53] <dmick> that secret is one established from time to time between the cluster daemons
[19:53] <dmick> I would expect it to generate a new secret "eventually"
[19:53] <dmick> how long has this situation existed?
[19:54] <mgalkiewicz> rbd list does not work for about 20h at most
[19:54] <mgalkiewicz> have not been working*
[19:55] * lxo (~aoliva@lxo.user.oftc.net) Quit (Ping timeout: 480 seconds)
[19:55] <mgalkiewicz> problem from the first gists is problably oolder
[19:55] <mgalkiewicz> checking logs
[19:56] * lofejndif (~lsqavnbok@82VAAFF56.tor-irc.dnsbl.oftc.net) Quit (Remote host closed the connection)
[19:57] <mgalkiewicz> at least since 27th of july
[19:57] <mgalkiewicz> logs from 27th complain about secret_id=0
[20:00] * BManojlovic (~steki@212.200.241.106) has joined #ceph
[20:00] <dmick> I don't know enough to diagnose the problem; it seems likely that restarting the cluster would help, if you can afford to do that.
[20:01] <dmick> if you would rather understand how it went wrong, I'll try to recruit some help
[20:01] <mgalkiewicz> I would be greatful
[20:19] * guerby (~guerby@nc10d-ipv6.tetaneutral.net) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * andret (~andre@pcandre.nine.ch) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * EmilienM (~EmilienM@191.223.101.84.rev.sfr.net) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * alexxy (~alexxy@2001:470:1f14:106::2) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * Ludo_ (~Ludo@falbala.zoxx.net) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * ogelbukh (~weechat@nat3.4c.ru) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * jamespage (~jamespage@tobermory.gromper.net) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * Fruit (wsl@2001:980:3300:2:216:3eff:fe10:122b) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * joao (~JL@89.181.156.247) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * gregorg (~Greg@78.155.152.6) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * lightspeed (~lightspee@81.187.0.153) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * al (d@niel.cx) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * laevar (~jochen@laevar.de) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * vhasi (martin@vha.si) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * lurbs (user@uber.geek.nz) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * Meyer__ (meyer@c64.org) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * gadago (~gavin@2001:9d8::223:54ff:fee2:f41d) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * ajm (~ajm@adam.gs) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * markl (~mark@tpsit.com) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * jefferai (~quassel@quassel.jefferai.org) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * DLange (~DLange@dlange.user.oftc.net) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * nhm_ (~nh@65-128-130-177.mpls.qwest.net) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * _are_ (~quassel@2a01:238:4325:ca02::42:4242) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * morpheus (~morpheus@foo.morphhome.net) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * Ormod (~valtha@ohmu.fi) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * wonko_be (bernard@november.openminds.be) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * benner (~benner@193.200.124.63) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * raso (~raso@deb-multimedia.org) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * SpamapS (~clint@xencbyrum2.srihosting.com) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * liiwi (liiwi@idle.fi) Quit (resistance.oftc.net reticulum.oftc.net)
[20:19] * MarkS (~mark@irssi.mscholten.eu) Quit (resistance.oftc.net reticulum.oftc.net)
[20:21] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[20:23] * hijacker (~hijacker@213.91.163.5) Quit (Ping timeout: 480 seconds)
[20:23] * EmilienM (~EmilienM@191.223.101.84.rev.sfr.net) has joined #ceph
[20:23] * joao (~JL@89.181.156.247) has joined #ceph
[20:23] * alexxy (~alexxy@2001:470:1f14:106::2) has joined #ceph
[20:23] * gregorg (~Greg@78.155.152.6) has joined #ceph
[20:23] * nhm_ (~nh@65-128-130-177.mpls.qwest.net) has joined #ceph
[20:23] * _are_ (~quassel@2a01:238:4325:ca02::42:4242) has joined #ceph
[20:23] * lightspeed (~lightspee@81.187.0.153) has joined #ceph
[20:23] * guerby (~guerby@nc10d-ipv6.tetaneutral.net) has joined #ceph
[20:23] * andret (~andre@pcandre.nine.ch) has joined #ceph
[20:23] * tjikkun (~tjikkun@2001:7b8:356:0:225:22ff:fed2:9f1f) has joined #ceph
[20:23] * ajm (~ajm@adam.gs) has joined #ceph
[20:23] * al (d@niel.cx) has joined #ceph
[20:23] * markl (~mark@tpsit.com) has joined #ceph
[20:23] * benner (~benner@193.200.124.63) has joined #ceph
[20:23] * gadago (~gavin@2001:9d8::223:54ff:fee2:f41d) has joined #ceph
[20:23] * ogelbukh (~weechat@nat3.4c.ru) has joined #ceph
[20:23] * cattelan (~cattelan@2001:4978:267:0:21c:c0ff:febf:814b) has joined #ceph
[20:23] * jamespage (~jamespage@tobermory.gromper.net) has joined #ceph
[20:23] * jefferai (~quassel@quassel.jefferai.org) has joined #ceph
[20:23] * SpamapS (~clint@xencbyrum2.srihosting.com) has joined #ceph
[20:23] * raso (~raso@deb-multimedia.org) has joined #ceph
[20:23] * Fruit (wsl@2001:980:3300:2:216:3eff:fe10:122b) has joined #ceph
[20:23] * laevar (~jochen@laevar.de) has joined #ceph
[20:23] * vhasi (martin@vha.si) has joined #ceph
[20:23] * jeffhung (~jeffhung@60-250-103-120.HINET-IP.hinet.net) has joined #ceph
[20:23] * liiwi (liiwi@idle.fi) has joined #ceph
[20:23] * lurbs (user@uber.geek.nz) has joined #ceph
[20:23] * MarkS (~mark@irssi.mscholten.eu) has joined #ceph
[20:23] * Meyer__ (meyer@c64.org) has joined #ceph
[20:23] * DLange (~DLange@dlange.user.oftc.net) has joined #ceph
[20:23] * Ludo_ (~Ludo@falbala.zoxx.net) has joined #ceph
[20:23] * morpheus (~morpheus@foo.morphhome.net) has joined #ceph
[20:23] * Ormod (~valtha@ohmu.fi) has joined #ceph
[20:23] * wonko_be (bernard@november.openminds.be) has joined #ceph
[20:31] * NashTrash (~Adium@mobile-166-147-116-105.mycingular.net) has joined #ceph
[20:31] <NashTrash> Hello Ceph'ers
[20:32] * hijacker (~hijacker@213.91.163.5) has joined #ceph
[20:33] <NashTrash> I am hoping someone can help me with a network related config issue for Ceph. I want to move network traffic internal to the cluster onto a separate network from the one clients access. Is this possible?
[20:34] <joshd> yes, use the cluster_addr or cluster_network config options for cluster-internal traffic
[20:35] <joshd> the client network analogs are pubilc_addr/public_network
[20:35] <NashTrash> Cool. Thanks. Do I need to specify both _addr and _network?
[20:36] <NashTrash> And, does this only apply to OSDs or everyone?
[20:36] <joshd> just one of them
[20:37] * sjust (~sam@38.122.20.226) Quit (Remote host closed the connection)
[20:37] <NashTrash> Alright. Right now my cluster is all on the public network. Is there a way to gracefully move to these new settings? I.e. can I just change the ceph.conf file and restart the services?
[20:37] * Meths (rift@2.27.95.52) Quit (Server closed connection)
[20:37] * Meths (rift@2.27.95.52) has joined #ceph
[20:38] * sjust (~sam@38.122.20.226) has joined #ceph
[20:38] <joshd> that should work for the osds I think
[20:38] <gregaf> yeah; I think so too
[20:38] <joshd> the monitors shouldn't change, their ip is well known
[20:39] <joshd> it only really matters for the osds
[20:39] <NashTrash> Excellent. Thanks for being so willing to help out.
[20:39] <gregaf> the other nodes don't support separate networks; they do all their work on the public one
[20:39] <joshd> you're welcome :)
[20:41] <joshd> mgalkiewicz: that could happen if the process connecting to the monitor could not connect to it for a long time, and the keys rotated a couple times before it could reconnect
[20:41] <NashTrash> One other problem that I have been having is with the cephfs client. Everything installs nicely and works great if I do mount manually. But, if I try to mount via fstab nothing happens. Nothing in the logs, mount -a is the same. I have to do something like mount /mnt/ceph for it to work. Any thoughts?
[20:42] <mikeryan> what's your fstab entry look like?
[20:42] <mikeryan> specifically i'm interested in your options, dump, and pass sections
[20:43] <NashTrash> 192.168.51.249:6789:/ /mount/ceph ceph name=foo,secretfile=/etc/ceph/secret.key,noauto,rw,noexecnodev,noatime,nodiratime 0 2
[20:43] <mikeryan> try setting pass = 1
[20:44] <NashTrash> that would be the second to last param?
[20:44] <mikeryan> the last
[20:44] <mikeryan> the one that's 2 now
[20:44] <NashTrash> cool. I will give it a shot. Thanks.
[20:45] <mikeryan> actually according to fstab(5), it seems like 0 is the correct value to use
[20:45] <mikeryan> paraphrased: file systems which should not be fsck'd should use the value 0
[20:46] <mgalkiewicz> joshd: hmm but the client listed in logs is connected all the time and works
[20:47] <mgalkiewicz> joshd: I have restarted all mons as well
[20:48] <darkfader> throw in _netdev might also be an idea
[20:48] <darkfader> so it gets mounted at the right time
[20:50] <NashTrash> mikeryan: Now I am not sure which to use :)
[20:52] <dmick> 0
[20:52] <darkfader> 0 0 for network fs
[20:53] <NashTrash> cool. Thanks
[20:54] <mgalkiewicz> joshd: is it safe to restart osd now?
[20:57] * allsystemsarego (~allsystem@188.27.167.57) has joined #ceph
[20:59] <dmick> mgalkiewicz: it should be.
[20:59] <dmick> (joshd is having lunch)
[21:08] * steki-BLAH (~steki@212.200.241.176) has joined #ceph
[21:11] * BManojlovic (~steki@212.200.241.106) Quit (Read error: Operation timed out)
[21:27] * lxo (~aoliva@lxo.user.oftc.net) Quit (Read error: No route to host)
[21:37] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) Quit (Read error: Operation timed out)
[21:38] * lxo (~aoliva@lxo.user.oftc.net) has joined #ceph
[21:48] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) has joined #ceph
[22:05] * allsystemsarego (~allsystem@188.27.167.57) Quit (Quit: Leaving)
[22:09] * Ryan_Lane (~Adium@c-67-160-217-184.hsd1.ca.comcast.net) Quit (Quit: Leaving.)
[22:17] * sjust (~sam@38.122.20.226) Quit (Read error: Connection reset by peer)
[22:21] * sjust1 (~sam@2607:f298:a:607:baac:6fff:fe83:5a02) has joined #ceph
[22:35] * NashTrash (~Adium@mobile-166-147-116-105.mycingular.net) Quit (Ping timeout: 480 seconds)
[22:45] * s15y (~s15y@sac91-2-88-163-166-69.fbx.proxad.net) Quit (Ping timeout: 480 seconds)
[22:48] * s[X] (~sX]@ppp59-167-157-96.static.internode.on.net) has joined #ceph
[23:00] * s[X] (~sX]@ppp59-167-157-96.static.internode.on.net) Quit (Remote host closed the connection)
[23:01] * Ryan_Lane (~Adium@216.38.130.167) has joined #ceph
[23:04] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) Quit (Ping timeout: 480 seconds)
[23:11] * loicd (~loic@brln-4dbc27d1.pool.mediaWays.net) Quit (Quit: Leaving.)
[23:13] * mgalkiewicz (~mgalkiewi@toya.hederanetworks.net) has joined #ceph
[23:55] * Cube (~Adium@12.248.40.138) Quit (Quit: Leaving.)
[23:55] * s[X] (~sX]@ppp59-167-154-113.static.internode.on.net) has joined #ceph
[23:56] * EmilienM (~EmilienM@191.223.101.84.rev.sfr.net) has left #ceph

These logs were automatically created by CephLogBot on irc.oftc.net using the Java IRC LogBot.